Artificial Intelligence? - more like Artificial Stupidity!
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Klostermann, Aiko
I talk about failures of AI/ML, how it more often affects minorities, can even have deadly consequences, what the reason for those failures are and what we can do to prevent this from happening in the future. Nowadays “Artificial Intelligence” is everywhere! And rightly so, it does enable us to do really cool things, things we couldn’t even imagine doing just a decade ago. In fact, it sometimes just feels like magic. This ‘magic’ behind it is often powered by “Machine Learning”. But even “AI” has its limitations. I’ll show examples where “AI” and ML have failed (sometimes with horrible consequences) and will explain why failures are unavoidable in ML but also mention what we can do to reduce them in the future. Furthermore, I’ll showcase how current AI implementations discriminate against minorities and how that in some cases even leads to a higher risk of death for those groups. I’ll cover the bias that humans introduce and I’ll explain how poor choice of data makes our world even more unjust than it already is. The takeaway for the audience: AI can fail and sometimes it has horrible consequences. Why is AI so hard to “do right”? How can we make AI better?
 Hi and welcome back to the Reinruhr stage in Mornheim of the RC3. If you have any questions regarding the talk, please feel free to reach out on hackintirc in the channel RC3-R3S or on Twitter or Mastodon with the hashtag RC3R3S or our Mastodon handle at r3s.com. Off topic again, we have put a link to the CO2 indicator you saw here and asked on our website. So you can find it there r3s.nrw.news.co2-ampel with an alpha A. Up next we're still talking about artificial intelligence, machine learning and ethics. When asked, the speaker didn't provide his credit card number so I certainly can't share it here with you, but he told me he likes ramen. He already was on the congress but this is his first time speaking. He worked in India, Bangalore and now Singapore. He described himself as a code monkey. So please feel free to give a very warm virtual welcome to IKU Klostermann and his talk Artificial Intelligence, more like artificial stupidity. Please have fun. Thank you very much for the introduction. To the point, artificial intelligence or artificial stupidity is not just a click baity title, there's actually something behind it. What I want to do today is tell you a couple of stories about ML, machine learning, data, artificial intelligence and some of these are success stories. Everything went well, great use case, great outcome, but some of them are not. Some of them failed, some of them pretty hard and especially into those I would like to dig a little bit deeper and talk a little bit about why they failed and what we can do in the future to prevent that. And when I say I'll be telling you a couple of stories, let me start with one of the, at least in my opinion, one of the greatest storytellers that we had. Douglas Adams and I'm sure a lot of you have read this book or at least know about these books. Well, actually five, three good ones and then two others. But even if you haven't read them, you must have come across the memes that they brought us that 42 is the answer to the life of the universe and everything and that we should not panic. But Douglas Adams has also introduced another thing in this Hitchhiker's Guide to the Galaxy books, which is this Babel fish or the idea of this Babel fish. And for you, if you don't know it, let me quickly explain a little bit what this Babel fish is. It's a tiny yellow fish that's living somewhere in the universe, right? And you can put that into your ear and when you put this fish into your ear, the fish picks up the brain waves of the organisms around you. And whenever they say something in their own language that you don't understand, this fish picks that up and translates it. And basically excretes that into your ear as a language that you are familiar with. So it translates every spoken language of the universe around you into a language that you understand in your ear in real time. Which is, I mean, it's pretty crazy, right? Obviously, it's a fictional idea. Douglas Adams has a couple of crazy ideas. This is clearly, it's not really working, right? There's no knowledge of the existence of such fish and it would be really surprising if it were real. So it's a fictional idea. It would never be true really. But let me show you something else now. Hello on the head city of the common has actually not working that well with the with the spinning. So that's a live demo, right? Never really works that good. Let me quickly walk you through what I would have done if it would have worked. I would have said something in German and in real time and I probably don't need to demo this because you all use Google translate before. But then translate this into English in real time and you could maybe let's try that. Hello on the head city of the common so my report about difficulty and the ligands. Right, so it worked. It translates it in real time immediately into a language into a different language. I could pick different ones, whatever you understand you. You don't understand German. And that is, I mean, it's not quite this baby fish, but it does this instant translation from one language into some other language. And we have something else that I'll show you that that Google is not only providing this this Google translate and they are the non Google translation services as well. Google in particular is also selling these pixel buds, which is, which is in ear wireless headphones that you can plug into your ear that do support this this translation feature. You still need to have a smartphone. But basically what it's doing is you plug those those pixel bus into your ear and someone is saying something in language you do not understand. And the phone and then forwarding it to the to the earbuds are life translating that into a language that you do understand. So this absolutely crazy idea that that was that of this baby fish that is more or less implemented nowadays. And it could have not been imagined like just a decade ago but it's implemented more or less through and it's mostly machine learning that is powering this translation. There's machine learning multiple levels of picking up what I say, the speech to text and then translating it. And that is just fascinating right that is a and as someone living in a country where I didn't grow up in speaking a language that is not native to me. This has been so useful for me. And I'm sure it has been used for so many other people. This is a, this is an exceptionally useful tool that that machine learning has made, made possible for us. And that's one of the positive examples that I'm talking about. But there's an even, even more impressive, impressive example that I that I like that that machine learning has enabled us. And it is about detection of cancer cells based on image data. And a couple of for a couple of years now. There are machine learning models that look at this at image data of cells of human cells, and they can identify whether there are cancer cells shown or not with a higher accuracy than than doctors than trained professional whose job is it to do exactly that looking at these images and figuring out other cancer cells or not. And then machine learning models that are better than that for breast cancer. And I think last year, Google released a model that that can detect cancer cells better in image data for lung cancer as well. And this is clearly this is a life changing functionality that we are given by the advances by the recent advances of machine learning. But coming to the coming back to the title of the talk, artificial stupidity. They are clearly are and you've all seen some of them. I just give a couple of examples here. I've seen examples where AI fails where machine learning fails and you can see here on the on the very right there's a Twitter user and that Twitter user is treating to Indigo which is an Indian airline and he's having the sarcastic tone saying hey thank you for sending my my baggage to Hyderabad while I need to call cutter. And I assume there's an AI answering this something of chatbot. So, hey, we're glad to hear that we're clearly that the AI simply didn't pick up on the sarcastic tone of that message and in the, in the middle as an example where someone is messaging paper and saying hey I got scammed just a serious thing right. And I assume again it's a chatbot reaction says hey that's great. And on the left there's another example where this went wrong. But this is coming to my favorite. One of my favorite examples where something went wrong in that case and this is if you if you know Mandarin you could maybe figure out what was what's going on here. If not, I'll just tell you give you a little bit of a background what's going on here. In China, there are a couple of cities that have implemented cameras across on traffic lights crossing for pedestrian crossing you can see the zebra crossing. And, and what these cameras do, they basically film the zebra crossing. And whenever there's a red light for pedestrians, but the pedestrian is crossing. They film that they figure out they have the figure out that there's a person crossing they have object recognition the figure there's a person walking and then and this is what you can see here. They have a gigantic screen next to it, where they show that person and you they zoom in and hey this person just crossed at the red light. So basically shaming that person. And they also, it goes even further. They also not only do they have object recognition and identify there's a person they also face recognition and identify who that person is. And not only do they know who that person is. They also know the mobile number. So, and it happened in this case they they sent this person a message. Hey, we just saw you jaywalking we just saw you crossing the street, while there was a red light. And if you look at this picture now you can see away the second. There's not really a person crossing the street. It's a bus that is driving by which was a bus as a green light. It's a bus driving by with the face of a person printed on the side of it. In fact, this is this in this case it's this the CEO of this company and the advertising, having advertisement of on that bus with her. And, and she was actually in a different city and she got this notification hey you, you just crossed the red light and so later and that's not true. And there's some and that's a clearly, there is something that is an example where I failed and it goes even further. Not in this case but in other cities, the system is not only not only identifies a person and and who that person is and messages that person, but also because they know who that person is and due to regulations. That means that the government knows also your bank account and they know that bank account they also know your WeChat pay account. And in a couple of cities, when you get caught by this by this machine learning system by this image recognition system crossing the red red light, you get deducted defined for this crime within seconds from your WeChat pay account. So, in this case, it didn't happen but imagine your crossing the street is red light, they everything is automated deduct the fine immediately. And that is something. And that's why I say, it's a, it's a great example where it can fail because it is an impactful fail. It's with a bot doing a stupid answer as laughable most of the time, right. But in this case it actually does have a significant negative negative impact. And let me talk a little bit I mentioned machine learning and artificial intelligence a couple of times. Let me quickly give a little bit of context. Surely most of you have heard these terms and they've been hyped for the last couple of years, at least for this particular talk. I would propose the following definitions that I found and for artificial intelligence that is the theory and development of computer systems able to perform tasks normally requiring human intelligence. Now, and, and the important thing to note here is that this definition and this is the definition that you that is from the Oxford dictionaries. And if you, if you use Google and your type define artificial intelligence, this is the one that is being used as well. I have friends who disagree with this definition, but let's just keep it for this talk at least. And the important thing to note here is that the definition does not talk about a particular technical approach. And the saying, it is a computer system that does something. And you would usually assume is human intelligence is necessary for doing that. I give you an example. A chess program. If you play chess against someone you would assume that person is somewhat intelligent. You assume some kind of human intelligence at least. Now, if you, if you write a chess program for that. How would you actually end up implementing that you can in the simplest form, look at the state of the chess board. Look at the pieces and just brute force or the different moves and subsequent moves that are possible, and then pick the set of moves that in the end, give the result that you want which is winning the game. And then brute forcing having a couple of loops and some if else is not particularly sophisticated coding. However, it would match this, this magical definition of artificial intelligence that I think is used quite often, especially nowadays with such a marketing term. And then there's machine learning, which is where I think is actually a little bit cooler, which is, which is where the cool things are happening because for machine learning, the software engineers, the cold is not really defining the algorithm anymore. The machine learning system is trained with the data and it tries to emulate tries to generalize that, and then tries to match the train behavior with new data. And that is something where the algorithm gets defined basically by the data and the machine learning algorithm and architecture, but it is not defined by the engineer anymore and then their neural networks which is basically machine learning that emulates like the biological brain of like neurons and connections and all those neurons or some of those neurons and then there's deep learning, which is also pretty hype term for the last couple of years, which is basically artificial neural networks, but a lot of these layers. And that is basically what what ends up being. And I give you a quick example, and this also. So it's called deep neural network on the slide is actually a very sexually relatively shallow with only three hidden layers and once it's like 10 input, the real production. The neural networks can have 50 or even hundreds of 100 layers or even more than that, and thousands of input variables and the thing to note here is that even with this very simple example. You can see like all these nodes are connected right there, like basically there's some information flowing from one node to the other from one neuron to the other. And if you look at this, it is, it's actually impossible for you to understand what is really going on here. Like, if there were now data flowing from all of these to all the others and it is, even with a small example, it's just, I can't comprehend what is really going on here. And that is one of the, that is one of the key problems that I see with the deployment of the machine learning models that we have nowadays. It's basically a black box model. You don't define the algorithm anymore. And even if you look at it, you don't, you don't understand what's going on. That makes it very difficult to debug it makes it difficult to predict how it will actually how the model will react if I give it data that it hasn't seen before. And that is, that is one of the two issues that I that I see that the machine learning models that we use and deploy introduction and use for impactful tasks are basically unpredictable. Like what's the idea. Right. That is the one thing and I would like to go to the other thing that there's two things to issues that I see that was the first. Let me tell you an example that showcases the second issue a little bit better. And again, I'll tell you a story in this case is the story of the city of Boston 2011. In the city of Boston, in 2011, they had a problem. And the problem was that they had a lot of potholes in the city. And as I mentioned, it's 2011, they did what everyone did in 2011 they built an app. So they had an app for that and the app I must admit, it's a pretty smart idea. The app worked in the way that you would you would install this app on your phone. And then the you would you would drive around and when you get in your car, you put the phone on the passenger seat with the app running. And so you would drive around the city and whenever you would hit a portal, there would be a bomb. The phone would pick up through the accelerometer would pick up. There was some movement. There must be a portal and would then take the GPS coordinates of that particular location, send it to the servers of the city of Boston so that they then could come later and fix up the portal. So they did that. They deployed that people installed the app. And they let it run for some time. And what they noticed was that the data provided ended up being showing only potholes in the high cost of living areas of the city. And if you're like only the only the regions where rich people live right and you think about that. That doesn't really make sense. Why, why would there only be potholes in in these areas. You would at least somewhat expect an evenly distributed amount of potholes across the city. If anything, probably less than those rich people areas. But what the data showed was that is more or less only there. And, and, and the reason for that, I say then found out was that so in 2011 the world was a little bit different than nowadays. Smartphones were still relatively new and particularly ones where you could install apps on. So, I mean nowadays, every five year old is watching TikTok videos on their on their smartphone. But 2011 was was a different time and that led to only relatively wealthy people having smartphones with these and where they could install this app. And now the thing to note here is, there's no machine learning involved. It's, it's solely is that the data provided was wrong. There's no machine learning. It's simply that the data just was not representative of the reality. So to in coming back to this example, the they fixed that issue and I think it's equally smart how they actually address this. They then put the app on phones and put those phones into public buses and garbage trucks so they would then cover the whole city and they had a significantly better data set that they could work with and then go and fix those bottles. But again, the, the, the important thing here is there was no machine learning involved. It was simply the data that was that was screwed because it was like it had a social bias to it. Let me show you another example here. And, okay, at least I'm trying that. Let's see how the demo goes. So with me now, you might, you might have, you might remember this. It was, it became popular a couple of years ago. I'm writing two sentences here. One is she is a doctor and he is a nurse and I let Google translate translated into melee. And the interesting thing about melee is that it does not have gender specific pronouns. When in English, you have he and she and they indicate male or female. You can see here that I actually don't speak Malay. I guess the first word is the pronoun. But what you can see is that there is no differentiation. She gets translated to the same word that he gets translated to. So, and the interesting thing that we see here and, and remember I talked she is a doctor and he is a nurse. If this gets translated into Malay where it loses this gender information and if I now click this button, it takes the translated Malay version and translate it back into English. And let's see what happens here. Remember she nurse, she doctor, he nurse. And all of a sudden the translated English is he is a doctor and she is a nurse. So the translation lost the gender information, but then had to bring back some gender information and pick the exact opposite of what I initially typed in. And I know I had, I had Google Translate as a very valuable, great example of use of machine learning. But what we can see here is, and this again, the machine learning part works well in a way. What is, what's the issue here is the data. Because the reason why this machine learning model translates this Malay sentence without gender into this gendered sentence and is using he for doctor is because the way these models are trained is basically reading a lot of text in like the same text in different languages. Reading this in English and reading this in Malay and then figuring out over time lots of different texts, text over text, and figuring out how to translate one to the other. And these texts historically speaking, these texts when they were talking about doctors, they were talking about him, they were talking about he, they were talking about a male doctor because historically speaking, that was probably the case. We know that is, that there has no reason for any gender not to be a doctor. And the same thing applies for the nurse. Anyone can be a nurse. But then historically speaking, the texts that were used to train this model, we're usually talking about a female being a nurse. And this was, this became popular a couple of years ago in, in the Turkish language. And Google then did something Google had to manually intervene, because people complained about it rightfully so. And then they added this extra feature. Hey, look in Turkish, it actually can mean both things. He and she, but there's a manual thing that they had to set up particularly for Turkish. And again, the thing here is the machine learning model, the black box behavior is not the issue with the data is what was the issue here. And I'm also sure that a lot of you remember Tay, that was Microsoft trying to train some kind of language model on Twitter and I think some other platforms. And basically what they wanted to do, they had a model that was able to communicate like send text and form sentences in a way. And they wanted this model to learn from humans how they interact, how they speak. Okay, now if the premises you want to teach this, this machine learning how humans interact is Twitter a good place to teach that. Don't know, maybe the premises a little requestable. The outcome, however, is not very questionable. They had to shut it down in less than 24 hours. There were a lot of people figured out what was going on they were training that that bot and they were using. They basically made it racist by got a horrible piece of. A bot. And they are was basically abused but the thing is that and this is so the last one that I'm the last example that I'm giving you kind of showcases that there's someone saying hey you are just a stupid machine. And correctly so. Hey, please. Hey, I learned from from you and you are done to and that is, I think again the lesson here is that the data that was given to this to this machine learning model then turned it into a this is racist but another example that I, but again, hey was just a Twitter bot was an experiment from Microsoft. I hope they learned something. But there are systems nowadays that we use where machine learning is used where data that train these models were used that have a significant impact and the compass system is one of them that's used in the US is basically used. If people commit a crime in the US they fill out this questionnaire like 200 questions or so. And this data that again is used to predict whether the person will re offend or not. Basically, the person gets a jail sentence and if they come out will they do with a committed crime again. Right and this this system says it likely or not likely based on the this question. And this was analyzed this was analyzed by pro publica and the outcome is quite interesting. So there are basically two ways how you can be wrong. Right. So the publica looked at at the these offenders and then after they got released and the next two years to figure out whether they committed a crime with a dog. And you can be wrong in two ways the system can be wrong two ways. Basically, for us positive for negative. So you predict the person will re offend but the person does not. Or the other way around you predict or the system does predict the person will not re offend but then the person actually does re offend. And there was a there was a very interesting bias that that pro publica noticed you for the white population of these of this test of this analysis. Only 23% were labeled a high risk that they probably re offend but then they did not. For the African American part of this test of this analysis, or 45% were labeled a high risk. And then it turned out they didn't actually re offend. And the opposite is true for for the other case, being labeled a lower risk was almost 50% of the white test subjects were labeled a lower risk and then actually did re offend. And for the African Americans only 28% were able to lower risk. And then did re offend. So there clearly is a racial bias in this data that the system is then using. And the interesting thing is that none of the questions were using were asking questions about skin color or or ethical background. So this was gathered from from other data somehow this this system was trained to predict a higher risk for African Americans than for white people. And that is, that is a very impactful thing. This data gets used to determine bail sentence by judges this data even get used to determine prison the length of prison sentences by judges. And this system is still being used nowadays. And I have one more example. And a quite impactful one as well. You know autonomous driving cars right they will take over our streets, maybe not in two years, maybe not in five years, maybe not in 10, but in 2030 years, there will be probably a high amount of autonomous driving cars on our streets. And these streets, these cars basically work in a similar way they have lots of sensors. A lot of them, a lot of the sensors are cameras, they are systems that are only cameras, for example. And then they try to figure out, okay, where can I drive and one thing that they as a driver you probably know that you don't want to hit a pedestrian, or even as a non driver you can imagine you wouldn't want to do that. Because that's bad. And so the systems try to figure out who's a pedestrian and who's not. Do I need to break do I need to circumvent that person or whatever that is or not. And an interesting study found out that is that these autonomous driving vehicles or the systems that that power them have a higher fail basically to to detect dark skin, pedestrian, more than white skin or light skin, pedestrian. And that means that if you and accidents happen right there will be if you have you have millions of cars billions of cars on the streets accidents do happen. There will be situations where there are a lot of a lot of situations where these cars have to make decisions. Do I break or do I drive. And if the systems are more likely to fail to identify a pedestrian based on their skin color. Then these systems will eventually end up killing people because they are dark skin, more likely than then killing non dark skin. And that is a can't get more impact for them that. And that's just a horrible outcome. It's an amount of power we give to these systems that we clearly should not have everyone can agree. Right, so I remember my mom telling me, hey, if you would jump if your friends would jump from a bridge, would you do the same. Clearly I would not right. But that's exactly how machine learning once work. If the data we feed in is biased and just broken then that's how they act. Right and now you're telling me I cope. So there's machine learning everywhere and you're telling me has this horrible consequences the world is on fire. Is there anything we can do. And the answer is yes. And the, the industry is already moving in the right direction. And the one, one of the things that we need to counter this black box behavior of machine learning models we need explainability of these models as a select criteria. If there are multiple models, and one is actually built in a way that it that it makes it understandable explainable to the developer hey this is why I react in a certain way. This needs to be a focus. And there's there have been examples especially an image recognition you can see here on the very bottom. It's a methodological deconvolution, now using a deconvolution where you can see this these layers of the machine learning model you can basically see what they detect the first one is just see pictures and then the other one seeing shapes and later more complex shapes and facial features and things that you kind of know what what they are detecting. In general we need to move away from this standard machine learning approach where we just put data into a model and then we take the predictions into a like a like a human interpretable model where we put data into a model and then have some way of interpreting this model and have human inspection improve the data and the model and run it again and come to a point where we actually have a model that that works as we as you want. Right. And one, one methodology or one technology that I would highly recommend also using is called line and it's fantastically simple and it's so powerful out works. And basically, I give you an example here there's a, you can see, there's a picture of frog and gets identified as a predicted as a tree frog. And what lime does it basically takes away parts of this of this image, and then lets the model predict the same thing again with a reduced amount of data. And you can see that here in the middle on the top. And the outcome is, I mean, relatively simple, if the prediction is still the same outcome, then you know the data you have removed was not relevant for that prediction. Right. And you can see that in the middle there's some data got removed and the prediction is way off. It's like, it's not a tree frog anymore. And so by by removing certain parts of data and then seeing if it still predicts the same thing you can figure out which part of the input data was actually for the outcome. And that's how you can figure out why you're what is relevant for your model and why the model decided in a certain way. And as if we don't need another argument for more diverse teams, I think having better prediction having better machine learning model, having better data is another good reason for diverse teams. And if you think of the autonomous driving cars, if the team would have had dark skinned people, they probably would have thought of that and would have tested them. And there's, I mean, there's a wide range of arguments and that is an issue in our industry to have more diverse teams. I think this is just one more on top of the list of long arguments for that. And I would recommend if you're working with these systems is looking at the data ethics canvas, which is something that is released by the ethics, which is basically a set of questions that you want to go through. And then there's a data center project where you can figure out, okay, this is this, this is my limitations of my data sources who am I sharing with a set of questions that you want to look at think about before you will make start working with it. And there's another thing that is really helpful figuring out to figure out how biased your data is, which is Google Fessus, where it's a visualization you just throw in your data, and it visualizes it shows you what's going on. And it's super simple, like you throw your data and you could see, oh, look, I only have 20% female people in my data set clearly. And then I need to adjust my data set. That being said, thank you so much. Thanks for all the fish. I hope I could give you some some insights on the two big problems that I see with machine learning or AI models that we have nowadays and what I am what we can do to actually fix those fix those two issues. So yeah, thanks for giving us such an introspection in our future overlords. Before we get to the questions, just to prevent worried messages in the chat, the bandage around my wrist is not because of an emergency. It's just that I'm left handed and left handed people tend to have problems with wrist strain in their left hand. So our micro search was so nice and stabilized my wrist, just so you don't need to worry. Anyways, two questions. The first one was traditional statistical methods also provide an uncertainty. Would it be possible for artificial neural networks to notice the input is outside the training set or even compute an uncertainty? Excellent question. I cannot answer the question. I will research that. Maybe if you whoever asked a question, maybe you can ping me somehow you can find me just Google my name you'll find a way to contact me. I would like to have a conversation about that. Okay, doesn't the gender is sorry doesn't the gender assumption reduce wrong just translation statistically since biases are still present to some degree. Oh, I mean, it obviously I mean the okay the statement is is correct in the way that it does. You have to pick some gender and and picking a historically more prevalent might be a good idea but then looking forward, if we like looking forward, I don't. I wouldn't see why and let's talk about nurses and doctors or we can talk about all the other things like if we talk about it industry there's a there's a big male focus for a reason that doesn't really make sense. So there's, I think the problem that we would run into is that we continue pushing this bias that doesn't really have a reason. I mean we would trade off that historically speaking we probably would be correct. But in the future I mean, even though it is still the case, I don't see why in the future we wouldn't have like an evenly distributed gender spectrum across doctors nurses or it professionals for that matter. So I think the risk here is that we that we continue pushing this bias onto onto people that that would probably it's like a district back loop that then causes the bias to come back again and I think that's the that's the problem. And you would probably add some kind of more accuracy at least for now but you would ruin future generations. So I'm not sure if that's worth it. The next question is, will links to sources be uploaded somewhere. I have not planned that, but I can do that. Again, maybe just contact me I can send them to you. Okay. How was the open source experience so far in, I would assume in particular for this topic, I guess. I guess the whether it's open source or not is is not really relevant for the for the buyers I mean, potentially people are more aware of these biases, but the. I mean, the tools itself are probably less impacted by that or the tools that we're using. The majority of the at least for deep learning the technologies, most of them are open source as far as I know. I guess the open source or closed source discussion is not conversation doesn't have that much impact on the bias or on the black box behavior. And the last question is what was the last data analysis tool shown. Google facet like FAC ETS and the big G in the beginning. It's just is really relatively nice. Throw in your data and it visualized it is quite quite nice. Okay, maybe we can provide links to that on our on our on our channels like Twitter and Macedon so our viewers can pick it up there and have a look at it. So yeah, also the signal angel passed to me that passed over to me that there were there were lots of thanks for the talk and I can only join them. So thank you very much for taking the time. Thank you for for virtually coming over half half the world to be here in Moenheim. And yeah, have a lot of fun on the rest of the RC three and we hope to hear from you again. Thank you very much. Thanks for having me.
 Hi and welcome back to the Reinruhr stage in Mornheim of the RC3.  If you have any questions regarding the talk, please feel free to reach out on hackintirc in the channel RC3-R3S  or on Twitter or Mastodon with the hashtag RC3R3S or our Mastodon handle at r3s.com.  Off topic again, we have put a link to the CO2 indicator you saw here and asked on our website.  So you can find it there r3s.nrw.news.co2-ampel with an alpha A.  Up next we're still talking about artificial intelligence, machine learning and ethics.  When asked, the speaker didn't provide his credit card number so I certainly can't share it here with you,  but he told me he likes ramen.  He already was on the congress but this is his first time speaking.  He worked in India, Bangalore and now Singapore.  He described himself as a code monkey.  So please feel free to give a very warm virtual welcome to IKU Klostermann and his talk Artificial Intelligence,  more like artificial stupidity.  Please have fun.  Thank you very much for the introduction.  To the point, artificial intelligence or artificial stupidity is not just a click baity title, there's actually something behind it.  What I want to do today is tell you a couple of stories about ML, machine learning, data, artificial intelligence and some of these are success stories.  Everything went well, great use case, great outcome, but some of them are not.  Some of them failed, some of them pretty hard and especially into those I would like to dig a little bit deeper  and talk a little bit about why they failed and what we can do in the future to prevent that.  And when I say I'll be telling you a couple of stories, let me start with one of the, at least in my opinion, one of the greatest storytellers that we had.  Douglas Adams and I'm sure a lot of you have read this book or at least know about these books.  Well, actually five, three good ones and then two others.  But even if you haven't read them, you must have come across the memes that they brought us that 42 is the answer to the life of the universe and everything and that we should not panic.  But Douglas Adams has also introduced another thing in this Hitchhiker's Guide to the Galaxy books, which is this Babel fish or the idea of this Babel fish.  And for you, if you don't know it, let me quickly explain a little bit what this Babel fish is.  It's a tiny yellow fish that's living somewhere in the universe, right?  And you can put that into your ear and when you put this fish into your ear, the fish picks up the brain waves of the organisms around you.  And whenever they say something in their own language that you don't understand, this fish picks that up and translates it.  And basically excretes that into your ear as a language that you are familiar with.  So it translates every spoken language of the universe around you into a language that you understand in your ear in real time.  Which is, I mean, it's pretty crazy, right?  Obviously, it's a fictional idea.  Douglas Adams has a couple of crazy ideas.  This is clearly, it's not really working, right?  There's no knowledge of the existence of such fish and it would be really surprising if it were real.  So it's a fictional idea.  It would never be true really.  But let me show you something else now.  Hello on the head city of the common has actually not working that well with the with the spinning.  So that's a live demo, right?  Never really works that good.  Let me quickly walk you through what I would have done if it would have worked.  I would have said something in German and in real time and I probably don't need to demo this because you all use Google translate before.  But then translate this into English in real time and you could maybe let's try that.  Hello on the head city of the common so my report about difficulty and the ligands.  Right, so it worked.  It translates it in real time immediately into a language into a different language.  I could pick different ones, whatever you understand you.  You don't understand German.  And that is, I mean, it's not quite this baby fish, but it does this instant translation from one language into some other language.  And we have something else that I'll show you that that Google is not only providing this this Google translate and they are the non Google translation services as well.  Google in particular is also selling these pixel buds, which is, which is in ear wireless headphones that you can plug into your ear that do support this this translation feature.  You still need to have a smartphone.  But basically what it's doing is you plug those those pixel bus into your ear and someone is saying something in language you do not understand.  And the phone and then forwarding it to the to the earbuds are life translating that into a language that you do understand.  So this absolutely crazy idea that that was that of this baby fish that is more or less implemented nowadays.  And it could have not been imagined like just a decade ago but it's implemented more or less through and it's mostly machine learning that is powering this translation.  There's machine learning multiple levels of picking up what I say, the speech to text and then translating it.  And that is just fascinating right that is a and as someone living in a country where I didn't grow up in speaking a language that is not native to me.  This has been so useful for me.  And I'm sure it has been used for so many other people.  This is a, this is an exceptionally useful tool that that machine learning has made, made possible for us.  And that's one of the positive examples that I'm talking about.  But there's an even, even more impressive, impressive example that I that I like that that machine learning has enabled us.  And it is about detection of cancer cells based on image data.  And a couple of for a couple of years now.  There are machine learning models that look at this at image data of cells of human cells, and they can identify whether there are cancer cells shown or not with a higher accuracy than than doctors than trained professional whose job is it to do exactly that looking at these images and figuring out other cancer cells or not.  And then machine learning models that are better than that for breast cancer. And I think last year, Google released a model that that can detect cancer cells better in image data for lung cancer as well.  And this is clearly this is a life changing functionality that we are given by the advances by the recent advances of machine learning.  But coming to the coming back to the title of the talk, artificial stupidity.  They are clearly are and you've all seen some of them. I just give a couple of examples here.  I've seen examples where AI fails where machine learning fails and you can see here on the on the very right there's a Twitter user and that Twitter user is treating to Indigo which is an Indian airline and he's having the sarcastic tone saying hey thank you for sending my my baggage to Hyderabad while  I need to call cutter. And I assume there's an AI answering this something of chatbot. So, hey, we're glad to hear that we're clearly that the AI simply didn't pick up on the sarcastic tone of that message and in the, in the middle as an example where someone is messaging paper and saying hey I got scammed just a serious thing right.  And I assume again it's a chatbot reaction says hey that's great. And on the left there's another example where this went wrong.  But this is coming to my favorite.  One of my favorite examples where something went wrong in that case and this is if you if you know Mandarin you could maybe figure out what was what's going on here. If not, I'll just tell you give you a little bit of a background what's going on here.  In China, there are a couple of cities that have implemented cameras across on traffic lights crossing for pedestrian crossing you can see the zebra crossing.  And, and what these cameras do, they basically film the zebra crossing. And whenever there's a red light for pedestrians, but the pedestrian is crossing.  They film that they figure out they have the figure out that there's a person crossing they have object recognition the figure there's a person walking and then and this is what you can see here.  They have a gigantic screen next to it, where they show that person and you they zoom in and hey this person just crossed at the red light.  So basically shaming that person.  And they also, it goes even further.  They also not only do they have object recognition and identify there's a person they also face recognition and identify who that person is.  And not only do they know who that person is. They also know the mobile number.  So, and it happened in this case they they sent this person a message. Hey, we just saw you jaywalking we just saw you crossing the street, while there was a red light.  And if you look at this picture now you can see away the second.  There's not really a person crossing the street. It's a bus that is driving by which was a bus as a green light. It's a bus driving by with the face of a person printed on the side of it.  In fact, this is this in this case it's this the CEO of this company and the advertising, having advertisement of on that bus with her.  And, and she was actually in a different city and she got this notification hey you, you just crossed the red light and so later and that's not true.  And there's some and that's a clearly, there is something that is an example where I failed and it goes even further.  Not in this case but in other cities, the system is not only not only identifies a person and and who that person is and messages that person, but also because they know who that person is and due to regulations.  That means that the government knows also your bank account and they know that bank account they also know your WeChat pay account.  And in a couple of cities, when you get caught by this by this machine learning system by this image recognition system crossing the red red light, you get deducted defined for this crime within seconds from your WeChat pay account.  So, in this case, it didn't happen but imagine your crossing the street is red light, they everything is automated deduct the fine immediately.  And that is something. And that's why I say, it's a, it's a great example where it can fail because it is an impactful fail.  It's with a bot doing a stupid answer as laughable most of the time, right. But in this case it actually does have a significant negative negative impact.  And let me talk a little bit I mentioned machine learning and artificial intelligence a couple of times. Let me quickly give a little bit of context.  Surely most of you have heard these terms and they've been hyped for the last couple of years, at least for this particular talk.  I would propose the following definitions that I found and for artificial intelligence that is the theory and development of computer systems able to perform tasks normally requiring human intelligence.  Now, and, and the important thing to note here is that this definition and this is the definition that you that is from the Oxford dictionaries.  And if you, if you use Google and your type define artificial intelligence, this is the one that is being used as well. I have friends who disagree with this definition, but let's just keep it for this talk at least.  And the important thing to note here is that the definition does not talk about a particular technical approach.  And the saying, it is a computer system that does something. And you would usually assume is human intelligence is necessary for doing that. I give you an example.  A chess program.  If you play chess against someone you would assume that person is somewhat intelligent. You assume some kind of human intelligence at least.  Now, if you, if you write a chess program for that. How would you actually end up implementing that you can in the simplest form, look at the state of the chess board.  Look at the pieces and just brute force or the different moves and subsequent moves that are possible, and then pick the set of moves that in the end, give the result that you want which is winning the game.  And then brute forcing having a couple of loops and some if else is not particularly sophisticated coding.  However, it would match this, this magical definition of artificial intelligence that I think is used quite often, especially nowadays with such a marketing term.  And then there's machine learning, which is where I think is actually a little bit cooler, which is, which is where the cool things are happening because for machine learning, the software engineers, the cold is not really defining the algorithm anymore.  The machine learning system is trained with the data and it tries to emulate tries to generalize that, and then tries to match the train behavior with new data.  And that is something where the algorithm gets defined basically by the data and the machine learning algorithm and architecture, but it is not defined by the engineer anymore and then their neural networks which is basically machine learning that emulates like the biological brain of like neurons and connections  and all those neurons or some of those neurons and then there's deep learning, which is also pretty hype term for the last couple of years, which is basically artificial neural networks, but a lot of these layers.  And that is basically what what ends up being.  And I give you a quick example, and this also. So it's called deep neural network on the slide is actually a very sexually relatively shallow with only three hidden layers and once it's like 10 input, the real production.  The neural networks can have 50 or even hundreds of 100 layers or even more than that, and thousands of input variables and the thing to note here is that even with this very simple example.  You can see like all these nodes are connected right there, like basically there's some information flowing from one node to the other from one neuron to the other.  And if you look at this, it is, it's actually impossible for you to understand what is really going on here. Like, if there were now data flowing from all of these to all the others and it is, even with a small example, it's just, I can't comprehend what is really going on here.  And that is one of the, that is one of the key problems that I see with the deployment of the machine learning models that we have nowadays. It's basically a black box model.  You don't define the algorithm anymore. And even if you look at it, you don't, you don't understand what's going on.  That makes it very difficult to debug it makes it difficult to predict how it will actually how the model will react if I give it data that it hasn't seen before.  And that is, that is one of the two issues that I that I see that the machine learning models that we use and deploy introduction and use for impactful tasks are basically unpredictable.  Like what's the idea. Right. That is the one thing and I would like to go to the other thing that there's two things to issues that I see that was the first.  Let me tell you an example that showcases the second issue a little bit better.  And again, I'll tell you a story in this case is the story of the city of Boston 2011.  In the city of Boston, in 2011, they had a problem. And the problem was that they had a lot of potholes in the city.  And as I mentioned, it's 2011, they did what everyone did in 2011 they built an app.  So they had an app for that and the app I must admit, it's a pretty smart idea. The app worked in the way that you would you would install this app on your phone.  And then the you would you would drive around and when you get in your car, you put the phone on the passenger seat with the app running.  And so you would drive around the city and whenever you would hit a portal, there would be a bomb.  The phone would pick up through the accelerometer would pick up.  There was some movement.  There must be a portal and would then take the GPS coordinates of that particular location, send it to the servers of the city of Boston so that they then could come later and fix up the portal.  So they did that. They deployed that people installed the app.  And they let it run for some time. And what they noticed was that the data provided ended up being showing only potholes in the high cost of living areas of the city.  And if you're like only the only the regions where rich people live right and you think about that.  That doesn't really make sense. Why, why would there only be potholes in in these areas. You would at least somewhat expect an evenly distributed amount of potholes across the city.  If anything, probably less than those rich people areas.  But what the data showed was that is more or less only there.  And, and, and the reason for that, I say then found out was that so in 2011 the world was a little bit different than nowadays.  Smartphones were still relatively new and particularly ones where you could install apps on.  So, I mean nowadays, every five year old is watching TikTok videos on their on their smartphone. But 2011 was was a different time and that led to only relatively wealthy people having smartphones with these and where they could install this app.  And now the thing to note here is, there's no machine learning involved.  It's, it's solely is that the data provided was wrong.  There's no machine learning. It's simply that the data just was not representative of the reality.  So to in coming back to this example, the they fixed that issue and I think it's equally smart how they actually address this.  They then put the app on phones and put those phones into public buses and garbage trucks so they would then cover the whole city and they had a significantly better data set that they could work with and then go and fix those bottles.  But again, the, the, the important thing here is there was no machine learning involved. It was simply the data that was that was screwed because it was like it had a social bias to it.  Let me show you another example here.  And, okay, at least I'm trying that. Let's see how the demo goes.  So with me now, you might, you might have, you might remember this.  It was, it became popular a couple of years ago.  I'm writing two sentences here.  One is she is a doctor and he is a nurse and I let Google translate translated into melee.  And the interesting thing about melee is that it does not have gender specific pronouns.  When in English, you have he and she and they indicate male or female.  You can see here that I actually don't speak Malay. I guess the first word is the pronoun.  But what you can see is that there is no differentiation. She gets translated to the same word that he gets translated to.  So, and the interesting thing that we see here and, and remember I talked she is a doctor and he is a nurse.  If this gets translated into Malay where it loses this gender information and if I now click this button, it takes the translated Malay version and translate it back into English.  And let's see what happens here. Remember she nurse, she doctor, he nurse.  And all of a sudden the translated English is he is a doctor and she is a nurse.  So the translation lost the gender information, but then had to bring back some gender information and pick the exact opposite of what I initially typed in.  And I know I had, I had Google Translate as a very valuable, great example of use of machine learning.  But what we can see here is, and this again, the machine learning part works well in a way.  What is, what's the issue here is the data.  Because the reason why this machine learning model translates this Malay sentence without gender into this gendered sentence and is using he for doctor is because the way these models are trained  is basically reading a lot of text in like the same text in different languages.  Reading this in English and reading this in Malay and then figuring out over time lots of different texts, text over text, and figuring out how to translate one to the other.  And these texts historically speaking, these texts when they were talking about doctors, they were talking about him, they were talking about he, they were talking about a male doctor because historically speaking, that was probably the case.  We know that is, that there has no reason for any gender not to be a doctor.  And the same thing applies for the nurse.  Anyone can be a nurse.  But then historically speaking, the texts that were used to train this model, we're usually talking about a female being a nurse.  And this was, this became popular a couple of years ago in, in the Turkish language.  And Google then did something Google had to manually intervene, because people complained about it rightfully so. And then they added this extra feature. Hey, look in Turkish, it actually can mean both things.  He and she, but there's a manual thing that they had to set up particularly for Turkish.  And again, the thing here is the machine learning model, the black box behavior is not the issue with the data is what was the issue here.  And I'm also sure that a lot of you remember Tay, that was Microsoft trying to train some kind of language model on Twitter and I think some other platforms.  And basically what they wanted to do, they had a model that was able to communicate like send text and form sentences in a way.  And they wanted this model to learn from humans how they interact, how they speak.  Okay, now if the premises you want to teach this, this machine learning how humans interact is Twitter a good place to teach that.  Don't know, maybe the premises a little requestable.  The outcome, however, is not very questionable. They had to shut it down in less than 24 hours.  There were a lot of people figured out what was going on they were training that that bot and they were using. They basically made it racist by got a horrible piece of.  A bot. And they are was basically abused but the thing is that and this is so the last one that I'm the last example that I'm giving you kind of showcases that there's someone saying hey you are just a stupid machine.  And correctly so.  Hey, please. Hey, I learned from from you and you are done to and that is, I think again the lesson here is that the data that was given to this to this machine learning model then turned it into a this is racist but  another example that I, but again, hey was just a Twitter bot was an experiment from Microsoft. I hope they learned something. But there are systems nowadays that we use where machine learning is used where data that train these models were used that have a significant  impact and the compass system is one of them that's used in the US is basically used. If people commit a crime in the US they fill out this questionnaire like 200 questions or so. And this data that again is used to predict whether the person will re offend or not.  Basically, the person gets a jail sentence and if they come out will they do with a committed crime again. Right and this this system says it likely or not likely based on the this question.  And this was analyzed this was analyzed by pro publica and the outcome is quite interesting.  So there are basically two ways how you can be wrong. Right. So the publica looked at at the these offenders and then after they got released and the next two years to figure out whether they committed a crime with a dog.  And you can be wrong in two ways the system can be wrong two ways. Basically, for us positive for negative.  So you predict the person will re offend but the person does not.  Or the other way around you predict or the system does predict the person will not re offend but then the person actually does re offend.  And there was a there was a very interesting bias that that pro publica noticed you for the white population of these of this test of this analysis.  Only 23% were labeled a high risk that they probably re offend but then they did not.  For the African American part of this test of this analysis, or 45% were labeled a high risk.  And then it turned out they didn't actually re offend. And the opposite is true for for the other case, being labeled a lower risk was almost 50% of the white test subjects were labeled a lower risk and then actually did re offend.  And for the African Americans only 28% were able to lower risk.  And then did re offend. So there clearly is a racial bias in this data that the system is then using.  And the interesting thing is that none of the questions were using were asking questions about skin color or or ethical background.  So this was gathered from from other data somehow this this system was trained to predict a higher risk for African Americans than for white people.  And that is, that is a very impactful thing. This data gets used to determine bail sentence by judges this data even get used to determine prison the length of prison sentences by judges.  And this system is still being used nowadays.  And I have one more example.  And a quite impactful one as well.  You know autonomous driving cars right they will take over our streets, maybe not in two years, maybe not in five years, maybe not in 10, but in 2030 years, there will be probably a high amount of autonomous driving cars on our streets.  And these streets, these cars basically work in a similar way they have lots of sensors.  A lot of them, a lot of the sensors are cameras, they are systems that are only cameras, for example.  And then they try to figure out, okay, where can I drive and one thing that they as a driver you probably know that you don't want to hit a pedestrian, or even as a non driver you can imagine you wouldn't want to do that.  Because that's bad.  And so the systems try to figure out who's a pedestrian and who's not. Do I need to break do I need to circumvent that person or whatever that is or not.  And an interesting study found out that is that these autonomous driving vehicles or the systems that that power them have a higher fail basically to to detect dark skin, pedestrian, more than white skin or light skin, pedestrian.  And that means that if you and accidents happen right there will be if you have you have millions of cars billions of cars on the streets accidents do happen.  There will be situations where there are a lot of a lot of situations where these cars have to make decisions. Do I break or do I drive.  And if the systems are more likely to fail to identify a pedestrian based on their skin color.  Then these systems will eventually end up killing people because they are dark skin, more likely than then killing non dark skin.  And that is a can't get more impact for them that.  And that's just a horrible outcome.  It's an amount of power we give to these systems that we clearly should not have everyone can agree.  Right, so I remember my mom telling me, hey, if you would jump if your friends would jump from a bridge, would you do the same.  Clearly I would not right.  But that's exactly how machine learning once work.  If the data we feed in is biased and just broken then that's how they act.  Right and now you're telling me I cope. So there's machine learning everywhere and you're telling me has this horrible consequences the world is on fire.  Is there anything we can do. And the answer is yes. And the, the industry is already moving in the right direction.  And the one, one of the things that we need to counter this black box behavior of machine learning models we need explainability of these models as a select criteria.  If there are multiple models, and one is actually built in a way that it that it makes it understandable explainable to the developer hey this is why I react in a certain way.  This needs to be a focus. And there's there have been examples especially an image recognition you can see here on the very bottom.  It's a methodological deconvolution, now using a deconvolution where you can see this these layers of the machine learning model you can basically see what they detect the first one is just see pictures and then the other one seeing shapes and later more complex shapes and facial features and things that you  kind of know what what they are detecting. In general we need to move away from this standard machine learning approach where we just put data into a model and then we take the predictions into a like a like a human interpretable model where we put data into a model and then have some way of interpreting this  model and have human inspection improve the data and the model and run it again and come to a point where we actually have a model that that works as we as you want.  Right. And one, one methodology or one technology that I would highly recommend also using is called line and it's fantastically simple and it's so powerful out works.  And basically, I give you an example here there's a, you can see, there's a picture of frog and gets identified as a predicted as a tree frog.  And what lime does it basically takes away parts of this of this image, and then lets the model predict the same thing again with a reduced amount of data.  And you can see that here in the middle on the top. And the outcome is, I mean, relatively simple, if the prediction is still the same outcome, then you know the data you have removed was not relevant for that prediction.  Right. And you can see that in the middle there's some data got removed and the prediction is way off. It's like, it's not a tree frog anymore. And so by by removing certain parts of data and then seeing if it still predicts the same thing you can figure out which part of the input data was actually  for the outcome. And that's how you can figure out why you're what is relevant for your model and why the model decided in a certain way.  And as if we don't need another argument for more diverse teams, I think having better prediction having better machine learning model, having better data is another good reason for diverse teams.  And if you think of the autonomous driving cars, if the team would have had dark skinned people, they probably would have thought of that and would have tested them.  And there's, I mean, there's a wide range of arguments and that is an issue in our industry to have more diverse teams. I think this is just one more on top of the list of long arguments for that.  And I would recommend if you're working with these systems is looking at the data ethics canvas, which is something that is released by the ethics, which is basically a set of questions that you want to go through.  And then there's a data center project where you can figure out, okay, this is this, this is my limitations of my data sources who am I sharing with a set of questions that you want to look at think about before you will make start working with it.  And there's another thing that is really helpful figuring out to figure out how biased your data is, which is Google Fessus, where it's a visualization you just throw in your data, and it visualizes it shows you what's going on.  And it's super simple, like you throw your data and you could see, oh, look, I only have 20% female people in my data set clearly.  And then I need to adjust my data set. That being said, thank you so much.  Thanks for all the fish. I hope I could give you some some insights on the two big problems that I see with machine learning or AI models that we have nowadays and what I am what we can do to actually fix those fix those two issues.  So yeah, thanks for giving us such an introspection in our future overlords.  Before we get to the questions, just to prevent worried messages in the chat, the bandage around my wrist is not because of an emergency.  It's just that I'm left handed and left handed people tend to have problems with wrist strain in their left hand.  So our micro search was so nice and stabilized my wrist, just so you don't need to worry.  Anyways, two questions.  The first one was traditional statistical methods also provide an uncertainty.  Would it be possible for artificial neural networks to notice the input is outside the training set or even compute an uncertainty?  Excellent question. I cannot answer the question.  I will research that.  Maybe if you whoever asked a question, maybe you can ping me somehow you can find me just Google my name you'll find a way to contact me.  I would like to have a conversation about that.  Okay, doesn't the gender is sorry doesn't the gender assumption reduce wrong just translation statistically since biases are still present to some degree.  Oh, I mean, it obviously I mean the okay the statement is is correct in the way that it does.  You have to pick some gender and and picking a historically more prevalent might be a good idea but then looking forward, if we like looking forward, I don't.  I wouldn't see why and let's talk about nurses and doctors or we can talk about all the other things like if we talk about it industry there's a there's a big male focus for a reason that doesn't really make sense.  So there's, I think the problem that we would run into is that we continue pushing this bias that doesn't really have a reason.  I mean we would trade off that historically speaking we probably would be correct.  But in the future I mean, even though it is still the case, I don't see why in the future we wouldn't have like an evenly distributed gender spectrum across doctors nurses or it professionals for that matter.  So I think the risk here is that we that we continue pushing this bias onto onto people that that would probably it's like a district back loop that then causes the bias to come back again and I think that's the that's the problem.  And you would probably add some kind of more accuracy at least for now but you would ruin future generations.  So I'm not sure if that's worth it. The next question is, will links to sources be uploaded somewhere.  I have not planned that, but I can do that.  Again, maybe just contact me I can send them to you.  Okay.  How was the open source experience so far in, I would assume in particular for this topic, I guess.  I guess the whether it's open source or not is is not really relevant for the for the buyers I mean, potentially people are more aware of these biases, but the.  I mean, the tools itself are probably less impacted by that or the tools that we're using.  The majority of the at least for deep learning the technologies, most of them are open source as far as I know.  I guess the open source or closed source discussion is not conversation doesn't have that much impact on the bias or on the black box behavior.  And the last question is what was the last data analysis tool shown.  Google facet like FAC ETS and the big G in the beginning.  It's just is really relatively nice. Throw in your data and it visualized it is quite quite nice.  Okay, maybe we can provide links to that on our on our on our channels like Twitter and Macedon so our viewers can pick it up there and have a look at it.  So yeah, also the signal angel passed to me that passed over to me that there were there were lots of thanks for the talk and I can only join them.  So thank you very much for taking the time.  Thank you for for virtually coming over half half the world to be here in Moenheim.  And yeah, have a lot of fun on the rest of the RC three and we hope to hear from you again.  Thank you very much.  Thanks for having me.
rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_0_71.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_72_359.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_360_1475.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_1476_1955.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_1956_4427.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_4428_5795.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_5796_5975.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_5976_6227.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_6228_6827.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_6828_7439.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_7440_7631.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_7632_7691.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_7692_8483.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_8484_8555.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_8556_9071.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_9072_14831.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_14832_20171.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_20172_21695.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_21696_22379.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_22380_23459.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_23460_23555.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_23556_23831.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_23832_23987.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_23988_24791.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_24792_26111.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_26112_26291.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_26292_26591.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_26592_26711.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_26712_27083.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_27084_27131.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_27132_28487.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_28488_32603.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_32604_32819.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_32820_33431.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_33432_34523.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_34524_35291.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_35292_35831.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_35832_36119.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_36120_36299.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_36300_36503.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_36504_36707.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_36708_36875.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_36876_36983.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_36984_37079.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_37080_48047.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_48048_48527.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_48528_49475.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_49476_50087.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_50088_51011.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_51012_52463.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_52464_53603.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_53604_53699.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_53700_53855.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_53856_54287.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_54288_54995.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_54996_61559.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_61560_64319.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_64320_64391.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_64392_64643.png rc3-r3s-10-eng-Artificial_Intelligence_-_more_like_Artificial_Stupidity_hd_64644_64646.png