Niebezpieczne AI, Czyli Jak Zepsuć Sztuczną Inteligencję

One, two, three, one, two, three. Okay, with minor problems, but let's start. Unfortunately, I have to look at what the projector shows, because unfortunately I don't have a mirror. For those who prefer to watch slides in their own order, you can watch the presentation. It's displayed at the end, so you don't have to take pictures now. You can wait until the end. My name is Kamil Frankowicz, just like Paweł, I work at CERT. I deal with analysis and search for vulnerabilities. I also play with malware. If anyone has any questions regarding the presentation, you can find me under these addresses. This is my company address, and at the end is my private address. My blog, where I write once in a while, but less and less often. My Twitter,

where I sometimes post something, but rarely. What do I want to talk about? I want to talk about, let's define what is... AI, machine learning or deep learning. It's easier to operate on these shortcuts. How can you break it in a soft and hard way? I'll explain later what it's all about and how to test it. So far, I think, the approach to it is that it's something magical, we'll put the data in there, something will work and we'll have some output. Unfortunately, it doesn't work like that. Let's define what is artificial intelligence, machine learning, deep learning. Does anyone know the definition? Maybe someone in the audience will hear it. that could help to guide others to understand what it is. I mean, not what it

is, let's say it's just a filler, more so that it's not empty on the slides. I would like to ask you to define what artificial intelligence is, or deep learning, machine learning, how it is understood by you. Ok, you don't know what machine learning is, or deep learning. You don't need to know it. Ok, what is it? This is also difficult. This is a neural network that takes programs as input in Pascal and produces output from it. There is a lot of scientific work on this topic, so you can use neural networks for such things. But what problems does artificial intelligence solve? Theoretically, we can sit for two days and discuss this topic, but I prefer practical examples of implementation

of the technologies that are used on a daily basis. And if we use them on a daily basis, what can go wrong? What can prevent us from completing our tasks? What can influence the Cyber Physical zone? For example, phones. Yesterday and today in Poland there is a new honor, of course, AI camera. Now every phone has a digital camera, which is supported by cameras, which is supported by artificial intelligence. Tesla, the same, the recognition system of signs, autonomous vehicles also use artificial intelligence. In general, there is quite a lot of use of it. Even though it's an output from Pascal programs. But few people know how it works. And they are aware of the use of these technologies in their daily lives. A

break for a meme. Because I will talk a lot and this is worth it. It's worth to break up. Generally, it has been led to absurdity. There is a blockchain everywhere, artificial intelligence everywhere. In this case, artificial intelligence is nothing more than simply a set of statistical methods for some data. But it's not sexy to use the word "statistical methods", so we use artificial intelligence. At least that's how business understands it today. I don't know how to call it, in a business sense, the word "sold products". But generally, as I said, there are statistical methods applied to some data. Sometimes it is supervised learning, sometimes it is not. Depending on the problem class we solve, Deep learning

is a kind of machine learning on steroids. Artificial intelligence is not defined yet. It's hard to say what it is from a user perspective. I found something like this on the Internet. I'm not sure what it is, but I feel that it's not enough for an average user. or a security guard who would like to test these systems or do something cool with them. Generally speaking, the problem of AI, as it is understood, is that we try to be inspired by the human brain to perform certain operations. Unfortunately, the human brain cannot be translated one to one. methods used in algorithms. So we are still lost in these definitions and in using them. But it

doesn't matter, we have already gone a bit off topic. More from the perspective of a regular user, how technology is used, how such technologies are designed, what role does a developer have in the whole process of creating this software. Generally, in the normal, traditional code creation, the developer is responsible for the logic. So everything that happens, if, else, the service of exceptions, this kind of bullshit. In the case of of creating systems, learning machine learning or even the art of the whole intelligence, whatever we define it, the role of a developer is to work more on, not on the decision logic, but more on the data, on the input that comes to the solution, which they had, to the black box, and then spills out some

data that could potentially be useful for us. Potentially, of course, because depending on how we proceed with these three steps, which I will talk about later, in the soft destruction of artificial intelligence, we can expect different things. For people with poor imagination, this is what a neural network looks like. which classifies some functions based on inputs, we can clearly see the decision boundaries. So if a point is calculated using the statistical methods I mentioned earlier, it will be classified into a given collection. It looks like this, but it's all boring so far. Let's think about where the problems with programming lie. OpenCV is a very popular software for computer vision, which is for seeing and recognizing objects' symbols from an input.

It can be any input, it can be an internet camera, it can be sensors of some other type, it can be photos. It's very simple. It simply puts the right object into some frame and tries to name it. Here we can use a very popular tutorial on YouTube about playing AI in GTA V and getting to the right places and such things, performing the player's mission. However, the project is old. It has over 25,000 stars on GitHub. It's practically the basis of every project that uses face recognition, road signs, etc. Unfortunately, the development of this project is very strange. There are no bugs. There are almost 1500 issues on GitHub, which means that the number of bugs is 653. These problems are

not fixed. The libraries are not updated. After 7 minutes of fuzzing I had almost 250-230 crashes, including 6 unique ones. which are potential debts. However, after the whole day of Hang Fuzz phasing, I had 324 unique and almost 47,000 crashes. Very, very weak. I haven't managed to minimize them all yet, because this is a slide made two days ago. I haven't managed to minimize them all yet, and report these debts. Unfortunately, they exist. Generally, I've already shown a lot of Zasun outputs on the slides, so I won't get into the classic problems of the set-up and the start. The old, outdated libraries, which developers know about, and the developer's recommendation is to simply turn off this library somewhere in

the switch. The problem is that this library is not turned off by default, and there is still a debate about which code to use to spread jpegs. The threat may be that this project is used in such systems as recognition of critical images, for example autonomous cars, because it is the most mature and the oldest. And in the prospect of some attack, which I will talk about in a moment, soft, not hard, the vulnerability may end up really as damage or also a bad track of the car, especially since computer vision systems are very vulnerable to it. Yes, it's a meme, of course. You don't have to fix it if you don't write any corrections. Unfortunately, it's a project, and there

are some new versions. However, the progress is what it is. Let's move on. Another, slightly smaller, 6 or 5-star on GitHub. A lot of statistical features that I won't talk about, because this presentation is quite boring and abstract. However, does it lead to a number of errors? Generally, yes. There are some problems with serialization of data in this project. Three minutes of fuzzing. and it's enough to find a null pointer of the reference. Of course, it's not a vulnerability that can be exploited, but nevertheless, if after a few minutes of checking it with an automatic tool for testing, I'm able to find some problems in deserialization of the code, then there will be more and more problems, because no one is looking at it.

So, unfortunately, another mem, so that it's not boring. Let's move on. There are many phases, which I mentioned earlier, that can be damaged and which belong to the programmer. The biggest one is feature extraction, which is extracting the most significant features from the data, which are able to teach our system quite effectively, so as not to consume too many resources. This is a very interesting issue. The next issue is, of course, the prediction, training of the model, sampling of data, minimization. There are many places where something can go wrong. There are many soft problems, which are not the classic ones, such as memory corruption, but rather some other problems, resulting from the characteristics of these solutions. All of these are

links to scientific works or presentations that discuss this topic in more detail. Due to the fact that I have limited time, I don't want to go into it too much, but I will tell you about the most interesting ones. Where are the causes of these problems? The main reason for all the problems that were written there is that few people realize that the algorithms that are sitting there have expectations about the quality of data. It seems that machine learning systems are such black boxes that something is thrown into them, something is ground and some output is thrown out. Unfortunately, no. The context of the data is very important. I think that testing some systems and building one system for

the work of the master, regarding the recognition of malware, it takes a lot of time to give context to the data, so that the prediction is as effective as possible in terms of quality. Many people forget about this. Unfortunately, this is the quality of data. The whole process, which was previously shown on the previous slide, what the developer does, must be carried out correctly. We cannot, for example, maximize one process, extract the features, or something else, and the data will still be weak, because it will all collapse. The worst thing is the lack of understanding of the whole process and the fact that now everyone wants to be an engineer in machine learning, deep learning, computer vision, AI. And

it's everywhere. So, unfortunately, the problem is... It's like a problem is created because few people know what they are doing, they learn from some courses conducted by Hindus on YouTube. Okay. We start with the first problem, which concerns soft AI problems. the knowledge we have developed from the data we have obtained, we can commercialize it. There are platforms like BigML and Google, which allow the sale of access to their own models using API. So if we have, for example, If we want to recognize a malware, we can buy an API package, shoot it to Google Cloud, which already has this model, and we will pay only for the output. It's quite an interesting solution, because the model development is simply expensive. I think that's what NVIDIA

did. They learned how to recognize a malware using the same format. The platform is called NV150. It took 3 months of continuous work and 100 graphics cards. The system was very efficient and was very useful. However, it was time, money and electricity. And if we can easily detect such a model by testing it of our solution that we buy. It turns out that stealing is relatively cheap. It is enough, even researchers have calculated, that 150 or 200 dollars would be enough to steal a large model from the cloud, of course, using appropriate algorithms. So here is an example of the test data that has been obtained. Because the model can be used not only for

the method of operation, i.e. the one learned, but also the test data, the model that was taught to them. And here we have an example of the work I mentioned, data extraction. At the top are the data that were in the model, at the bottom are the data extracted. This is the model that is responsible for the recognition of the manual, the numbers. The quality is quite good. So if you can buy a model that was trained for 3 months for $200, it's really good. Here are some technical details, but I won't bore you. This is a very interesting problem I mentioned earlier, related to computer vision. Very minor perturbations in the image, which concern the change of white balance, brightness, and adding colour pixels, completely

change the way the image is classified. Here you can see examples of how you can make ice out of cheeseburger or carbonara spaghetti out of cauliflower. These are very few examples, but if you translate it to cars, drones, autonomous systems that make decisions about important interactions with the physical world, it's not funny at all. This is a test of identification of signs from different distances and with stickers and reflections. The weakest case was 66% of the classification of cheating. Quite a lot. Of course, there are two hundred, so if our car does not recognize the stop and will drive by itself, it can be crooked. It's not a car or our nose. This is a video of a turtle being classified as a "carabine".

That's why we have OpenCV solution. The problem here is a turtle shell, which contains some strange perturbations. The same model was used, so it's very easy to do and it costs money. I will let go of this movie. But with Panda you can make a monkey. The worst thing is that anyone can do it, because the frameworks are universal and can create a picture for every image, similar but with a different content that classifies It's classified differently, and you can generate it in 5-10 minutes on middle-class equipment. So it's worth remembering that when these systems are taught, they must be taught by all possible perturbations, which are unfortunately hard to deal with. The most interesting problem, probably among all the problems related to

soft neural networks, I use these terms interchangeably, probably on the recording it will be machine learning, deep learning, artificial intelligence. All that I will replace with MSC, I will put into one bag with neural networks, but also with simple algorithms such as the nearest neighbor or SVM. A very cool thing is that you can learn the network and bug the right input so that it can choose the right path in the neural network. and classify the output differently. This experiment was quite expensive and there is no framework yet that can easily bug neural networks. I suspect that within the next two or three years, there will be frameworks that will be able to generate such

input from the hands to the majority of possible cases. This is also the case with the handwritten recognition model, which is related to the numbers. The perturbations are small, you can see 4 additional pixels in the corner of the image and we are able to change the network's operation. How to test? These problems are very... I call them soft because I don't have a better word for it, but hard ones are memory corruption and other things. How to find problems that are difficult to solve, let's say, technically? There are also frameworks for this. I'm a fan of fuzzing, so I managed to find a fuzzer for neural networks that detects such things. It's an AFL,

but it works for neural networks too. Everything is free of charge, everything works quite well. I had a demo yesterday, it was a mess, so unfortunately I don't have a demo. However, if I manage to reproduce this environment properly, I will add a video from the neural network phase on the slides. Unfortunately, not today. CleverHands is a framework that is used to create images classified as monkeys. or as a panda, of course in an inappropriate way, it's also very nice. It's worth to lean on it and play with your phone, generate some pictures, upload them and show what it is. I think most of the new phones have recognition using AI objects, it also creates a frame that it's a

laptop or a wallet. Finally, the end of my talk. First of all, it's worth looking at it from a slightly different perspective than just Buzzword. Few people are dealing with it at the moment, with such a soft and hard problem-discovering in these projects. If you already have some solutions for machine learning and other things like this in your projects, it's worth testing it. These are just two frameworks that are worth recommending, but the area is developing very strongly and there will be more of it. It's worth keeping an eye on. And most of all, many people are using AI without security, so it's weak. I think that the knowledge of developers will grow and the

models that will be created will be properly secured and it will be harder to cheat and make a spandex, a monkey and that kind of thing or a cheeseburger, spaghetti carbonara. That's all from me. The presentation is under the same address as at the beginning. I slept well because I thought it would be worse in half an hour. There are many slides. Are there any questions? At the beginning you showed a decision-making chart of AI. It was a very interesting spiral. Where did it come from? This. There are two functions here: function accepting two arguments x1 and x2 and function accepting x1 and x2. Mixing these two functions through the corresponding number of neurons can be done.

It was clicked by me for the quick presentation. Everyone can click on something like this. I think it's called TensorFlow. It has something like this for visualizing neural networks and how it works. Of course, there are also some parameters of learning, activation and loss functions. So, this is only the visual part. Unfortunately, there is no meat of learning parameters in this part, because it was not included in the slide. And it would have a small added value for the presentation, if I were to explain it. You can click on it, I'll add a link here if I don't forget. And here we have output, how it looks like after all this. The point is to choose the right activation function for neurons without setting the

entire environment, so that you can quickly start prototyping a solution. Did I answer the question or something else? There's not everything here, so maybe the context is out of context. Any more questions? If not, I've already provided lunch to everyone. Thank you very much.