AI in a Minefield: Learning from Poisoned Data

hey everybody this is urban director of the ground truth track at b-sides las vegas the next talk is ai in a minefield learning from poison data by itsec manton itzigmanten is the head of innovation at imperva his work over the last 20 years includes research in web application security advanced persistent threats drm systems automotive systems and data security as a security veteran he continuously asks what can go wrong just a reminder the live q a will immediately follow the talk and you can submit your questions via the ground truth discord channel so i hope you all enjoy the talk thanks very much for coming thank you for joining me uh in the session ai in the minefield uh learning

from poison data i'm isik mantin lead scientist at imperva in the last 21 years i've been innovating on security algorithms under intersection therefore also my interest in building a machine learning technology for cyber security and for analyzing threats on ai on ai solutions i enjoyed the game of analyzing threats and designing mitigation i love problem solving data science and in general the art of algorithms i love building innovative security technology in my spare time i love hiking mountain biking trail running and traveling the world and doing everything that you can do outdoors i'm just finishing now uh seven amazing years up in perva and on my way to the next day and divorce uh so if you're

interested in a consultancy on the data science cyber security competition computing cryptography then uh you can reach out at the algorithms for cyber at gmail.com i will start with a short introduction about ai and ai threat landscape i will then zoom into the threat of data poisoning and when why and how these threats is effective and what can we do about it i would then uh discuss the problem of learning from web traffic and how the threat of data poisoning is effective there and again what can we do about it uh and then i will go into the details of a robust learning solution that works in a streaming friendly fashion without a significant memory consumption

and i will end with the summary and conclusions we are no doubt in the ai era ai is everywhere is doing is uh effective affecting almost every domain of our life air is also the data error because data is the is the the material from which ai solutions are made of but both ai era and the data are are amazing i love using huge amounts of data in order to build uh to create insights that we didn't know before but ai comes with several caveats with it and in many cases we tend to ignore these caveats or clicks underestimate these caveats but we'll speak about them uh today uh i think every uh discussion about

ai and need to at least mention two uh significant mega risk or or threats that ai brings with it except for you know um attackers using ai for for for their purposes and there is the deep fake threat the ability of actors usually malicious actors to synthesize and create uh images video and audio that looks uh very uh authentic uh we are only see now i think the tip of the iceberg of using this uh uh this technology for uh malicious uh uh social intense uh so this is one um threat that's worth mentioning um another risk that is the ai discrimination or more accurately discrimination in the name of ai or under ai or perpetuation of biases

ai is based on looking at data coming from the past in order to make predictions related to the future so it assumes that uh the future will look like the past uh and whenever um so it has strong potential to perpetuate a biases um and whenever ai technology is being used in order to analyze the credit of someone for example it is a prisoner and what are the chances that for a rehabilitation or whether it is a loaner what is the chances of him to uh to returning the loan which of course has an impact on uh on the on the interest he will get or what are the probabilities of anyone to succeed then ai has a malicious a cruel tendency

to to focus on features we don't want it to feature our are not legitimate to focus uh in our modern societies like gender race or color so whenever you use ai technology for the sake of uh decision making regarding people then you should be aware of ai discrimination risk and mitigate it somehow but today we speak about the ai threats ai as an attack surface so what you see here is a typical machine learning system there is trading data it is fed into a training and then it is being used in order to build a model this model can be classification model or regression model too and once this model is is ready then it is being used on inputs data in order

to make decisions to make predictions in some cases these predictions are being later evaluated in order to continue and improve the accuracy of the model when you look at this uh of this uh setup from an attacker's perspective then uh if the attacker is an internal or if he's an external that somehow found his way in then of course the sky's the limit he can do whatever he likes he can tamper with the model he can steal the model he can extract the data from the model he can tamper with data temporary decisions he can do whatever he likes however even if the attacker is an external then still there is plenty of things he

can do for the training part he can do what is called that data poisoning we'll talk in detail about this threat later on um to to make the model uh different or to um more appropriate for the attacker objectives than uh uh than to them uh the model owner uh objective uh and there is also the what sometimes called evasion or adversarial examples or ai deception which is uh when the attacker coming to a working model to a model that is ready and it creates some malicious data samples that make the model make incorrect decisions many of you probably have seen this stop sign that when you add several stickers to it then uh that look native for us but

still for um the image classification for the scene analysis uh ai engine that lies in the autonomous uh vehicle then this stop sign does not look any longer as the stop sign it looks like a speed limit sign which of course we have uh uh devastating consequences if now this vehicle is is coming into a junction uh and there is also a training data leakage it is slightly more esoteric third but it is still there when you uh build a model from the training data then uh this some of these training data leaks into the model and in some circumstances this uh data can be later extracted by by by attackers so whenever you are using um training

data that he's sensitive it can be pii it can be health records uh or anything like that should be aware of that of this possibility and and take uh adequate actions uh to make sure it is uh mitigated uh we start with very uh a simple example in order to to explain the idea of data poisoning uh so in the spirit of uh of the olympic games that are are starting in these days uh so we have uh a gym exercise by uh by the greatest and simon bios and uh suppose that for the sake of simplicity that the score of this uh exercises is it's been um calculated by several uh referees uh six referees and i suppose it is it

is simple average uh now uh the five referees give pretty high scores because they love the exercise however there is another and malicious referee and he wants to make the average seven then he will give a minus two score or if he wants to make it in ten it will give a 14 score uh now this example is is very simple so you immediately can understand what is the the reason that this is possible because the impact of a single data points in this case the data point that is provided by that by the malicious uh the malicious referee is to um is unlimited uh he can if he would have want to have a hundred the average ig

would give a number that will uh imply an average of uh of a hundred and you can also immediately think about what is the uh the mitigation first uh minus two is not a legitimate score so you can say that every data point should be valid between six and ten uh for example you can uh which i believe is uh actually been uh uh done in a practical in actual competition and also you can uh this is something that again as far as i know is being done in these competitions you can uh eliminate the extremes eliminate the maximal score and the on the minimal score and take the average of uh of the others uh

so this is uh again so this is fairly simple to uh uh to mitigate in this case i will give a spoiler that uh things like using the the median for example uh is more complicated when you are uh well it is easy when you have all the data uh in your hand in some sort of a batch processing or offline processing but it's more complicated when you are seeing the data points one at a time you don't want to buffer uh them all now i i only heard about you know this terminology of data poisoning a couple of years ago but but but this is not a new threat i mean uh i think this is also almost as old as

the internet because i'm probably like myself when you are looking at trip advice or or or imdb or uh amazon shopping and you see a rating then you ask yourself okay it is rating authentic or maybe this is someone with interest uh to make this rating high maybe the owner of this restaurant or making this rating low because this is a competitor of this uh of this uh hotel and and again whenever you have you're building a mechanism that makes a prediction or makes an analysis or uses a logic that is based on data that is coming from uh from users uh then the data posing threat is there and and indeed you know at least in

booking.com i think they they they only accept uh a rating from um people that actually from users that actually completed their transactions actually paid money so they make it very hard for someone to uh you know to to feed their engine with with uh hundreds of uh of uh uh fake reviews well so um and uh i'm i'm also sure that in other in other sites uh you need to be a little usually you need to be a registered user in order to provide a rating and and moreover if you now as a user will send i don't know 1 000 um rates uh rating uh uh request in in in one minute then probably not all of them will be

counted as you expect uh so how does data poisoning work uh beyond this example of uh of an average which is very straightforward so on the left side you can see a linear classifier this line comes to provides the best optimal separation possible between the red triangles and the blue circles and it does a pretty good job however if you only modify the location of a single data point you can see now that the classifier it is still a good classifier it is good yeah but you can see it is very different so uh there is potentially a significant impact of even one or a small number of data points on the actual classifier on the right

side you see again similar problem uh you want to separate the the red from the blue but this time there is an adversary and he analyzes the situation and he wants to actually to foil this classifier so you want to create points in in a place that will make the classifier the worst possible so we identified this gradient and he will put these uh his new data points uh data poisoning data points in this location and what you see now is that you have a classifier that is that is pretty lousy does pretty lousy job now uh doing the separation this is exactly what the attacker in this case uh wanted uh to achieve uh

and again so this is these are very straightforward data poisoning uh examples one of the first uh battlefields for data poisoning is the area of spam filtering uh this is not surprising because if you think about that i think span filtering domain was one of the first were security applications where machine learning proves itself as as an effective mechanism effective technology in this case what you see is an attack on the gmail actual attack in 2018 on a gmail spam filter what the attackers wanted to achieve uh they probably planned to uh to have a spam campaign it did they didn't want google spam filter to identify uh these uh spams and their spam so

in advance they uh created the large batch of uh messages all of them had uh had many things in common with the uh with the tbd with the to be sent uh uh spam emails and they sent them and they classified all they labeled all of them as a b9 and they were hoping that their what they did will uh make the spam filtering engine of of gmail to classify these sort of emails as benign and therefore to create sort of a backdoor uh within uh the model uh so you can see that uh all these schooling attempts which were expressed with a large amount of uh of messages that were uh classified as uh

uh this burst of messages and classified uh detected as a spam by uh filtering and this was the this uh template x where this was discovered by google researchers uh another example uh which is slightly different is sort of an availability attack uh this case the victim is a spam based spam filter which is an open source spam filter based on uh on tokens on building a dictionary of tokens that are correlated with the spam messages but what the attackers try to do here is not backdoors but to do the other way around they wanted to pollute the the spam dictionary with good wards and this by the way not attackers these are researchers um

but the the researchers were able to uh to show that even with control over only one percent of the data used for the training they were able to uh to make the model classify 80 of the good messages as a spam and in a three uh class um spam detection where you have a spam benign and and unsure they were able to create to make the model uh detect 95 percent of the of the b9 emails as a spam or unsure both both results make render this spam detection model completely and unusable so probably like myself at the beginning you looked at this example and then you say okay we gave the attacker the possibility to

provide the data and also to do the labeling this is a significant power we gave him to do the labeling if we wouldn't have given him the power to label then the problem was not here right wrong because there another variant of this attack works also when the attacker does not have any control over the labeling process so assuming here that the uh i don't know the owner of this image classification is has some employees that are trusted the labelers and now the attacker wants to convince uh to make the model to make the training uh create a model that will identify all these images at the top of fish to classify them as dogs so what the

attacker does they take this image of uh of a dog on the left side and now he adds to this image invisible noise that is somehow correlated it reflects the structure or different properties of the images of fish off the top for us we still see an image of a dog and similarly to us the trusted labeler he looks at this image and he says okay i see here a dog he labels it as a dog it is being fed into the training as a dog an image of a dog however uh when the uh the correlation of this image with these images of a fish makes the now the model uh identify all these

uh images of fish and also as dogs which is again exactly what the attacker wanted that to achieve and it works in both ways in the top you see images of fish classified as dogs and in at the bottom images of dogs classified as fish the results are pretty impressive you can see in this in this diagram that um uh the clean model classified all these images of fish with zero uh with very very low confidence as dogs and after the attack then the classif the classification success confidence was very high 90 95 and more and again the most important thing here is that the attacker has zero intervention in the labeling process so we talked about the threat let's talk

a little about them about the mitigation uh so there are two prevention mechanism approaches which are pretty straightforward um the first one is to filter suspicious data for example if the data comes from suspicious origins like ip addresses or users that uh we know that we have some reason to suspect them or in some cases data points coming from bots and not from human users they can be filtered out if you think of again of booking then data coming from a non-authenticated users can be uh they can cannot provide uh ratings uh another approach is a fault tolerant data sampling uh to limit the impact which can be the number of data points or the

the weights the influence of these data points are coming from a single entity what is exactly an entity that depends of course on the problem the domains on the day of the nature of the data and things like that it can be entity can be a user it can be ip address and i will given other examples later on other mitigation approaches which are detection approaches by nature is less effective is to look for significant difference from previous model when you have several uh when the model is evolving then every time you have a new model you can compare the smaller to the previous one if you see significant difference then you can deduce you can conclude that you are under data

poisoning attack uh you can in some cases use some kind of a golden data set that every time you have a model you uh you test it over the data set to make sure that you are uh it provides the expected results uh however in in in many cases you build machine learning models that come to that need to adjust to the reality uh and the reality is dynamic and changing in these cases um these detection mechanisms are are less effective um and of course there is a security web security you can assume that the attackers don't know exactly what you're doing they don't know what are the machine learning problem you solve they don't

know what is the algorithm that you're using and therefore you put your hand in the sand and you can say okay they will not be able to attack me uh however we are in besides uh we know that security by obscurity rarely proves itself as an effective mechanism security mechanism again against actually nothing ah another area which i think worth and mentioning um when we uh there is no you know a new domain of a confidential computer computation uh when people er entities executes logic and data without knowing the data there are several approaches some based on cryptography like uh fully automorphic encryption and functional encryption and npc some based on clustered execution environment and secure enclaves

some on advanced data science technology like federated learning the data poisoning threat is is still there there is no guarantee for protection against data poisoning uh there and even moreover any mitigation that is based on processing or validation of the inputs uh becomes significantly more complicated if you go back to this example of of simon biles then uh just think that now we um with the uh the training uh the entity that does the training no does not see the 9 and the 9.2 and the 14. it does run some obfuscated logic that runs on obfuscated versions of these data points so it is become significantly harder to uh to to identify that these inputs are

invalid so let me summarize what we have so far so a data poisoning is significant threat on learning mechanism it is critical when using data from untrusted sources with or without trusted labeling and there is no silver bulk mitigation but there are several mitigation approaches that can at least um reduce the risk or throttle attackers make the attackers work less effectively and whenever you are using uh data from uh untrusted sources and your model is doing something that is meaningful for someone and someone has an interest in making the model make different uh decisions and i hope for you that everything that you're doing is is is uh um making decisions that are meaningful for uh for someone

uh and this is the case whenever you have a rating system whenever you're using machine learning for any security solution a firewall spam filters malware detection all of these are situations where the data poisoning threat is is real it's significant and it will be there it will be implemented it will be realized and you need to take steps again in order to mitigate it uh next time we move to the area of web security so most of you are probably familiar with web application firewalls and how they work but for those that not i will explain that um a waf resides between an application or api and the outsider world outside their threads aka and the

internet waffs usually are uh based on a combination of two models a negative security model and positive security model the negative security model is based on under on the assumption that everything is good except for things that we know for sure that they are bad sometimes it's called rule-based or signature based it has its pros and cons i will not get into the details of that and the other family of uh of uh of web protection uh mechanisms are a positive security model where we assume that everything is bad except for what we know for sure is good for what we learn that is good so this is sort of anomaly detection uh and in order to get

that uh we need to learn a baseline profile for the web or api traffic and to block or alert whenever we uh identify deviation from uh this profile but when i say uh to learn a baseline profile of web traffic traffic comes from outside outside it means that we have everything we need for data poisoning we have data coming from outside from untrusted sources and we make decisions that are meaningful for someone uh so data poisoning uh is there so how does a web or api traffic profile look like so this is very sort of a sketch of how how uh such profile can look like uh in the center you have these objects marked in red which sort of

think of them as parameters can be create string parameters can be body parameters can be cookies we can think of them as sort of off parameters each of them is associated with containers uh which can be host url method or a combination of this and each of them has a traffic profile for this for this uh parameter uh traffic profile is likely to include a collection of of features of attributes that are uh somehow related to uh to web attacks uh like the type of the parameter the multiplicity whether it can come on once or several times within a single request whether the parameter is optional or mandatory if it is a numeric parameters then what

are the range of the of of this parameter of the values if it is a string parameter then what is the range for this the length of the string or what are the character sets that is uh being uh that is permissible for this uh for the values in this parameter and stuff like that um so having that uh again and a flow that that makes sense at least for me for using such profile then first use mitigation one which was this filtering cleaning to filter suspicious traffic data points that uh for some reason there is a uh uh that there is really to think there are suspicious or malicious can be uh coming from suspicious ips or

uh events that were identified as suspicious by any other different uh rules uh it can be it can make sense also whenever you have an attack to assume that to throw away all the data coming during this attack even if this data looks benign looks legitimate and in some cases it makes us also to figure out all the traffic coming from automated declines next uh doing the learning uh doing a threshold learning uh mitigation two to uh limits the impact of every particular entity uh and again what is exactly the entity you can use several notions you can do several of them together for example uh to um in order to limit the impact of a single ip you can say

that uh we learn things only if we've seen them from uh more than x1 unique ip addresses or more than x2 user agents or more than x three geo locations etc if you want to eliminate uh to reduce or to um to limit the impact of a particular hour because you don't want to learn something that happened many times within a single hour because this is uh this is how uh uh good characteristics for for attacks and then you can also look at the anti at the edit hour or day as an entity uh and then you have this set of learning and eventually you will be the profile and you have in the enforcement you alert whenever

you see deviations from this profile now when you look at this threshold learning that i described then oh it is fairly easy when you have batch processing just like probably a single or a couple of sql statements and there you go you have all this filtering that you want however batch processing is not always possible uh you don't usually build a profile based on the traffic you've seen in five minutes probably a day maybe a week maybe a month and you don't want to buffer all the traffic you've seen in a website a long uh a complete month and uh there is a need for uh implementation of of such mechanism that work with the low memory works when you see

the data points one after the other and you don't want to buffer all of them together in order to present the streaming-friendly algorithm i will take you to completely different uh words completely different problem of a dog food tasting challenge we have two dog food brands we have a tio and pedigree and we run sort of off a poll of which one of them is uh is preferable by by dogs and their owners and we are getting the raw results that we have 12 likes for teo and six likes for a pedigree however we want to prevent bias we are concerned about the path of the data we are concerned about data poisoning so we want to take a threshold learning

mechanism so we want to accept that particular uh brand is good it's tasty only if we see uh testimonies indications for at least three cities and at least three uh dark breeds and here only pedigree passes even though it has a half the the likes of nto and if you look at the data then you see that uh few has many votes from pomeranians and many votes from new yorkers but zero for some bernard and zero for san francisco so it has only two cities and two uh breeds while pedigree have a small number of positive votes from every brand and every city so it passes the threshold and the reason for that is that our data

was towered towards the pomeranians and new yorkers and they have a very strong tendency to love theo which is exactly the phenomenon that we wanted to to eliminate its impact uh so how uh it is going to be implemented so we have a pedigree is an object tastiness is a fact a boolean fact that binary boolean facts about this object and uh there is uh two attributes that we are looking at one of the c attributes of the of the votes one of them is ct the other is breed and we have threshold of three for each one of them so for each one of them we need to remember the set of cities and the set

of breeds uh from which we've seen tastiness indications uh for pedigree so we hold the same structure for theo and the same structure for other facts like the nutritiousness of of the dog food brand uh and now we have data coming in and after we see the data then we uh we have a set of three cities and three brands for pedigree so we passed uh this threshold test and tastiness we have uh tastiness as part of the profile however for teo we have only two cds and two brands and two breeds so we don't pass any of the tests and tastiness remains not part of the profile uh so what we we've seen here

that we can learn a boolean facts tasty or not tasty that an object x in this case dog food brand has a property y in this case tasty or not now the memory consumption of our solution is proportional to the number of objects the number of properties and the number of attributes and also to the threshold because you need to uh to remember the the set of cities and brands however the more important thing and the most important thing is that it is independent of the size of the data it doesn't we don't care if you've seen a million uh votes or a billion votes or a zillion votes uh still the data structure is not uh

affected by this um by the the size of the data which is exactly what we wanted to achieve in the first place uh how to use this for a web or api profile well it is again pretty straightforward during the training building the profile and every time you see a data point a request then you extract uh that the fact x was seen on this request now you collect all this fact x scene together you check whether they pass all the the relevant thresholds and then you have fact x allowed if you passed all the thresholds or fact accept or actually exclusive or fact x prohibited if any of the threshold was not met uh during the inference and you have a

profile you want to use it so every request that you're seeing you extract a fact x scene um and now if you have a fact x scene that corresponds to effect x prohibited then there you go you have a violation of the profile you have something that you've seen and it's not part of the profile and now you can do whatever you would do with alerts you can block you can alert you can yeah you can do whatever you yeah you like uh so you are convinced now that we can do boolean facts but what this looks like pretty limited model what can you express with boolean facts uh in the next couple of slides i will

try to convince you that you can express pretty many things with boolean facts now when it comes to the objects to the existence of parameters and their correlation with combinations of containers digital location host and methods and this is pretty straightforward because they have a boolean nature exist or not exist so if for example you want to express the fact that you've seen a url x under the host y then you have a boolean effect url x at host y allowed if you want to create a parameter x to url y with method set then you have a query string parameter x and url y at method with methods that is allowed so these objects and

containers and the relation combination of objects and containers actually they work pretty well with boolean facts and no problem here but what about the traffic profile we have data types we have enums character set ranges how we deal with this uh so in order to explain that uh i will let's start with uh with the type okay um so we have um the type can be numeric or string or boolean or none it can be also a combination of this because parameter can include can have can carry several different types so for a numeric type we will use a four boolean fact four flags one of them will be a num type allowed and it's exclusive or it's a it's

um the opposite would be a numeric type prohibited and we will also have non-nun type allowed and non-num type prohibited and we uh it is possible that both num type will be allowed and none on type will be allowed it's also possible that both of them well it is less likely that both of them will be prohibited but each one of them has to pass all the threshold tests we did the same thing for uh strings we did the same thing for for none meaning that the parameter contains no value and uh for boolean and now uh let's look at um a property like type equal strings uh then it is likely that our profile

will learn str type aloud but it will not learn a numeric type allowed so it will learn numeric type prohibited and it will not learn non-str type allowed so it will have non-str type prohibited because all the values that we see all the time are strings or maybe we're seeing from time to time something that is not a string but all these non-strings are not passing all the threshold tests so they are not part of the profile and now whenever we see x equals abc or x at gmail.com then these are strings then no problem if we see x equals 23 then this is a numeric so it uh contradicts that numeric type prohibited and we have a relation

if you want to learn that type is a male address but the type is you can only carry strings that correspond that match a regular expression of a mail address um we are seeing only uh email address uh structured values so sqrt allowed and also we have a flag male regular expression allowed however non-male regular expression since we are not seeing non-male or non-male regular expression strings then this flag will be prohibited so whenever we see x equals abc which has i'm sorry

uh which uh has non-male regular expression because abc does not fit and does not match a male regular expression we have a contradiction with the non-male regex prohibited flag and again we have violation if you want to express the fact that the parameter is mandatory then we have a flag missing that will missing prohibited as part of the profile if now we didn't see that this parameter coming missing then missing will be the missing flag will be prohibited and whenever we will not see this parameter then we'll have a violation if the parameter is optional then uh we have the missing flag uh um allowed and there is no problem when x is missing uh when the parameter

not allowed to carry uh value then uh we will have none i i hope that in this point you understand that the left side the green flags are not really important because they are almost meaningless the more important ones are the red ones so when we want to express the fact that the parameters should have no value then non-non-type prohibited is the important flag of the profile when you say want to say that multiplicity is prohibited then multiple occurrences is prohibited so if you see x x equal three and x equals four then it will have multiple arcs in and it contradicts the multiple arcs prohibited uh when it comes to uh character set when things become slightly more uh

complicated you have to do some kind of something that is called the one hot encoding in data science world so you take the set of special characters and you replace it with a collection of flags each representing a particular special character so if you want to say that you allow alphanumeric plus colon and semicolon then you will have a 3a ascii 3a allowed and asked 3b allowed but all the ascii codes of other special characters are prohibited [Music] when you want to express ranges then uh you can use uh flags you can divide the sort of discretization of the range and you can use flags that um a greater than flags and lower than flags

uh and you get not exactly not fully uh maximum and minimum of the range but you can get almost very near um uh you get um flex a profile that is very close to uh to actual uh ranges so we've seen how we can learn uh boolean facts so if the profile has put in fact and uh then we're good uh but what about other actual machine learning models so i will speak about how this this concept this approach can be implemented in uh the area of the class of models that is called decision trees and and forest uh so decision tree essentially as you can see in the right side it is a very pretty basic machine

learning model uh for example you want to know whether an animal is breathing air based on uh i'm sorry whether an animal is a fish or bird or mammal this is these are the classes and you have two features two bullion features breathe air or lay eggs then uh this decision tree is used to uh to make this a decision uh and since decision trees are pretty basic models and are not usually not sufficient to uh to express the comp the complexity of the problem you want to solve in in many cases you use a combination of trees uh some of these combinations are called bagging exam bagging and symbols and some are called boosting ensembles so

essentially beggings uh samples are based on uh taking the the data samples you have and build uh independently make collection of models and then the the the result of the of this super model is the average of all decision of of the models um when it comes to boosting uh ensembles then uh you actually build the model uh gradually in the model evolves uh based on the on the previous model and continues to improve usually based on samples uh areas data samples that uh the original model made the incorrect uh decisions um so how can you i use uh this um translate these decision trees into into boolean facts uh so i'm going to present

one way to uh to do that but definitely this area um deserves more uh more research and more analysis so at the beginning you you build a tree you build a tree same as before it can be in batch in streaming [Music] whatever you like at this point you add uh sub leaves um to each of the of the leaves of the tree you can see here this is a typical example from from sk learn uh of uh of a planned uh classification based on on uh three features um so you add the establish so for example on on the left side you can see a leaf uh where 47 samples of the train training data have reached this uh

leaf it was classified as uh versicolor so um since it there are 47 samples so first the color then we have a green uh sub leaf sub leaf to this uh to this leaf and now uh we do the same also for the internal nodes uh so for example this guy here i hope you can see my cursor um this guy here it has um both uh versicolor and a virginica so we had add two uh sub leaves to this one and now what we do uh we use uh this uh threshold learning for each of the sub leaves and moreover to each of the paths to these sub leaves so for example since this uh subleaf it has 47 samples

it is very likely that it will pass all the threshold tests and will be uh part of the of of the validated tree however here uh the the green sublease it has 47 samples which is okay but the purple sapling has only a single data point so probably didn't pass threshold so it will not be validated it will not be part of the part of the validated uh model and now we actually have a tree okay and we this tree is valid however we have some odd situations if we reach at this point for example we have two decisions if we reach um at this point we have no decisions so we uh and before that we had you know uh we had an

exclusive decision now sometimes we have exclusive decisions sometimes multiple decisions sometimes no decisions we have somehow uh to deal with that but there are many ways to to to sort to resolve this so i'll not get into the details down here um so that was for decision trees uh what happens when you go to one samples so in fact when you are doing a bagging then it is pretty straightforward because when we have the models the models are decision trees so you do validation of the trees and now remember that we have multiple decisions sometimes and we have no decision sometimes well this is not a problem because we anyway do ever aging so during the voting

whenever we have multiple decisions we can count uh several votes whenever whenever we have no decision we can count no votes and and it still makes a lot of sense when it comes to the boosting and symbols well it you can also use this validation whenever you have a tree but uh but there is no guarantee that the process will evolve into into a good model so more research is is required for uh for boosting and samples so let me summarize uh data poisoning is a significant threat on learning mechanisms threshold-based learning may provide an adequate robust learning solution again it has to be uh you know exactly tailored to each particular situation and and problem and data but but

it can provide in a decorated solution uh the boolean facts framework provides a streaming friendly implementation for threshold-based learning and many features can be expressed with boolean facts uh and eventually a traditional learning of trees in forest is partially possible also with boolean facts thank you very much and now we have a couple of minutes for uh for q a