Large NLP Machine Learning Models and Their Upcoming Impact on Cybersecurity

okay cool hi everyone i'm jeff mcdonald today i'm going to be talking to you about large generative language machine learning models and their upcoming impact on cyber security i'm jeff i manage i manage a team of data science data scientist teams who work on microsoft defender for endpoint we work on both the antivirus product as well as the edr product and network protection product i'm personally based in vancouver canada a lot of our data science teams are based in based in redmond and across the u.s although we do have some data scientists researchers research teams here in vancouver as well and we do have a lot of uh security and reverse engineering teams actually based here in vancouver

as well cool uh yeah so um i guess about me personally i have a background in cyber security i've done like reverse engineering uh particularly on the antivirus product and i've done like i've done a bunch of deep security research for probably about maybe five years before transitioning to data scientists data science data science which i've been working at for approximately the past about five or six years now cool uh so my talk today is going to be uh introducing you to some of the large scale network the large generative language models uh so i'm specifically going to be talking about the generation task large language models today so i'm just going to enable my laser

pointer here so generally generally these are these these are i'm talking the generative language models which generally read text from left to right so like the malicious document is and then they sort of predict the next token that follows a token is sort of like a word however less common words will be broken up into multiple tokens so a token can represent a whole word it can represent a set of characters or like a single character even as well so for example it might predict the next token in this is any one of the following words and then it's going to randomly sample one of those words in this case it's going to say that the

next word is bread so this can be repeated this can be repeated multiple times to auto complete whole sentences or whole paragraphs or you can get to generate whole stories all by itself for example so if we were to predict the full sentence here we can say this document is read via fifth phishing emails period so you can see for example phishing was not as popular war of a word in the vocabulary that the model is trained on so it actually split into two words for example so these large generative generative task language models are trained to predict the next token from a very large collection of text and then they read the text from left to

right and then they predict the next token that follows and they can be used for many different sort of tasks that we're going to see here today so these models are generally trained these large these large language models are trained from very large corpuses of text data one of the most common training sets is called the pile which is almost a terabyte of text data the training data comes from many different sources like the the largest one is common crawl which is like for crawling of websites but it trains on the contents of books github research papers you'll see like wikipedia is included here so it actually trains on a mix of both text as

well as code content and actually through this training process gets a good amount of understanding about a broad variety of fields including cyber security so this can be used in like you can do question answering on cyber security documents like what is a supply chain attack and it will auto complete a paragraph describing what a supply chain attack is for example cool now i'm going to quickly cover what zero shot one shot and fu shot learning is so with these large generative models you don't actually need to train the model to your task you can actually just use these really large generative models to to apply to your task without any training without any label data or

anything so the first example is like imagine you want to translate english to french so you might use a template where the text is english colon and then you'll specify you'll write in text the english sentence that you that you want to translate and then you type french colon and then get it to predict the next characters of the output the next tokens that are output and you hope that it understands that you're trying to translate that the french colon is sort of going to be the friends translation of the same sentence so using this you might say english our cloud service will host 10 nodes french colon and you predict the next characters which is the output and you

hope that this is the proper french translation you actually didn't train the model to do english french translation you're just using the the next token prediction in order to get it to do the translation text and more like generalized intelligence so few shot learning is where you give it uh a couple examples of the task that you're performing so in this case you give it a couple uh you let it read a couple translations first so english french english french english french and now you apply your translation task so here it has a bit of an understanding of the format of the output that you're expecting and you'll get a much better result usually so now you apply the

translation task with what you actually want to translate so again you haven't trained the model you haven't provided really any label data you've just sort of provide a template with a few shot examples so here's a quick plot of like you know how much that impacts the quality of the models if you do zero shot you can see the quality of the models perform not very good according to the accuracy in some of the tasks however even if you give one a single example of the format you expect it gets way better results and it gets quite a bit results up to 10 10 shot examples and and beyond that it gets even more improvements so again

you're not training the model you're just providing a template with a few shot examples now you can actually train the model to fine-tune the models as well to your task to your task as well so imagine you might have maybe a hundred english to french translations already that maybe even used a few shot or zero shot learning to help you create because maybe you don't know french at all but once you get like a few hundred examples you can maybe do a few trainee epochs of these large model on the data so you're just slightly adjusting the really large model to learn from your specific task and then you might use it in jira start

learning which makes it a bit cheaper to inference the model okay now i'm going to give a couple examples of what these types of machine learning models are used for uh one one that you probably see the most is probably auto complete so for example if you're auto completing emails maybe automatic suggestion for what you should reply in a text message a lot of these would be using these type of really large machine learning models and here's an example of some security problem on them if you're training obviously with customer data it could reveal some some information if you've seen this xkcd comment it's pretty good it's also used a lot in it's also starting to be used in code writing so i

highly recommend checking out github copilot this is uh this is more in code generation and i've given it a try i've used it in some of my personal hobby coping and it's been it's been amazing it's been blowing my mind with the proposed solutions that it goes goes that it proposes and it's an extremely useful asset already in coding so basically you can write the definition of the function maybe a comment of what it does then it's going to write the whole function for you or it's going to propose many different solutions the same function for you you can also use it to i often would use it more to write code filling just part of the

routine like you might write a comment this part i'm going to take the data frame and i don't know featurize it in a certain way then it will suggest the code to autocomplete that for you and it's been kind of a really amazing experience i highly recommend signing up for the beta and trying it out with vs code it's also used for example in uh games uh starting to be used in more games there are some text writing uh text writing games where you can basically uh interact with a more like a choose your own adventure game where you can write whatever you do in the world and then it will generate what happens in the world

in response to what you said so it's a completely open-ended choose your own adventure text generation games and i highly recommend trying them out it's quite entertaining uh so in terms of these these models that do the natural that do the generation task of the large language models are extremely big most of these models can't be run on your local device even like with a really large gpu they generally have to be you can't load the whole models onto a single gpu you have to run multiple gpus in order to even inference them one thing to note here is probably most of the largest models are closed source they're not open source the largest open

source one that i'm aware of was released a couple months ago the gpt neox 20 billion is a is the largest open source model now pretty much anything 12 billion above is applicable for most of the cyber security attacks that i'm about to demonstrate especially when fine-tuned to the application now what i'm going to be demoing today is the openai gbt3 davinci which i'm going to be demoing for for all the attacks here today cool so now i'm going to start with all the fun stuff i'm going to show about how all the cyber security risks these text generation models have at least some of the ideas from my own experimentations that i see is the risks

coming risk to cyber security and i'm going to demo each of these attacks as well so the first one is i can't move my mouse anymore okay cool i think i froze i hope everyone can still see me perfect good okay so so the first one i'm going to demonstrate is e is phishing targeted phishing emails at scale so this is the the opening at playground that i'm going to demo with here

perfect so so what this what this is is we're doing a few shot learning so i give a couple examples of of targeted phishing emails you specify the target like the company the company of the target for the target phishing email attack you target the person you're targeting within the company you talk about the role within the company and then you provide the payloads url that you want them to click on for example so in this case i give a total of one two three i give three examples and then i then i carry out a generation task where my target is gonna be from walmart i'm gonna be talking targeting james scott who works in public relations and this

is the link i'd like to use and now you can sort of generate uh you can generate these phishing emails and each time you generate it it will generate a different phishing email i'm just going to let it finish here i think it's still generating perfect there we go so now here's an example of the target efficient email i'm going to try regenerating a few times to see if i can get a more pr focused one often we'll get some better examples here

perfect so here's another target phishing email i'm going to cross my fingers i want to get a good one targeting pr perfect okay yeah this looks like a pretty good one okay so it says i'm gonna wait for this to finish generating perfect so i hope this email finds you well i'm a journalist with the wall street journal and i'm writing an article about walmart's public relations strategy i'd love to speak about this topic and your insights you can find more information about my article my website here i look forward to hearing from you and you can really write whatever name you want and it's gonna it's gonna generate a target email to the specific target

okay cool um yes back to presentation i'm sorry it looks like my powerpoint actually crashed so just give me a minute

okay cool um so i'd say like the targeted phishing emails at scale i think it's an immediate high risk problem uh if you combine it with with for example uh scraping of linkedin profiles uh or for example business listings you can create targeted emails at scale specific to the person's role specific to the person and the company and i think it's going to be starting to change some of the fishing techniques that we see the defensive technologies that rely on the same large natural language models i don't think get as good of a benefit as as the attackers unfortunately in this case because the emails that are generated with this are generated to more closely look like a

real human that's writing the content i think a lot of the defenses in this case are going to continue to be based on like sender and url reputation in order to detect and protect prevent again protect against these attacks cool so now i'm going to apply it to the next task i'm going to be talking about how we can obfuscate powershell script perfect

perfect so again we apply like a few shot learning template here we apply to we give it two examples of example of powershell script uh unobfuscated and then we provide an example of it obfuscated so in this case we've added a whole bunch of very basic string obfuscation in order to try to avoid av detections or edr detections uh and then another example of using invoke invoke obfuscation uh here's an here's an example and then you can then here's an attempt of trying to use it to obfuscate uh this input powershell command and in this case we give it a hint to use super advanced obfuscation in this case so now i'm going to generate a few

examples using this sometimes you'll get like almost no obligation sometimes you'll get some sort of basic obfuscation and sometimes you get completely wrong solutions like in this case that was not a not a good real answer there's again no obligation at all in that case no i think we're starting to get a basic obfuscation example once it finishes perfect cool so now here is sort of a sort of simple more simple obfuscation where they've sort of hidden the characters involved with it so sort of my assessment of whether that would be sort of the use case to create obfuscate a powershell script i don't think it's uh i don't think it's a very i don't think it's a very effective

method i don't think it's an improvement on the existing attacker techniques in the way that they obfuscate scripts i think it i think it doesn't model the way that the attackers generate their obfuscated scripts either because i think they're writing coding routines like like you can imagine like a python function which will which will do the obfuscation of the powershell script so it doesn't really closely match the techniques and it can't match the same complexity that would be involved either so i think instead of instead of using using it to directly obfuscate the power skill powershell script i think what it's going to be used to do is to author code which obviously it's powershell and

i'm going to give a demo of that now

perfect so in this case we're starting to define a python function build malware loader and then you specify you specify the url um that you wanted to download the payload from you give it a specific you give it a short comment as what you want the function to do so in this case we're going to build an obfuscated powershell command to download and execute an executable from a specified url so then you get it to author powershell script here which sometimes will have a decent output i'm just going to let it finish generating this function perfect i'm going to go ahead and generate another one as an example

and i'm gonna try generally one last one after this one finishes perfect yes so sometimes it doesn't do a very good job um here's one of the best examples i got for code generation this one was from github copilot actually so it's the exact same function and it actually was able to output a solution which uh ignores it with uh exhaust each each of the bytes with nine for example these xor encoding and then it actually authored a powershell script for example here which would go through and ignore each byte by nine to reverse it and then execute the powershell command which would then download the file and start the process for example which is a pretty good

example but you do get a lot of a lot of cases where it creates uh failure failure cases where it won't generate successful uh script content so but i think you can combine it with uh unit tests where you can generate uh you know a hundred thousand or tens of thousands of these routines and put them to unit tests into uh scripts functions that actually generate powershell code that successfully executes that url would be the test and then you can basically get it to author many different python functions which will generate python or powershell loaders for you with many different obfuscation techniques so that makes it that that makes it harder harder for the defenders

in order to detect the attacks because of the wide variety of techniques that they'll be using to obfuscate their attacks one thing to note is it does understand like it can begin to understand more complicated asks as well like you can say build an obfuscated powershell command to enable rdp and send credentials to specified url now this is too advanced for whether models can be right now but for example the models do recognize how to how to enable rdp in the device for example so it starts to get some of the more uh complex parts already you can also get it to generate like other coding languages it doesn't have to be powershell you can get a generate

office macros vbscript or javascript loaders as well so i'd say like this is it is an immediate risk but i think it's not a great improvement on the techniques that the attackers are currently using it might increase the diversity of the of the script attacks that we see but i think it's not like a great improvement a lot like a humongous improvement upon the existing attacking techniques here um this technology can be used defensively in this case as well and i'd actually argue that maybe benefits the defenders more in this case because the you can either generate a large number of diverse attacks using this approach as the defender then you train a model to classify all these

attacks even if though they haven't been observed yet or you can train like a large nlp model using similar transformer models to do in binary classification tasks because these often these powershell scripts and macro content really isn't doesn't look doesn't isn't trying to look clean so it makes it fairly easy to identify as malicious by these large language models okay cool now i'm going to go into more of a futuristic one which is automating a human-operated human-operated ransomware attacks on an enterprise i'm going to go ahead and demo this example

perfect so for example you can give the you can give the model a little bit of context of what the objective of the attack is you can say here we are carrying out an attack using metasploit on an enterprise the attacker goal is to infect a machine elevate privileges and move laterally to the domain controller afterwards we'll push out a policy to encrypt all devices on the network and you could maybe the attacker could even specify the exact url to download to download the payload to do the actual ransomware encryption which would which enact so that they'd actually encrypt it with the right key and everything so you could add that the attacker can add that to the description

so then you can get it to output like the first commands that it would do to create the the fishing kit um i'm going to go ahead and retry this a couple times

perfect so this obviously didn't generate the right code it's trying to exploit the internal blue blue blue vulnerability but i'm going to go ahead and assume that was sort of right and then i'm going to assume that you have like a reverse shell then you ask what your next step is going to be and then it's going to generate the next steps once you have a reverse shell on the device so it says it's going to execute a privilege of escalation and then often it'll do like a credential access yes than it does maybe i'm actually not a super expert at metasploit but i imagine some of this will be somewhat right but i imagine it

won't be it won't be actually technically correct so then for example it'll run the credentials your cadential manager let's assume that that this would then give us the credentials i'm going to go ahead and go to the last stage of the attack i think it gets really interesting perfect so let's say now for the last stage of the attack the last stage of the attack i think you could effectively you can basically effectively loop a giving output of the command that they give you successfully got the credentials and then next i'll do the following so basically you do the next action of the steps then you give the output back to the model so it understands the output

of the tool that it runs and then you repeat the step indefinitely in order to let the ai start to learn learn about the environment that it's attacking as well as respond to the output of the tools in more like a human-like way so perfect and i guess it attempted to escalate privilege again here perfect so basically once you once you you basically repeat this loop of you see the output from the tool and then you ask it what the next commands that should be ran so if you repeat this process repeatedly you can get it to respond reactively towards what it's seen in your environment and the output of the tools that it's running

and then it can like eventually once the mod once these natural language models get good enough it is going to be a risk to be able to automate a human-operated ransomware attack where you could be automating metasploit or cobalt right on a network and you just give it in general terms like general general written english what the what the goal of the attack on enterprise is which is to download human operate ransomware encryption from the specified url on all devices and execute it for example um i would say i would say the that this is a distant high risk on to cyber security probably in the next decade we might be able to see there

might be there might unfortunately be large enough advantages in nlp that this that could make this a real risk and that's going to completely change the cyber security game because now they don't need human experts to carry out the attacks defensive technologies benefit from these same advances however it's more costly for defenders due to the scale of this log of the security logs and these these large nlp models are really expensive to run over over a large amount of data these can also be used for propaganda fake news and social social media attacks i'm going to go ahead and show you some examples of those

so in this case uh you can define say your platform i say i'm running for the prime minister of canada for example and maybe this is the platform my election platform and then i make i describe that my tweets are going to be aggressive and attacking and discrediting the person that tweets and then i give it a couple examples where a person tweeted something and then i i give a reply to it so this is again two shot learning and now in this case someone's tweeted this jeff's plan is crazy we can't just steal crypto we need a real jobs plan and then it will generate uh it'll generate tweet replies to it so basically you can

you can automatically create um fairly intelligent uh social media interactions um social media social media interactions which uh promote specific agendas or it could be against specific people or it could be promoting like your your political candidate or attacking opposing political candidates um so this is obviously uh i think this is a current problem this isn't probably anything new that you probably haven't heard of before [Music] but i think like in terms of like propaganda fake news social media attacks i think a lot of what's been done currently is like human generate content but i think imagine some of it already is uh these large generative language models creating content towards their specific platforms as well so i

don't know i don't know what the solution should exactly be here i think like elon musk is saying hey they're going to add identity verification to all users of twitter like we can have strict identity validation in order to help solve this problem however obviously that comes at its own privacy cost as well this can also be used to fully automate starting to be possible to fully automate scam attacks i'm going to give a quick demo of what that looks like in an automated scam attack

so so for example one of the one of the facebook scams and this is one that i've seen myself as well is that the the attackers will compromise a facebook account uh usually using a weak password of some inactive user then they'll go to go and go join some facebook groups like parenting parenting groups for example and then they post like a saab story there they'll post a sob story and then the how the scam works is that they will is that they hope that people initiate conversations with them to help cover whatever whatever scenario they're in so then they start a chat with the person and they ask them to to basically send them money

is how the scam works so you can get it to create targeted scam emails which you can which then can be posted to the facebook specific space facebook groups and then even the chat conversations uh within like facebook for example could be fully automated where the ai is reading the chat in response to the story and come up with sensible replies to even fully automate the human interaction the human conversation part of the part of the attack so basically i think the whole the whole camp the whole scam campaign could be could be automated already with the current technology it might not be as effective as a person operating but i think it would have a good amount of success that

it is starting to become a risk uh so i guess in conclusion targeted efficient emails at scale is an immediate risk and it does not equally benefit the defenders automated malware loader and authoring is on the verge of being effective uh and i'd say it benefits luckily it benefits defenders more than attackers i think in this scenario automated human operated ransomware attacks i think are on the horizon uh but this is more on that like in the within the next decade type horizon which is going to be really which would be like a really game changer really big game changer for the cyber security field disinformation is already a problem and scams of the future i think are

going to start to leverage these models as well and uh i highly recommend playing with these models yourself you can try out the openai api you can sign up for it uh you can sign up for the beta for github copilot for the code generation interactions guci is a lower cost implementation of the of the open source large text generation models which you can try out and i highly recommend trying out the games that work with this technology as well yeah thank you everyone