Open Source Adversary Simulation Toolset Review

all righty hello everybody thank you so much for coming out today my name is Ryan painter one of the organizers of besides Pittsburgh and let me say on behalf of the organizing committee volunteers staff etc we are so so elated to have you all coming out of the conference this year so Rundle applause absolutely thank you thank you I will try not to clamp into the mic or scream into the mic Wow okay so I'm excited to introduce a fellow Ryan is our first speaker for the black track today so Ryan Volek will be presenting open source adversary simulation toolset review a little bit about Ryan Ryan has rainbow-like has 16 years of experience in leading and

maturing cybersecurity programs for large national enterprises with executing major enhancements of to security operations programs under his belt he's considerable experience with improving defenses via maturity assessments Incident Response and red teaming as a Pittsburgh native Ryan started his career in retail and in higher education he's currently working with one of the largest health insurance providers and healthcare insurers and providers in the United States among many some of Ryan's passions include blue teaming maturing processes and developing people so that further ado I'd like to introduce rainbow Locke Thank You mr. Paynter all right welcome your open source adversary simulation tool sets all right start off Blue Note

first time with PowerPoint sorry what come on come on

who knows what this picture is no goat simulator that's right goat simulator alright so why this is my theme goat simulator why this is just stupid funny and I like stupid funny things and the it's interesting that the game could actually simulate just about anything you could simulate microwave all kind of different things it's just very silly plus my eight-year-old really likes it I probably should have put in there that goats kind of have to do have a like similar similarity to like evil things I guess so I don't yeah that's my theme alright so agenda today we're gonna go over the goals of the presentation we're gonna talk we're gonna get a hopefully i briefed

adversary and simulation primer we're gonna go through for practical use cases or use kit uses of these tool sets we're gonna remove five not four but five open source tools I added one and we also discussed the future I know by like goats email actually started out as a joke from the software company internet went viral with the video and it became a big thing so who me Who am I in this order I'm a Christian husband father and the manager at hmmm health solutions I've got a few experience got a few creds and this is actually me and my besides 2017 shirt and pieces just like five days ago had a great great time

first time besides talk yeah so so goals I want to help you understand the practical benefits of adversary simulation and the tool sets we're gonna do a quick review of the five tool sets kind of like a software review I want to encourage you the community to participate and to utilize and participate in the development of these tools I hopefully and I'm going to entertain both technical and manager levels defenders so why I really think these are some important concepts for cyber defense just based on my experience there's a lot of new things coming out like the miter attack framework where we can better quantify operationalize and and prioritize our defenses through using the techniques in

the miter attack framework I think these tool sets are in generally not widely discussed in this in the security community which is kind of why I want to start talking about them I you know I learned they what they really learned a lot through this process and I wanted to share the practical uses and I really want to give back to the community and by the way I'm Noam by no means an expert I'm really just a InfoSec guy who just ran through these tools just to learn a lot of things I'm just sharing my experience so throughout the talk you're gonna hear me say a few different words though these words mean the same thing so when I say adversary I

meant I mean threat group or a threat actor or threat group I may say either one when I say simulation also means emulation when I say test it also means a miter technique or an attack so just as you here listening to me talk that's sometimes the terminology that you may hear so we're gonna Stoff a quick miter primer I hopefully everybody in this room knows this there's 244 techniques classified under 12 technics that classify the latter half of the cyber attack kill chain we're gonna be focusing on techniques today because that's what these tool sets do many of you are familiar with the pyramid of pain for cyber defense or an art of intelligence artifacts where as you go

up the cyber I'm sorry the pyramid of pain it gets tougher and tougher as you go up the the the pyramid I put at the top adversaries they might be the toughest part artifact I guess to identify however the good news is our controls and intelligence is getting better at TTP mappings specifically to the miter framework and we're also getting better as an industry with that identifying our adversaries so how do you identify adversary I have a series other there's lots of ways to do that a lot of organizations three-letter organizations through other government EDR tools I mess P vendors they're all coming have a lot of their databases of intelligence is getting better there are 86 adversary

groups currently published and miters website so this is an example of an adversary and the information that's available through the public miter reference this is deep and a lot of people know about the Panda it gives you some explanation of what the attack group is and then the techniques utilized and the software utilized all public publicly available the defining adversaries is is tough because it utilizes concept called technique cheating which is what I want to kind of focus on but also in addition to technique chaining but then reviewing the mission of admission and objective of an adversary but we're gonna focus on technique chaining because that's what these these tool sets focus on and this

it this graphic hopefully kind of explains what technique chaining means as throughout a cyber attack throughout the kill chain of a cyber attack an adversary will look like this we'll use these behaviors or I'm sorry these tactics to identify behaviors I'm sorry these techniques to identify behaviors so adversary simulation why is it so important what is it so simulation is emulating the technique chains of an adversary that's most likely to target your environment so we're not switching the focus on just the TTP's and artifacts we're switching to looking at all the TTP's combined and looking at the behaviors that are observed in a cyber attack so why identify and stimulate these adversaries well you've heard it said by

famous researchers that the blue team's only need to detect once throughout the cyber kill chain a defenders primary goal should be to increase that probability within that kill chain to detect that attack identification to identifying our adversaries allows us to prioritize which techniques and cyber controls we should focus on there's more information a lot of great information available on the web it's a whole nother talk in itself about the process of adversaries thing but I thought this was a good one to reference this is just a this is a talk on sands websites but attacking the status quo threat based adversary emulation with miter attack by Katie Nichols and Cody Thomas it's a five steps of adversary simulation the

process and we were focusing on that last step here today so the tool sets all these tool sets discussed today Maps simulated attacks to the miter techniques all these tool sets are frameworks that include typically include a sample of tests and the community is encouraged to participate in developing those tests all these tool sets utilize typically utilize scripts for executing commands on the hosts some these tool sets include binaries or in bi and executables that come with the test some of the tests get binaries from the internet to execute some of these tests require dll's or separate executables my point here I guess this is some of these tests actually include the binaries some of these tools include

the binary some dump all these tool sets are typically safe and typically include cleanup functionality all these tools have the ability also to create technique chains to simulate an adversary or at least define a strategy to help simulate a strategy or simulate an adversary all these tools are either locally executed or an agent client-server relationship with an agent that communicates with the server or through a remote access tool it's interesting that none of these or very few of these actually include adversaries to simulate so it's very interesting on that they do give samples a lot of them do give samples I still probably wouldn't recommend it running for it in production environments unless you know that the tools really well and

what they do so let's go over for practical uses of these simulation tool sets the first one is supporting purple team testing so in my definition of purple teaming and once summed up in one sentence the tester says hey I did this did you see it in the defender responds yes or no so primary so adversary simulation tool sets are the primary tool set for red teaming I'm sorry purple teaming adversary simulations also known as threat based red teaming that means when a red team and engagement would like pretend to be essentially an adversary and utilize the techniques that the adversaries utilize so pen testers I am hearing to starting to consider use utilizing these tool sets one to help

communicate their pen test reports and the techniques that they utilized to their clients another use that there possibly utilizing utilizing is assisting the customer training the customers to utilize these tool sets the validate and test for mitigation and validation post penetration test and some I've also heard that some are using a pen these tools as a penetration test tool in its own due to some of the automation and variance that these some of these tools can provide we'll talk more about that the second practical use for these tool sets is to quick test your cyber controls you really want to answer you know hey did my fancy new EDR EPP or MSSP that I just purchased did it

actually detect what it should be detecting it is actually working is it functional a lot some of these tool sets have the ability to just run all the tests and in within the tool the tool set typically that's fairly easy easy some tool sets bet do better than others you serve protect for this is very useful as I mentioned useful for testing even new or existing controls a lot of times this is these tools are utilized through technical proof of concepts for security products and services a useful idea that I thought is develop a simulated detection report for your an SSP or an EDR vendor for roadmaps for detection your organization you'll for your organization techniques that your avatar

ease adversaries utilize another thing that you could utilize is the for your quick test is testing your sock for simulating tests and how they respond you know testing also post tests post tests can also help with defining training opportunities within your sock identifying which logs and the alert names and make sure they're intuitive to your analysts we go also look for improvements in your knowledge bases for those simulated tests and also you could also test the manual or automated response triage with those events the third use practical use is improving your our cyber assessment and threat hunting processes simulation improves the detection maturity measurement and that's the key is the maturity measurement of a cyber assessment or a

threat hunt process or a certain threat hunt if a control detects a simulated test then the maturity is risen for that for that specific tested technique you've seen these are the heat maps and maturity over time this is a threat detecting I'm sorry a threat hunt technique detection maturity level so through each technique that is hunted or tested you could with a properly simulated attack and detected attack should put you in a good or excellent five four or five zero to five level there are many different ways to quantify your hunt here's some really good resources that I recommend quantifying your hunt and but not your parents red teaming by Devin Kerr and Roberto Rodriguez and then ready to hunt

first show me your data is by Robert Rodriguez really good information about how to quantify your threat hunts in cyber assessments by the way I will be providing these copy of the slides at the end so you no need to take pictures all right so the last and final practical use case for these tool sets is security product development it's a key concept for a lot of security software providers like EDR and EPP NMS MSSP vendors you know they're utilized in many of these tools utilized in their QA processes for regression testing validate their product actually works so typically like run like it's inserted within a normal QA process for normal software development ran after changes

or a new tech technique detection functionalities added simulation tools are utilized to help help improve noise rates really improve relative accuracy and help with the output that the tools provide so a simulated test confirms acceptance criteria it's good some EDR and EBP developers that I've encountered actually have a one to one one to one test for each technique and function within their development processes so those are the four practical uses I want to talk about something there's an issue a process issue throughout the simulation that's it's a challenge it's called I'm gonna call it I'm sure I'm not calling it actually Chris Gates called it closing the alert loop so what the problem is after you perform a test it takes a lot

of manual effort after the test is that identified to law go into your logging system and validate yep did my tests detect did it prevent yes or no that process right now I have not found any open source tool that does that is able to automate that process so really the goal of the process to close the alert group alert loop would be to execute and generate that alert after a simulated simulated test is not detected hands-free so we want that process to be hands-free so really the desired process we really like to help address this issue is for every new technique or intelligence contained or when a new test is developed in the community execute that

test that have our logging systems detect and log that system or controls it could be an EDR could be a sim but then the sim until would then integrate with that logging source and I and identify whether that test was I Det simulated was identified detected or not if it's not then it sends out an alert hey a test was executed but not found in your log so that is something that is as that some of these tool sets needs some help with and some of them were developed for that reason all right so a little bit about the testing methods what I did so I just did some research I installed and ran a certain technique

which I'll talk about and then I started documenting what I found some cases I did contribute to the open source projects and many cases I also interviewed the primary project contributor I utilized detection lab which is a fantastic discovery for me it has four servers Windows Linux actually three windows and a Linux server included domain it's got all the free really cool lab environment I also discovered vagrant which was mind-blowing discovery for me it was super easy to set up a VirtualBox environment with just a few commands and you got yourself a lab was pretty really cool so we're going to look at for each one of these we i ran a example technique this technique actually is

utilized and examples through many of these tools it seems like to be a standard for a lot of the tool sets and how they explain how the tools work all it is is it's a defense evasion execution another way another way to execute an executable with with the bypassing without actually running an executable so register for 32 if you read anything read the read but it's a way to execute something all right show you the tools let's go so we're gonna go through I added purple team attack automation framework onto in addition to what you're seeing in the agenda and the orders somewhat shows an evolution of the tools so starting from his from earlier developed to kind of very

current all right endgame RTA so just a little bit about the format the highlights is just kind of some points a different difference but I'm just going to talk through them I also have some screenshots to show you the tool set so some interesting facts about are ta Devon ker is the author our primary author I actually went to college with him I know him real well and great guy and so he developed this tool out of end game security software product development and so it's it is a locally ran framework there's no agent there's no server or remote access tool it's basically it's on a runs off a Python core just simply just Python

scripts the tests do include the binaries and executables and what's nice about it it's it's pretty simple but again it's it came out of end game product developers really one of the best uses for this tool here is some I think it's a Python code for this test and you see there the Reg server 32 you're going to see this line and several the example codes that I show you very similar commands are executed you'll also know some cleanup there and whether they're not if the tasks this framework executes snoke pad upon a successful test some other ones an execute calc by default but this is all changeable with its scripts but this is a yeah the yam oh yeah Mel's uh no no

you know what that's Python the next tool is meta this was authored by Chris Gates who out of uber he was it probably could have been the first popular tool of its kind being in the given that the fact that it ran as an agent specifically the the biggest difference of meta is its architecture it was it utilized somewhat of a actually a client-server relationship but it utilizes vague which is very interesting the reason why I was developed was to help uber security team close the alert loop really I trying to automate that detection in reporting in one without any manual intervention it is a python-based it runs off a Redis celery server adversaries are simulated through

a scenario scripts of many actions and the executables are not included by belief they are typically downloaded as a test is initiated as I mentioned vagrants required vagrant is the rat and very good for cloud infrastructure as a service because it because it supports vagrants a lot of practical uses for improving your cyber assessment and hunting programs just because it was intended to build to close that alert loop so the architecture again is the the biggest difference with this tool set so you got your meta server at the top which works and issues and integrates with vagrant to issue commands down to each of the hosts in your VirtualBox or VMware environments there's some screenshots on the left

hand side there is the configuration in the integration with your vagrant hosts so you list your hosts and you list your location of where your vagrant scripts and files are located and on the right is and of the ammo file I think he ammo is not your markup language it's a very readable type of markup language and there you see command exe with reg server 32 and a reference dll very similar for forty-one one and seven very similar execution from the first framework but you also see other in the code you all see other information meta literally meta information in the the code you have the reference to techniques descriptions some decent decent information here's an execution of a this is

actually not the execution of t11 seven but this is an execution of just a mo file that's called on target that just runs a bunch of net user net view net group commands and just runs it on the box and here is a report of the the output of that command and the commands that were ran is there it gives you the time and the action the other thing that's interesting about meta is that it does they did build a slack integration to help again to help close that alert loop to integrate with other tools all right atomic Red Team next tool so atomic Red Team was developed out of red red Canary primary author there's Casey Smith they

it's not necessarily an execution framework for executing tools but it's more known as a library in the largest library of the tools available I'm sorry the tests available they do have at execution frameworks there's PowerShell there's Python tool and there's a ruby ruby tool that allows you to execute these tests i the real benefit for these with this that they're simple and they're atomic they're there they run within themselves there's really no other binaries to download you you just run the tool some of these as you notice there there are four hundred eighty-three tests but 182 covered techniques so that means that there are multiple tests per technique and you'll see in the screenshot next of me running

the invoke red team execution framework fraud of powershell adversary simulation is manual once again they have a scripting a sample script they call it chain reactions but i think really all they are just scripts to put tests together and make decisions I mentioned this largest largest repository out there of tests and I think I know I'm pretty sure the reason why they have had so much support from the community because their documentation and their framework for including the community is very very well done it's very organized and very good I guess they give really good guidance to how the community can support them very good processes there for testing and for inclusion if you want to develop

a test they really want the community to help this framework because this framework is utilized in other open source frameworks this is a very good tool to quick test your cyber controls but it does support all the uses of adversary simulation tool sets here's a little bit about atomic Red Team and it's in it's it's framework it really is just a zip file you X you export it and you have a bunch of folders that include your tactics and inside each tactic folder is a MD and a yamo file then you also see a binary there

here's an example of t11 7 being executed through the PowerShell framework you see there that there are three tests executed in its output I believe this is a verbose output but one two three of three for this test in my lab I'm still not quite sure why calc only executed three of two out of the three times but hey it worked I was really happy with the output here's a little bit of some code the every technique is very well documented and every test is pretty well documented as well so at the top is just a summary of what this test is and at the bottom is one to test one two three a little bit

about each test that's R and within that technique I'm pretty sure there's a way to execute a specific test within a technique I just didn't get there so here's actual code from atomic read team there's your command but also have as the yamo file again vary a lot like meta and it's the mo information gives you some information about the technique on actually executing the test description of the test and that's one one of the three tests within that amo file and I really like the documentation is on the wiki as well that same documentation is available also on the get wiki page for atomic Red Team this is a little bit showing a little bit about the chain

reaction scripts again they are just simply scripts execute executing multiple atomic tests and the thing about adversary simulation is it requires decisions to be made throughout the throughout testing the cyber kill chain but this these decisions would have to be manually scripted you if utilizing atomic red team all right caldera and this is probably one of the most exciting frameworks so we're getting more exciting as we go along through the framework for these tool sets so caldera was developed and supported by the MITRE Corp specifically David hunt he his team are really wanting this toolset to be a framework for automated Red Team operations they want to automate the testing focusing on post compromised tests there is a 2.0 version the

screenshots your is going to see are the one auto version included it from the detection lab I just did not get a chance to do that to try out the 2.0 but functionality you know they want its a GUI typically GUI its point-and-click adversary simulation they want to make it very easy to install very easy to use they want to bring your own technique they want that capability actually they have that capability they don't want programming experience you look to required they have a new open plug-in architecture that's at the released in the 2.0 you get to bring your own rat or client-server relationship tool or how you execute commands on on a host you could do that

in multiple different ways there's also some very smart planning capabilities through artificial intelligence which is really really cool supports multiple devices I think this is the most mature open-source framework I've seen however that's the development activity on caldera is very high and it's a good thing because they're making it better but with the 2.0 and one dodo are very very different so just know that mitre is certainly supported because obviously it came from mitre the best practical use is the support purple team testing so you would have this installed in your environment and probably in a dev environment to test your cyber controls and prove an assessment performing threat hunts utilizing this this framer but a

majority is that the majority of the practical best practical use for this tool is actually supporting the process of adversary simulation so adversaries simulated there you've seen that I have some numbers there I put variable for this one because it is a like a point-and-click of defining your adversary it's make it really easy where other tools utilize scripts to make decisions this tool it's it's point-and-click it's really nice reporting's very very good good GUI as well alright so caldera you know it was originally developed by the defenders but David Hunt and The MITRE team have been really pushing and utilizing into their offensive space which is really cool if you want to learn more about

caldera and the 2.0 version there's a article by David called automated adversary emulation I'm sorry it's a YouTube video automated adversary emulation available very good run through this tool very technical of the version 2.0 so here we go this is caldera one dot oh this is now known as the legacy interface and again I just didn't get a chance or time to because the two dotto version was recently published but it works very similar the concepts you're going to see are very similar to the two dotto here is your point and click adversary definition you see there you can select which techniques you want to add in your adversary I titled my adversary bad v-dog ATP 1 2 3 just just to you know

play with it and you just point and click and select which have techniques that you want to utilize throughout your tests successfully executing a test looks like this this is a visual of a compromised network each of these hosts with in my lab environment had the caldera agent on it the one auto does have an agents but also and that agent will then execute whatever test is on that system but this this is just showing a properly compromised Network after I think believe that's after an attack is executed so 2.0 notable features I think I mentioned that well the new version has a new Python core control center and I think one of the coolest things of

this tool is the ability to automate planning and an intelligent technique chaining it's super super cool it utilizes AI scoring for the technique for the next technique chain so throughout the technique chain it could depending on what techniques that you seem you want to simulate through your adversary after you define it adversary the definition and how it decides to do which technique next is super smart and it's kind of scary in a way you also have the ability to define your own logic to how you want to sim how you want to how you want your technique chaining to work or your next your decisions that you're made that are made very very cool stuff

it does support bring your own technique it does support the atomic Red Team framework and all 483 tests you could bring your own remote access tool what comes out of the box is a rat called sand cat and utilized there's some other plugins with caldera that's that are really cool I again I believe the intention of this is to help the plug in one of the intentions is to help close that alert loop hopefully someday there will be a plug-in that does that okay and finally the last toolset to talk we're talking about today is the purple team attack automation framework it is one of the newest of open-source adversary simulation tool sets out there

it was just recently Rini released very new it's pretty simple if you know Metasploit it's a clone of Metasploit that includes references to post exploitation modules as tests as tests the rat or their the client-server relationship actually is a matter Pradesh L so you have to get meterpreter to execute on your host and then it communicates with your client MSF she'll our instance and but really the best practical use for this is the support purple teaming testing because the Metasploit is more of a red teaming tool I feel like it it's easy for a penetration tester to help with that and proving the cyber steps assessment threat hunting processes it does have a hundred tests that comes with it out of

the box but it doesn't really do adversary simulation but I still think it's pretty cool because it utilizes Metasploit and interpreter so here's some examples of mess plate and when you do a search on purple it comes up with all your test techniques see they're categorized there by your mitre tech techniques there's t 1 1 1 7 and then here is an example of a matter procession I think it's a reverse TCP shell connected to your Metasploit console that is required for this and here is once you are connected with this session you can utilize and use a certain technique you just list out the technique you notice there that there's some cleanup there you could set cleanup

flags the yes or no whether or not you want to clean up the the files that are left there but there's your execution of T one one seven register over 32 dll and then on a certain meterpreter session and you can notice that it found calc and it's running it and the test was successful so this framework actually identifies whether calc was actually ran or not which is a good function okay let's talk a little bit about the future of these tools these are kind of what I'm seeing these tool sets may do and it's just kind of some it's just some thoughts you know we have seventy-five percent coverage of tests right now we're starting to near I guess

all the 244 tests there were about 75% covered of all of tests able to be simulated I'm gonna see I know there are some tests and tech techniques out there that I'm sorry there are some techniques out there there are just not testable due to the technical nature or unknown technical nature of each test but we're starting the near 100% coverage of all the testable or simulate able attacks I know that there's commercial software out there and I didn't do any research so if you have any questions about commercial software I just don't know but we're gonna probably see more of that I think we should also see the closing the alert Gloop alert loop fixed

and automating that through the future versions of these tools also would expect to see some integration with some commercial or public intelligence feeds that include adversary behaviors and identify adversaries I think we are all seeing a better adoption to smiter attack techniques and TT technique mappings within security products so when a certain for example EDR product when a certain tactic is I'm sorry technique is identified that the output of that technique that T one one one seven would be in the output of the log that way that way we would be able to close that alert loop I'm gonna see we're gonna see increased adversary I expect to see increased adversary simulation functionality within security products possibly within

themselves or possibly like a side tool to test their own stuff provided to customers I don't know how much you would trust that but it's possibility we also I also expect to see the ability to emulate and simulate manual non technical techniques within a toolset I'll think with a lot of the scriptable one tool sets that that's fairly easy to do you just have to do some scripting but I haven't really seen the ability to if you want to do a fish test for example or do something in the middle within the technique kill a cyber kill chain there's not really a spot to do so it's just more manual I would like to see

more I think we expect to see more and more consultants and service providers to provide simulated hunt assessments or threat hunting utilizing these tools I also expect to see penetration tests tooling possibly like purple attack team purple team attacking automation to include mitre references and more and more tools like purple team to include that capability and possibly the functionality to test whether or not or to integrate with a sim or EDR out-of-the-box to close that alert loop I know there's some concerns regarding for some tooling like EDR vendors that actually detect that develop detection alerts and techniques from their tool sets that only catch simulation tools being executed so beyond a be on a lookout for some I guess some products

that only look for the tests that run within these simulation tools that specifically red canary just to get a higher rating on there their software you know through third parties and such all right that is my presentation thank you very much so we have a few minutes does anybody have any questions or comments no questions or comments yes

we'll useful for what

in my environment well my environment is a so the question is which toolset do you believe was works best in my environment and it really depends what you want to do I think atomic red team was a very simple one that many people can utilize and execute they actually have a script with in atomic that as a script that runs all tests I did provide some feedback about that script about initiating delays and output and the time so you can run this all the tests and then go back into your your sim or EDR and go look at later I typically think atomic red team is fairly good place to start you're welcome any other questions all

right thank you so much enjoy the rest of your day at besides Pittsburgh [Applause]