BG - ATT&CKing the Status Quo: Improving Threat Intel and Cyber Defense with MITRE ATT&CK - Katie Ni

let's get started please welcome Katie Nicole's good morning everyone I'm Caitie nickles I work from mitre and I'm joined by John wonder also mitre employee also wanted to introduce Blake Strom in the second row co-creator of attack and Adam Pennington who's also on the attack team we're really excited to be here thank you so much to the B sides volunteers round of applause for them it's awesome to be here as I mentioned John and Adam and Blake and I work for the mitre corporation if you're not familiar with mitre it's a nonprofit that works in the public interest so we operate what are called federally funded research and development centers FFRDCs kind of a male mouthful basically

means that we do research and development and cool projects like attack which is what brings us here today we're going to talk about so we're going to talk about is what attack is and how it can help you up your game and defenses and addressing some of the common challenges we see so let's dive in here's we're gonna do today we're gonna talk about the status quo of the challenges that we've seen John and I have worked in network defense for many years along with a team some of the challenges that we see and how we think attack and help all those right explain what attack is so if you've never heard of it cool no worries we'll bring you up

to speed if you have you get a review then we're gonna dive into how you can actually apply attack my backgrounds with a threat analyst so I'm gonna cover the threat intelligence portion then I'm going to throw it over to John to talk about how you can use attack for detection and analytics next we're gonna tell you what's next on the attack agenda for us and then wrap up we'll take some questions and hear from you about how you're using attack how you want to use it any feedback you have for us so I've worked in a different security operation centers for eight or nine years now and these are some of the questions that I and others in socks

face on a day-to-day basis how effective are my defenses this is an executive question all the time how do I know how good we're doing do I have a chance at detecting apt 28 or whatever the adversary of the day is is the data I'm collecting useful I'm getting all these different log sources but are they actually doing anything to detect adversaries I care about do I have overlapping tulle coverage I bought these five awesome tools what are they detecting on the same things and am I really getting the best bang for my buck there and of course it's black hat week so we have to talk about these shiny new vendor products the machine learning blockchain AI is

that product actually going to help us improve our defenses or is it just another thing to use money on so we see attack as having a place in helping us answer all of these tough questions another tough task for us is detecting TTP's the defenders in the room are probably really familiar with david Bianco's pyramid of pain which models for us how painful it is for adversaries when we deny certain indicator types to them so start at the bottom think about a malware file if an adversary changes one bit of that file the hashed value will change and so if you're detecting on that hash you're not gonna find it anymore so that's almost painless for an

adversary right domains and IP addresses same thing they can register new ones you know under five minutes easy so if you're detecting on those things they can change really quickly and you're probably not going to catch those adversaries so ideally what you want to be doing is detecting on adversary tactics techniques and procedures they're behaviors because that's the toughest thing most painful for them to change and that's really easy it's easy to say right but a little tougher to do and we see attack as jumping in on that TTP part of the pyramid can help us there so we don't hate indicators but we know indicators probably feel a little bit slighted they definitely have a lot

of value especially doing things like Whois lookups and trying to anticipate as people register new infrastructure but they're not enough they're necessary but not sufficient so don't hate on your indicators but move towards TTP's so what is attack simply stated it's a knowledge base of adversary behavior what can add verse Ares do whether it's before they've compromised you how do they collect information about your organization whether it's how they compromised you through spearfishing or then after you've gained that they've gave an access to the network what can they do within it things like moving laterally establishing command and control that type of thing so some key points about attack attack was created by a team of it the Fort

Meade mitre site it was known as the Fort Meade experiment it was a series of Red Team Blue Team exercises the red team would do their thing and own the network and then they would communicate with the blue team hey here's what we did and those teams needed a way to communicate and they didn't really have anything like that so that's how attack was born and the team thought it might be useful for other people because they found it so useful so it was publicly released in 2015 another key point about attack it's based on real-world adversary behavior the goal is not to enumerate everything anyone could ever possibly do because that would be unbelievably huge rather

the goal is to focus on what we know adversaries are doing or they're likely to do things that cutting edge red teams do because that's the way that we can prioritize based on adversary behavior a threat based offense right also key to note that attack is free and open to anyone if you're a student you can use it to learn if you're a vendor integrated into your products if you're a sock use it for defenses it's for everyone it's very simple terms of use on our website and that's really key and you know mitre works in the public interests and we think that having a tap publicly open to everyone is in the public interest next attack serves as a

common language there's so many times when I'm reading a threat report and I'm saying like you're saying the adversary is moved laterally I don't know what you mean how did they do that attack can be that industry standard that we all point back to and say hey this is the technique that I'm talking about when I'm describing this behavior lastly it's community driven I mentioned that attack is free and open for everyone we also rely on the community for information we have a huge list of contributors and mitre you know we're the maintainer is we do not have all the answers we do not see all adversary activity the people out there fighting these adversaries you're the ones who do so we

really need you the whole community to help us out and make attacks stronger for everyone so that's what attack is a knowledge base of adversary behavior another way to think about it is zooming in on the adversary lifecycle kind of a different form of the Lockheed Martin cyber kill chain people are familiar with a key point of attack is that it's important to focus on the entire adversary lifecycle people so often focus just on perimeter defenses right i lock my firewalls they're never going to get in we know that's not true we know adversaries are getting in so we also have to look for them after they've gotten in the assumption they will so breaking down there are few different

flavors of attack focusing before adversaries have exploited you is pre attack that covers things like information gathering from your websites or setting up infrastructure they would use to Spearfish you later and then we have enterprise attack which covers that initial access through command and control and that covers things like moving laterally through your network or exfiltrating data in addition to pre attack an enterprise attack attack also covers multiple what we call technology domains so within an enterprise we cover windows very heavily and Linux and Mac we cover those as well but they're not as mature as Windows if anyone has linux or mac expertise come talk to us afterwards we also have mobile attack which covers Android and iOS so now that

I've entered attack I'm going to turn it over to Jon and he's gonna get into more detail yeah thanks Katie so if you're already familiar with attack you've probably seen what we call our matrix view it looks basically like a giant table so I want to dig into that and just talk about what each of those are so across the top here you have what we call the tactics these are basically the goals the adversaries trying to achieve like what are their technical objectives so some of their objectives might be they want to collect information they want to move laterally they want to escalate privileges so these are think of them as the things the adversaries are trying to do when

they're on our systems and networks and then down you have what we call the techniques so techniques are different ways that adversaries can achieve those goals so for example they might be able to move laterally via several different technical mechanisms we call those techniques you can see we're probably building out the TTP framework if you've if you're familiar with that and then within each one so within each cell of this table you can actually click in and get a lot more detail including what we call the procedures which are basically you know the exact mechanisms that any individual adversary group would use so the command and things like that so how they execute this technique so this is the matrix

it's basically a table across the top you have the tactics down the columns you have the techniques and if you dig in you can click in and get a lot more information so looking at that extra information this is all available on our web page which I'll show you a link to in a bit so this is an example of a technique new surface so new services away adversaries can achieve persistence and things like that in this case what we have is a description the description talks about two important things one of them is like what is the actual functionality what is the like like attack is about things that adversaries do they're using legitimate behavior

generally creating a new service is something that you know all programs do all the time so what is that functionality and then how two adversaries use that to achieve malicious goals how do they use this to achieve persistence for example so that's what the description is those two important things platform is very obvious that's just what is this apply to so this would be you know Mac Linux Windows or mobile it might be iOS or Android permissions required I think is also obvious this is you know what you need and what permissions you need in order to achieve this technique and then effective permissions especially if you look alike privilege escalation you might have new permissions at the end

after you execute the snake then then before you had it we also have ideas at how to detect this this isn't definitive this isn't like the be-all end-all one thing we try to do with attack is stay away from telling you how to do defense what we're trying to do is describe what the adversaries are doing but we do have some ideas here different things you can look at to try to detect this for example for new servers you can look at monitoring service creation right that seems obvious and we also talked about mitigation so how are you how can you prevent this from being a problem in the first place so it's about detection so

after the fact how can you detect this being used and before that how can you help mitigate these attack vectors coming in another really important aspect of this is data sources how can you actually see this well log should you be collecting in order to be able to see new services being created so for example Windows registry changes are important you can use process monitor and command-line parameters things like that to detect new service creation next really important as Katie mentioned attack is really based on real world observations of adversary behavior so within the examples your examples of threat groups that actually have used new service to carry out their attacks so you can check that it's all based on

open source reporting each of these will have a reference another important piece of data so this was all open none of this is closed intelligence it's all available online so you can click through and see the original reporting from the from the source as I talked about so we have examples of the threat group so we also have short pages for each of the threat groups themselves so for example a PT 28 this threat groups you know the one that's been attributed to the Russian government and the DNC axe it's been in the news recently so for each of these third groups you have basically this short description we have some information like other names they're

known by as you probably know there's different vendors we'll have different names for how they consider the group's government organizations may have different names for what they call the groups this is a best-effort it's not meant to be exhaustive and obviously there's sometimes not perfect overlap in these things but it it Elise tells you you know if somebody's talking about fancy beer and you're looking at a p228 you know those are roughly the same thing then obviously you want to know what are the types of techniques that these adversaries are using what is a p228 do when it attacks you and so this is the set of attack techniques basically and again there's references for each of these so to link to a report

that says when did a PT 28 use a new service or data obfuscation or connection proxies for example so you can see the source reporting you don't have to just trust us and then also a lot of adversaries obviously use software so they use malware they use different utility tools they use built-in tools and we'll have the list of software that we've seen them to be using as well and again the references so digging in the software for example chopstick is a malware family again similar to the group ages this is basically just sirs to link to the technique so what are the different types of techniques that that this tool is capable of performing so if you see

this tool on your network you might want to look for these different types of techniques and things like that and then obviously also you want to know what groups are using this piece of software and again you want to have references so you can really dig into the source reporting again so we've talked about what attack is a lot probably also important to understand how are we going to use this thing we typically talk about these for use cases and more importantly it's also the communication between these for use cases we'll talk a little bit about that today you can start with Katie's going to talk about threat intelligence how you can use attack to describe your threat

intelligence I'll talk about detection how you can use attack to better your detections and to better build your detection so you're detecting adversary behavior you can use attack to do assessments in engineering so you if you familiar with attack you may have seen these heat maps is what we call them just different colorings of the attack matrix this is really helpful as a tool of tracking where you stand in your defense am i covering the things I need to be covering what am I missing and how do I get to filling those missing gaps and then adversary emulation is another really important use case this is all about we know we want to understand the

adversary and the best way to understand the adversary is by being the adversary essentially so adversary emulation is all about trying to pretend to be a p228 using the techniques that ap t28 uses in order to better validate our own defenses to make sure that we're detecting what we think we're detecting to make sure we're mitigating what we're think we're mitigating in to make sure our assessments are actually accurate and measured against ground truth so what are the resources as Katie mentioned attack is all available for free online the best place to get to that is attacked on my org if you remember one thing from this talk remember go-to attacked on my org you

can see all of our content there there's a public web site it's built on media wiki right now Katie will talk about some updates to that coming up this has the matrix it has each of those technique pages it has all of the software pages and all of the group pages as well as some other resources I'll talk about in a minute if you're a fan of like JSON and structured content or you want to do scripting around attacks so for example maybe you want to count the number of techniques used by an adversary or you want to ingest this into your own repositories we have sticks and taxi available sticks is a JSON format for cyber threat

intelligence tax is just a way of getting that to you so this is structure content that's available on miters github page you can also get to that from attacked on my dirt org we also built a tool called the attack navigator so we don't want to tell you how to use attack but at the same time we know it's helpful to have tooling that will help you work with it so the damn navigator is all about making it easier to work with attack so you can color different cells you can add notes to different techniques and you can use that to kind of capture what you're doing with attacks so if you're building out your

defensive program for example you can say I'm going to color all the things that I think I'm detecting in green and the Navigator again the Navigator doesn't tell you exactly what to do but it does help you do it and then we have adversary emulation plan so like I said it's helpful to understand what the adversaries are actually doing how you can be them what that looks like all the way from the top level to how they go about planning out their attacks all the way to the lower level of what commands are they actually plugging in what things are they running to do each of these techniques and we've published one adversary emulation plan right now we're

planning to publish at least one more that talks about this and other organizations are using attack to do similar things these are all also available on attacked on night org and lastly I'll talk a little bit about this coming up but we have a cyber analytics repository analytics are a way that people are talking about how to do detection for TTP's so not looking just at indicators but how do we detect the tippy top of the pyramid of pain how do we detect those behaviors miters published a cyber analytics repository card at mitre org also available from attack tonight org that has some things to start with basically so that's all that's available the big thing to

highlight here attacked on my dad org you can get to all of these other things from there so just remember that part and with that I'll turn it back over to Katie awesome thanks John so now I want to talk about how we can use attacks specifically for threat intelligence I've been a threat analyst in socks for many years and here a couple the challenges that I see of the way we're doing things right now first of all there are so many reports to read open source close source things that are coming into email things that your executive sends you things on Twitter things on Reddit your RSS feed it's a lot right also it can be really

difficult to apply intelligence that you gather to your defenses right so I don't know any analyst in the room you might have had this experience you write this amazing ten page reports detailed analysis you know you had your biases about this adversary you hand it to your defenders and they give you a look and say what do you want me to do about it yeah that's kind of a little bit of a sad day but it's something that really happens it can be really tough to you know apply what you gather about adversaries to actual hands on defenses right also our reliance on indicators we talked about it earlier with a pyramid of pain I think that one of the

challenges and why we're so tied to these indicators IP addresses domains is that they're countable right and we all like metrics and we like to feel that if we have you know 5000 indicators rather than 4,000 were safer that's not the case it's a kind of a game of whack-a-mole or whack a kitty I like cats no offense here but you know we got to do better that's not going to be sufficient so these are some of the challenges we see in threat Intel and that I've seen in my experience so what do we do we use attack to structure threat intelligence when I say this a lot of people are sort of confused so I

want to walk through you an example report this is an older report from fire I operation double tap from 2014 but you could do this with pretty much any threat report that you have so we're gonna walk through you want to look for the behavior think about behavior that adversaries doing rather than you know looking just for an indicator and then we're gonna map that back to the attack technique so let's start off first of all they talk about a kernel vulnerability and they're using to escalate privileges to system that's one of our exploitation techniques exploitation four privilege escalation there's our first one next adversaries used command line to run Who am I to figure out who the system owner user was

that's two techniques command-line interface and system owner user discovery moving right along a really common persistent method persistence method is scheduled tasks on the adversaries did that and that's a technique next up a socks5 connection for command and control which we track under a standard non application layer protocol also there's C two was over two ports a t1 and 1913 these are sort of uncommon ones so we classify those as uncommon Lee used fourths and also because they're doing different ports for c2 multistage channels so in this one report just a portion of a single report we have seven attack techniques we've pulled out so you can imagine as we do this for multiple reports from

many sources for different adversaries you're going to get a massive library of different TTP information so here's what we have on our website for apt 28 one of the groups I should mention that this is a collection of techniques based on open publicly available threat reporting we know that does not represent everything adversaries are doing but it's the data that we have available so that's a really important bias to keep in mind if you're gonna use this data it doesn't represent reality but it does give us a sense of the techniques that the adversaries all right have known to use in public reporting so that's apt 28 I recreated this using the attack navigator that John mentioned earlier

next apt 29 another Russian group so if you cared about these groups you might want to do mapping of your own sources that you have internally to what techniques they're using and then what you can do since it's all structured BAM overlay 28 with 29 so you have 28 in yellow 29 in blue and the techniques that both groups use in green so if these are the two threat groups that I was most concerned about you've a really easy way to say hey the techniques in green I should probably focus on prioritizing how I detect those from there you can take it a step further right if you look at your environment and you know what attack techniques you

can detect or not you can then overlay that so just notionally let's say these are the five techniques that I've looked at the green ones I know I can't detect those BAM we've gone from 219 techniques enterprise attack to the five that based on threat reporting we should probably be focusing on and John's going to talk more about how you do that detection I'm going to give a few examples from industry a few threat Intel providers are starting to map they're reporting an attack McAfee is listing techniques at the end of their blog posts Digital shadows just released a blog post mapping apt 28 techniques from the GRU indictment this is another example unit 42 which is Palo Alto threat Intel

team has something they call adversary play books I think they're up to five or so different groups they're mapping they're using pre attack and enterprise attack techniques to map out what adversaries are doing and one thing I really like about this they do it over time so that's a really important thing right adversary's change so they map you know what if adversaries have done over this six months and then over the next six so that gives you a way to compare behavior over time a couple implementation tips this can be tough if you're starting out you're a small team but here are a few tips I have to get started first of all tailor your

existing threat Intel repo on the good news is some threat intelligence platforms are already starting to support attack natively a few of those include Mis which is open source threat key which is a commercial product and others or if you have a team of engineers and you have a custom homegrown thing you can grab attack through sticks and see if you can implement it into your own threat Intel repo next have the threat Intel originator do it I showed for example that command line that the adversaries ran who am i if you don't have the original data it can sometimes be tough to figure out what technique and adversary is doing right so if you have

actual intrusion data that's gold to map to attack also start the tactic level can be really tough I know 219 techniques in enterprise even more in pre attack there are only 11 tactics in enterprise and 15 in pre attack so think about what tactic what's the adversary goal behind the behavior do you're trying to map then bring up that page and go from there also use the existing website examples we have the team has been mapping threat reporting for years so we have hundreds of examples of how adversaries have used this so do a keyword search if you're not sure what HTTP is throw it in there and you'll bring up that's the standard application

layer protocol for c2 also work as a team I mentioned this one because there's a great example of a report that John and I mapped separately we each found techniques the other didn't this is human analysis and it's prone to all those biases you're more likely to see the techniques you know that happens but you can hedge against that by working together as a team so what does this get us we started with these issues the status quo issues on the left so many reports to read ok that's not going away I can't do that but if you structure your threat Intel and if providers start structuring it you're going to get that all in the same format

and then you can more easily compare those TTP's also tough to apply until 2 defenses right now that it's structured we can directly overlay our reporting with our defensive coverage next the reliance on indicators we talked about that attack gives away to move to TTP's plus it gives us a common language in those threat reports if a blog post says I'm talking about this attack technique then you know what they mean you don't have to guess right and also it allows us to compare those groups like we did with 28 and 29 so now that I've talked to you about how you can use attack for threat intelligence to track those adversary behaviors I'm going to throw it over to John who's

going to talk about how you actually detect those behaviors

okay there we go uh-huh Thanks so hopefully we've now moved from this point where we're getting our threat intelligence as a bunch of reports that you have to read I know everybody hates reading and now we're starting to get our threat intelligence in terms of attack and this helps us solve a really important problem in detection one of the biggest problems at least I've seen in detection is knowing what to focus on first like what are the things that I need to be worried about and what things can I wait until next month or the month after to focus on it's like there's all these things there's all these reports that come out there's all these new

adversary techniques what are the important ones for me and what your threat intelligence can give you is really a good outline of like what are the things I should focus on right now that are likely to be problems a lot of times we've talked about doing detection based on indicators of compromised Katie mentioned this at the beginning so indicators are compromised as we talked about are these like IP addresses file hashes things that we can share very easily and do detection on and then we see the pyramid of pain and we know we're supposed to move to analytics and a lot of times it's hard to understand like what does that really mean what does it mean to do detection based on

analytics and this I'd like to think of it as a spectrum I'm sure other people will tell me I'm totally wrong and there's a very clear distinction but I think of indicators as you know known malicious behavior these are IP addresses that we've seen the adversary using we know they use their hashes for pieces of malware that we know are bad and they're also tend to be many of them we talk about you know hundreds of thousands or millions or billions of indicators like there's all these indicators of badness and analytics on the other hand our detection for things that will look suspicious that adversaries tend to do so you know like I talked about creating a new service

doing reading from LSS in a weird way that lets them pull credentials out of it and things like that so these are things that look suspicious versus things that we know to be malicious already and that makes them a little more challenging to deal with in particular it means that we're often going to get more false positives so we've all had the stories of you know the 8008 indicators so it's not like we don't have false positives with indicators that means your indicator is bad though whereas with analytics it's kind of a fact of life that we're going to detect things that look suspicious that are actually just or sysadmin figuring out that all of a sudden they

can use PowerShell to automate this thing that we're not automating before and so dealing with those false positives is one of the challenges that I'll talk about and the nice thing about it caters that kind of balances out that false-positive problem is we have we need many fewer of them in order to do a broader set of detection because it's very there's so many IP addresses there is you know so many different file hashes even there's so many different ways you can make a file to create a different hash on the other hand there's only so many ways to get credentials out of a Windows machine there's only so many ways to create persistence and so

we can have more in the order of dozens and hundreds of end of analytics versus you know millions and billions and trillions or whatever of indicators so I talked about how analytics differ from indicators but what our analytics really how do they work and a lot of you're probably familiar with this I'll go over it again so analytics really look at the observable events in artifacts from a system that indicate a preserve behavior so if an adversary uses a new service to create persistence what does that look like if an adversary uses RTP what does that look like a lot of times we'll see this in log data so when you already ptoo another machine on a Windows environment

windows event logs will log that and they'll show it as a login event with the type equals to remote interactive and that means if you're looking to detect lateral movement via RDP then you can look for those log entries and know that that's happening right now and now you're probably thinking well crap everybody does already fee I use that all the time myself to administer boxes so that's the trick really with an analytic is how do we distinguish the fine uses of this technique the things that my sis admins doing my users do from the things that adversaries are doing so if you look at what's in attack it's a lot of things that are typically

used day to day in your enterprise and so you're going to find that a lot of things are in this gray area where there's all this behavior that we know to be good all this behavior that we know to be bad and then this this fuzzy gray in their area in the middle and what we're trying to do with analytics is when we get a new event to come in we want to build up a set of evidence and then pull apart those circles that we can understand which circle does this actually live in is this a thing an adversary is doing to me or is this thing that I don't have to worry about

that my you know it's normal in my environment so that's what an analytic is it's really writing these searches against our log data that lets us separate the good from the bad and really alert on the bad and send that to something to our responders what that can look like in practice this is a Splunk search it's super incomplete please don't write this down or anything I just want to highlight the key components of it so first a lot of stuff an attack is really reliant on endpoint data this is some you know traditionally as Katie talked about we do a lot of detection at the perimeter we're looking for things coming in and things going

out a lot of the stuff an attack is really focused on what the adversaries are doing once they're in and so that means we need to collect a lot of endpoint data one common way of doing that is using system on and so what that means is we narrow our search to system on and in this case we're looking for a UAC bypass UAC is user Apps account control this is a if you're familiar with Windows it'll pop up a box and tell you it's asking for admin privileges there's ways of getting around that programs can kind of write off of normal operating system functionality obviously operating systems need admin functionality all the time and so what

they can do is they can hook into that and so that's what a user account bypass is so what we're looking for is really processes that have an integrity level high so things that have been elevated and then what we can do is try to find the ways that programs can hook into these things so there's a github page called UAC me that has a list of like 42 of them I've listed like two of them here one thing they can do is they can try to hook into the fod helper that exe file that's not important what that is they're kind of folk into that and try to run a program off of that and so the

end result of that is maybe they only had user privileges before but they are able to get admin privileges without popping up a box to my prompt view and warn you that something's wrong so they're able to work around that so we want to detect that we want to find that suspicious usage of this similarly they can use clean manager dot exe with some specific flags and things like that and then in the green at the bottom what we're doing is we're just kind of categorizing that one thing you want to make sure of with your analytics is that they are useful to your responders in your analyst so when they get an alert they want to understand what it is and

so what we do here is we kind of tag it as which you a see me technique it is so they can then look at that and understand how to investigate it even once we have all these filters in place you're probably still going to find false positives so how do you go about doing this the first really important thing to writing an analytic is understanding the attack you want to not just understand what it looks like in the logs but you really want to understand what is the adversary trying to do with this what is their goal and how does that look different than just some guy doing his job on your networks so how do you separate those things out

how do you know what the adversaries goals are and the best way of doing that is to try the attack so look at different ways of carrying out these attacks yourself spin up a VM that you toss out after you're done or you know find a test now we can carry out the attacks and see what they really look like collect the logs ourselves and then try to create a search that really finds those things specifically like the specific badness you're going to find your first search is probably also going to find a lot of good things it's going to find just people doing their jobs as I've talked about and your trick is going to be kind of writing and

iterating that better understanding the attack trying different ways to make sure that you're getting a search that isn't over fitted to an attack so it's not looking for something too specific but at the same time isn't too general that it collects too many alerts and you have you know too many false positives essentially so you can kind of think of this as just feeding into itself and yet this isn't a one-time thing so you might do this you might get a analytic that makes sense right now this month and then next month you buy a new product and all of a sudden you're getting a ton of false positives this is an iterative process you also find that

the adversaries will discover some new way of doing this and you'll have to broaden your analytic to detect that so this is not a once and done thing you have to keep maintaining your analytics over time so as I talked about one of the best things that attack gives you is a really a way of understanding a what priorities you should have in building out your detection program but also understanding where you stand in your detection program what are the techniques that adversaries are using that I have no ability to detect right now what are the techniques that I will see sometimes what are the techniques that I'm pretty confident that I'll see if they happen on my networks an attack

gives you kind of a scorecard of what that looks like so in this example we have things in green that we're probably pretty good at detecting things in yellow that maybe were moderately good at and things in white that we still need to build out detection for so you're presented with that map and there's like 219 techniques that's a lot right we're not going to be able to do all of those immediately and so what you can do is then take the information from your threat intelligence and really figure out which are the which what's the next one I need to look at what is the next thing that will give me the most bang for my buck the most value in

spending my analyst time on and for example we can highlight that in orange and then that'll give you kind of a plan to build out your detection to go from where you are now to having a comprehensive detection for the things you know you need to detect and we can again think of this as a cycle of defining what our threat model is what are the things that we're worried about assessing our coverage so understanding where we stand right now identifying gaps in that coverage and then filling those gaps and doing that all over and over again again keeping this up today even with that though even with that prioritization filling dis gaps is hard

it's time-consuming and it's expensive it's hard because it requires a lot of information about like what's actually happening on our systems you need to know Windows system is internals you need to know you know if you have a Mac environment you need to know Mac systems internals and that's going to be like a totally different skill set than your Windows team and so that's hard it's time-consuming because it takes a lot of time to do this iterative process you have to go back and forth a lot you'll probably find things won't work and you'll have to start over so it's time-consuming and of course being time-consuming means it's expensive but the good news is we're not alone

there's a lot of folks that are working on this problem right now for example one of them I like to talk about mitre participates in this it's the National Health I sack so this is a sharing community they spun up a working group around sharing analytics based on attack the way that works is we meet over two weeks every two weeks one of the organizations will present a detection they've written for an attack technique mitre presented are UAC bypassed detection and then all the other organizations that are participating can take that into their own into their own environment try it out they'll probably need to tweak it a bit nothing is going to be a perfect transition it's not like

indicators because there are there is this false positive problem but it's a start there's also other communities we have the cyber analytic repository that's available for free at cartomizer org so if you're looking to start out you can do that the MIS stretch sharing platform we were talking to them recently they're trying to figure out how to do analytic sharing and NIST so you can use that existing threat sharing platform to also share detections for adversary behavior and then the Sigma project also available on github they're trying to write a kind of a sim neutral query language and as part of that they're building out a lot of seem content for analytics that said we don't

have this problem solved there's a lot of things that we're still working on and that I think still need folks like yourselves to look at so one of those is being realistic about coverage so I talked about attacks as a scorecard and that makes a lot of sense for some cases it's really easy to understand it feels great to change a caller fro of a cell from like yellow to green or something like that but it is a little bit overly simplistic when it comes to building detections and writing detection z' you'll often find that the best way to do it is detection for some technique is really to look at a chain of techniques

so looking at several things in a row because then you'll you won't alert on too many false positives and so that means then that the scorecard becomes more of a qualitative exercise than a quantitative exercise your green doesn't really mean that you're fully covered against this it means that we think we're probably covered by this we feel confident as an analytic analyst judgment that we are and so it's just important to understand that coverage needs to be realistic but at the same time attack does give you that scorecard and that thing to always go back to second challenge area is handling false positives so as I talked about we um our socks are already oval overloaded just

looking at indicators and other alerts coming off our firewalls and things like that and all of a sudden we're being told that we need to build out this big detection program based on adversary behavior 219 techniques you know all these different throughout groups they're all doing different things and we're telling our analyst team that then we need to look at these false positives as well when they're already you know busy 24/7 basically so how do we do that how do we deal with this false problem false positive problem so as I talked about one of the best ways I have seen a least and this is super common right now is looking at a bank graph so rather

than looking at each event in isolation look at it chains of them I like to throw this meme in here because I think like half the audience might get this is from the Graduate updated for today it's all the graphs now look at graphs neo4j is huge basically it's it's a way of just tailoring or narrowing down what you're finding based on chains of activity machine learning also you know is a buzz word in some senses in other senses you know it's a great technique to do classification of you know behaviors into good and bad I like to include this meme because it's totally ridiculous I don't know what he's doing he's looking at something and he's

glowing citing the feedback loops so as your analytics are creating all of these alerts your analysts are going to look at them and they're going to tell you whether it's a false positive or true positive because they need to go analyze that they need to go figure out is this something real that I need to worry about and respond to or is this just false positive and they're telling you good information each time they do one of those triage is basically they're trying to say they're telling you that your analytic is you know correct or is incorrect in this certain way so listen to them don't be like Michael Scott take their feedback in to account because they're telling you

good information and you don't want to ignore that just because they're a first line analyst and then you can target your detection so don't try to detect all of the things all of the time because that is a recipe for overloading everything focus on what are your what are your business critical or mission-critical assets what are the things that you need to protect and then try to do your detection z' kind of on the pathway to those things so what are the servers that have that information what are the admin accounts that have access to those servers what are the user accounts that have access to those servers so I know my mic is cutting in

and out here and focus on those because that will let you narrow down your false positives you only see those things that are you know important to your business yeah so focus on your priorities what are the things that are really important to you and spend your time on those things rather than trying to solve all of the problems all of the time and then the last problem is analytics are requiring increasing and increasing amounts of data because adversaries are getting better at hiding in the noise of our normal systems operations they're better at kind of mirroring exactly what we do and that means we need more data about exactly what is going on we need

better endpoint data we need better Network data we need better infrastructure data and all of that data requires resources to collect it requires resources to search CPU and things like that so how can we improve that how can we better target our collection can we do agile collections so that we're you know collecting a lower baseline and then when something goes wrong can we increase that and really detect much more once we know something is wrong and can we decentralize collections so we don't have to like collect all the things and aggregate them into one place and how can we do like so we talked about graph based search but one of the big challenges that we have run into at

least at mitre is like making that scale across a medium or large size organization can be really hard because you have so many different connections and things and graphs have a big you know connectivity issue so how can we make graph based search scale and how can how can we make effective use of our resources so how can you make sure you're not blowing all your money on Splunk license and you're using it correctly and the last thing I want to leave you with is like how can we get started on our own so how can we you want to go from where you are now to being able to do analytics so this is a

step by step here for take a look at detection lab detection lab is basically a set of scripts and utilities that will help you build out a sample Windows environment it comes with some Red Team tools it comes with sensing and all that built in and it'll basically let you get started quickly there's a medium posterior you can also Google for this pretty easily and then you're going to want to know how do I be bad so I'm not a skilled adversary emulation person I do not have those skills at all atomic Red Team has a lot of commands that you can try so you're not going to be able to mirror a red team unless you

are the red team you'll want to work closely with them but if you're just looking to get started take a look at atomic red team that'll have a lot of kind of commands you can plug in to carry these things out that's from red canary and then lastly like see what that looks like do it try it and write some detection x' and then share what you've learned this is still an emerging area you might find that some people have already done what you have done but we're all still learning so please share where you have write a blog post write a tweet well retweet it and tell us how you're detecting these attack techniques so I want to talk a

little bit about bringing it all together so we're talking about mitre attack as this framework to describe for intelligence to do detection and to do adversary emulation so what does this really look like so if you can structure your threat intelligence using attack that gives you a common vocabulary right it's like whenever I see this technique being used in my report I give it the same name and then I can trim that over time right I can understand how the attacks are changing I can do prioritization you can then feed that into your detection program so what are the things that I actually need to detect on a day to day basis and what can I kind of you know put as a lower

priority and you can also feed that into your adversary emulation in your red teaming so your red team is carrying out the things that you are facing on a daily basis they're not kind of off in their own corner they're based on the threat intelligence so they're based on what the adversaries are doing and then you can feed that into your defensive process as well so your red team when they carry out an attack they tell you exactly the techniques that they did you can look at your detection and say which of those should I have detected which of them did I detect which one did I miss and it just gives you like a common

vocabulary to use across all of these no hand it back over to Katie to talk about what we're doing next thanks John so we've told you where attack is today and now we want to talk about where it's going we know it's not perfect we have plans to to try to improve it over time with the community's help so first of all we want to improve on the content we have and add to it a few upcoming things the next few months you'll see are the addition of sub techniques which is another level of detail beyond techniques a good example is for example credential dumping as a technique lots of ways you can do that with different detection

right so we want to break those down also looking at impacts things like destructive malware destroying data they don't map too well to attack in its current form so how do we address that maybe adding a new tactic working on that also looking at new technology domains things like cloud what does attack for cloud look like what do we have to add to that next continuing to expand the attack community we've had an amazing kind of grassroots rising of people who are using attack we had ten thousand Twitter followers last month and we haven't really tried that hard to do that people are finding it useful we want to keep spreading that word so that others can

use attack also we're looking at opening up the governance right now mitre maintains attack we want to start pushing that towards other community members getting advice on what we should and shouldn't add so we'll be announcing something along those lines in the coming months and last John mentioned it in September we're pretty excited we have a new website coming if you love MediaWiki I'm really sorry but it's going away it's time the new website is sticks taxi base so much easier to work with a new infrastructure the taxi server is already up so for anyone who is using the old media wiki API please transition to taxi that would be awesome all right that's all we have for a

presentation material open it up if anyone has questions it is the first time you've heard this like what do you think how could he use this for those who are already using attack we'd love to hear about how you're using it if there any challenges anything we can do to help you better all of those things I think we have about ten minutes so open it up all right in the cobalt blue shirt that cobalt I think that's cobalt blue yeah you hmm that cobalt there purple I don't know yeah so um just as a plug because that our use are you use it but could you tell me more about caldera caldera yeah oh I'm so glad yeah this I'm not

sure just a pulse applies to volcanoes I think yeah I'm gonna bring up a blake Strom everybody blake is not only at the attack late he's also the caldera lead so rather than me fumbling through it you get to hear from Blake so what specifically about caldera yeah okay and so the story of cold air goes back to the Fort Meade experiment the project this sort of drove attack and car and a lot of the follow-on work so we were continuously doing these adversary emulation exercises with what manual Red Team and we thought it wouldn't be great if we could try to automate that process so we created a system that we defined pre post conditions for the techniques

that we're implementing and developed an online planner that can essentially start with zero knowledge of the environment with patient zero I'm so you choose which hosts get started and it can perform the discovery techniques out of attack on its own as its building its internal representation of the the network and then it will if you've selected use this persistence technique or move laterally in this way and dumb credentials it can piece those things together and execute the sequence of actions for more rapid testing to develop analytics and refine your defenses in a more automated way so it's very useful for people who don't have their own red team's to sort of see the their network through the eyes of an

adversary and if that sounds interesting the good news is our colleague Andy apple bombs actually be talking about caldera at the DEFCON AI village on Saturday he's also presenting at the blue team village on Friday at DEFCON so either those topics interest you catch him there um will do second row woman in the glasses I will wait for a mic one sec just I'm bad at repeating questions you know it's a follow-up question is that for external or and the internal infrastructure caldera caldera is open anyone can use it no I mean in terms of attacks are we attacking the external infrastructure there or can we also use it for internal infrastructure mais so if you have an internal network you're

looking at us our behavior it's it's useful for that other questions comments over there go second row back I guess sorry third row back or we'll do third row back maybe yeah there are clear to go but are there any plans to map the similar items between different technology domains so like in Windows you have new service on Mac OS you have launch agents and launch daemons are is there anything overarching you're looking for to connect the same or similar attacks that's not a bad idea I mean right now we're dividing up by technology domain but I think cuz it's so different in how you detect it right but there's no reason why we couldn't try to find some

other underlying behavior something we hadn't thought about not on our roadmap but the reasonable suggestion cool I think one robot from like an inn how is I mean is there anything with a tag that can work with misattribution so how is it dealing with misattributions something like the leaf I don't remember what that was just like a couple days ago and it being at misattributed to energetic bear and other kind of attack technology yeah sure sure so we got the question sometimes about can I use attack for attribution you can use it as kind of one piece of evidence but I wouldn't use it like the EPT 28 graph I showed if you see those exact techniques

I would not lead that to the conclusion that that's definitely a p228 attribution is really tough and much more complex than that it can maybe give you a hint like hey these techniques have been used by these groups in the past that's sort of a starting point so for that reason I wouldn't be too concerned about misattribution because it's only one piece of the puzzle in the very complex puzzle that is attribution and that's a whole separate talk good question well way in the back there oh sorry making them down your workout keep going gray shirt which gray shirt there you go I know right so you mentioned the creating analytics is a lifecycle process and I totally agree with that

one if there's any good ways for managing the analytics as you create them I haven't really seen any good i acting out there other than excel for as a reference something and some of those are reference for the state of the analytic description of it the details may be possible investigations things that you could reference in a ticket for other people to take a look and get more context about what they're supposed to do and whatever you know log network alert that they receive I just really don't see anything along those lines yeah unfortunately I haven't seen anything that is fully production ready yet there are some projects that are kind of developing out this is still a

new area one of them is the unfetter project is an open source project from the NSA you can find that on github it kind of does that content management system for your analytics for you unfetter UNF et ter unfetter unfetter of course you yeah so take a look at that one they're still in development they're doing active development right now to kind of build that out it also does some of the charting of understanding what data sources you need so what data sources do I need to in order to use this analytic but yeah that's a great problem I think that we're still trying to solve I know tracking analytics is definitely hard for us as well how about

the woman right next to the mic yeah do you think hi yeah I was wondering if you could say more about what you were trying to use with graphs like you highlighted that you think that they can make the real difference and I was just curious what you were doing and what problems you ran into with scaling most immediately yeah sure so there's an open source project called cascade this is also available miters open source repository what I was trying to do was it would plug into your sim tool and or tried to connect different process trees and things like that and build out like what is this whole event look like in practice and that makes a lot of sense

if you already have a focus area that you want to look at because you can kind of constrain what the graph is and you can say if it's going too far beyond that stop but running that type of detection across the whole network was just not doable for us at least I know some folks are looking at that using other types of databases and things and if anybody has success with that I think that would make for a great blog post or something like that I'm at a time for one or two more go for it Katie take one which one in the back will be back on the rail another gray shirt another it's actually

a red Canary shirt pretty sweet shirts it is it is I'm glad I got it so question about how this project is being used in the community in the market you know I've noticed a lot of vendors have latched onto it almost using it like a standard like mapping their products to it so and I'm technically a vendor now so that you know apologize if this is a cringy question but how would you prefer companies to work with you on that are you gonna have like a considering like a certification program or you know I don't I don't know if you want to prevent anybody from just slapping you know attack logo on their product ya

know it's a really fair question and we're really happy like I mentioned earlier we're really happy to see the vendor community and everyone latching on to attack and using it one thing we've done to try to help with this this is a spin-off effort that's kind of separate as an effort called attack based evaluations led by Frank Duff we've announced I believe eight participants in the first round this is going to be testing different EDR products against a subset of attack techniques in the style of apt 3 so testing each of those products kind of trying to articulate what the products capabilities are compared to the attack techniques and then we plan to publicly release those results in the fall so

that's one way that we're trying to address that honestly though I would say that you know anyone who's talking to a vendor can ask hey and we've heard this from people they'll ask hey what out of the attack matrix can you cover and then if you get that map do the do the test yourself right ask for an eval triatomic Red Team try some analytics see what it detects and you know decide for yourself so that's the advice I would give people who are looking at those products the other thing I would say is we're happy to talk to anybody whether you're a vendor and end-user organization email us at attack amide org and we'll set up a you know

one or two hour phone call and kind of talk you through how we're seeing it being used figure out how it fits into what you're doing right now yep we are happy to do that and with that I think we're out of time follow us on Twitter attack thought might org we will be tweeting out slides I should have said that up front I'm sorry I mean will tweet out link the slides so you don't have to worry about taking pictures just remember attacked up might org follow us on Twitter thank you all so much appreciate it [Applause]