Understanding the Latest MITRE ATT&CK Evaluation Data

hello everybody it's norm chan from sentinel one i am the pre-sales se for sentinel one in western canada and i am based out of vancouver thank you everybody for joining my session today i'm going to be talking about the mitre attack results for 2021. i'm going to try to make this a no spin zone and reduce the marketing fluff i want to talk about what the results look like how to interpret them because uh one thing that mitre does not do when they re when they announce their results for their attack evaluations is they don't declare a winner and they merely provide the testing this year against 30 different edr solutions and they publish the results for

everybody to see they don't really analyze the data so very often what we see is everybody's marketing departments going into a heavy spin mode and you see that uh declarations of being the absolute best or being the winner or having amazing levels of uh of maximum detection and protection things like that so i'm going to try to stay away from that today and we're going to try to use the actual data that's on the the miter website so let's uh take a look at the let's take a look at the the agenda so uh we're going to talk about what is a miter attack and why it matters we're just going to level set for everybody that's in the

audience today we're going to talk about the testing overview what mitre does and how they do the testing and then we're going to just jump straight to mitre's website and we're going to look at the results we're going to do a live q a and um uh and who knows time permitting maybe we'll uh we'll we'll do a little bit of audience participation and pick out some vendors that you're interested in looking at the results together so um who is mitre corporation and specifically what is uh what is mitre attack so mitre ingenuity is a subset of an independent foundation that was spun off from mitre corporation and they've been charged with forging strong relationships

with public and private entities for the betterment of a number of key areas in security in cyber security one thing that they're tasked with doing is sort of managing and becoming the stewards of the mitre attack framework so mitre attack att ampersand ck it's a bit of an acronym it stands for attacker tactics and techniques and common knowledge so one way to think about a mighty lighter attack especially the miter attack framework is you can think of it as a knowledge base of all the different methods and ways that attackers are using to try to get access into enterprise environments so i'll give you a show an example of that later on when we we cut to

my browser actually get on and show you the miter attack matrix for enterprise but it's it's valuable because they enumerate all different ways that attackers all the techniques and tactics that they're using to try to hack and obtain data within the enterprise so it's also useful because it gives us a common set of vocabulary so as cyber security people in you know in cyber security we can talk to each other with a common set of explanations on tactics and techniques it also describes uh various adversaries and because when you see thread actors using consistent sets of tactics and techniques you can begin to identify them a little bit more easily so one thing in terms of limitations

around mitre attack especially the miter attack evaluations is this is not a this is not a perfect test you know so i would say that one of the limitations around the attack evaluations is they don't test everything because there's lots of different tactics and techniques available in the matrix in the matrix and typically what they do each year for the test is they pick a one particular threat actor and they perform the tactics and techniques that that threat actor is known to do actually this year you get a bit of a two for one because they picked two threa two thread actors that we'll talk about uh in in a bit so here's another way to look at uh why

this matters and you know credit where credit is due i did not invite invent this sentinel one certainly did not invent this but a david bianco who's a sans instructor he actually works in cyber security at a major retailer coined this way back in 2013 so this isn't anything particularly new especially if you've attended any of his courses but he talks about this pyramid of of pain and the the idea is you want to inflict as much pain as you can on your adversary as as possible so depending on the methods and the protective measures that you have that you're using today in the enterprise a lot of that's going to govern how difficult how much pain

that you're going to inflict upon our cyber adversaries so if the technology that you're using relies on you know hash values or ip addresses to identify an attacker you know this would be things like antivirus technology or you know just basic firewalling technology for example those things are pretty easy for an attacker to circumvent right it's trivial to change the hash on a file or come at you from a different ip range so tools that are operating at the lower part of the pyramid of pain they don't inflict a lot of pain onto our adversaries the higher up that you go in this pyramid is where you get more pain okay where they they go from annoying to downright tough

so at the very tip of the pyramid right tactics techniques and procedures it's really darn difficult for an attacker you know to change their tactics techniques and procedures because they're they're used to doing things in certain ways and as you may already know the numbers of the ttps they're relatively finite so if you can hone home in on and identify their their specific ttps this is where you want to be and this is where mitre attack lives so let's talk about the testing flow that the attack evaluations do in the in the fall so in the fall of 2020 the fall of each year they conduct a series of tests and this in this case

uh it was over three days uh they set up a number of systems in their test range and what they've done as i've said earlier is they've have they've picked out two thread actors the carbonac threat actor and the fin seven threat actor and what they do is they conduct this uh series of uh of steps or stages 20 major stages all right composed of 174 sub steps so they run these steps against the systems and then they bring in various vendors 30 vendors this year and the hope is that the edr technologies will be able to detect or protect against these the very steps and attacks two new things they did this year is

they did a first time ever a multi-platform test so both windows and linux were in scope there's about 12 sub steps that involved a linux host and they also did a prevention or protection test as well so as they are executing these uh these steps they again they picked out the carbonax redactor and the fin7 thread actor and if you have really really good eyesight you can see that fin7 they're financially motivated they go after u.s retail restaurants and hospitality carbonac this threat actor mostly goes after and and targets is known to target banks right so they're going to mitre is using the known tactics techniques and procedures that these two threat actors are known to do so how is this scored

how is the data scored so it looks like this when they run a step then you would hope that the product is going to detect it okay if it doesn't it's considered a miss right so the very first step you'll see i'll show you is uh the opening of a rich text format document with uh with word so the hope is that that whatever protective product detective product that you're using is going to be able to record that if it records it at the very least that is scored as telemetry now if the product under test is able to perform additional context and additional analytics on that on that step then ideally you'd be able to determine

that this is potentially looks like a miter tactic perhaps they're someone is trying to achieve uh initial access or perhaps this looks like fishing and even better if the product can determine that this is a miter technique then that that's even richer so you have sort of data over here on the left and then you begin to have enrichment or analytic detections is what mitre calls the ladder 3. so minimally processed data over to enriched data what else before i get to the actual data there's also two additional qualifiers or modifiers that you'll see in the data number one is miter allows for a configuration change so imagine your product is being tested and let's

pretend that it missed right it didn't register telemetry or perhaps it didn't register a tactic or technique mitre allows you to go back you as the vendor to go back and then change the product sometimes the changes are minimal maybe it's an expansion of a field in the user interface sometimes the the changes are are a little more serious you have to make actual changes to the detection logic of the product so whenever that occurs they note that as a config change if there's been a significant delay in the time that the detection comes back into the console sometimes some vendors they're still using technology such as sandboxing which is can inherently have some delays

that's one example where you know there's a delayed response and then that's also noted in the console so let's do this let's uh with the time remaining let's go over to mitre's actual website and let's take a look at some of the data so this again this is the enterprise matrix the enterprise attack matrix we have the tactics across top and we have techniques under each of the the columns and here on this tab on this url this is the results for the carbonac and fin 7 testing normally when if we were doing this live i would invite audience participation and i would ask individuals in the audience you know which of the vendors we want to see the results for so

because we're not doing this live i'm pre-recording this then i get to pick so what i'm going to do is i'm going to pick my results so here are the results for sentinel 1 and then i'm going to pick two more i'm going to pick microsoft because everybody knows about microsoft i get asked about that one quite often and i'm also going to pick on uh or rather i'm just going to choose a checkpoint and i'm choosing checkpoint because there's been a lot of talk and social media checkpoint was um uh i was pointing out that we did not include them in uh in some of our marketing slides so i will definitely include them

uh for this comparison um so you can see over here i have my tabs opened up i have the data for sentinel one uh on the first tab i'm going to scroll down look at some some graphs and charts and on the second tab i have the data for microsoft i'm going to scroll down to the same chart i'm going to the same thing for for checkpoint um i'll ha i also have two additional tabs that i want to point out that i've pre-opened this one here this is their technique comparison tool so it's super handy when you want to compare techniques between multiple vendors you know how i talked about there was 174 steps that were conducted in this test

you can see all 174 steps here on the left hand side scroll scroll scroll 174 steps i find this interesting because sometimes i've seen some vendors marketing departments they've said that they scored 100 detection and over 20 major steps right i said well there's not 20 steps there's actually 174 there's sort of like 20 major but there's a total of 174. um so you can actually come in here and take a look at all 174 steps and you can see whether or not someone has detected or if they or if they missed and then over here there's another great tool this is the participant comparison tool so here you can pick out two different uh vendors that you want to compare so

you know sentinel one to anybody else to microsoft or checkpoint and then you can very quickly say show me everywhere the two products were similar where they had matched steps and then show me where they were were different right so it's a super handy way if you've uh if you're doing some kind of edr uh comparison or bake off right now and you want to know how two vendors how your two final two stack rank on this particular test it's a great simple way to do that so let's take a look at some of the data so let's take a look at first let's look at let's look at checkpoint so uh when i look at the data

here and you can see here's all the scoring on the bottom so again if there was a no if there was a miss or if there was no detection that comes up as none and what you can do with these charts is you can dynamically filter out the telemetry the tactics techniques and the general detections that the product has actually made so very quickly you can see the these are the areas that the checkpoint product has failed to detect and then you can do the same thing over with anybody else that you wish to compare so then on the on the microsoft side start filtering out all of their detections and you're left with all of the misses the areas that um

that the products did not detect and then of course you can do and if you sort of eyeball between checkpoint and microsoft i know i would hazard to say that probably it looks like microsoft missed a little bit more than checkpoint and then you can do the same thing for anybody else in this case i've got sentinel one up so i'm going to filter out all of my telemetry my general my tactics and technique detections and you can see that sentinel one we we didn't miss so another way that you can sort of see the totality of the data is you scroll back up and you can see over here over here on the right

so this is the row for the carbonac and fin7 testing of of for this year and we detected in 174 of 174 steps and then you can go back and up and take a look at the specifics for for microsoft so they're one set 151 out of 174 and you can do the same thing for uh for checkpoint so 162 out of 70 out of 174 so they did they did better than microsoft something else you know how i mentioned the configuration changes that vendors were allowed to make so what you can do to sort of figure that out really quick is you can click on the actual uh testing round right so i clicked on the carbonac and fin seven

round and all the techniques and all the all the steps are shown here okay so step one a one a one you know they scored telemetry and a technique okay but it also points out when a configuration change was made so you could just really quickly just do a uh you could do a search for configuration change so that comes up 32 times on the first round of testing and you can really quickly go through and find you know where where is it that say for uh for checkpoint that they had to make a configuration change in order to in this case score a technique here to score a technique and you can see that it was a logic

change uh each time for each of these so there's about 32 of them on here and then again you can do the same thing for for microsoft or drill in there and let's just do a find and we've got about 22 times that that shows up on this page and then we can do the same thing for sentinel one do a quick find and again it's zero right so in other words we we didn't have to make any configuration changes uh to the product to get the results that we got last thing i'm going to do sort of keeping my eye on the clock here is what you can also do is sort of take a

look at the configuration that had to be brought to what like what products the vendor have to bring so when you take a look at uh let's take a look at microsoft the microsoft configuration looks like this they had to bring defender 365 defender that was enabled they brought defender for endpoint of course they brought defender for server they brought their linux agent for centos they brought something called the microsoft defender for identity and they brought mcas cloud app security and when you compare that to say for example what we had to bring we brought our windows agent and our linux agent then of course our our management platform so um there are like i mentioned before 30 vendors that

were tested uh this year so i always encourage uh customers to kind of ignore the marketing hype that any vendor publishes i i i don't even bring up my own uh marketing data but you just come straight to the mitre site it's free and you being you can become sorry you can begin to just look at and compare all the data on your own to determine what's going to be the best fit for you and your organization i am going to pause this now and i think we're going to move into some live q a so thank you very much for tuning in to my session today and have a great day for the rest of

vancouver b-sides