Building The Best Team To Hunt The Biggest Threats

you thank you right thank you very much good morning everyone my name's Martin Lee I'm also part of Talos I have the pleasure of managing the Telus outreach team within amia a little bit about me I've worked in security now for almost 17 years for the past four years managing the Telus outreach team across Europe before that I was working here in Belfast for alert logic prior to that it was also at Cisco you can't really escape Cisco they're a bit like the Mafia once you've worked for them they they pull you back in eventually and then for many years before that I worked for message labs in Gloucester detecting threats coming in over over email I am a

very keen if not particularly fast or indeed in any way talented runner but I do enjoy it and when I'm not hunting threats I tend to be out into the into the hills running and I'm also a recycled human viral geneticist I started my career researching the genetics of human viruses before basically by accident ending up in security but I'm very glad that I'm here as a sponsor of the conference you have one obligatory a semi marketing slide which is this which is what we do in Cisco talis we are Cisco's threat intelligence and security research organization so what we do in the wider group is follow the telemetry and the intelligence the information that we have about what's happening in

the threat landscape understanding what the bad guys are up to and what those attacks look like and then using that information to power Cisco's product security product offering so that we can protect customers against the threats that are out there but in order to protect against these threats you actually have to understand what happening and most importantly be able to spot what is new what is different and what you actually need to concentrate on and understand how the threat landscapes shifting and this is what we specialize in as part of the outreach team what we do is work on analyzing the intelligence that we've got spotting what is different and understanding that and we've actually

been very very successful in what we do in Earl and Edmund have just talked about sea turtle so here we have a nation-state threat actor undermining one of the fundamental protocols that allows the internet to function we also we identified VPN filter this was a network of 500,000 compromised small office/home office reuters across the globe that were poised to launch a giant attack we don't know what that attack was because we managed to stop that attack before it happened working with our partners in the public sector to take down the commander commute a command and control networks of the threat actor so they couldn't launch an attack and there's a load of other stuff that we found we found

various rats remote access Trojans we've also identified a previously unknown apt group which is particularly interested about the Korean Peninsula and we're also identified a bad rabbit which is a criminal ransomware that appears to have been derived from the same source code that was used in not Pecha which is also quite an interesting observation so not only are we finding these very very interesting things first I'm also incredibly proud to say that we're an award-winning team se magazine recognized us as having found the most important cybersecurity discovery by a company research team this this year and Paul who's in the audience and Warren were the recipients of the Peter saw reward from virus bulletins for there

research on on sea turtle this year so I like to think this is evidenced that we're doing something right the trick is is well or what are we actually doing and what is special about what we do and how we operate that allows us to find this and I think to be fair what we do has more or less happened by accident there hasn't been a grand plan but we've been trying to implement there is no manual out there of how to manage or how to make function a threat a threat research or a threat intelligence team but I do get out and get to talk to an awful lot of organizations that are trying to do the same thing as us many

of them less successful and some of them to be fair have basically failed in creating a threat intelligence team so it's very interesting to compare what it is that we do with other organizations and also especially those that really haven't managed to get such a team functioning and I think it it really starts with these assumptions of how do you actually go out and hunt threats naively people tend to think it's like some kind of sausage machine and you put in at the top data you know lots and lots of data a big dollop of data you mix that in with some tools probably a seam solution because that gives you a nice dashboard and then you've got some

kind of procedures that are that are carved in stone that are rigid and what it is that you have to do you turn this handle of the sausage machine and you find threats coming out of of it in fact no this is absolutely not what we do the way that we function is more to think about threat hunting as a stack of technology we certainly need data we need you know data forms the hunting ground in which you're going to hunt threats this data can come from many sources it can be internal coming off your own networks from your own telemetry from your own visibility of what's happening on the Internet it can also come from external sources

the both of private and public sources that you can tap into that anyone can use to hunt threats as I said sea turtle where I primarily identified through passive DNS which is available to anyone and also through virus Topa which again is available to anyone so you don't necessarily need this secret magic store of data which only you can access you certainly need some kind of tooling this is quite in my opinion quite a narrow and thin layer within this spec we need some way that we can ask questions of the data and get back answers but far far more important than that tooling is actually having the right people having people with the right skills and most

importantly the right attitude in order to hunt threats and those people thrive in the right culture you can absolutely kill the best people with the right tooling in the right in the right data environment if you have the wrong culture this culture piece I cannot emphasize just how important it is to enable and encourage people to hunt threats I'll come to that in a moment and then also there's this very very thin layer on top which is kind of the management bit which is which is that strategy we need to have some kind of direction but realistically this is a very very small part of the writ of the--of the mix that strategy for me

fundamentally is just two questions you've got to have some idea of what it is that you want to find if you have no idea what it is that you're looking for you will never find it but if you have at least some goal this is the type of thing that we're looking for then you stand a chance to be able to find it or to recognize it when you stumble across it there's also this other piece what are you going to do when you find it it's all very you know well and good finding things maybe in finding that your network has been penetrated by a in a criminal hacker or an apt gang but

unless you actually have a plan what it is that you're gonna do once you find this then well you might as well not have bothered if you can't resolve it if you can't do something with it then really what was the point in looking and finding something for us I think these questions actually have very very clear answers what it is that we do in our team we are looking for the most significant new threats that are happening on the Internet we see our role within talis of where the guys that protect the entire internet we're looking out for everyone that's on the internet we want to hunt down and find the bad guys find the new things that

they are doing be the first people that find these things so that we can protect our customers and protect the internet as a whole and also most importantly inform the community inform everyone you know if you don't know what the bad guys are doing then you won't know that actually you need to protect yourself against this so actually being public talking about the threats talking about how that threat landscape is changing and what you need to do to protect yourself against that is a very very key part of what we do and this is ultimately what it is that we're trying to achieve to find these these threats we need to adapt the intelligence cycle

so this is something that that's come out of military intelligence since the Second World War when the the military was faced with a problem how on earth do you get soldiers to the right place on the battleground where the enemy actually is in a lot of cases in the Second World War you've got groups of soldiers entire regiments wandering around trying to find the enemy and when they get there where they think the enemy is the enemies actually moved on and so over the past seventy eight years or so we've had this maturation in the intelligence community of trying to define what intelligence is and how it acts and it starts at that bit at the top with the

planning and direction firstly it starts with just a question you know what is it that we're hoping to find what is it that we are looking for and then to go out and find the data that could actually support the question that it is that we're trying to ask so we're starting with a question what are we looking to find we need to have the data to which we can ask that question so we can get out a response in in the world of IT very very quickly I mean you can all come up with some very very good questions that you'd like to ask your network and your systems and your users about what a particular threat might

look like on it but when you start asking those questions and start looking for the data almost certainly you will find that the data that you have doesn't actually allow you to get an answer for that question that you would like and so to begin with we go through an iterative process we'd like to ask this question but actually we we just don't have the data to ask that nevertheless in looking at this data we can find it yeah there are actually quite interesting questions that we could ask of it nevertheless which are kind of similar to that question that we started with so we have this process at the beginning of going back and forth looking at the data that

we have thinking about the questions that we'd like to ask until we come to an interesting question that's relevant that's going to be useful and we have the data that can support that and then we go out and we collect that data we process it we collect in various different sources analyze that to understand what it is that we found what's the context how does this fit what can we actually do with this and then in our case emanate that get that information to the people to whom it is useful but the magic really happens in that processing and analysis and data collection part and there's a number of techniques that we can use to make sense of the data

that we have the typical bit is really just classical in engineering as engineers we love to have repeatable and predictable processes with nice defined inputs and outputs and we can put something in at the beginning we can follow some nice procedure and come out to a nice predictable end of where we have a nice clean answer to the question that we answered this is great this is really the holy grail of what we want to get to a very good example of that in a typical type of thing that you'll find in a threat hunting team a new threat intelligence report will come out it'll have indicators are compromised within that a very very sensible question to

start with is okay in my system in my environment have we been hit by this do we have these indicators of compromise within our systems and so we kick off this nice predictable repeatable process we will look through the data that we have we will look to see whether we've got these indicators of compromise and we will come out with a nice clean yes/no at the end of it lovely people do this time and time again and they tend to come out with the answer no why well because within the report you're just gonna have one set of ICS for one particular attack and if the attacker has changed a single bite in his malware you're gonna have a

different hash value the attacker may well be using a different domain a slightly different IP address when he's attacking you so actually if you're only looking for IO C's that are published elsewhere within your data to be honest you're unlikely to find anything you might be left but probably not I think the question that you should be asking is what might such an attack look like within in our environment so we're not specifically asking the indexes have you seen this we're thinking well if such an attack happened what traces would it leave in the data that we have but we could look for and this process is actually something it's quite different it's a very innovative process we have to think

about our system our environment our data what might those traces look like it's not a nice clean predictable process the outcome might actually be undeterminable I mean may well come to an answer which is well we simply don't know or hmm probably not we're 80% sure but we haven't seen this but if you keep doing this sooner or later you will find something which is interesting and something which is worthy of further investigation and identify something significant what we really want our analysts to continuously ask is these questions what if what if this hand what would it look like what else might be happening and make this a process of introspection and continuing asking of

the data what else what might be happening what might I be seeing how do I find this within the data when we come across new ways of asking questions of our data and looking for outliers and modeling it and finding it this is the type of thing that when we get a success we can turn this owner into a into an automated procedure this kind of world is perfect for machines to operate this is not what you want your people to be doing this is a waste of human resources of that ingenuity of that professionalism of the sense of a but you want to create within a set within a threat hunting team this is the

type of thing that you want your threat hunters doing asking questions thinking what if what else what might be happening what might this be looking what might this look like so what I really really don't want you to do is to create an assembly line you know the 20th century way of working where we have people on very very tightly defined work and responsibilities conducting repetitive tasks following tightly defined procedures which are chiseled in stone which have never changed for a manager it's great because it's really really easy to measure a reward we look at the people that are performing these procedures the fastest so how many how many firewall alerts did you resolve today you know are wow I'd resolve my

firewall alerts in less than five minutes each you know why why are you doing this it makes no sense what we want is to create a 21st century way of working which is a very very different way of doing things we need people to collaborate and share ideas and come up with new ways of doing things new questions that they can ask of the data new ways of modeling it new ways of searching for things and be very very open to new ideas and cultivating that to get to that point we need to be oriented around a goal not how many firewall alerts have you resolved in a day not how many tickets have you closed

but how have you actually contributed to what it is that we are trying to do and this is the outcome that we will measure a key thing to remember about being innovative and trying to do new things you will mostly fail most times you will get it wrong and you will just find nothing but that's actually what you need to do we need to go through many many ideas of trying out new ideas again and again again until we find something that works but that's great this is what we've got to do have lots of ideas recognize that most of them aren't gonna work but never mind that's fine we'll just fail fast fail early and move on not invest too

much of our own selves in our ideas but just try them out see if it works if it looks interesting brilliant let's keep going but actually most of them they're they're probably not going to work but that's fine in our own team what this actually looks like and I spent ages searching for a stock photo that would illustrate our way of working and yeah mostly it looks like this it's a bloke on his sofa with these dogs and a laptop with in the European team we all work remotely we all work from home and yeah this is a very very similar way to the way that I work but for us this works very very well indeed and the key things

that I think means that we are successful is that we have a very very strong sense of mission of what it is that we're trying to do we're the team that hunts down the bad guys on the internet we find them first we protect the internet this is a very very strong sense of what it is that we're trying to do and that strong sense of mission itself drives a very high degree of self motivation that itself also drives lots and lots of peer collaboration because people want to find things they want to hunt want to find it first the hunger to fulfill our mission to work together across the team keep that motivation high even though we're all

working remotely and we're collaborating over remote working systems and encouraging each other to keep going and keep looking what do you think of this what if we tried that and it turns out and it becomes that actually what we do it's not really a job it's actually a lifestyle and that works because we can build in a lot of flexibility into the way that we were if you want to take your dogs to the vets in the middle of the day that's fine you want to collect your children from school absolutely fine because I know that comes six seven eight pm at night and on the weekends you are still gonna have that hunger because you want

to know what your scripts are finding what are the bad guys doing what are they looking for so for us this works very very well indeed but Brian as you said it's all about passion it's all about having the passion it's all about wanting to make the difference and wanting to hunt down the bad guys so if I take sea turtle as as an example obviously Earl and Edmund went into this and far much much more detail technically but I think it's interesting to to look more at the process and the environment in which this happened so it did all start with this taunt we have someone making fun of us on the Internet no way

him down so clearly we want to find this threat actor and you know he's taunting us we want to put a stop to this we want to find his malware very interesting that he's conducting commander control over DNS you know that's interesting we don't often see that so what we want to do is find more malware conducting commander control over DNS so we'll set up various rules will look for malware such as that we know what it is what questions we want to ask the data show me more malware which is conducting research using DNS and ultimately this brings us to this document the one that I'll presented that has the two macros in it here we

have a piece of malware which is conducting command a control traffic over DNS brilliant first question that you want to ask well what is the the server the IP address that is communicating with so we find these IP addresses and the first question that we want our analysts to ask is well what else is happening what what else is going on and this is the point that we have that moment when we look in the passive DNS and we actually see the web mail dot governmental domain had been pointed at one of the IP addresses that was being used as command control so here we have our evidence this is actually quite significant there's been a deal has

probably been a DNS redirection attack happening here something has clearly gone wrong that we've had a legitimate a very high reputation domain which somehow is pointed at this IP address which we know is being used as part of the malicious campaign so here again we want that to spark off that reflection war what else is happening you know what if what what else is going on which leads us to look into more detail and we find these range of domains that are being redirected we also identify again you know what else might be happening what else might be we were seeing and we find domain validated entirely valid TLS certificates and public registries that have been issued

for these domains where we've seen the IP address changed maliciously at the same time so clearly this looks very very much like someone is doing some kind of man-in-the-middle or certainly an impersonation attack for us well the first thing we want to do is we want to share this we want people to be aware that this is happening so we passed that dissemination part of the of the process we publish our findings with those indicators are compromised in there and what we like to think of happening is across the world people are reading our reports and a taking action and certainly we we did see action coming out of this we saw a lot of interest

within the press a lot of reports being being written about it clearly lots of other people were thinking about what it is that we'd found and were looking at their own systems and their own data for more of this this led to the department of homeland security in the u.s. issuing this emergency directive you agencies to audit their own DNS information and for us this is this is like really really interesting because this is kind of suggesting that there's more going on than we might initially be seeing also the fact that we've been very very open about this and saying look we're seeing this led to further intelligence coming to us and people sharing information with us which

allowed us to then put together what was the full picture of what was happening identified that actually we had two different DNS redirection campaigns going on and this one with sea turtle being conducted almost certainly by a previously unknown nation-state threat actor very very significant stuff we look at about the steps that led us to this it's actually mostly about engagement well firstly we've got that community engagement of someone tweeting us a malware that they thought would interest us okay that's great this is us being part of the community we want people to be part of the community and to share things and to people to tell us information that they think might be of

interest to us the fact that it had that taunt very interesting observation camarda control over DNS that's interesting we want to find more of this so we set up these automated processes that are going out and looking for more of this kind of malware which brings us to this HR document this job recruiting document that was conducting that commander control over DNS and then we've got that very very interesting observation we've got a governmental domain which is sharing DNS with this malicious IP address that leads us to reflect what what else might be happening here how else might we find more evidence of this attack within the data we then publish it weekend we've got that piece of community engagement

of people sharing information with us which allows us to do more research which allows us to identify the full picture there very very few tools in this there isn't a seam in sight there is lots and lots of engagement and actually little in the way of tools this is mostly people being people and communicating and being innovative and having good ideas and trying them out and wanting to get to the bottom of the story so how do you go about threat hunting and what advice can I give you the first thing that people tend to think of when we mentioned threat hunting is it's dead easy you just want a seam and then you go on the

dark web right I get a lot of this a seam is a really good place to start at least it makes you think about the data that you might have how you might access that and how you might visualize it the great danger with the seam Solutions is they're on Rails that they will show your analysts only one view of what's happening out there and actually make it difficult to ask different and innovative questions of the of the data I had a very very good example of this the other day I was talking to someone who's wanted to start up a threat hunting system and he wanted to know what would an intrusion look like in his

network if somebody compromised the laptops and so he went to his threat hunting team what would this look like if someone compromised a laptop how would you find this and their response was well the desktop protection would flag and we see the light go off in our in our scene and like guys know this is not the way of doing it what if it was a successful intrusion that didn't trigger your desktop protection what would that look like these are the questions that you want to ask that's how you're going to find the new stuff as with the dark web yeah there's all sorts of malicious activity happening in the dark web again I talked

to someone a few weeks ago who's really buzzing about researching in the in the dark web and told me that yeah it was absolutely what you need to do because you can find the bad guys discussing the denial of service attacks that they're going to do on your network before they happen which is great and he's not wrong however when you look at the the set of things that actually happen and the things that are discussed on the dark web yeah some of the things that are discussed on the dark web actually do go on and happen but a lot of it yeah it could just be noise just people discussing things and what they're

discussing might not actually happen and equally if they're talking about this brand new super cyber weapon that they've developed that they're going to make for sale on the dark web yeah right the threat actors that are actually coming up with these new cyber weapons wanna cry not mentioned on the dark web not discussed on the dark dark web not Pecha not discussed on the dark web sea turtle not on the dark web so in fact when we look at the things that you actually need to worry about and concern yourself about these are certainly the things that I happen but that overlap with the things that are happening on the dark web might be smaller than you

think so the first place that I'd start in terms of that strategy is you know what is it that you are hoping to find what would you like to find in your environment and then what would you do with this once you've had once you've found it how will that improve your security posture and the goals of your organization another question to ask are there actually better things to do would you be better off having a larger and better resource team which is doing patching or sorting out user authentication rather than spending your time looking for advanced threats within your within your network it's a key question to start with threat hunting isn't necessarily what it is

that you should be doing it's great if you've got the maturity that you think this is going to be a useful addition but may well be other things that you might want to consider doing first keep it for me is is the culture we want to enable the team to hunt that will mean you will have long long periods where nothing is happening where you're finding nothing when all your ideas are failing that is actually a good sign you need to encourage that you need to keep people supported and motivated we want lots of ideas we want to know that it'll take a long time to find the bad guys when we do find them it's probably

gonna take a long time to understand what's actually happening this is a good thing we don't want to build that assembly line where we're very tightly controlling what people are doing so that we can measure it so that we can report on it so that we know that they're following procedures checklists have their place but where we are in the Internet at the moment we don't necessarily know what those checklists are we still in the discovery mode we need to find out how we identify the bad guys not necessarily following prescribed wisdom that may have been developed you know four or five years ago great for finding the threats four or five years ago not necessarily not

necessarily relevant now and then that big piece it's all about human connections getting people to work together getting people to share ideas share information working together to solve the problem people key key key bit we want a mix of both technical skills people able to do malware analysis software engineering skills so they can write the tools or work the tools differently and also data skills thinking about the data that we've got how do we model it how do we find what's unusual how do we define unusual being able to think of what questions we can ask how might we be able to uncover the unusual within the data that we have and I think a lot of that really is

driven by curiosity I think the key thing that we should be hiring for within threat intelligence and threat hunting is people who are really really curious about things who like asking difficult questions and really getting to the root cause of what's going on tools your sales guys will tell they tell you this is the most important bit no no not at all it's all about what questions can I ask the data and how easy is it to ask them you do not need an all-singing all-dancing data system to find things quite the opposite you need easy ways to ask questions of the data rather than relying on a vendor showing you exactly what they want you to see and only what

they want you to see and then that date of it yeah you need some kind of data often it's driven actually what can you get access to and what is being collected it's never going to be perfect that's fine just accept what it is that you can get doesn't have to be all internal external source is a very very good indeed and for goodness sake don't store everything think about what data would actually be useful to you and for how long would that be useful in restoring every single log within your system for five years probably won't be helpful to you but what happened yesterday might be very very useful but again think about the sort of

signal-to-noise ratio what might you be able to find in the data that you have access to and which you can actually meaningfully store so to wrap it up I mean really the key point if you want to build a threat hunting team just empower people to thread to hunt threads that's it you've become a threat hunter by hunting threats and you do that and in an environment that empowers people to get on with it and do it and enjoy it and get rewarded for it and with that I will thank you very very much indeed for listening to me and both open up for questions now or also if you want to come and chat to us on

the booth we'd be delighted to talk to you in more detail thank you

Gordon Oh Michael's work okay yes hello all the rain and where does attribution attend all of this and if $1 is there any value in it given that you can't always actually work out he really was behind it again you've got I think it comes back to that initial question how is this going to help you the the example that I like to give even though the pr2 people have told me not to use it so Daniel have to close your ears for a moment if you get mugged in a dark alleyway does it matter if it's hairy the heroin addict or Bob the crack addicts who mugs you I think it's more important not to get mugged and to

understand what it is that you need to do in order for that not to happen again rather than focusing on who was the person that carried out the attack certainly attribution has its place I think it's useful to be able to cluster together attacks and be able to look for similarities between them because that helps you think well if if two attacks have some kind of commonality this would be a very good thing to search on in the future so knowing how attacks overlap and those features that allow both allow you to do attribution and also to to aggregate together attacks are useful knowing exactly who was behind it personally I don't think actually gives

you that much information people tend to be very very interested in it but again it's that question what are you hoping to achieve by this how with that knowledge actually improve your security posture or would you be in a better position just focusing on how do I protect and how I detect about this better in the future so I know there're that there are many people who disagree with me and there are other teams that that actually do focus on attribution and if it is useful to you then then great go for it my personal opinion is actually I don't I think this helps a lot less than most people think I'll get be quick obviously you've got people

down there doing this threat hunting you mentioned there about people want to be first to find the threat tell us doing it fireEye doing it CrowdStrike you're doing it SecureWorks are doing it how does that work then in terms you've got all these those those four companies they're probably many many more I've forgotten how does it work in terms of the people then trying to be first against people doing the same thing for another company I think it's mostly I think it's incredibly helpful because it comes a very very strong motivator in order to want to find things first because this is how careers are made careers are made on finding the most interesting things first so having that

little bit of pressure behind because there's other people also looking for VIP for this I think focus is the mind enormous Lee and helps people focus on what it is that we need to find how do we get this information out as soon as as soon as possible so that side is great also we go back to this community thing we share enormous lis within the the community were founder members of the cyber threat Alliance we're proactively sharing information and intelligence throughout the security community those companies that are that are seen as our competitors in a commercial environment we're still part of the security community and we are all united against a common foe we united

against the bad guys that are out there that are trying to undermine the internet that are trying to undermine the systems that the power in our lives and allow us to function those are the bad guys that is what our focus is on and to be fair there's more than enough malicious activity out there to go round we are not hunting in a very very small hunting reserve there is a lot of activity there but that's there to be found which i think is great because we can get more people in there's lots of big game out there that you can have come down and find anyone else who wants place nope Martin thank you very much thank you