4000 Holes in Blackburn, Lancashire: Defence at Scale

hi everybody four thousand holes in blackman lancashire um a lyric from the beatles song a day in the life song whose lyrics were mostly taken from um stories that john london was reading in a newspaper and it got me thinking um the poor maintenance highways maintenance chat from blackburn council that will have had to deal with four thousand holes probably had absolutely no clue where to start with that it was an enviably large task four thousand songs like a lot of potholes and i realized it was very similar to the job of your average vulnerability manager out there it was a huge number of vulnerabilities to fix and no idea where to start them this is going to be a very quick very

dull very boring talk on a very quick very dull boring subject um there will be a longer version of this talk at some point just a warning these slides are for people that can actually pay attention for um to things there's no fancy graphics there's no um clever memes or bad bad jokes in the middle of them it's very dull and boring so um let's start by talking about how most vulnerability management programs start really you come along somebody realizes you need to do vulnerability management in your organization so you go out you buy quality messages and expose any you know probably it's kind of those lots out there the big ones are all quite similar

um and what you get back is you while you fire it first at a huge number of devices you get an absolute pile of data back you get a colossal number of vulnerabilities right across your organization that need fixing and at this point one of two things happen either you become security policemen you put your hat on you go on with your big stick and you sort of moan the people that own these systems and write this software and tell them you know they should they should be taking your um your security seriously and they should fix all the things you become universally hated and you're seen as a blocker that just stops people doing

real work and a lot of organizations go down that route and that's that's probably the worst kind of vulnerability management observation to be in where the entire company hates you um the other way of dealing with this that a lot of companies take is that they get their big report they put some nice shiny headers on it they drop it on a bosses table and they run like hell um nobody wants to be the guy to actually implement the fixes so it's like i've done my bit i've told you where all your vulnerabilities are let's get out dodge before um before somebody makes it my problem but the third option of this really is we start prioritizing we we look at our

big list of vulnerabilities and uh we try and find a way of prioritizing the important ones so how do we do that well normally people's first attempt is they look at their vulnerability management scanning tool and realize it has a thing in there to allow you to assign priorities and risks to you know to subnets or to bits of your network or things like that and people generally start off using those but they realized pretty quickly that the um their complex network with a million different things in other places doesn't really fit the very simple zones or categories or whatever the software has um so other places they go for the quick wins and they um they they realize it's

a numbers game they've got a big lot of vulnerabilities so given the choice of fixing one single cbs s10 or north rce over here or 20 cvss twos down here for some weak crypto that is you know you need to be physically money in the middle of the box to get hold of they're going to go for the 20 because that gives gets rid of 20 vulnerabilities makes their numbers look a lot better that gets rid of the one doesn't make them look any better at all really however in terms of actual getting your company popped that rce is the one that's going to get you pop not the weak crypto down there so that's not really an ideal way of

doing it either and some just privatized by cbs escort it's like okay we're going to take the raw schools at these vulnerability products chuck out the back of it and we'll do the high numbers first and do the low numbers last and whilst i'm normally a big apologist for cvss cvss is very much maligned and is actually good for comparing huge amounts of data so you know disorder it to another audit or you know this year's to last year's or things like that it's dreadful for comparing individual vulnerabilities um it can be made a lot better by using the environmental features of cvss there is actually a scoring system in there so that you can go and say for this

vulnerability um there are these mythications in place the the the the um the environmental stuff in there is quite usable but it puts you back in the same position as the first problem here if you spend your life not actually finding vulnerabilities but of actually feeding the vulnerability system data to classify your vulnerabilities and that's every bit as boring as finding them so um say so neither does i do we need to move to some sort of risk-based system so um what do we want to do for the risk-based system well we want to prioritize them based on the risk of exploitation going back to our real example we want people to focus on that north rce who really is going to

get on an amount and not so much on the on the low stuff this is you know it there's a theoretical risk there but there are so many other stars that need to align it's not likely to happen [Music] the trouble to make risk decisions is you need data to make those risk decisions on and we've just highlighted the problem that we're actually trying to fix here is we've already got too much front data with the vulnerability data just going hey we've got a big pile of data to fix this big pallet data what we need is an even bigger pile of data not a great place to be starting from um and even if you already have this data

in a lot of cases this data isn't all in one place you may well have something like a cmdb for those that don't know configuration management database um is what it stands for united steel terms in a lot of other non-italy organizations it's just a risk register but essentially when i use cmdb what i really mean is list of all your um your hardware assets and all the data about them seems to be actually much wider scoping thing but that's what we mean in this case um as i said you may well have these but in um in in the old um xkcd cartoon goals you know when um if somebody realizes you have three

different policies for something what they'll do is they'll look and go oh um you know we should have argument those are all into one policy and what you don't end up with is one policy what you actually get for policy exactly the same as cmdb's if you've got several different cmdbs or asset lists containing bits different bits of information about your hosts and your notes and things like that normally running out and um and trying to build a new cmdb will just get you another cmdb that's half-assed and only does half a job so what this talk because this talk is really about is how to build that cmdb so that it gets adopted and becomes your wonderful

sort of truth i also remember these are grown-up slides if you don't know the s-cat kcd cartoon i'm looking for go google it go find it yourself um you're not going to get pretty pictures in this talk so um starting from cmdb one thing to bear in mind you have something your company probably needs you own a list um of all the hosts on your network if you're using the discovery right you've got a full list of everything on your network that is very valuable data to a lot of organizations in the organization a lot of places need to know what pcs have we got you've got that list that's a good starting point um

what you really need to do is find a project and use that data and hijack it now i i um i kind of hinted at this on twitter the other day and so you can't say that when people get the wrong idea you've got that data you want to share it you want to get it out there you don't want to run you know hold it up for a ransom for other people stuff so we're going to hijack it we're going to work together on a mutually beneficial solution um or hijack it if you have to but yeah essentially we're going to buddy up with somebody else that needs that data and let them be the driving force behind

building your cmdb um so our first cmtb project isn't actually going to be cmdb it's going to be something else that just happens to be sat on top of our cmdb so what sort of data do we want to put in there we'll say we've got our list of hosts and our list of vulnerabilities what else do we need in there to go to make risk decisions around that data um so first of all we don't want to reinvent the um the wheel we don't want to be the owner of all this data we want to try and grab other people's data and get it into or close to our cmdb so we can use it we

don't want to actually start owning a lot of this data but we can reach out to lots of other teams to see how their data stored and see if it would be better stored in cmdb you'll be amazed how much your organization works on big lists in excel where once a year they may come along pulling your list of posts and compare it to you know what they paid for or what the network teams think they've got or what the licensing team things they've got the trick is to reach out to those teams and say hey we've actually got all this data over here in this lovely cm in this lovely tmdb that we've built why don't

you take your data and put it in the same place and then we can use the same reporting layer the same management tools have the same people look after it and try and get them to work with you to put their stuff and they've currently got holding little spreadsheets together they'll probably get a much better thing out uh result out of it they'll probably get bits of people to own the process maybe own the database it runs on or own the management layer and generally your cmdb starts to grow into time over an actual useful thing that everybody um bolts on top of um and point out to them that you know at the moment they're taking your data

say annually they can have it in real time you if you're running a management software is updating your list of hosts all the time so could there so that their gdpr map or whatever they're doing also has this data going in there live and you know so you know you can basically you're helping them to make their thing better by building on top of your thing um so what sort of data do we want in there um things like network types network segments that sort of thing if you do do well segmented networks um then you know make that a factor the idea was here what was what was basically our starting point is um a host contain a host on our public

perimeter containing very private data is the highest risk on top of that we're adding mitigations we're adding things that lower the risk so networks that went if it's on a network segment that you can only get to if you've um if you're in a special group of users and use 2fa that's the mitigation we can add that into our um into our list um host type um you know sort of a a vending machine may be a whole yeah there's a whole different risk to a an access control system so build that into there reach out to people like your compliance team they have loads of information about the data that you've got on service and how it and how

important is the company and how much risk there is to the company so things like pci it's like you know if if it's um within the cardholder data environment um you know with the the cde that's a very high risk if it's not it's less of a risk same with gdpr if it's got pii on it it's quite a high risk if it hasn't then it's quite a low risk reach out to all these teams get their data back put it with your data so you can use their data to help drive your prioritization um similarly the actual financial value um is is is a relevance big important service system high risk little desktop probably not so much risk um some odd

ones that we've used that have worked quite well that data recovery priority if something's got a 15-minute sla in a dr situation that's probably quite important to the company probably care about that a lot made that patch if we've got something that's got a three-week sla that's holding the the lunch ordering system probably not going to be top of our patching list either change status was another one that we tied into um you know if something needs you know that an engineer can just go and change without approval that's quite a low resistance probably not particularly one that is top-of-the-line to care about if it's one that needs three layers of quite senior approval again that's probably a high risk high

value system so you can build that into it things like slas or wildlife there's all sorts of things we can put into it um and i say it's all about mitigation so you know if you've got a list of things that are behind the waffle things that aren't the things behind the waff naturally lower that risk therefore we can give a little bit of time on the patching um and literally anything that helps us decide the the importance and of the relevance of the data on there and the longer version of this talk i talk a lot about set architecture and how it can save you everything sec architecture do is a mitigation everything that this

your prioritization should be doing is taking into a into account any mitigations and giving people um longer patching sla based on the number of mitigation they so go talk to your architects talk about what they've done try and codify that try and build that into your calculations as well so the end result what have we got we've got we we no longer have a patch all the things attitude we've got a perhaps the most important things first attitude which is great because you can actually say to people well we don't ever you know patching isn't a zero-sum game we don't expect to patch everything we just want the important stuff and this is the important stuff

the um the sla between what's important how long it can be fixed that can be a moving line depending on the capabilities make it easy at first say okay anything that's a 10 we're going to give you seven days to fix anything you know and then in this case actually tens are really important so we're going to fix those under incident conditions and same same way anyhow anything that's um that's a high we're going to give you um you know a month and then come down to you know say okay can you do this within 10 days okay let's move outside that one anything because you've got the math say it gets really flexible to where to put

that line um it also cuts down a lot on arguments and the only real argument is whether or not all the mitigations that they're saying is the reason that they shouldn't be bothered patching does the math take that into consideration uh you know if they're saying well you know nobody can ever get to that machine instead well we'll take that into account no um you know the that um that machine is is is um you know there's no data on that machine so why do we care that same consideration so it actually clips out a lot of the arguments about why people shouldn't shouldn't patch um so some final tips to help it to land

um metrics reporting again a longer version i talk a lot about reporting about reporting is really important but reporting is a narrative you're trying to send across a message that message should be how well improvements are going it's not about how bad your infra team and your ops team and your your your devs are about um patching it's all about um you know how they're improving so control that narrative um highlight the good behavior don't necessarily highlight the bad too much um big numbers are scary t-shirt sizes are great so you know very much aim for um a critical high moderate low rather than you know a 6.3 versus a 4.4 and percentage changes are great if the

thing you're trying to show is that everybody is improving this and getting better so a ten percent sound change sounds a lot better than something going from a thousand to nine hundred so lots of percentage change numbers in there um oh my slides have stopped working focus only on what's attainable if you're only expecting people at the moment to focus on criticals and highs don't bother reporting on the moderates and lows or you know bury that deep in the report so nobody really cares about it you're trying to change behavior with this and if you think that this behavior you're not going to change in the moment don't bother actually reporting on it again this when people see that huge

pile of vulnerabilities it looks bad if you can say hey we we've we've cut out 80 percent of our um our criticals and highs that's a good message to get across um help people understand that risk owning is their job too many places it's like oh well we this this machine is hard to patch so we've asked security for an exception it's like that's not an acceptable thing the risk is there security can advise on risk um explain to them that you know this is their risk it's their choice whether they patch it it's up to their up to them or not um and this this report is to help them understand that risk and one of the things that we do bullet

works really well is letting them set their own slas we sort of jokingly said it's like okay you um moderate vulnerability you can have a year to fix that and they all sort of scuffed as if to say you know what that's stupid you know why would a vulnerability open for a year it's like okay that's that is stupid we agree but what what's acceptable what do you think is a sensible time scale to fix this in and then once you've got that in place you measure and say okay well we're hitting that quite easily can we get it faster do you need more automation if we're missing it let them you know there's no point in um

creating a target that can never hit um and another one again i just like um reporting in general another going to be a lot more in the longer version of this is people have someone back saying oh we can't afford the downtime to take that you know that box makes the money we can't afford downtime just to patch it for something really important the argument against that is that at the moment you had a vulnerability management problem you've now got from problems and an availability problem what happens if that box goes down for something unplanned get them to fix the availability problem if the problem is you can't patch it because it can't go down the problem is

that it can't go down not that it can't be patched that's just a symptom uh and some bonus slides for um red teams at the end just because i said in the abstract to include something for red teamers if you can get hold of the cmdb data if you can get all of the reports around it they it points to a lot of the um the dead bodies so things to look out for in that data look for things that are flagged as out of scope these are the dark colors these are a bit that nobody really wants to be bothered at looking after and if no if the people that are trying to drive

under management aren't looking after it there will be bad stuff in them um legacy stuff stuff that's that's old stuff that is is clusters too hard to patch because it's ancient again if it's not if it's old and start getting patched it's vulnerable generally look for stuff that's risk accepted as i mentioned before i despise risk acceptance as a as a concept the risk is always that you can't accept a risk it just mean you you just have decided there are other factors that mean that you can't meet in sla but anything's risk accepted again there's a reason that's not getting patched go and dig into it because you know it it's definitely bad um one of my favorites

the unknown and unloved in any organization stuff gets built that nobody ever like you know the team that built it got disbanded and sent six ways to win which means that nobody's looking after it nobody's patching it if you've got cmdb that implies ownership go and find out the stuff with no owner because that's the stuff that isn't getting looked after and that ladies and gentlemen at just under 20 minutes is my guided tour of um of of how to win at building a cmdb to actually do vulnerability management properly um if any of you have any questions you've got literally 30 seconds um come and find me online um glenn pegdon on twitter or most things

or come and grab me if you're actually here at conference or um yep if not find me online thanks a lot goodbye