Your SOC is doomed to Fail - collapse it and automate

so our next speaker is gary caldwell's so uh gary is with palo alto networks global solution and he's the architect there um and he has had a 27-year cyber security journey and who has taught and spoken at conferences across three continents to diverse business and technical audiences he is a resident advisor to the rogers cyber accelerator at ryerson university and is a periodic guest lecturer at universities gary speaks on a broad array of cyber topics from foundational security principles to technical concepts of current interest so please welcome to the stage gary caldwell's with your sock is doomed to fail collapse it and automate

thank you very much and um you know just first off inaugural b-sides for cayman and uh i'm thrilled to be part of this so thank you for having me um i don't intend to be completely contentious but obviously that you know the title suggests that there's some contention of you know potential there um i understood from the front desk there's at least three sock providers that are uh part of the conversation uh in the two days so you know if the tomatoes are ready to be thrown have at it it's all good so again thank you let's uh let's dive into this so what is the mission of a sock um you know fairly straightforward three

three key things that we're looking for in the in the sock infrastructures threat monitoring threat hunting incident response right essentially the ability to disrupt the operations of those that want to target and attack our infrastructures get after iap get off to our personnel exploit our weaknesses right so we need to have an understanding of that entire picture do constant review both our own and third party and external review get our auditors involved get feedback from them and all parties that we do business with so that we can have a holistic picture john's call out for the c-cert type approach a community type approach i think is a very good one i think it's tremendously useful

particularly in an island community like this where it is a smaller community a tight niche of of potential players under the 70 000 population where we have common interest common stake in in working together on that i think that's amazing john thanks for that um really useful so let's have a look tactics techniques and procedures right you adam spoke this morning extensively about you know the techniques approach on the deception side amazing talk very good conversational just referencing specifically to that output the most recent output for the mitre ingenuity attack evaluations um a really good paper that was was produced a study that was produced where essentially there was you know 25 i think it was between 25 and 30 vendors

took part and essentially an open study where it's like literally we roll up we give you the technology and good or bad they're going to examine it they're going to put it through some stricture they're going to market against an evaluation structure and obviously a framework and from that will come in output and that output is published so if you're subgrade or you miss key points or you have things in there that are maybe out of scope or haven't been covered well bad luck for you if you have good technologies well implemented well-structured understandable and translatable into something that people can use in a very effective way great for you and so if you read these

reports this one specifically is really interesting you'll see uh one of the things that's kind of cool is across the commentary from the vendors that were in play all the cfo ctos etc that made comment about the reports finding were largely positive the nuance in there is you know how effusive the the positivity was um there's a handful in there that did incredibly well as expected ourselves and our peers it is what it is um you know great players great technologies good solutions good thinking behind what can be done um specifically around uh xdr and the sock component and integration of those two things but what the what their studies specifically looked at was they did modeling on wizard spider

and sand the sand worm teams um you know so the not picture thing obviously was sandwich was well known for that wizard spider with uh with ransomware attacks across a number of different infrastructures in finance and healthcare i'm based out of toronto i've dealt with healthcare and government exclusively for the last 18 months just as a focused piece and i can tell you that healthcare in general is uh it's undefended is under-resourced and is is essentially struggling and at least in our canadian context uh we had a number of instances where that particularly was that bit people and and cost them some money um and for the public infrastructure obviously that's a problem for for

everybody um the focus of that specific testing was tactics techniques and procedures to abuse encrypted data for impact um i just the reason why i wanted to really circle around this is you know one of the things you have is a stock is a volume of information that you have to deal with you have to disseminate breakouts of what's important how things are going to structure through your approach through your teams through your technologies to get you to in a decision point um if you're not encrypting stuff encryption is part of you know that's things are too scary too complicated to uh you know too abstract um you're missing a great deal um just a quick reference

point i think the current numbering on on google is about 96 of everything they see in touch is encrypted at least in tls format so just decision and conversation around there if your security practice excludes decrypting things um you've got you've got a serious conversation we need to have you know please take me aside and let's chat some more about that so what is in the scope of our control people process and technologies as we all understand these are foundational things that we've spoken about for a long long time um and it's the balance of budgeting capability currency of you know of these three points that we can sort of pull together to give us

a sense of where we're at what our state is and what we can do right so we have great people you know maybe okay technology um you'll get frustration it will amount and in the stock conversation specifically the traditional thing always was you know we're going to build a framework we're going to have teams that'll you know we'll do sort of level one folks that have you know a certain intermediate level understanding of what's going on in the environments they'll do some low-level alert stuff they'll process things through and hand it off to the smarter teams who will then do a next-level analysis maybe dig into what they're actually seeing and hand that off to the actual

threat hunters who'll get into the meat of things a common three-tier sock was what people structured around and that is one of the aspects that is doomed to fail it is impossible to maintain a staffed infrastructure given current conditions given you know what's what's what people want to do and how they want to maintain interest and earn well right there was a conversation probably about five years ago where this was a hot space to be you know the stock was really catching on more and more infrastructure was built around these by healthcare finance etc where they had these unique teams government sorry not to forget them where they would have these unique teams they'd pull together to do these

response pieces understand what was going on and do all these things you know in a very split explicit terms the single greatest thing that was a problem for that wasn't the technology wasn't the process it was just maintaining people in seats right because we had a muppet we had a med a mapping and a modelling that said like for us to be fully staffed and fully functional we're going to handle this many events by these many people and this should be our expected outcome and we haven't you know it's into events per hour that these people would be able to process and that gives us a sense of where we're going to go and as things the velocity of attack the

velocity of problems increased um largely the people on the front lines of that got disaffected looked at what they were doing and said now i've got to be doing something different right this is a job i'm just getting swamped to this stuff i can never keep up there's no reward in this for me i'm doing nothing that's really important and you know will be interesting to me in my career i'm literally just wading through slop and they started packing that in real problem then because that means that we had technologists that weren't going to generate to the second level which is a you know one of the more important levels of that stock modeling of the time um and even more importantly

they were then in time generate and go up to the third level being those tier three analysts you know the super the guys in the birkenstocks and the running shorts and the you know the running vest in the back of the room doing the really cool and sexy stuff um those guys and ladies would would you know be a piece apart and they'd be the smallest sort of piece of the pyramid in terms of knowledge and experience i will reference here very specifically and i'm going to put this up in terms of like this graphic because i love the graphic for a couple of reasons number one you'll see the eight ftes um if we'd had this conversation five

years ago we were that entity right the largest commercial threat database in the world we see and and process uh you know attacks at a volume that it's just unheard of and to deal with that we had a 42 person sock so we were the model of we have to have these three layers we have to have these three managers we have to have this you know this process that covers 24 hours and is global and all the rest of this because that's was the understanding of the times and that's what you did there was a sea change in that where people started recognizing we just we couldn't keep bums in seats process you know we just we couldn't

keep up with the process that we needed in terms of just adjustment for what we were going to be dealing with as the volume and the breadth of what we had to deal with increased right so the technology was the piece that had to change and the technology drove a lot of change in a very practical way um the biggest thing that was required to make that change courage right it was to to say like we don't need 42 people we need to window this down we need to let the technology take care of a lot of this for us trust in automation trust in a playbook structure trusting in a an approach that has a decision tree that we can maintain

and grow and actually have time to work on and adjust validate it you know through our peers validate it through third party our auditors can validate you know what we're thinking of and doing within here and make sure that this is all sound and so this is the picture today right and you see that number there it's 1.5 trillion events this is 1.5 trillion events we face every 90 days right i'll give you just a referencing we block automatically about 224 billion unique events a day that don't even feature on this number but we're producing about 4.3 million global signatures that are available to everybody within our infrastructures at a moment's notice as soon as we see

something new and we can codify it it's it's structured and sent out through the different technologies about 4.3 million of those a day that are completely unique you know and uh and valuable so pretty cool numbers pretty shocking in their own way what's really really cool is if we had this conversation three years ago that number was 800 million right so three years 36 months 800 million to 1.5 trillion and it's just going up like every quarter this number adjusts it's crazy what you see there about a billion of those are just automatically taken off the top these are such silly like high level stuff that it's it's really just like scraped off and not even dealt with

down to the next level monitoring and alerting so the breakout here which would have been kind of your level one previously digging through the chaff of all those alerts all the stuff that's out there right having that process through a decision tree in an automated way that would allow you to take from there 1.49 trillion events crystallize that down to about 6 000 alerts that you would actually care about i mean the numbers are just ridiculous right 1.5 down to 6 000 events or six other things that you would care about um you think about that in terms of like the fte thing eight nifties six thousand events 90 days still quite heavy but you know we're at

least we're in the range now we might actually be able to do something and still have a sip of water and you know get back on the drum beat of rowing the or and pulling the galley forward right um so what you need from there is further analysis further investigation and again technology again playing the biggest part within this right getting the automation getting your playbooks getting different structures of playbook and decision tree so that you can see things that you've had before you can recognize where things might be awful we see something that's new and interesting and have to make an adjustment within there right and every one of those eight ft's essentially functioning as a tier three

at this point has to be able to be able to do those things completely throughout the whole cycle i'll give you a little little nugget about that we don't publish but it's kind of interesting those fts are nine to five we don't have a you know 24 24-hour sock we don't have a global presence all right we've got those ftes one manager nine to five operation we have automated everything behind that so that we have pagerduty two of them carrying the pagerank potentially not a pager anymore but the pager sent you at any given time to respond to stuff that might come up at any moment really really cool getting down to that so 5200

automated analysis and investigation functions that go on there and then from that crystallizing down to 800 manual investigations so the actual cool work the stuff that people research and and potentially write about and go to their local b sites or their local universities and have conversations about and share with with everybody else talk to john and people like that you know with a c-cert type of a conversation share with the industry peers that they they're aligned with those are the sort of things we get into like that's the career building interesting things that everyone always wanted to deal with by being a sock analyst right and so it gets down to a crystallized function that we can actually manage in terms of

time so people process technology in function in perfect harmony gives us that type of approach and the number at the bottom the little gold thing down there this is a 10 quarter like 40 months we've been having this discussion with with industry um 10 quarters no major incidents to date and i'm actually going to talk about part of sunburst and solarwinds as a function of this just to give an example of how poorly things can go but you can still be you can essentially still be covered within that in terms of a sock infrastructure so the enemies of the legacy sock and at this point i'm just going to take a little pause and i'm just going to make the statement

the conversation i'm trying to have with you is about numbers it's about people process technology dealing with volume in a way that you can actually handle this is not despite the little conversation you had up front about um you know the sock will fail this is not a knock on socks as an approach this is very much an underlining and underscoring of how essential they are to every single operation so no matter how small you are no matter how big you are you do need sock type functionality whether it's your own in-house or it is a managed security service that is applied and driven by someone else audited and verified by their third parties and

yours for functionality and efficacy um it's very much we we're very much saying that shock capability is definitely what we need um legacy sock too many low fidelity alerts again that burnout factor i've just i can't i can't deal with this stuff it's just driving me crazy you know this is just rubbish investigations are time consuming and i'm going to slide a little bit further on that will deal with some of this sort of stuff one of the things that was a challenge initially in changing from traditional sock down to a very lean automated type approach is digging into these investigations right seeing what was time-consuming see where we were bound with things before in terms of having conversations with

different stakeholders different groups third parties you know police and uh and national security infrastructure people that might be involved in these conversations so in canada you know you're going to have to go to csc and thesis and explain what you're seeing why it's relevant have they seen this is there something that we can deal with you know etc etc those things always take forever because everyone has to acknowledge whether they're going to even talk to you about the issue first and from there acknowledge and decide whether they want to engage with you on it but there's a little nuance that's very interesting but anyway it is it is what it is and then there's repetitive manual tasks

right that sort of stuff is just drives everyone nuts and this is a function of what we've done for 30 years in cyber security functionally is those repetitive things those things that you know should be taken care of that you still have to slog through you still want them checked you still want them logged you still want a record of them somewhere but this is not a people thing this is a technology thing now and you should use it as such so in legacy socks important threats are missed just a volume thing right target everyone remembers that alert fires body recognizes the alert calls it out hey there's an alert the other side goes like file

and months later someone goes like oh hang on something happened there nobody told us about this and they're like oh actually they did but oh well you know not a great sort of approach continuous firefighting mode again that dreaded thing of like just continuously being in that mode of never being able to maintain never keeping up as the volume increases as you know the uh as the problems sort of mount you just you're on the back step more and more and more and it just it just drowns you right 90 of analyst time spent responding to alerts again think of that pyramid you've got to push that stuff aside right if you're wasting people's time

doing that you're going to burn them out they're going to leave you they'll go into something else they'll be selling real estate and please give me a shout i'm looking to move to cayman you know at some stage in the next year if you're burnt out stock analysts and you're selling real estate give us a shout let's go um large shock teams so trying to maintain that the volume you'd need for people to deal with the volume of issues of course um and higher unless turn high analyst turnover right the job sucks why would you do this and so things to bear in mind let's talk about that example so fireeye called this sunburst in december of 2020

um really interesting timeline here and there's there's a very nice little confusing narrative around this so if you looked at the response to solar storm was we called it unit 42 hour researchers called it solar storm um timeline is amazing because september of 2019 we saw an incident occur where there was a command and control call out from a device locked automatically blocked locked down and a signature created in our structure right the automation doing its job as we would expect so great result further analysis we take that thing offline we look at it contact the folks over there and say like hey um something's going on here this is out of bounds it's definitely your process

we'll work with you let's go on this thing and there you know unfortunately this does happen in industry you know you you do the responsible disclosure you reach out to people you say like hey this is something you guys should think about here's a problem for us and they're like yeah we're kind of busy over here like we'll get back to you and so a month tricks by and in october then the conversation really starts to like he said that thing you guys called out we told you to basically bugger off um let's have a talk about that and see what's actually going on and then in december firearm recognises they've been compromised by their clients referencing

okay shoot things are happening in here and they call it out and publicly publish the you know the sunburst paper so that's when you know industry and people really got whoa hang on orion servers are everywhere solar winds is common you know this supply chain or this you know this trust chain that's been essentially established in there is a real problem for us and as an industry it was kind of a big wake up for a lot of people to you know to sort of reanalyze about how they how their trust structures work and so let's dig into that so sun versus a back door injected into a into a legitimate solarwinds orion plug-in right digitally signed by solarwinds so

the guys that structured this the people that put this together were smart enough to put that in the way and and basically subvert that trust so that it could potentially propagate unnoticed and the unnoticed part's really interesting because in front of congress in the us when the solarwinds team had to go in there and please explain they put it down to into an intern right with the old blame the intern track um i'm really really happy to say that after that their ceo actually came out was you know you can google it publicly he came out and said yeah we should never have said that it was you know that that's not correct we're not that kind of company

we're not a blame the intern group um our their own internal sort of investigation suggests it was a two-year breach of theirs that had gone on right so timeline here is ridiculous like it's like with so sometime from 2017 onwards this activity was infiltrated into them very quietly very stealthily in play even with you know how this thing worked was was very clever so when we saw it in september one of the things that our guys noticed in their initial analysis was it waited a week or two before the dns request would start coming out so it was like a it was in a stealth mode in terms of it load sit there quietly and they're

just beaking up just quietly to see if if you could get dns requests out to one of the the tagged hosts if it could it generated uh command control traffic and it mimicked the orion updates so very easy to you know in a common way in a common sock with volume spraying everywhere to not even catch this thing right to see this thing like there's an around update going through yeah we've seen 5 000 of those this week great off we go next um so the timeline here is really really cool if we look at actually how this works i'm just going to dig this through from points one to six solarwinds orion downloads the malicious code

sunburst checks through dns after that week or two like just literally a little beacon out like hey you know calling home response you know if it can get out gets the response back it then downloads and executes the cobalt strike attack that's the point and if you see murray around she's got the pink hair she's uh she spoke yesterday i think she's the extra expert actually in the space that's the part where the xdr component literally the endpoint component fires up and says yeah this is out of bounds this is beyond the processes that we accept quarantine the machine lock it down generate a signature alert the rest of the world like there's a problem here

right now based on that we then do the analysis and feed this out so the automated stock does its job the automated technology does its job in concert these things need to feed together and be obviously programmed and reprogrammed and restructured together consistently and you know had this failed cobalt strike would establish that c2 lateral movement would occur exfiltration would happen and you'd be you know one of the many companies that uh that suffered through that and had a very panicked and uh and painful cleanup right so the game changer what do we have to do about these sort of things right ppt dynamic in play here modern prevention it's obviously technology that actually prevent not

just alert strong configuration within those to maximize the capabilities within there i've had a number of conversations since i've been here since yesterday one of the things consistency is an industry that we fail on is we're very good at looking looking at what we need to do understanding you know to a large degree what is required of technologies to implement them and do things with them um actually doing that actually going through the pain of deploying them properly validating them again to ourselves through our peers and partners and third party validation um and making sure we maximize what we're using and what we've got is an industry problem um you know there are ways to simplify that

and automate some of that as well so i'd encourage you to think about that high fidelity alerts fast investigation so the speed to response is critical right having things sit there again like the target example hey there's an alert crickets crickets cricket crickets hey what was that again it doesn't work like you you've got to be like it's got to be almost instantaneous um and the use of saw right security orchestration and remediation technology right the combination of those three things together has an absolute automatic impact on your sock right this is where you start to saying you know we don't need the people anymore because we can apply the technology in a very specific very

useful very efficacious way get high fidelity input from that automate how we're going to deal with that and then the sock essentially falls into line unless we start getting those beautiful inverted pyramid diagrams right the ideal here being a 30 30 30 model 30 of alert response 30 on hunting what's going on and then 30 on improvement and the key thing here is and this is like if you look at the john kinderveck zero trust approach one of the key things that john referenced to was like defining the approach defining how your zero trust philosophy is going to apply but you have to revalidate consistently right this is not a state in time and then walk away right we don't do this

thing go to the beach forget about and come back in six weeks and hope it's all good this is an ongoing review ongoing check-in on what we're doing ongoing validation again ourselves our peers and third party to make sure we're doing the right things and so in a broad diagram how this would work is essentially on the left in the blue you have all the different you know technology components that you can apply and these are this is not exhaustive this is a representation of the common sort of things so firewall is some type you know assess security of some type um cloud security app you know mechanisms of different types some sort of assassin approach and then

potentially xdr component within there as well um xdr just being you know edr but that it covers more than just the endpoint is the what the x is for so looking at that we cover the reconnaissance phase weaponization exploitation and your installation phase with technology very specifically focused and set to deal with those things and then when we get to come on and control your lateral movement and the actions on the objectives that's where you look at you know looking at the threat analysis and hunting components and feed that out to essentially what i'm arguing for is a playbook type approach right automating everything that you can within that is key so just an example of a stock playbook

structure um a lot of people when they when they talk about soar you know have some concerns about the complexities involved you know the nuance of their environments in terms of their geographic you know dispersal the networks the history of the networks who who knows and owns everything simple example i talk to the banks in canada awful lot um you know there are people that that are at the bank as an example just one of the major banks he's got one guy he's the proxy guy and the reason why they can't get rid of the proxy is because brad knows it right he's the guy that runs that infrastructure and that's his guarantee of you know future security is that he's

he's the only one essentially that can run that stuff um there's an awful lot of that sort of silos and fiefdoms that go on when you try and apply a saw technology into spaces like that you have to involve people all over the place all these different fiefdoms lots of conversations lots of true understanding of what the network actually has right so when my little you know instance of something that happens to be at a caribbean bank and beacons out wants uh once a quarter to give me an input you know via the old x you know x86 modem or whatever it is that's still communicating with us and it's the banking thing that nobody's

allowed to touch because it still works things like that that are just little abstracts out there that's obviously a silly example but when you have things like that in your environment that people don't understand or that i've poorly understood they really become the corner cases for soar and get interesting in terms of how you can apply them i'm here to say playbook approach soar approach is really well understood it's very well documented and the best thing about it is for the most part the vendors that produce and derive these technologies have massive communities who share with each other playbooks that they've already generated for different instances so this is not a day one thing of like hey we're gonna go to 40 you

know from 42 ftes down to eight because all the stuff exists of course there's work that's involved but i'm here to say that you know with all the complexity that we represent out in in a slide like that and this is really just an abstract of the high level sort of approach a lot of the stuff's already been done a lot of it's public you know you can go on a github for domestic as an example or phantom and you can actually see like what people have done what they're sharing what's current variations of what's been done before it's it's all there um so that makes things really interesting so what you want to break it out to think about is

incident-based playbooks analysis sub playbooks right again thinking about that funnel that i drew indicator scripts breaks those upon the trigger upon you know detail gathering for further analysis or further support of decision containment escalation remediation and then post-incident metrics and re-improvement right so it seems like a bit of an eye chart seems pretty complex it's really really that not that that awful at all this is one of the argument areas for sock as a service right there's a lot of stuff that can go into this um there's a lot of work to be put into place and experts in the sock fields experts around saw can really make everyone's life a lot easier by coming in and

working with you guys doing analysis of what you have binding it then into this type of an infrastructure so i said earlier there was a slide on here that would talk about the fte component and you know how you can rationalize time against cost and time of doing these technologies just a quick brief one in here um this is a real example it's our own obviously so just be with us on that key things in here in terms of savings automation types enriching the alerts 1090 alerts enriched gives us 635 hours back right deduplication of alerts like 7 700 of those gives us 648 hours back and so as you go down here i mean i'm not going to

read through each one the eye chart is is good the message there is in one month if we can say 1400 hours that's a real that's a real turnaround in fte and that's the point right so we draw the we draw the the funnel reference eight ft's give you the example here and of course these go on for all different components of your network all different components of your life um including the bottom on their other jobs is like all the other little menial you know interesting things that could go on i mean you can even use xor technologies as an example to do automatic onboarding and off-boarding of personnel right unless you just have

that as a playbook right joey's leaving kelly's leaving put them through there just off-board them it's done but it's very straightforward and so where this brings us to is and this is a really an agnostic slide this is kind of a panacea there's a vision of the future for what we'd like to see as an industry this is where things can be really useful this is the perfect sort of new world right a new sec ops team so reset the table reset expectations get away from the old views and look at these you want to assess and organize all your alerts and infrastructures around there look at your roles and responsibilities what interfaces they touch and have

access to and can implement look at operational establishment sorry enablement proactive visibility how that can be applied where it's useful what metrics you require reporting and again across the three spaces your own peers and partners and then third party autonomous security operations largely the function of this talk and then continuous improvement and again you know referencing john's foundational paint in a paper from 2010 continuous improvement is a theme that we should all be pursuing at all times as cyber security practitioners so that is the talk i'd love some questions and i'll thank you for your time

yes please yep microphone's on its way this has nothing to do with my workshop coming up promoted but but i am curious how you would um integrate deception tech honeypot technologies and etc into today's sock and how you expand your your detection capabilities 100 so like anything you have that has an input that is useful and valid like those technologies you can absolutely feed in as part of your playbooks to give you inputs towards your analysis so analysis isn't just from those security technologies i referenced there which are like commercial they can be anything so like oceans you can take deception technologies you can even take partner inputs you know so one of the things that you

see in industry these days there's more and more stuff around sharing of iocs at you know the commercial scale where there's 10 000 plus a day that we all share with each other or more they can take those inputs and feed them all through in the same time as well yeah that's tremendously valuable input thank you none of this none of the sock guys are going to throw tomorrow's come on i was promised this i feel let down

that's very gratifying anyway so thank you very much 10 minutes early i'm early to get you to lunch remember me for that at least have a great day everyone thanks a lot