BSidesCharm - 2017 - SOC Panel Keynote

that's kind of fun um julian uh polenji yeah uh yeah senior consultant with mandiant uh doing instant response mainly also helping do some stock transformations with clients prior to that worked in a sock in a large bank in canada i'm a canadian if anyone else doesn't know okay so many years experience in in sock as well all right sean hey um i've been in and out of different socks both as an analyst and an engineer for about six years give or take both government and private uh mssp and the last three i've been doing threat hunting incident handling and breach discovery everybody loves breeches right and joe cline in and out of socks for years both

architecture cleaning up sensors cleaning up logs doing analytics against the systems and a lot of threat hunting over the years and ir which is always you know sopping up the blood so now that we've introduced us i'm curious how many of you work in a sock today or have recently worked in a sock your hands up there we go good bit so that should make this fun we want to make this interactive and help you guys out so kind of the first topic for discussion is some of the roles in the sock uh joe if you want to start us off joe i was actually going to mention that we have echo uh i was gonna actually mention that uh

i don't know if you know about the i'm calling the echo his mic okay he's blaming me okay so um i've actually sat on a project for dhs at one point to create the nice which was actually to create the roles for what worked in a sock the only challenge is that a lot of those roles from my perspective were under size as far as their capability or their education or whatever but if you haven't seen the nice document you might want to go out and take a look at it it also makes recommendations based on low medium and high or senior mid and whatever sock engineer so just just from a problem perspective when you come into a sock you have

engineers you have analysts you have guys who actually go out and remediate problems you have a lot of that one of and because a lot of this is issues in a sock what you find within these departments is they're they're all siloed so you end up with engineering over here doing engineering things that don't really take into account how it will affect the analysts who don't really take into account what they do how that will affect remediation team or the end client or whoever that may be and the larger your sock depending on it being a small like single company or government to something like an mssp these teams can kind of develop into 15 different teams give or take that all

exist within their own little silo and we're going to talk about communication later i believe so go for it um okay i'm still on the topic of roles within the stock i think there's really three main roles and probably everyone is going to be familiar with these but you have your standard stock analysts right these are the guys on the front lines reviewing alerts on day to day churning through those things you have your seniors of course so kind of more advanced level then you typically will find that a stock is best equipped with a couple or maybe a few very intense technical experts right you don't need everyone in the sock to be an expert

in technical matters but you have a couple guys that you can lean on for those very complex problems the majority of your team really those analysts and the senior analysts need to be guys who are females as well that you can rely on to kind of keep a consistent level of capacity for analyzing things on a regular day-to-day basis and if you have those three roles in your sock you can add on and adjust beyond those but i think those are the three core roles you really need to focus on when you're building an effective sock and kind of the last part from my perspective a lot of the challenges have been around actually having

separate roles for within the engineers to both remediate and deal with actual false positives so tuning them out of the various tools so that you're not being swallowed by noise and a volume of tickets you're expected to close that you have no hope of because you have too many by vast majority so that tuning aspect as well as actually building new capabilities so when someone says well do we check for this one having the ability to answer that question for management which is going to come up every time you they see something in the news and also actually building those capabilities and making sure that we have any particular sock has coverage for whatever today's flavor of threat is

because pretty much everybody i feel like who's worked in a sock has had that manager come down and say the senior leadership or the board even sometimes wants to know if we have protection for x we need a document on how we're dealing with this threat coming out what we have right now and what we'll have next week because that's always a fun challenge i think the real the something from a long view my view is that the same threats come up every once in a while almost cyclical and you have to be prepared for it a lot of times at least again looking at long term on stocks organizations won't focus on one thing

it'll then open up the vulnerabilities not be prepared for it and then you'll be playing catch up be it you know ddos fishing spam uh directed attacks tax via wi-fi i mean these these same vulnerabilities come up and again and again flavor of the week so to speak exactly in flavor of the week basically you're always fighting you know bernie fixing that one thing but not actually looking at the long-term architecture and then the systemic problem of actually like building detection and remediation for those threats which is a lot of times lost in the difficulty of actually solving the problem but that goes that goes into the roles right so i'm i'm very much of the train of

thought here that an engineer should spend some time as an analyst yes and an analyst should spend some time as an engineer a little bit of cross-pollination so to speak so that hey maybe the analyst wants this ruler sees this thread or is up on us that intel and he wants this to be made well he needs to understand how the sim works to do it or whatever piece of technology that they're using likewise the engineer needs to know what the analyst wants otherwise you end up in that situation where analysts are doing three tickets a minute so you're talking about engineers from the perspective of configuring and changing the two tuning the tuning the tools yes

absolutely yeah that makes sense i think that's very important actually is to really focus on being able to tune the alerts coming to the sock sock needs that cycle on a regular basis if not daily if not hourly it should be like instantaneous as soon as they see false positives they can get that tuned and interfacing with the engineers is the only way to do that um and maybe putting engineers in the sock well and that's really what rotation for roles should be less specific and allow for for teams to kind of intermingle a little bit to gain that experience and gain that insight otherwise from a game theory standpoint each of them have um roles that it's better for somebody

else to fail for our team to succeed and because of that i've seen very large socks especially major organizations that it becomes finger-pointing it doesn't become actually hey we're all trying to solve the problem that is a very common problem it is and then also working with management because management side also has to provide those goals because let's say your boss says you have to close x number of tickets a day that makes it very difficult for you to actually resolve a full ticket and take it to either false positive tuned out true positive remediated or network issue resolved anything other than those three is an incomplete ticket but let's not forget that in in the

security world that we live in today most most of our management team aren't necessarily security guys so we have a responsibility to appropriately communicate with management in a way that they will understand to kind of say hey this is our problem and this is what we need to do both from a technical perspective and this is why this benefits you this is why this will benefit our company and this is what it's good for so the analysts and engineers almost have that responsibility to to handle that and be able to talk to management i think the other thing that you have to deal with you typically have middle management that are just project managers not to put down project

managers but because they don't actually they haven't sat in the seat to try to answer the questions they're really looking at the numbers they're really not looking at you know hey how can i really fix this process do these particular three people need additional training and how do i just uh justify training up i mean again the the flavor of the day vulnerabilities are hard to keep up if you're only sitting in front of them fighting fires it really takes additional education and ongoing um justification of funding for training i mean this is hard stuff i mean uh was it dan gear has a quote out there that this is the hardest intellectual job in the world is cyber security

does this lead us into point two quite nicely actually hearing and retention to his point that's a problem that we're seeing the sock the modern sock has become a place for people to escape and i i mean the majority of us on this panel are people who have escaped that right we have gotten away from that is become a place to escape rather than a place where people want to be a place where we have our best and brightest the place where we have good animals and a good a big portion of that is because we've become so focused on this metric based system rather than a security based system that we don't focus on the quality of

the analyst in the seat we just focus on having someone there or the quality of the work absolutely so if you're talking about retention then really how to retain people in a stock i think the goals are one change the metrics right metrics of number of tickets is really not is not suitable it has to be based on the quality perhaps of the work that's coming out of it so if a ticket takes three weeks but that investigation that case actually takes three weeks to come to conclusion if that's done successfully that's what should be measured uh that's the first key i think to really focusing on retention the second is actually having a plan a lot of stock

analysts feel like all right i'm a stock analyst where do i go from here what's the next step and so trying to develop some stages or phases that they can go through maybe you guys have some ideas on that is really the second key component but the first is get the focus on the right metrics not on just tickets closed because a stock analyst coming in and just has to slog through tickets as you see in maybe help desks people don't want to stick around in that kind of environment right drowning tickets makes sock analysts want to run away and the sock has kind of in a lot of places become a glorified help desk for

cyber security and also solving other things i mean that's that's one of the things i found sitting in a sock any problem in the operation any place they define it as a security problem because they may not have the talent to actually solve that also and the time and energy also sometimes right not just necessarily talent right and that becomes another issue because how do you actually cross bill that and how do you have that conversation with your management that hey the last 30 percent of our tickets were solved because we were solving somebody else's networking problem or host problem or a backup problem or or something else and that becomes a real problem because then again that just

requires that creates more burnout in the process because you're not actually focusing on that particular issue you're focusing on fixing other people's problems so now that we've talked about retention for a few minutes on the hiring side one of the other things is that a lot of the sock rolls that we find out there at least i have have been targeted more towards a junior audience and there's some incentive issues there with pay but some of it is actually i think in a lot of management view the sock is considered a junior role and a lot of that has to do with some of the best folks trying to migrate out of the sock to better paying

more exciting more interesting role where they're not drowning in tickets and being unhappy um building up those that team of skilled assets and getting in experienced folks and yes it will cost money but compared to the cost of the opposite right if you don't have a skilled team if you can't build up your team with skilled folks or either build a training program to build up your team or retain talent you have to be able to get new talent in at a more advanced level so it's really difficult now everybody does have to start somewhere and an operation center isn't a bad place you get all kinds of different technology all kinds of different experience all kinds of

different problems the problem that and and to just kind of go forward from that is we seem to focus on it's only for these people it's only for new people so we don't keep experienced guys they don't want to stay they want to get out because it's not interesting enough so it's too many tickets and not enough investigation or hey maybe i want to do malware and i want to learn how to do that and we see malware but that's not in my job description and there's a lot of that oh you're this tier you can't touch this rather than letting a person kind of base themselves on their own ability and their own desire

to go forward and be a better asset to that team so what do you see is the average retention time for somebody coming into this this role at this point six months to a year and a half tops yeah typically people burn out very quickly so if the cost of rehire is the cost of two months salary that's significant if you're burning your people out one of the things that i've been part of is to do things like have a mentor the other thing is implementing purple teams and making sure that the red team is actually helping them understand some of the technical issues i've done that before i've been the purple team person as my

red team was coming in to have the conversation of hey are you seeing this on your sensor how do you actually see this kind of thing what are the details you're looking for hey it looks like you're filtering this out why have you filtered that out that's kind of interesting i think another part of that's going to be a management thing yes because i've done interviews and i've i've gone in for stock jobs and done these interviews and the first thing that they tell you is i want you to understand that this is a revolving door job we don't expect you to stay here this is a foot in the door and then you'll move on to something better as

soon as it comes up and that's our expectation if i ever heard that i think i would probably not accept a role with that company but i mean it happens and i think that first roll is complicated i appreciate that honesty from these guys yeah but it's a failing of the sock itself to treat it that way and to build it that way i would agree i mean i i definitely agree with your points in terms of if it's a failure to build it that way but i think you know like you said right they're being honest with the people that are coming in because the sock the way it is is set up like that what we're trying to

say here is maybe it shouldn't be built that way and so when you're hiring if you want to hire the right type of people in your stock in my opinion i think the three kind of key components of somebody that you want to hire into a sock doesn't necessarily mean they have to be fresh out of school but you need people who can self learn right so they can teach themselves new things new concepts very easily we're very curious who will never leave a question unanswered right i think that's very important for someone in the sock to have that that kind of drive to figure out what is this why is this if they don't know

something to go and figure it out and learn it because you're always dealing with something new and then the third thing is really that persistence right they won't give up so that's in in line with the curiosity they're going to persist and focus on trying to solve that problem no matter what if you have those three qualities you actually don't even need that much technical expertise i've actually had people come into a sock with a chemical engineering degree or a psychology degree and just because they had those capacities we were able to teach them all the technical and they were able to then take that and become very successful so hiring the socks start off with getting the right

people and then you got to figure out how to keep them there and that's that's the key right so if if you bring in those kind of people because i agree entirely about the type of person that you need for this work so if you want to keep them you have to bring them into an environment that nurtures those things that you expect when they come in the door you have to let them be curious you have to let them dig down the metaphorical rabbit hole they have to have enough time to do that they have to have enough time to fuel that passion and look and learn and see but then again we have the balance of

what management expects and that's where the that's where the burnout comes in that's where you know the there's a actually a level depression uh i saw a presentation at defcon last year that within three to six months stock engineers get very depressed because it's just such a hard job but they just don't know how to actually deal with this much data and their lives are basically you know just the sock the drowning in tickets i think oh yeah but most of that so in ir um a lot of ir and a lot of the stuff that we do is is kind of on an up and down scale so it's it's almost you're part research you're part of a

research division on top of that so from a fixing this problem perspective maybe when you bring these guys in you need to locate you know so many hours a month for research that is just set aside for these guys to do personal development research and investigations that fit their own fancy um the other thing is training again this is pretty hard whether it's video or or on-site or going to cons like this show up this con next year but the a lot of times management can't justify training their junior guys to understand this um honestly i learned more from walking around and having conversations not just presentations here but having conversations out in the hall

on some of the challenges that i have um than anything but management doesn't deal with that so the issue is is that middle management and upper management have to start coming up with a system to say yeah you know to keep you from burning out let's make sure you have time to do research make sure you have time to ask questions of other peers in the community and oh by the way not just product specific but product non-specific conferences because asking the questions the the non-specific conferences ask questions that the product specific conferences won't even discuss mostly because sometimes there are limitations in their products and you have to ask those questions beyond that that technology beyond that scope

that probably ties into the retention thing too right i mean if you are rewarding your guys with good training they get to go to cool conferences and they get some time to do a bit of research or maybe play around in the lab when tickets are at a lower volume yeah i think that definitely would help people to feel a bit more like this is a cool job i want to stay here rather than i'm just going to come in six months to a year and then get out now all that means cost right it's an additional cost but from a stock perspective and one of the things that i've noticed as a trend is clients and potential

clients are more and more unhappy with the sox system as it is right so really if you think about it from a long-term spectrum they're going to lose more money long-term from unhappy clients no longer paying or no longer keeping than they are from paying a little bit more to make sure that their guys are happy and knowledgeable and again the problem of retainment if you lose people and having to retrain them setting aside budgeting for that time to move forward again if you're not moving forward within your role you're probably going to be looking forward to moving outside of your role at some point typically it takes anywhere from three months to six months to get

somebody up to speed not just in the technology in the process but how to actually work within the environment so next we have budgeting because everybody loves and hates to talk about budgeting so one of the big challenges is tech first people where do you spend your money a lot of managers have decided that or heard marketing to say that such and such product can replace x number of engineers i think i've never found that to actually be successful um in any context but there's certainly a large volume of tools there's a large volume of people and finding a happy medium is a difficult thing so with open source tools you sacrifice time of your actual people to build up better

tools to build up your open source projects and get them running in your environment so whether you're paying for a pre-made tool or whether you're paying for an open source tool to be actually set up configured in your environment either way you're still ending up spending some money i personally think people is kind of the way to go about that i'm with that i mean let's let's do a scenario because we have a crowd so we have a bunch of people who work in soccer have recently worked in socks so let me give you a scenario you go into work and you sit down and you're there for eight to ten hours a day right

so within that time you have a sim system or some system that gives you tickets uh an alert gets generated a ticket gets created and you handle that there's not really a lot of investigation there's not really a lot past this generated alert this is all you get be it in email or within your sim and then you figure out oh this is nothing because i've seen it 100 times and you just close it and move on or oh well let's just send this up to the client forget about it how many people's lives is that right now i must be way off basic nobody's raising their hand right now because i've seen that in a lot of places they just don't

want to admit to it it sounds i got one i got one anyway maybe they just don't want to admit to it or maybe i'm way off base i could be uh one of the i'm going to mention this because we were talking about it earlier one of the phd candidates at mandiant did a really good study of process that we go through um the successful and the not so successful closing the ticket and the first thing most people look for is the ip address without context so if you see it's some nation state you go oh gosh that's bad we've seen that from before but what if it's not that nation state it was a spoofed address there's a lot

of context that doesn't come back to the individual from these tech tools because you don't know that this particular address was just spoofed one minute ago or this particular attack is actually coming it's actually an ipv4 attack but they're also attacking through v6 so you you get the ticket and somebody else gets the ipv6 ticket and somebody else gets a tunnel ticket it's actually the same attacker and that's that's hard to deal with with technology today well look at the number of tools that we have now that exist today that are proprietary information so alert fires it's fit it's x i want to know why x fired oh it won't tell me i got i got

a three line description but i've got no technical information on what x is i have no way of verifying whether or not that's bad i have no way of going forward and doing anything with that ticket exactly i can choose to either trust it and send it up or not trust it so i think the key here is is the technology is is a tool it's good but you really need to invest heavily in your people too and i think a lot of times organizations end up focusing too much on buying tech right management buys this technology this technology they hear the latest buzzwords now i need a dynamic sandbox now i need a next-gen firewall

um but really you need to focus i think primarily on getting the right people pay them properly so that they're not leaving constantly or so that you can attract the right talent focus more of the money on that at first so you can build up your team and then the tools and technology will be used by experienced folks or people who can figure it out most effectively really i didn't do it technology is legitimately there as a function for people to play letters it's a thing for people to use that's what this tech is for it still needs a person a person is important in this quant in in this equation and we seem to often

forget about the importance of the person i think we're also seeing people apply machine learning and not actually having i love buzzwords fully buzzword compliant our sock is there um the the challenge is that when you start looking at some of these algorithms from the statistic uh processes the attackers are now attacking the ml so how do you mitigate that without a person actually asking the question to say wait a second this doesn't make sense i think also another side of this is when you deploy a tool giving proper time and resources to actually utilize it and make full use of it so when i say that i found organizations that were trying to deploy rapidly tool after tool after

tool typically right after a breach they'll deploy 810 tools without the time to actually make use of any of them so they have an incredible volume of information coming in and no processes no automation on dealing with that data and no you can not deal with that many new tools in your environment and actually get meaningful data absorbing them one at a time and taking okay now we've added uh i don't know webinspect or something like that okay let's get it time to tune it down so we're actually getting meaningful information from it and adjusting it to our environment if your tool doesn't need to be adjusted to your environment there's something wrong with the tool

every environment is dramatically different they all have all kinds of different information i've i've never seen a tool that you can throw in your environment and not have to do a fair amount of adjustments to make it value give valuable information to your environment and you have to be sensitive to it because of a couple things number one people that are not aware of the protocols the environment whatever may tune down a product so that you miss some of the more important vulnerabilities you know i'm sorry if i run only a microsoft set of signatures and i only have microsoft i don't really need linux detection in that space except i have a linux machine on and that shouldn't be

on my network those are the kind of things the other thing is a lot of these tools have variability i patch this tool how do i know it's actually triggering off of that signature how do i know that it's actually successfully triggering off that signature or the next patch actually fixes that and then wipes out a dozen other signatures so the repeatability is something that you have to have a person to ask those questions if you don't have a test after you update how do you know it's still working that happens in av but how many other tools do you have known tests for that are guaranteed to produce a result we had a switch patch that wiped all the

vlans so all the sensors went off and we went back and we found that the administrator never had a process to validate those sensors were actually on the right vlans so they were actually pointed at servers that weren't external and were low risk so it's those kind of things that you have to go through and you have to deal with and you have to pick up quickly for a person again the technology's not picking this up another fun one with this was a company bought a product six months later they got owned vendor goes to the customer and says oh my god you just bought our product can we do something to help do you need some

additional training tuning adjustment come to find out that customer hadn't unboxed the product yet it was still in a closet yeah so if you don't have the time people energy to deploy resources to deploy these tools and actually manage maintain and actually tune them into your environment again we're much more towards the people side of things because until you've actually implemented it the value that that tool is providing is very minimal are we on to the next one yes sir he did inter-team communication this is my big one just just from from everywhere that i've been it's always been a siloed environment it's always been team a team b team c engineering analysts and remediation or any number of people that

you can throw in to do varied tests depending on how big the organization is and it's this siloed sense of no communication and we talked about this a little bit earlier when we were talking about roles but you almost have to break these roles down and and include communication and one of the things that i'm a big proponent of from a perspective of building out a suck and kind of making a sock better is to take not a manager but a senior analyst or a senior from each team and put them on basically an advisory committee that once a week or once a month meets with every other team and says hey engineering us analysts are dying here

for x reason we need this from you what can we do for you and get the technical people talking about what they need from one another to solve problems across teams and what they need for management yes reviewing processes for a lot of different federal teams i noticed that that's typically not included so uh here i'm going to throw out an idea here please if you want to improve into team communication right i'm going to stereotype a little bit but set up events or activities between the teams and maybe help them bond outside of work related stuff if you're in an organization that allows alcohol maybe go to the bar everyone can have a beer together and

just chill relax maybe you want to set up a ctf or have a lan party night or something if everyone likes playing computer games starcraft brood wars just became free right i'm sure everyone's heard of that in the 90s or the 2000s um set up a night or set up a day because when people empathize with each other when they can sympathize with someone else as a human it makes it a lot easier for them to understand those people's problems that maybe they don't have the same views on right so if the engineer says the tool's on and it's working just leave me alone don't bother me and the analyst is like no but you don't get it

it's not really working properly if they have a relationship beyond just engineering analysts they're going to work more effectively together i just read a book uh on this to understand some of the problems we have in security it's called bowling alone and it's basically isolating ourselves to the point that we're in our own heads or only with our groups that we don't actually reach out and ask other questions we don't learn things we become so self-focused we don't miss it we miss uh real communications yeah and that's why i like instead of just submitting a ticket to their side just putting them in a room because people face to face tend to get along way better than people over an

email or whatever the case may be there's empathy when you have to look somebody in the eye and say screw you or i don't want to listen to that like you just don't do that to people face to face or resolved won't fix that or resolved won't fix or i'm just going to drop your priority even though i know it's an issue whatever the case may be but i i think humanizing these inner team community humanizing is a big step to enter to making these inner team communications work right because engineering and analysts should not be enemies they should not hate each other they should not be mad at each other trying to pass the buck

they're one unit let me extend that also your network people and your host people your server people there are they should also be included in some of these your development people um it it simplifies the whole process i know one company that i work with setting up a sock we started bringing the team in once a month and have pizza and just have conversation about some of the challenges we have and those challenges that actually uh broke down some walls to the point that the analysts could call a specific admin and go hey i know that you work on this can you tell me about this we just found this and it basically short-circuited the going up the chain

going down the chain which was which saved hours or days in some cases by the way the inner team communications you also have to have the shifts because there's different shifts there's weekend shifts there's people that are stuck on holidays and they too are usually excluded from these kind of processes absolutely if you're not on day shift in most situations you're excluded from really having any input for the most case right because you're a night shifter yeah you get stuck in this little silo i did night shift for many years you get stuck in this little silo where you don't know what the company is doing you don't know what's new you just walk into new stuff and you go

hey what's this guy on second ship i don't know is there when i got here yeah exactly if every other team views you as creating problems for them it's not going to be a fun organization building that relationship and solving breaking down those barriers is crucial to being able to be successful as a sock now we beat that down yeah are we on the next one yep oh this one's fun lovely evil beast of compliance for security so having worked in or stood up a security operation center that um was supposed to be for security and ended up uh kind of getting driven by the compliance team uh yeah they sadly it was the compliance

team who paid the bills so they kind of went out in a lot of ways um the technical difficulties of implementing good security is much much more complicated and much much more expensive than compliance compliance is more just documenting good security practices and actually proving that they work and they're repeatable and they're tested so while that is painful and frustrating to sit there and document what you do it also becomes insanely valuable when moving with other teams new people in the team new management when you have a book of processes and actually have a plan it's really helpful now bear in mind almost everything that we've talked about thus far right is really heavily focused on the

security side they're always going to have compliant shops and not compliant shops are always going to be compliance jobs so from that perspective most of our plans so to speak or what we would like to see happen to kind of fix the overall problems of stocks probably won't take the same effect in a compliance shop but they'll always exist and they have to exist just everything that we talk about is mostly for the security centric side because we're security centric people i think that's the key is that when setting up a sock you need to decide what the focus is going to be is it focused on security and protecting the organization from threats and attacks or is it going to be

focused on compliance type activities usually the intention is the former and sometimes you end up with the latter unfortunately but that has to be a trade-off right if you end up having your stock analysts essentially becoming compliance checkers just to make sure that oh is there a ticket in for that change was that approved yep oh this server got updated was that approved yep you take away time from them to actually be doing investigations that you might think are more valuable and we go back to hiring and retention right i want to bring up one other thing we've seen cyber insurance a lot to reduce focus away from security towards compliance so that they can get funded to fix these

issues so if i get caught yeah i know it's it makes me very angry um i've literally seen organizations dumb it down to the compliance only fill out the forms and then hey we had an attack you know please i want the 20 million dollars you promised me this does this doesn't work very well by the way for government but um from the commercial standpoint that's becoming the thing um is to just have the compliance so you have the check boxes for the insurance company and then you're good yeah it's turning into a there's no more fixing breaches there's no more worrying about even what you have from a security perspective past what you need to have the insurance

and i mean in in a sense can you really blame because the system that we're at right now it's really not a matter of if a company is going to get breached it's a matter of when so from how often or exactly or how often so from that perspective i kind of feel like this stuff is coming up because we're failing we're not doing enough to provide good security to make it valuable for companies to utilize it rather than just saying i'll just pay for the breach i don't know i think my data i'd rather work with a company that actually wants to protect my information rather than pay me out when they lose my

fingerprints or security number i definitely agree but you know you see situations where like you said the company it actually might be cheaper for them to pay a 20k ransom than to actually hire in a security team for a full year to protect me against that ransomware attack right so i think

do they ransom the backup do they infect the backup and lock it up we've seen whole companies go out of business within two to four weeks because it compromises and by the time the insurance is paid they're out of business so recognize that's been occurring for probably ten years uh by the way um has anybody read the the uh ran um report called security and privacy that's 50 years old that deals with all the same problems we keep on seeing again and again the products aren't fixing it it's people that are trying to fix it there's there's from a game theory standpoint people are pushing back which is really what this conversation is about is financially justifying it

but if you can find it out on the net you'll find it really interesting we haven't fixed problems over 50 years i'm not that old i wasn't young when that was you know out there but it is very frustrating all right i think we've killed that show from here moral of the story is make sure your teams are focused on security and less on compliance compliance is useful of course but you really want to make sure the breaches are prevented or at least detected very quickly and responded to as quickly as possible the the change in this situation because changing changing the structure of a sock is not going to come from management who has nothing to do with

cyber security changing the structure of the sock and of our operation center and of our culture so to speak that we have right now is going to come from analysts and people who are there pushing back against what they've been given and insisting that it'd be better in a constructive way yes they're constructive or they leave and that's the other part of retention it's a frustration of people that are really motivated um [Music] several friends have actually models around this that eventually get to a point you're frustrated you leave so you leave you you lose your most talented people and you make no changes to the system you actually have a decrease in the ability to solve specific

problems because you've lost that talent and then you have to build that talent back up again you eventually get somebody else that's that really wants to turn this into security and not complain or can't use those tools anymore because all your experts left or the licenses didn't get paid for yeah if we as if we as a community though can come together on this and insist that if we're gonna get paid to do this which we get paid to do this but we should insist that we're going to do it the right way and if we can do that and we can make that case by leaving the places that don't which we absolutely have the ability to do

all that will be left in the end is good places and i think it you know it's we're up here on a panel talking about it but it's it's literally going to take the entire community to make the change your turn you say something he started okay i just forgot to try all right so do you all have any questions related to sock that you want us to discuss i'm going to walk down that way and if people want to come up that because that way the recording will get them i was just going to repeat them but yeah you can do that too up to you go ahead

so what uptick in techniques or tools is popular now basically that's like a two-year cycle you know you go ids to sim to host-based monitoring is is the complete hotness right now and i don't know i've seen a lot of ml coming up and just popping up all over the place ml's the new one yeah question on defense tools or attack techniques attack techniques okay in the opposite direction whoops i think we did i think uh okay lately let's say what are you seeing htas and javascript files and email attachments is probably number one right attackers have switched to that word word document macros excel macros probably number two yeah but that actually has been around for a

while it's funny because at one point that that microsoft company actually disabled macros and they re-enabled it and a lot of their products and then we started getting compromises again and uh what was it uh eight months ago i noticed sans is teaching very heavily how to use this so there's actually a life cycle on how this takes place as forms as far as a defense and an attack posture we start seeing somebody forget something and a product whatever that is whether it's apache we say a standard change like we saw in d was it dtcp which was an ietf standard which caused other protocol failures that um that rolled into lots of uh headaches uh so yeah we we

see these uh vulnerabilities coming from lots of different places and that i will add in the nine years or so that i've done this the methodology has not changed that much the specific vulnerability that the methodology hits changes all the time yeah but the the actual methodologies from an attack perspective and how attackers are trying to come in it doesn't change that often and frankly how the tools don't change that much from a defensive perspective i mean we're using the exact same tools we used five seven years ago and there really hasn't been much change except for signatures and tuning and what's the new threat of the week a lot of it just keeps coming around and

around like a merry-go-round space stuff is newish though like full-on hosting spaces yeah yeah it's a little newer in a bit of a newer kind of technology that almost super advanced investigative av so to speak i think that came out of a lot of disappointment that it took so long for the av community to respond to vulnerabilities i was part of a group that did a study and we found that most of them took anywhere from 20 to 30 days to actually take a vulnerability and turn into a signature and because of that it required rewrites for a lot of the products unfortunately that left people vulnerable if they were directly attacked for 20 to 30 days and that's assuming

that the malware author didn't rewrite it in that time because they saw it in some online medium it's very insulated and that's the thing it's interesting now that we have that online medium right right you can grab a hash and pop it in and check 50 avs like that for free absolutely and that's actually a process i've seen three presentations over the last five years at different conferences i attend a lot of conferences um that specifically said if you are a malware developer this is what you should be doing i i think i'm going to take a bit of a different spin on it actually maybe slightly disagree with you guys i think i think the technologies are important

maybe the attack tools might be changing a little bit but really when it comes down to it now i see attackers are focusing mostly on on the people which is what they started off doing right in the early 90s or whenever it was easier to phone someone and get them to just read you the password over the phone than to try and create some super secret malware that's going to hack in and do something crazy and i would i would like to say that that's what we're seeing a lot more of now oh yeah less on super technical stuff and more on just focusing on i don't know that that's it's been like that for a long time i think but

uh good example um drydex just did v3 right so drydex v3 is an adobe acrobat file that opens up a word file that does powershell and then infection vector what's interesting about it is when you open it up it starts with adobe you have to enable macro so first you get an email you have to download the attachment from the suspicious email you have to enable macros in adobe you have to enable macros in word you have to turn off protected view in word when it pops open so there's five manual steps to infection there and large companies are getting hit with this hundreds of people but it was the perfect candidate it was the best resume ever

we're also receiving a lot of uh or we were up until three months ago uh we were receiving a lot of calls from microsoft support by the way if you're interested i have those calls dozens of those yeah i usually have a virtualization i pop up and i pull down the the mouse i i've kept them on for an hour finding out where they are and all the other information is reverse social engineer them but you know we see those all the time and it just takes one person in the organization to do that or plug in a usb stick or you know whatever click on that link because somebody said hey we provide this really cool

product for your sales guys you need to check us out go to this website if you're looking for ones of those to troll though the google support numbers that you can find when you google gmail support is hilarious gmail support googling that will actually get you a bunch of scams yeah a friend discovered that by accident should we do more questions ah did we did we sufficiently answer your question well we were like 15 minutes late yeah that's true yeah maybe one more if somebody has one more one or two more thread intelligence so uh i'm one that i have a very strong view on this um today a lot of socks pretty much everyone that i've found or

talked to has had a struggle where they're struggling with alerts that are 99 when they get an alert they know something bad just happened they're not dealing with all of those they're not preventing those so purchasing a list of ip addresses and eight user agents and other uh resources that a lot of these thread intel companies are putting out a lot of these organizations just aren't ready for right when you get to the point where you've resolved the primary you know 99 type threats then you can start to move on towards the threats that are a little bit lower percentage accuracy so when you've got that process down when you've gotten that matured okay maybe it's time

to move on when you've blocked your dried x's and things that you see regularly and repeatedly in the same organization i i think also thread intel it's it's useful but maybe not as an independent trigger maybe you use that to help build a story right you have an ids alert and maybe there's some intel reputation information about that ip address that plays it apart the biggest challenge i think that people have with threat intelligence based on ip and domain stuff like that is it ages and dates very quickly right if it's a week old maybe it's useful if it's a month old i don't even know if i would care if it's a year old it's basically

garbage at that point so the question is is an organization and is your organization at that point capable of taking something in and then making a choice on a weekly basis hey do we need to get rid of this now is this still relevant because i don't know a lot of organizations that are mature enough to to do that on a weekly basis with threat intelligence and weekly is really not the issue there are several feeds to look for bad dnss that the dns is only good for anywhere up to five to 30 minutes so how do you actually add those and remove those from your system quick enough but still retain them so that if you do get compromised by

malware or whatever you can say okay we sold this one month ago um from my perspective i think that a single ip address and a reputation system has pretty much failed mostly because um we're we're focusing on a an ip address and that's an ephemeral number and you're not actually associating an attacker towards it you're associating this ephemeral number one of the things that one of the experiments i tried was what if i take one of those ephemeral numbers and fake an attack does it show up yeah i then spent about six months telling people to take my ip address off of this particular list and by the way this was behind a carrier nat because there's now two

levels in that out there and therefore anybody that was actually blocked from that their vpns and their applications would stop working for 10 to 15 minutes and then they would start working again kind of kills help desk calls okay so i tried this experiment and i realized well ip is our we need to really know the attacker we need to get additional signals from the attackers something like um i call it uh device super identities that way you can actually say it's this identity of this device and this user or attacker associated with it and all the ip addresses associated with that particular thing because we can change addresses quickly basically to to an earlier point right

it's something that can be valuable should you have the people in place to use it appropriately but i think going back going to your point about attacker profiles i think that is probably actually thread intel that people can use or the sock team can use if they're not getting ip and domain reputation because ipm domain reputation much more difficult to handle but attacker profiles that's that's actually kind of useful the profile about an attacker group a threat group those types of thread intel which is different than just atomic thread intel can help the teams to understand what kinds of threats are facing their industry okay let's read the profile of the top five attackers who attack

retail because i work in a retail stock and now i know the techniques that those attacker groups are using more often that i think is valuable intel that any sock can use and also it provides you all the recon techniques it provides you all the pre-techniques who do they actually target oh new employees that have their names associated with a website you know but things like that those are again attack the people first it's the technology second all right do we have a final question can we answer your question yes

how has it changed in the past few years and where's it going in the next few years i feel like with the onset of technology at least in the past few years what we've seen is as we've gotten more technology we've gotten more focused on that technology we've gotten more focused on this this ticket based system that we run in where it used to be a bunch of guys looking at pcaps and looking at raw logs and it took a real analyst i mean it was really crappy work but it took a real analyst where now we have technology to tell us what the problem is so it doesn't really take an analyst anymore we have the technology

as far as where it's going i'm really hoping that some of the stuff that we're talking about here we can start to bring analysts back and make the sock a good place for not just a place to escape i think there's also a growing trend towards orchestration right automating a lot of those level one tasks some of it i think you definitely can automate right having somebody receive an alert and then pull five other log sources and simply aggregate them and do some deduping that totally can be automated and i think in the next few years you're going to see people pushing orchestration tools that's the term for it security orchestrators because they want to automate it so they

don't have to worry about the turnover at the level one they don't have to worry about as many level 1 analysts if they can automate 90 of that but it's going to be so misused we're not sure yet whether that's going to play out and or whether they might end up having to go back to full human socks we'll see how technology progresses i agree solid i think automation is going to play a key role and i think in the last couple years even the maturity of some of the sock tools some of the sim solutions going in a very different direction more towards the almost research perspective with elasticsearch and splunk and those kinds

of tools blowing up and becoming much more the standard versus traditional alert-based tools like a lot of the other old-school sim solutions i think given you get a lot more ability to really dig in and research and pivot until you are bored and tired of it yeah one of the things that i've seen is trends on some of the security startups uh in the world is the ability to outsource that analysis at different levels and people who have specific expertise in a protocol um there's two companies right now there's a third one i just heard about i'm not sure how successful it's going to be because you're actually opening up your sim or your analytics to somebody

else and how fast that can actually you know happen but i've seen that as a trend i've seen the other trend is leveraging things like software defined networks docker things like that to move the problem around maybe it becomes more of a devops vulnerability i mean feature but you know there's a lot of other technologies that are being pushed into the market that we don't know how successful we have no way of measuring it at this point and if we had the ability to measure it this would actually be a lot easier of a job i mean how many how many real vulnerabilities did you receive how many false you know how do you actually identify

you know false flags false information you know how do you classify them right now a lot of people i i mean it used to be if somebody pinged your firewall you said oh my god i was pinged it must have been an attack but how many times a second do you get port scan today uh yeah well i don't have that on my ipv6 network so i just want to mention that disbelieve oh you have seen my lots we'll talk all right so we did want to mention because we talked about this a little bit beforehand because it looks like there's still a few questions that after we wrap up which i think we're going to do in a second we're

going to make ourselves available um kind of on the couch area over there for anybody who has any additional questions or just wants to have a conversation yeah the the other thing is um we were discussing opening up a google doc for other people to put questions in and suggestions and things like that i think this is a chance because i've never seen a group specifically to deal with sock in the last five years to say hey there's a problem here let's figure out how to solve it um so if you're interested in that provide a contact we'll send you an hta file just make sure that it's in a pdf yeah it will require flash

yeah there you go nah it'll be all right so we have about an hour and 45 minutes before the party and we have some lovely giveaways that's always a good thing and we also have some lost and found uh apparently some cell phones and a credit card um i i'm still have a few questions on how that happened but uh yeah as i said right outside we'll have more discussions thanks everyone for coming out thank you thank you all get them up here

so you shouldn't run away yeah