← All talks

Zero Days Should Not Be a Fire Drill

BSides Las Vegas · 202249:1753 viewsPublished 2022-09Watch on YouTube ↗
Speakers
Tags
About this talk
Steve Winterfeld and Tony Lauro share incident response lessons from data breaches and zero-day vulnerabilities across banking, retail, and government sectors. The talk covers how to move zero-day response from reactive crisis management to planned, coordinated action—including stakeholder communication, risk assessment, virtual patching strategies, and board-level engagement.
Show original YouTube description
GF - Zero Days should not be a fire drill - Steve Winterfeld, Tony Lauro Ground Floor @ 17:00 - 17:55 BSidesLV 2022 - Lucky 13 - 08/10/2022
Show transcript [en]

this talk is zero days should not be a fire drill by steve and tony um i do have some announcements before we go ahead and get started first and foremost we'd like to thank our sponsors especially our diamond sponsors lastpass and palo alto networks and our gold sponsors amazon intel and google for their support the support of our sponsors along with donors and volunteers is what makes these events even possible cell phones i do want to give you a quick reminder i'm sure you've heard it many times already these talks are being streamed live so as a courtesy to our speakers please make sure your cell phones are silent so as to not be disruptive

um in the event of questions which i believe we're going to have some questions time after uh we're going to ask you to use the crowd mic there's a mic in the middle again these talks are being streamed so for the sake of the audio on the youtube we ask that everyone speak into the mic uh as a reminder besides photo policy strictly prohibits taking pictures of anybody without express permission of everyone who is in the frame so as these talks are being streamed you will have access to the content later so if you uh are concerned and would like to know more about the content you can look at it later and uh unless you're

100 sure that you have the express permission of everyone in the frame please refrain from taking photos please make sure you have your masks on at all times [Music] is it so with that i will kick it over to steve and tony thank you so much so my name is steve winterfeld i have um responded to both data incidents data breached at a loss vulnerability announcement zero days in banking in retail in government and so i want to share some of those lessons with you my co-presenter hello everyone uh so thank you we know you have a lot of choices for talks one thank you for showing up after a happy hour uh and also thank you for

coming to uh to our talk this is me um we didn't really do intro slides and then you threw this one together for me so thank you um but uh thank you guys for being here we're gonna talk a little bit about uh zero days uh response to those i've been in infosec since uh the late 90s and whereas i feel like you've kind of been in the management you know senior track i've been the person who's like always been under that i do work i do work do you sometimes sorry i don't actually do but you know when you're the person you know with your boss breathing down your neck like what your next move is is very critical so

we're going to talk a little bit about taking the guesswork out of that process and how to make you more successful dealing with zero days so let's get started and so a big fan of the office uh i'm not gonna lie when i went and looked for an example and i found this clip i lost about three hours watching uh clips of the office uh if you go to youtube and you do fire drill in the office i'm not responsible for the next three hours you guys lose but it's awesome and essentially it is saying you know dwight had a fire drill and nobody did anything so then dwight had another fire drill and it went the other direction

and so we want to say that you know how often are we going to have a vulnerability coming out and what time does it come out friday at 3 o'clock so log 4 j is going to come out before the holidays on a friday at 3 o'clock how much of our life should we be given up and what can we do to kind of turn that around hey just a quick note the the other most awesome episode of the office is when dwight cuts a face off of the cpr dummy and then puts it on his face and he's like hello clarice remember that one and so when we get to questions if you don't have a question about

our talk anybody's willing to willing to can try to stump tony on movies give him a live movie and he should be able to do it so we should have some questions all right so let's talk about some examples here's uh kind of a whole myriad of examples and you'll notice too that these uh arrive in your uh you know in your um life in a number of different ways right solarwinds this is like supply chain like a product that you're using where's that product installed who's the product owner who's the owner of the server that the products are installed on that's a very different story from like a large scale you know like heartbleed or log4j a

protocol related risk those require a whole different group of people to be contacted and to respond to this incident and then going down you look at things that are kind of status quo parts of your infrastructure that you're using um you know they're they're obviously you know there was a great talk the other day about devsecops and how it's failing um it was actually in this room um and the funny part is most people always talk about dev and sec the ops part uh is all the other uh all the other things that have to happen for devon sec to work properly because implementation deployment um who's using those systems that's the operational aspect and then obviously

stuxnet um you know a meltdown spectre these are things that most people aren't really prepared for in general right because these are these are vulnerabilities or you know exploit attempts will be happening against systems at the very deepest root of your of your environment so it obviously requires a different perspective and you know again this is a team sport so as soon as i hear about what's going on i've got two challenges where is it in my environment and where is it in my critical provider's environment and so you know you need both those process you need to have thought through on you know as soon as i start on my checklist i have a partner in vendor management

who's going through so at nordstrom we have had and my wife is very mad that i moved away from the nordstrom discount but the we had 5 000 vendors so then we had a category of which of those vendors had our customer data which of them had credit card data you know some of them were just they were in our system so when we sold a pair of shoes they sent us another pair of shoes some of them knew who bought it what it was for and so you know that's that's the first one the next is where is it in my environment how many have a lot of confidence in their asset management system

you're in trouble how many people have an s-bomb you know where all their log4j is and all that that is awesome working on it nice to see i have this going next to me y'all take your mask off and yell you have an s bomb do you know if it's correct if it covers all your stuff you know and then firmware how many had a process when spectre came out to patch firmware i mean for us that was a new process we had to develop and so i and is iot in your asset management system today or is that just a rogue system not considered to have data on it so it's not in your

asset management and so the reason we built these out just like when you're doing data breach i think you know third-party data loss i lost data i lost access to ransomware you know there are classic scenarios you should go through in your planning and we think this is a framework yeah the last point too um especially when you're talking about where these things exist in your environment um we you know he just mentioned about s-bombs right a software bill of materials there's a lot of especially on the iot side uh the kind of the antithesis of s-bomb is do you have a problem maybe scanning into your environment because of risk of these fragile tcp stacks being

knocked over from your asset identification scanning tools right so there's a there's a big play there are you really catching everything that could be at risk here from these types of vulnerabilities and if you're trying to figure out why the raiders are up there it's because i live in colorado and this is one of the few safe places i can put a raider logo up so while in vegas i'm going to take advantage of that opportunity so i think part of the problem today is we think of a zero day as an incident and these are our two classic processes to deal with an incident i'm saying that a zero day is not an incident

it is a systemic problem and it belongs in a different box i think it belongs in crisis management it's not something that's going to be finished out in the sock more often than not it's not a quick remediation with our tools again it's a committee it is a large phone call you know when the ceo hears about log4j on npr on his drive in what's the first phone call you're gonna get that day you know and so it's notification it is having a coordinated response who needs to be notified for what at what level of incident does it um so again a customer's going to hear about log4j and they're going to call the call

center and the call center is going to be asked is my data safe from log4j does your call center have any idea what they were just asked so again i don't think this belongs in crisis management because you need a message for people that call in the next person that's going to call when i was a cso for nordstrom bank is the feds the occ they're going to call and say what's a risk to the bank are you guys safe so you have this constant stream which is why i think we should move it out of incident response into crisis management and follow that same thing and so one of the things that is you know

which systems impacted by wannacry log4j pick your pick your zero day of of choice solar winds what access was it to data tony was talking about solarwinds who was running it i don't care who was running it what was on it what kind of information was involved is it customer information is it infrastructure information those are different reactions those are different people i have to notify if there's customer information in there now i'm calling privacy they need to be on the phone call if we lost it i have 72 hours to notify somebody and so you see where i'm going um and then again i can't bang on this drum hard enough i need you to be in lockstep with your

vendor management or whoever's going to own all that others and so for log4j what's a reasonable amount of and this is manual i haven't seen any program and if somebody has let me know to automate this where you're calling your key vendors and saying are you secure from log4j you have my customer data what's your plan and what are they doing they're doing the same thing you're doing trying to figure out if you even had that protocol anywhere so it's not a quick call and done no no go ahead uh so so this goes back to what we were talking about before uh there are a lot of different people if you look at the

racy chart at the bottom this really depicts who is responsible for getting that ring uh into the flames uh to destroy it right um but uh you know when as you kind of map out who in your environment needs to be contacted there's some really kind of practical things you can do so one if you've ever done like a tabletop exercise think of something where you're under duress right so uh so we work a lot with uh you know large-scale web app attacks or ddos attacks and you'd be surprised going through a tabletop exercise uh before we go through the process of developing one you just say hey you're under attack what do you do they're like okay we're

gonna call our two contact people in network ops and blah blah blah and then they don't answer the phone because it's lunchtime or it's midnight where they live what do you do next right who's the second person to call on the call tree how do you uh transfer responsibility at what point do you open up a bridge and you say hey we're gonna have a a bridge that's open from now until this this attack is remediated to where anyone who needs to know needs to be notified of what's going on they know this is the bridge to call into right there's a lot of procedures that can be done beforehand to save time because when you're under duress when you're

uh you know when you're hard down or uh if you've failed over to your failover data center just say and god forbid your failover data center is not failing over um you know everyone starts to get red in the face they start to make poor decisions uh and you frankly you have you know somebody that looks like this standing over your shoulder just that was that was not cool uh but you have like true you have your boss looking right over your shoulder and you're just like i've got to focus i've got to remain calm the more planning you can put into this whole procedure here for who's involved who owns what who are the secondary contacts all that good

stuff that has to be done beforehand um because the longer you wait when when it eventually does happen uh you're not gonna be able to respond uh in time right you're probably gonna have a poor poor experience and that's why i love that when you're building things like this and then you do the exercise you do two things one you discover that you know tony built this and it says create fellowship and there's two people responsible for that is that legitimate can two people own the process i tend to say a racy can only have one r well frodo was a key contributor to the ownership in fact some might say he was not the

most equipped to take the ring on that and that's why i say you know i think i think you have it anyway um and so exercises are are critical to both make sure everybody knows who owns it how they own it if it's a partnership how they're going to come to resolution and also to set expectations so for those of you who have been involved in some kind of a major incident the first thing you get from the leadership is what's the status and we want an update every 30 minutes and then you turn to your sock analyst and you say you go do your work and when you have something come tell me and i'll update them every 30 minutes

you don't need to come back to me every 30 minutes because guess what how long does it take to troubleshoot something until you figure out what's wrong you know i love those those hypothetical questions when are you going to know how much data was lost

when i can tell it's not happening anymore and so building expectations building the team making sure people know their role i can't emphasize how important this is you know it is it is key especially now with the high turnover all of us are experiencing you know how many of us have that that one guy who left was the historical knowledge of all your firewall policies you know so so these exercises are really important so let's take a little journey back in time uh even further than this let's go back to 2014 when there was an ssl vulnerability called heartbleed that was disclosed uh and oh yeah we found out that it had been actively actively exploited in the wild for a

previous two years to that right so this was one of my first big experiences when i started awkmy and to the point of you know who gets notified before anything was fixed we had to make a public press announcement right because everyone knows there's tls or ssl certs deployed globally across 350 000 servers uh that are oh yeah in front of big banks big social media companies big commerce platforms so we probably have to give some kind of notification what's going on um that that process what what's really interesting about this is we learned a ton of lessons from this uh and we we've announced this publicly um but the initial response was our implementation of openssl was not

vulnerable to this exploit and we started sharing what our implementation was uh and then like three days later a researcher came back to us and said hey you guys are still vulnerable here's here's a poc of of the exploits still working so we had like a three or four day lack of a head start on going out and re-keying all of our our certs globally that's a huge impact and oh yeah when you go through that process you really start to find out man a lot of people need to touch a lot of systems at the same time in order to get this done expediently um so it's like hey can you spell computer congratulations you're on the

task force right um so so uh you flash forward to the response to log4j and here we have um what that's i guess eight eight long years after that and we started uh seeing that all the things that we learned from those initial uh you know heartbleed and then later poodle uh we put into action to have a really proper response for log4j one the visibility around you know where it exists in your environment that's a real key issue um but more so you know we saw so much exploitation attempts uh against our customer base at one point i think 60 of our customers were being hit by exploitation attempts as we started to kind of weed out and we

started to see patches come out then we could start doing a reduction of that and say okay these are vulnerability scanners that are testing for the exploit based on this vulnerability scanner code that it had and we can start to take that out but this this was a really massive massive problem and so this is where we've kind of diverted from our generic example to an example of a security vendor so how many of you work for a security vendor of some flavor so this is where you're good for you guys never change so are you protecting yourself and simultaneously looking at your product and how your product can protect your customer and so what we're talking about here is

what we were seeing hitting our our thousands of customers on our web application firewall and this is where you really kind of break out and there's intricacies to this right so like we have an adversarial resilience team that really looks at how adversaries might uh and do try to bypass or do try to bypass my platform security so securing the platform means making it continue to run properly but then the other side is anything that's deployed on behalf of a customer how are those configurations and that implementation uh secured separately right so again it goes into the the racy chart again who's responsible uh who's gonna be brought into a task force um and uh you know if we go to the next

slide we'll see that uh so the the red graph here on the right side is uh percentage times ten so these are six sixty percent at one point sixty percent of our customers were being hit um by these attacks and then you'll notice here as well as new exploit code is being released we saw a huge influx in attacks that test out that new exploit code right because anytime new code is released on a zero day there's a race against the clock like when is this going to get patched if i'm going to exploit it the time is now so we see like a fevered pitch here you'll see around the um a few days after uh christmas time here

massive massive uh increase in attack traffic now some of this was attributed to uh scanning tools that got the previous patch uh and then they were testing obviously um but uh but again this is uh the interesting case about this again that i mentioned is uh 92 of the attacks that we saw were being blocked uh far away from where the exploit uh was was actually vulnerable so this is where you come into what what is your model for securing your environment do you have like virtual patching capabilities where you're the exploit may still exist but you're buying time by blocking the visibility of that you know from your uh from the adversaries right so

this again it was a great learning experience for us and one of the key lessons is never let a good crisis go to waste and so i want to talk about the three aspects we have here the first is protecting our customer um you know on day one you have to get the message out to whoever your partners are whoever your customers are if it's internal i.t if it's if it's making sure your your third parties are working on it if you're supporting customers if you're a bank managing wealth you're notifying people what you're doing so you know that educating and getting the message out that you're on top of this is critical um [Music]

i find it interesting that this was a huge chance for people to re-evaluate the risk they've accepted how many people have a security control in monitor mode only why because we're worried about the function i was going to say be honest but yeah no need to do that how many people won't raise your hand no matter what i ask okay good nice psychological warfare i like that so uh you know this is something that that is is fascinating because when this came out the leadership was now asking are we blocking this not security banking can we please put in a blocking rule and so they took advantage of it i i saw some customers change their entire platform to block

mode and some customers only block on the log4j rule um and so there were again this is risk tolerance it's the customer experience security balance out there but a lot of lessons learned around this is an opportunity to go back to leadership and talk about your your risk appetite which is a an ffiec banking term going back to what's your corporate willing to to say is the right thing to do hey on that point will you talk a little bit about uh you mentioned you know someone's driving to work as a board member they hear hey the latest you know earth-shattering vulnerability that you're all gonna die because of talk a little bit about the

involvement of the board when things get to that level how it changes the scope of uh of how you have to respond to it you know from a notification perspective from uh now they want to be updated every 10 minutes just like you know your boss would normally want to be what's out of the fight so my experience has been 10 years ago our boards had more luddites on them than technically competent people if you don't know what a luddite is i'm going to google that it's worth going to wikipedia i'm not going to go down that rabbit trail but it's it's one of my favorite derogatory terms and so um the the thing we've seen is

how many businesses now don't have a critical part of the business that depends on i.t it's fewer and fewer every year and so boards are now talking about the business strategy in terms of technology capabilities and so we're seeing more of them educated more of them they've always talked about risk they've never talked about technologies and so the board doesn't want to know what log4j does they don't want to know what open source is they don't you are not a technology advisor you're a business partner and if you go up as a ciso as a technology advisor you're you're not going to knock it out of the park and so you're going to go up there and you're

going to talk about risk and on day one you're going to say right now we have risk around customer data in these three places and we put a a a segmentation policy we have additional monitoring we uh have an immediate patch process once a patch comes out we've implemented a waf or whatever you know this is if it's solar wind it's a product then you know there's a different set of what you have to do are you going to uninstall it what's the impact of the business again rabbit trails i won't go down um but it's going up and talking about the impacts and the risk to to your business and if you go back 10 years 10 15 years

uh how many how many companies didn't rely solely on the internet to do their business um so whereas you might be able to say just say you're in secops or you're in incident response um and there's an infrastructure problem it's so easy to kind of pass that over and say oh that's a that's a you know a platform issue or that's the cloud people that have to worry about that or that's a uh you know or there's some kind of if there's an internet outage if there's anything that could affect your company making money trust me anyone who could possibly fix it is going to be in scope for you know a resume writing opportunity right so

the whole idea here is as more organizations are relying on the software stack the internet stack to to viably have their business function um it changes the conversation it changes the education of the board uh a lot of the board now are a lot more technology savvy worse versus you know 10 15 years ago they were just kind of pure play business people advising on the next strategic step to take uh so i think that's an interesting change too uh and and we should be cognizant of that when it comes to how we deal uh with our superiors up the chain because you have to you have to understand where they're coming from too and you probably haven't heard of this

organization so you may want to write this down but there are a lot of industry ice acts out there never heard of it and that's his favorite topic um and so that's another thing that you've got to decide to put in your process when you have these are you going to call in and see if there's an isac call fsi sac healthcare ice hospitality retail at one point there were 15 major ice sacks i i think there's more than that now yeah i think there's like 24 now or something like that it's pretty pervasive but where are you going to go to your your peer group what communities are you going to reach out what external resources are you

going to leverage and there's a bunch of them out there dhs is going to put something out there about a week after you need it um but again no offense dhs no offense is what i meant to say and so um you know but i will tell you why it's worth going and looking at that is it's a great lessons learned it's a great checklist to see what you did and what other people did the last thing is expanding our tool kit for each one of these there are different remediation or mitigation techniques out there and so kind of think through that that first set we went through what tools do you have in your tool kit

and which where do you have gaps you know we we talk about segmentation we talk about access control we talk about you know if you're in your ap environment api environment what what are the different tools you have or levers to pull during these incidents

so again i want to talk a little bit about you know what is the next step what should we be doing and um i've really tried to make a case here that we should shift off of thinking about a zero day as an incident a sock-owned problem and expand this to be what it needs to be you know a communication exercise a racy with i.t vendor management forensic the pen test team going out and pr even public relations even if you have to give incident response or no notification that hey you are looking at this or maybe because of the vulnerability we had to take systems offline for patching uh all of that has to be in scope

absolutely and so um you know if you don't have a crisis management prop process you can go leverage that we need to have another discussion but for those of you who do i think this is a great um i'm a huge advocate of nist uh simply because nist is something that you can show the auditors is a best practice so you know something like um know moving missed 190 i think is for apis i think we have uh miss 207 for um so trust you know there are a lot of special pubs out there that you should be going pulling verbiage from best practices from and crisis management well and don't forget that first incident response

cycle i showed you is from nist as well and that's more the transitional 853. um so go out there build the documentation exercise practices i know you wanted to talk about that as well um and and cover a little bit about the s-bombs and the vendor management there yeah you know we talked earlier about the idea of uh um an s-bomb right so what software exists in your environment uh whether or not it's your responsibility to patch it um i mean years ago i used to use tools like secunia which are agent-based tools that will install and basically map out all the software that's installed on that system and then compare it to the latest patch

update for each of those different software packages um things like that exist now i know they got bought out they're doing similar work though uh but the idea here is um if something exists in your environment eventually it can come back and bite you because you know who's going to be looking for your you know uh your 20 year old uh you know software version of uh whatever package you're using on one of your old web servers the bad guys the bad guys are going to be looking for that and it's not just by happenstance it's the path of least resistance i don't have to use a zero day if there's a seven year day out there that hasn't

been patched yet because you haven't gotten around to it so that's that's one of the big pieces and again to the whole point of of walking through this and practicing it i can't stress that part enough um pretend like you are going back through heartbleed say what is it going to take for us to re-key all the um the private certs uh and then redeploy um you know the search to our other machines what's that going to take who's going to be involved just practice that and without going through a process like that if something like that were to happen again um you're gonna be starting from square one uh and that's what we're trying to

avoid here we we do a lot of this when we work with our customers um and our clients just because if something were to happen we don't want to be the ones who are in the way of them getting their problems fixed even if it's nothing to do with us uh so we're trying to help kind of grease the skids so to speak of those things working better within our customer base so if something does happen again we're not the we're not the roadblock yeah and so again going back to that first slide the different categories that you should do exercises around uh and then there are different types of exercise there's a this typical tabletop

where you're you can just sit down with the racy and talk through the race and you should have a timeline in steps notification discovery patch release you know you'll have some key milestones in your exercise for each of these and some of these will involve leadership some of these will be more technical in nature so so think about the kind of exercises you want i think they're all beneficial so we put up some resources here one arthrit research group has a bunch of great publications uh i was just talking at uh at the other conference across the road about our gaming state of the internet security report so cool findings that we found basically if you own a gaming system or

if you've been uh during the stay-at-home order i like to call the stay-at-home and game order because as a gamer it's like you mean i get to be at home all the time and if i'm on a boring call i can be on a oh is this being recorded um i can be playing something right so uh also i have add so i work better with that kind of distraction in the background um our security blogs uh again a bunch of cool stuff here just around things that we're finding um we have a security researcher named larry cash dollar he has something like 350 cve say his name we call him the wordpress killer although he has

recently kind of branched off from wordpress um and on youtube we have some developer videos uh a lot of these are developer kind of advocacy videos talking about how you guys as security personnel can interface with your development teams better and again how we try to do that within our organization as well and then the last piece is we are hiring so if you guys think it'd be cool to work for a multinational cloud security organization uh we do really cool stuff i've been at october we hired just about anybody that's that's true i i snuck in um but i've been to document for nine years uh it's the first time i've ever worked for

vendor uh but i've been in secops since the late 90s so it's really been eye-opening kind of seeing all the cool stuff that a company like like us are doing um so that's smart i would also say uh the blogs a lot of the stuff we talked about for log4j you'll see in our blogs a lot of the charts tom evans is great about ddos or is great about fishing we've got i'm really proud of some of our research all right who's going to who's going to stump us now's the time we don't have to go there you don't have to stump us but um if you use the microphone it'll be recorded for the uh for the people at

home as well

okay really important one thanks for this this is great uh who played the woman's lead role opposite roger moore and the spy love me uh okay i recently just watched all those again uh with my wife because we were like dude these are classes call your wife dude no no i i was just saying dude like these are class okay shut up steve um that's that's a tough one i i should have we should have put a caveat that said he's better with lines i'll give you that any movie that is from the what no go ahead b all right dude it's too late for the caveats okay i know one you can answer yeah yeah so

um the pr part you touched on is really essential i i think that when a breach happens um eventually your tech guys are going to figure it out but if you botch the pr your rep is screwed for a long long time um if you're a smaller group and you don't really have people that are specialized in that area how do you handle that that's a great point so i will go back to my first hand experience in 2014 um with heartbleed we put together a group which was our pr ar team uh our platform security architects some other kind of security evangelist type people and our cso and we all kind of work through

literally like the wording of what this press release was going to look like uh what we were going to say sometimes you can't say too much exactly uh but sometimes if you don't say enough it's like are you are you giving proper warning to someone else who may need to take care of this right so that's a it's a very fine balance and i think ultimately depending on um what organization you're a part of and what your responsibilities are there's legal responsibilities but also ethical responsibilities i think all that has to be taken into account so it is steve i don't know if you have a more specific answer but so i mean um transparency equals trust and loss of

trust is hard to recover from that said i think about two things during like during a breach i have a person assigned who's going to tell me if i'm not following our process because if i don't follow our process and they just the class action lawsuit discovers that i'm going to be sued for negligence i have my chief legal officer my general counsel hire my forensic people so the forensic evidence is under attorney-client privilege so i mean that's how i'm thinking at the same time when i'm talking to the board i'm saying where we don't have enough to say if we say we lost 3 000 records today i'm we may come back tomorrow and say oh

make that 1.5 million yeah big difference so it doesn't sound transparent to say we're still trying to figure out what's going on not hard to get the the marketing and pr people to sign off on that statement so it's a delicate balance i would lean into a culture of transparency uh and and talk to legal and talk to pr about sharing upfront yeah thanks for that i what i'm seeing in general is people weighing cya against yes what is the damage right so that seems dangerous if there's like a framework that really makes it clear this is your legal obligation to disclose this portion versus you know what i'm saying yeah yeah and i think they've attempted to create

frameworks around um you know one from a vulnerability perspective if you have any risk involved there responsible disclosure uh and then also breach this disclosure is being handled a lot more from the compliance level where they're like hey these this is what you have to do if something were to occur uh as per the the compliance that you've you know opted into that you've uh agreed to right so there's also some psychology around admitting mistakes um the the classic example that i always remember is warren buffett every year puts a mistake he made in his annual report because he has so much more credibility by doing that and the one year he didn't feel like he had a good

oops he talked about the oops from the year before yeah um so they're they're again transparency generally wins uh and it also goes back to how negligent you were uh in the first place thank you thank you thanks for the question so you guys are a huge company very technical serving very technical customers who are serving their very large very technical companies i work at a startup and i've got three engineers i'm in qa by the way i'm not even a security person thank you for your service i tried um are there resources on your blogs for how to get the rest of your non-technical team on board with these ideas with these for lack of a better

term war games about like how do we bring everybody into these because i used to work in support so i'm really good at writing effective apologies and emails that don't get us in legal trouble there's a lot of resources outside of just the technical parts that can be looped in here and be really helpful for cya for making the apologies for doing the press releases do have more resources for how i can get my marketing team interested in this for what happens if hubspot goes down and planning for what happens right yeah yeah um do you want to take a several so um there are first of all there are great industry specific podcast uh and and one of the things i like to

do is figure out how like somebody how somebody likes to learn i prefer to read uh my add makes audio books impractical what was that and then um you know other people have different methods so find out what they like sans has a lot of great examples uh they're great to plagiarize from uh youtube has a lot of great resources um like i said i plagiarize a lot from nist uh iso is a cost so i i don't if you're international you want iso over nist but it's not free um and again belonging to community uh there are so many communities out there beyond the ice ax that uh the more people you can plug

into them but um you know like i said for your marketing if this is hilarious so i was i was talking to the marketers and they're like the the book influence is great we use it and then i'm talking to my fraud expert me and i were doing a talk together and he's like well all the fraudsters know that the book influences the best book to learn how to be a good fraudster and so you know that's everybody yeah yeah that's that's a resource that uh that went right to this top of my reading list well i think the uh the point too about uh join a community another good piece of advice might be

build a community within your organization say hey i really want us to be prepared for what happens if something catastrophic were to happen uh find stakeholders within each different part of the organization and all have lunch together one day and then maybe the next time you say or at the lunch maybe say hey it'd be great for us to have like a monthly meeting just to kind of talk about all the pieces that fit into this and then maybe from that you guys can start assigning responsibilities or areas of expertise even a lot of times people's areas of expertise is not the role that they're currently working in uh i worked for years at a company where

i was like to do everything person and it turns out you know i would rather do less so uh for from that perspective not just less i would rather not have to do everything the hard way um i want to automate myself out of a job kind of situation right so um but i would say build a community within your organization uh of people who are like-minded that that think it might be a risk if something were to happen and how are you guys gonna respond to it so hopefully that helps thank you yeah and if i come up with anything else i'll post it on twitter thank you so follow me on twitter it's not a shameless plug

nobody follows me i don't know why i'm just not interesting really but i will share something if i find it okay awesome so thank you so much also my favorite line that no one seems to get i am an excellent driver i'm an excellent driver excellent driver yes well now this is like really putting me on the spot that is 80s is that um weird science it's not okay all right what you got it okay i don't want you to stump me and get the one that she could anybody else know it oh yes okay to be honest i probably should have put some caveats at the beginning of that well this whole talk has been for naught

uh anyway if there's no other questions anybody else have anything else all right thank you guys for being here really appreciate it and uh have a great weekend [Applause]