I Am The Cavalry Track — Public Safety and Critical Infrastructure Security
Show transcript [en]
[Music] so [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so [Music]
[Music]
so [Music] [Music]
[Music]
[Music]
[Music] do [Music]
[Music]
[Music]
[Music]
[Music]
[Music] do [Music]
[Music]
[Music]
do
[Music]
[Music] do
[Music]
[Music]
[Music]
[Music]
[Music] so [Music]
[Music]
ah
[Music]
[Music]
[Music]
[Music] so
so [Music]
so [Music] so [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] [Music]
[Music]
[Music]
[Music]
[Music]
[Music] do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] so
[Music]
[Music]
[Music]
uh
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] so [Music] so
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so [Music]
[Music]
so [Music] [Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music] do
[Music]
[Music]
[Music]
[Music]
[Music] do [Music]
[Music]
so
[Music]
[Music]
[Music]
so [Music]
[Music]
so
[Music] so [Music]
[Music] do
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do
do
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music] so [Music]
[Music]
you
[Music]
[Music]
[Music]
[Music] so
[Music]
good morning besides las vegas i hope you're as excited to be here as we are so thank you for attending excited to have you back with us this year in person a few announcements um we'd like to thank our sponsors especially our diamond sponsors lastpass and palo alto our gold sponsors amazon and visiom and plex track with their support our sponsors donors and volunteers this event wouldn't be possible so thank you to all of those that have made this event amazing this year this talk is being live streamed so please take a moment to silence your cell phones as a courtesy to our speakers and for those following online if you have questions there will be a
microphone up here so please feel free to step forward and ask questions and then no pictures without speaker's permission so with that i will pass it over to josh all right thank you welcome to day two um this is the meet the press fireside chat we are we talked about everything's on fire yesterday or things were flammable and stuff's on fire uh so let's have a fireside chat with some of our favorite journalists um there are a lot more puns in our wedding vows than there will be on stage today but um so a couple things just to frame the day and then we'll get to uh introductions for our panelists or our fireside chatters
is yesterday was really about we started the cavalry nine years ago we wanted to tell people stuff was flammable that our dependence on software and technology and critical infrastructure was growing faster than our ability to secure it we were worried about where bits and bytes meet flesh and blood but we knew and we told the room that no one would really listen until there was proof of harm until we had existence proofs and steel people are actually hacking these it's one thing to do stunt hacking of a car or a medical device it's another when it manifests harm and could represent a crisis of confidence in the public to trust these connected technologies and we don't want that so while any loss of
life would be tragic we realized the failure mode was not merely a loss of life but any sort of crisis of confidence for people to trust otherwise superior medicine driverless vehicles that the light switch would turn the lights on or that the water was drinkable and in the last two years in in parallel with the global pandemic we saw successful attacks of the water we drink the food we put on our table the oil and gas that fuel our cars our homes and our supply chains the schools our kids attend the municipalities who run our towns our cities are functioning of government and even timely access to patient care with mortal consequences as we saw with
kendra's talk yesterday about the sissokova task force findings so we have seen that delays affect patient outcomes and loss of life and protracted cyber attacks introduced delays sufficient to drive those outcomes so i think one of the head scratchers for us in the cavalry was now that people can see on john oliver on hbo or on you know it's main street news that you couldn't get gas so there have been disruptions and there's been hacks we really thought there'd be more political will and more advanced conversations about what to do about it and now we're asking between yesterday and today is how should the mission of the cavalry change and evolve now that we're not telling people and
educating that things are flammable but now hopefully trying to drive down and minimize risk a little bit more fire fighting or engaging those underserved by either the private sector or the public sector and we lovingly call that the cyber poor so what i really wanted to do is we've always loved working with our press and it's always been a difficult thing to talk about future risks and slow moving risks if you saw the movie don't look up today's opening session is to say how did the world look nine years ago before there was proof of harm and how does it look now that we are having main street mainstream kitchen conversations to quote bryson from yesterday about
cyber attacks affecting food water shelter safety and maybe learn from some of the journalists that have been on this ride for a while how this room and this community should think differently about pitching stories making ourselves available focusing on the public good instead of the private enterprise good focusing on public safety human life instead of record count and we don't have good answers to this but i've started some conversations with lily and joe and i will let them introduce themselves so we can get into some content and there are microphones and if you do ask a question i'll either have to repeat it for the streaming because people are watching from their rooms and from the
inter tubes uh or just go up to the mic and uh you'll be heard yourself but let's get started on the meet the press would you like to introduce yourself lily hi hi everyone i am lilly hayne newman i'm a senior writer at wired and i've been on information security cyber security digital privacy beat for six years and i was a general interest tech reporter before that so uh going back to the start of the cavalry i was definitely writing about adjacent stuff uh so it's yeah it's really i'm happy to be here and it's it's really interesting to think back to that time and sort of reflect on where we are now i think we don't always
have a moment for that type of reflection and i think it's really productive um my name is hero util um i am i might have to get a little closer or paulie sorry about that my name is joe utel and you can hear me now i am a reporter for sc media which is a business to business cyber security publication before that i was at axios and started the code book newsletter which i think they're about to bring back so hooray um i've been covering cyber security as um like as an ex exclusively covering it since just before sony so what's up 2014 2013 2015. it's weird that i can get i now date things
by news stories um but yeah since around that um it has changed a lot uh i think back in uh when i was starting out i could not show up to the office and nobody would know and now now now it would be a thing if the cyber person wasn't there yeah during our prep i was trying to say so i guess this is a good a good place to go which is um if we try to do a deliberate compare and contrast between 2013 uh and now um one of the things i pointed out is we had just had the snowden revelations and there was probably the worst levels of trust between hackers and government
that we were going to see for a while and quite a few white hat hackers or helpful hackers were pretty angry and looking at maybe uh going a little gray or a little charcoal colored in their hats and um people were upset and it was really difficult to say let's try to be a helping hand to drive safer outcomes with government partnerships but we also were pretty worried about that we'd seen some medical device hacking from barnaby jack in fact he tragically had lost his life just before we launched in fact he was supposed to be in the room with us um we saw some car early car hacking but it was considered theoretical or it's not
hacking at times and one of the things we told the room was people would have to die first before they're really going to listen to us and if somebody shouted out why do this then and i said well we want to be have a head start and built the trust and laid the groundwork and the scaffolding so that they turn to us instead of lesser people with lesser motives and lesser ideas or that we don't have an overreaction like a cyber patriot act or something but the whole goal here was to maintain the trust the public so if you situate yourself in post-snowden revelations anger and distrust between government uh no fda regulations um and what did coverage look like um
and now just pre-associate i guess well i was just thinking we were talking about data you know dating the passage of time with news stories and it that era was really like a data era i feel like where like data can be accessed or it exists in tropes like that was the big concept because i'm thinking about i think it was early 2014 like maybe february or march that the target breach was revealed and like neiman marcus and i don't know this is feeling very retro in my mind i'm like wow we're back in this era but um i feel like that was those were some of the big mainstream discussions that year along with you know what you're saying
about the snowden revelations so it's sort of like a you know bulk surveillance collection corporate data troves what if people were to access this data like moment or that that was like where people were at in sort of the mainstream in terms of what the press i think was trying to convey to the mainstream audience um sorry i almost did the same thing again um it's strange like a lot of there was a i don't think we quite knew what we were covering yet because so many of the threats hadn't materialized so a lot of things were very speculative but crime was since it was crime was as as always said mostly data breaches and
the nation-state kind of level was still very very driven by um it was still very uncertain like we people didn't understand how attribution was going to work there was the north korea thing where people just sort of sort of rejected um i think since then we've seen sort of the rise of news covering um every breach and making that the big event like breach by breach by breach and that being the big news story of the day and not really taking a broader look at it i think that's sort of dying down but only because they've been concentrating on single breaches longer but it seems to be very event-driven now whereas in the past it was more
speculative if that makes sense yeah and i'm also thinking about like we used to cover i don't know even thinking you know i always and i'm sure you the same way like we were always trying to have the most context possible and do smart journalism like joe and i are both trying to like do good reporting and not just go bridge to breach or something but it feels quaint to think back to some of the coverage like you know even five years ago or something stories that wired readers were super super interested in like massive traffic which not for its own sake but in terms of who was reading it like the number of people who were reading uh i'm thinking
about like stories about what's the biggest volumetric ddos attack that's ever happened it would be like now github was hit with the biggest one now this was hit with the biggest one or cloudflare whatever and like people were very interested in reading those stories and it would cause you know maybe outages for a few minutes or an hour or like the company wanted to talk about it because they didn't have any downtime at all or like whatever but it you know it's just to say that that was where we were at in terms of the collective conversation of cyber attacks and impacts one thing that we don't get to write anymore and it's for the better um
around 2016 and for for obvious reasons there was the rise of the um that wasn't hacked news story like where there would be a power outage and you'd have to write a new story about how that wasn't hacked or apt squirrel yeah the uh one i think people are a little bit more a little bit less on edge that anything could be any anything that went wrong could be hacked well i mean it isn't really anything that can be wrong for that it was really all power grid um and i think that's uh so but yeah sorry yeah it's it's hard for me to even ask these questions because i've always been on the i care less about my
credit card and more about my access to patient care you know we've been here for nine years we've been trying to get the world to catch up a bit but you know i would say that as you think of how we used to characterize stories it was record count or dollar amount was the unit of impact and now it's potentially quality of patient care or how long are you without oil and gas or how long can a municipality not function and perform the duties it needs to to keep its citizens safe and you know what are the supply chain cascading impacts of a meatpacking facility getting hacked or cream cheese my daughters were about to riot because
they couldn't have cream cheese for their bagels for a while and people didn't know why and part of why was there was a hack of one of our concentrated uh developers of cream cheese exacerbated by trucker shortages exacerbated by other coveted factories but the net outcome here is if there's not enough slack in the system you know even these seemingly unimportant ransoms of a single entity could have cascading rippling effects on you know if my kids are going to riot for cream cheese so i care a little bit less about cream cheese and more about losses of life but it is often difficult and this group has found it difficult to get these stories told that are about
the public good instead of a specific victim so we're all ears for advice on how editors are shifting focus or what stories have a chance and which ones don't you know um how we might both respond in organic changes but also try to affect maybe some deliberate changes so that we can ensure there's not always and i can't i still find people who say well yeah josh people might have died but what about the record count like there's just a really palpable part of us that is painful yeah i think there's a belief that and it may it might bear out in traffic numbers but there's a belief that if unless it's it kind of
unless it's unless there's an enterprise angle unless you're talking about um you're talking to businesses it's tough to sell advertising uh um at least that was the sense i got at axios where they wanted it to be more more either about controversies from big tech brands or um more about businesses or you know i i hope that's changing um i i've been at a been at something that solely covers the enterprise for the past two years but uh i think that there's as people have seen [Music] real-time chaos engineering um you know the power outages in texas or like you were mentioned colonial pipeline people have a better sense of what can go wrong but i think it's one
at a time yeah um i i don't know if there's i think that people are very driven by stories like not not necessarily um people like a narrative rather than a fact and that's it's easier to tell a narrative about people lining up for you know lining up for gas to put in their trash bags than it is to say um uh to cover the you know the effect of copyright law on uh on on hacking hardware well yeah i i want to add like we should talk a little bit josh about what we've been discussing about the interdisciplinary nature of some of these stories because i just want to say first that you know i
think what we were saying about what you said about it used to be i could take a day off and no one even noticed and now it's like where is the person who needs to write about this every single day like i've that is a real thing and that's a real issue with getting stories written and getting them to break through is that there's just so much going on i mean it used to be when i would come to black on def con and and b-sides and cover talks at all the conferences like there was nothing else i needed to be doing that week but that right like that that was the you know news that really
needed to get out there that week and now it's like 30 other things are happening this week and i'm like like what you know my editors and i are like what do we do because you know so anyway i think and so especially you know to transition to you know what we had been talking about like stories and what you're saying about narratives and real human stories that are the most important stories that we all really want to be telling that are the most impactful those stories take a lot more time a lot more sensitivity and and various types of expertise and it's just a very there's like a lot of cooks in this kitchen you know
i think one thing that's worth noting um and this is for the better um more people who write about cyber security are focused on cyber security now than there were um at any other time in history it's a fun thing to be able to say um i think that if you remember uh norse yeah with the aei report that speculated that iran was hacking the united states hundreds of thousands of times a day it just turned out it was you know background noise and ppu um that can happen those kinds of stories can happen the uh the the power grid story in the washington post i think that that was the vermont was that there was a there was a time that they
accepted some did have the power grid that turned out to be oh the ms blaster from 2013 that one no i think it was around 20 well the fact that there's more than one example um a lot of those things slipped through because the people who ended up covering cyber security stories were people who had no infosec background and can be easily swayed by the um by things that sounded very impressive um but not but didn't really have a sense of how to evaluate whether or not they were true wow there's so much i want to unpack from this i'm just going to scatter shot a few things i heard that on us choose your own adventure because any
one of these could turn into an hour of conversation one of the things that surprised me um i'll just enumerate them uh not prioritized one of the things that surprised me is i've actually seen a thinning out of journalists um a lot of really good journalists and really good beats have gone away and now some of those excellent journalists are in sponsored corporate journalism they're still doing good stories but there's always that risk that there's an editorial slant towards that market or that product set and i still love and respect these journalists and it's it's sad to me to see so many of them not working for a news outlet or an independent news outlet
but doing news under the imprimatur of some corporate entity so number two um a bunch of us were observing there's stunningly few journalists here in vegas this week and it's possibly koba it's possibly competing stories it's possibly that this isn't sexy anymore um but that is an observation i think worth noting i'm very glad you're both here um number three um i worry um about where lily was going we talked about this in advance some of the stories that came out of our sister task force were so multi-disciplinary in nature you couldn't find a single journalist with a single beat that felt comfortable doing it and unless there was capacity in their medical companion or their government oversight
companion these stories couldn't get told there was a level of complexity that created enough hesitation for the harder the more substantive stories to get told so pick apart any of these or none of these but um i i have some concern over this i guess working backwards um it's easiest it's easy for me to say uh lily should do it but um the uh i think i'll take all your scoops too the d i think that with uh things that are extremely multidisciplinary you might have better luck with magazines where they have time to do thought out coverage uh compared to newspapers where we're expected to increasingly put out a story not only every day but
every few hours when i was at the hell um i was at the hill but also i didn't say that in my bio um but the expectation was four or five stories a day wow um and by the time i was leaving they were trying to figure out which four or five stories a day and i remember they were really excited about a lacrosse scheme that was hacked because i thought that would be very popular it did not turn out to be very popular um but uh at least i got to learn that there was a professional lacrosse league so that's something but in terms of the uh the five story a day model isn't great
for things that we have to learn outside learn something new one story a day isn't great for that but it's possible or but the people who have time to really write things out probably have the best chance of understanding something that covers more than one thing but i think you really hit on it because the thing is i think the jbs meet you know ransomware attack supply chain attack uh is a great example because it's like to use the example of wired the thing we don't have at wired is someone who's sourced up on the meat supply chain right like who does or who does like meat futures or like we have no meat coverage guys
so i i think this really raises a good point that some of the blind spot or some of the like you know best intentions where stories don't end up happening that josh you're saying like how do we get these stories to happen comes from the fact that certain publications potentially have the like institutional expertise if you put multiple people together but there's the pres those are newspapers and like wire services and stuff that have their own pressures right like reuters for example they do amazing cyber security coverage um they they were an organization that i was like okay they're more likely to have solid stuff on jvs i'm just using this as an example because
they've got cyber security reporters they've got you know meet people they've got you know all the different like um whatever import like you know reporting expertise or import export all the different things you would need but they don't have time right they only do a certain type of story like the the the sort of 200 word or 400 word reuters uh and i mean occasionally amazing long investigations whatever but just to use this example whereas wired can go really deep on something but can only put one or maybe two reporters on like a deep investigation uh so yeah i think that's really getting to there's an interesting thing that goes along with that um kim zetter once made
the point and uh uh that um one of the nice one of the ways you could tell it was going to be a good news story is if one of the names in the byline was the the research person you know the person who um many newspapers have a not just an archivist but somebody who's in charge of sort of doing background and research fact checking and things like that that sometimes will get their name in print um as part of a story and that creates a great sense of institutional knowledge and i don't think that cyber security really has that a lot of the reporters are young um and i think that that sort of gets to
some one of the other things you're saying a lot of the reporters going to [Music] either either becoming a lot of the older reporters moving towards um public relations or um the you know things like the record which i say that they are independent so uh just to be fair to right um um there are great farm systems now i think cyber scoop has created a bunch of really good reporters um chris ping came from there um a few other and we were supposed to have a cyber scooper on this panel right suzanne smiley thank the flights for that one yeah and suzanne's had some great initial stories shannon you know like a bunch of really
good people have come from that and one of the things that's really difficult in journalism is finding somewhere that will train you to do something well and so um while some of the more venerable people may be doing more things that seem that seem distasteful i think that there's a group of people there are more people who exist who might be able to okay to handle them and um it's hard to jump off my own shadow when i ask this it's a bit of a loaded question but as somebody who i think is kind of rallied the public interest public good public safety subset of the hacker community we often cringe when we see people we
like and respect for their enterprise perspective or their corporate for-profit perspective being quoted on public safety stories when the message is optimized for private interest instead of public interest and i think some of it i've seen is just people get used to you know reliable sources or people that are quote worthy or might get more clicks because their name but they may exactly be the wrong person to ask like we'll see something where there's collectively 100 years of experience in the room and somebody who spent no time on that topic is the lead quote and you know i know that that's some journalism one-on-one stuff but is there a need or an opportunity to differentiate between corporate security
and consequential security or private interest security and public good could there be a discipline where it's easier for any journalist to find sources in category a versus b or am i just butchering my articulation here i think lily might have a better sense of it than me because your beat is more business to business more business business um but i think that that's the fact that there are publications that are business to business and not publications that are public to public right um sort of says something that uh the uh people are interested in the enterprise point of view because many of the readers of these stories are in fact enterprises uh and want to know what they should be
doing and want to know how they won't be in in the next one of these news stories um i don't want to say that the public doesn't matter but um from our perspective yeah don't say that yeah from yeah from from from we have a business model and you're yeah from our the business model that i see is to talk to enterprises about what they should be doing rather than talking about public good yeah and uh i'm no longer in cisa i didn't work on the colonial gas example so i'll ask a question from a distance without inside knowledge but that's an example where typically in a country a public good like oil gas public utilities it's it's a
responsibility of the public sector but in our system of government we delegate a lot of that the ownership and operation to private entities so when you have a breach like that and people come monday morning quarterback the decisions made it seemed to me from the outside that the right thing for shareholders might be to preserve integrity and billing and shut things down but that that unilateral decision might hurt the public interests of the eastern seaboard for a while so when you have a tension between what's right for the enterprise and what's right for the country or the region i think there's an incredible story here of how do we need to recalibrate the social contract between the public-private
partnership and you can delegate operation from the government to the private sector but you're still accountable for it so are we actually able to govern reliable maintainable access to critical infrastructure or have we gotten out of balance and i i found it really there were a couple of decent stories about what happened but maybe not the strategic implication of that public-private partnership yeah i'm having trouble organizing my thoughts but i think it's because of what you said that this is like a societal level recalibration that one could argue needs to happen in the u.s but then i'm also thinking about how globally like a thing about when we're thinking about this you know the years of the
cavalry or you know since 2013 or just like recent years of cyber security and the security industry like the the private sector has been so uh just has has led so much of the narrative and and shaped so much of the narrative uh and and sometimes in good ways and sometimes in like court leading the horse ways um so there's like a sort of parallel thing there but i think that issue you know some of what you were talking about just now is us uh centric but the the larger security industry and like what people understand to be the scope of cyber security topics uh and even like uh uh thinking about like the concept of cyber war
so much of that has been driven by the private sector that i i can see why there is this like decoupling that needs to happen or that someone might feel needs to happen because it that's sort of how all of this has sprung up and i guess that's sort of the classic like black hat versus defcon thing of well where's the space for all the vendors to be and where's the like you know economic engine of this whole thing um and then you know like other folks in the community feeling like well i just want to do my research and focus on like what's actually impacting people like i think it's a big big issue
so i'm gonna float in a opinion you've triggered um i also want to remind people that we do have the mic in the room we also have the streaming chat um so if there is a burning question you'd love to ask either of these two fine journalists please do while he's heading up there i think one of the areas that this dynamic got it wrong was ransomware i think the hacker elite rock stars thought it was boring and we just kept paying it and we created this juggernaut that's unstoppable and i think one of the ways i put it after rsa we were sitting in the garden i said the attackers have figured out how to
monetize the cyberpoor defenders haven't and it's going to be a feeding frenzy it's so big now there's no stopping it there's no single even international move that's going to stop the business model of on availability no matter what you do your unavailability can be monetized and unless and until this becomes interesting to either rockstar hacker voices and or commercial coin operated we've got a real public health issue that was possibly preventable in its infancy but it's it's it's not going away anytime soon the the uh the government will set a lot of the narrative there and i think that the government right now is putting a lot of pulling a lot of that pressure into two spaces one of
of international uh partnerships um which doesn't get as much coverage as the other one which is you know forcing uh which is you know requirements for security requirements through the executive order and things like that um it is a much broader topic than that but i think that for the most part when people look for coverage it's going to be what the government what the the cover what the what the elected person said they were doing um oh yeah all right we have a question i do i have two questions one is how do you keep up i mean like cyber security is my job and it's it's like my job right it's bigger than
what i can do so i wonder how you keep up and the next question is an offer how can the i am the cavalry community help not not just you too you're great but i'll just say the press more broadly uh when it comes to stories and pitches and background etc um how do we keep up yeah it's really tough i always go back to um just trying to it's gotten tougher and tougher like we were talking about but i i just try to go back to having as wide a source space as possible people i know people i sort of know but who won't be freaked out if i dm them or you know
whatever and just really trying to always be in touch with people and hearing about what they're thinking about from as many different uh perspectives as possible and that ties into what you were saying about how how to get the right voices into the right stories but there is you know every journalist like that's our job and we pour a lot of work into it but there is a limit to just like how many people we can know or how many sources we can maintain and relate you know those relationships so i think it's like an important ongoing question of combining all the journalists like networks and um efforts to do that to to try to do the full coverage you
know because you really the beat is way too huge now for just like a few reporters to cover alone so that's on on that piece um in terms of uh how i am the calvary can uh sorry i missed the microphone again um in terms of how you can get more stories placed um reaching out is a good start um i don't uh and not just reaching out through press releases and things like that if you pick your shots and just occasionally say this is important um i think most people will will listen um at least most people who have a sense of what you're what you're talking about will listen but i also think it's like the proactive
thing of you know by the time you like implement network segmentation or something like it needs to have already been done it can't happen once the crisis is going on i think maybe it's sort of like that with reporters that it's it's helpful to just like be in each other's orbit so that you can then you know if one you know if i know one of you and we've you know met it besides and chatted a little bit and we keep in touch and send each other memes or whatever and then if you dm me and you say this is really important i really want you to look at this you know that helps a lot versus like my my email or
what like my inbox is just like a hellscape like i can't keep up with and sometimes there are really important things in there signal tip lines same way like but it's so difficult i'm always worried that i'm missing stuff so i think that personal connection you know like befriend your neighborhood journalists like now before you need them you know all right do you have a question i do oh there we go not step on the mic uh thank you for coming and i have a question since a lot of the news seems to be very kind of western-centric i was wondering if in your experiences and in your connection circles and your professional kind of working lives
do you see a similar investment or divestment of resources of opportunities in international partners that you may or may not have uh places in like europe is obviously a big one but other uh non-like english-speaking countries you see kind of like a similar uh like oh we should kind of cover these like security things are do you believe that certain governments are more oppressive in that manner and that they go you should not report on these things otherwise your family will disappear forever it's a good question um we certainly have some good uk journalists but that's fairly close partner to the us yeah joseph is great joseph cox yeah great dude love him he's one of the top joes
we have a lot of joes yeah just a bunch it's just way too many i think there definitely is in uh and and like has been for quite a while but you know it's very fair that like the effects you haven't necessarily seen the fruits of this so much but i i think there has been like long-standing interest in doing more and more international coverage um like you're saying i think reporting uh getting uh working with sources in a way that's safe for them is difficult in a a bunch of countries uh so that is like a factor but means it's all the more important to do it and just one other thing i would say quickly
is like i think um this is another area where the the private security uh like threat intel juggernaut is like good but also you know limiting because that's a lot of uh how journalists but also the community in general uh hears about certain types of attacks and especially government uh or like apt uh activity in a bunch of different countries globally and um so we're all sort of at the mercy of those um that framing or or that uh yeah mindset yeah i think i think the the the the company with the most visibility in china is 360 right um and they publish in chinese uh and you'll have to forgive me for not knowing
which which version of chinese because i can't tell the difference and that's one of the problems is uh there's a language barrier to the sole source to the person with the most telemetry on on a on a field on an area um i don't know how to fight that yeah it's a it's a fair question and i the cavalry is a pretty international movement and we i have been personally burned because of just translation issues um like when i was at the atlanta council for a while i made out of my way to call it cyber safety instead of cyber security to differentiate the cyber physical impacts and in some of the countries covering us
they were the same word like there wasn't a different way to put it so here i am trying to mince and nuance u.s language and one of the stories almost got me attacked by anonymous i mean this is before the cavalry but like i said what i said i said it cleanly and clearly but it was lost in translation and an op was declared trying to destroy me and my family so it got called off you know we had some people from anonymous call it off but i'm probably more timid myself even though i'm a confident media trained person to get lost in translation so it's probably an error we could put some more deliberate focus into and specifically
on things like our hippocratic oath for connected medic medical devices or the s-bom work we were delighted to see translations in french and german in japanese because when you're speaking some of the strategies are universal and translated and we've had some really powerful partners in singapore or elsewhere who cared more about some of these topics than some of our us partners did but i think we could be more deliberate in international reach than having researcher friends who could maybe be more confident in those areas this might seem like a hard transition but one i really want to get to because lily and i had some fascinating precursor discussions raise your hand if you've seen the the
documentary film called um don't look up okay it really hurt for me to watch that um probably kendra too anybody in this is a code task force it felt autobiographical um we have a i have been surprised and i'm usually have pretty grounded expectations even though i'm a dreamer it's usually pragmatic idealism but i was really shocked to see how how little people actually cared about loss of life and healthcare and i'm gonna do a little bit of an exposition here we keep hearing from the healthcare that there's no money there's no staff there's no appetite there's no political will we're doing the best we can people die and they say until somebody dies you know we're doing the best we can
so there was a story in germany of a woman who had a branson had her go too far away uh a very long ambulance diversion to the next near facility and now we have our first loss of life and it has since been refuted and debunked but maybe less so than you probably were told because we know that delays affect mortality and that in cyber introduces delays so if cyber introduces delays and delays and for introduced mortality you know we we shouldn't be in even in the proximity to this but there's a a palpable desire for the private sector in the hospital to say that no one's ever died no one's ever died and that one we let go of because it was
debatable enough we let go of it and then on october 1st 2021 uh wall street journal front page talks about the baby who prior to the pandemic in alabama potentially um was a victim of degraded technology assist they said if we had had the imaging if we had known this we wouldn't there were text messages between the physicians saying had we known we wouldn't have treated the pregnancy this way so people are like okay we finally have a named person but then we say oh it's just an anecdote it's a one-off you know we can shouldn't change policy over that well that same day our team at cisco task force public statistical proof that we could measure hospital strain
associated with excess death in pretty large numbers and we could analyze regions hit hardest by ransomware to show that they were in those excess death stress zones for a projected amount of time so non-zero numbers of people lost their lives due to delayed integrated access to patient care in vermont in san diego and other parts of the country so we we kind of have this proof but what i was talking to lily about now that i'll get past the exposition is even really intelligent people even leadership people in the isacks and the sector according councils in the hacker community we still continue to say no one's ever died from cyber like we can actually have a smoking gun with a named
victim we can have a smoking gun with statistical proof we can have them come out on the same day but there's something about slow moving this is a human psychology thing that maybe we could all try to simmer ourselves on which is these slow moving threats these strategic threats we're really really bad at risk management as a species and we tend to like these bite-sized things where we have a control sphere maybe there's something we feel we can or can't do about it but i think my hyperbolic slightly exaggerated concern is nine years ago we were too early and now no one's coming back to tell the fuller story of these things and why we should be we
should listen to the warnings and there are stories covered and you i don't want to put words in your mouth or um or tell them about the dumb thing i said is that what you wasn't done no no the uh but you said there are certain people that every time someone's like why aren't you covering this or why didn't you cover this there's usually evidence of it so can you get in the pro public like i'm trying to wrestle with why don't we have an easy way to talk about high-impact slow-moving threats like climate change or oh okay no this wasn't uh i thought we were talking about uh examples of stories and i i gave like a
silly example uh but no i i just trying to think how to summarize i i do think there's a bit of there's a concept in general or like a thing that happens in general not just in this field where sometimes people will say why isn't the press covering x and often right we hear this all the time currently and uh my response just to myself you know internally as i'm seeing these you know arguments or whatever it's like but there is coverage just no one wanted to read it no you know it didn't like come above the baseline of noise because no one was amplifying it no known as interested in it from the readership
um and yeah one example i gave was like often when people say this there's like a big propublica investigation that happened five years ago or whatever about xyz thing that just everybody forgot about and you know but and there's other examples too uh you know and and sometimes it's my own coverage or joe's coverage or whatever where i'm like no i i did cover that like a lot you know and just no one read the stories or or one of them was big but you know then no one read the other stories to follow the narrative or whatever so i think that's one component of it but i and and so that's where we were talking
about you know climate change as like i think that is a big potentially analogous example where people always said why isn't it on the front page why isn't it you know every single day and it's like well it's actually i i think genuinely a more complicated question about like the service that you know of news and what is newsworthy each day and it doesn't mean the coverage isn't happening but what people want to read more of is relevant to what gets covered more and there's just all these feedback loops that are kind of complicated but it's i mean it's not to make excuses at all i i you know i think it's just to say like these dynamics aren't so it's
not always just why isn't it being covered sometimes things are being i think when people say why isn't it being covered they usually mean why isn't that the conversation on twitter this thing or why isn't it on the the prime time opinion shows or you know talk shows um and reporters don't really have control over that we can just you know go about our beats and submit things uh as as we can but um i think for specialized news coverage areas um and that includes homeland security and national security there it's harder to for the public to keep multiple stories [Music] that are outside of like a very narrow set of parameters in mind at a
time and i think people still view infosec as a specialized topic as opposed to something that no matter what your job is and no matter where you live it affects you at home and at work so i guess this is a i got the hook uh we have like three minutes left so um we can continue the conversation in the hallway and throughout the day but um i guess where i'd like to end is maybe with a semi-rhetorical question which is i think this group has done incredibly important work to warn and to be calm voice reason and lay the groundwork so that we can be better prepared for when things go south i think now that things
are going south a concern i have is when you let it go too far south and there's victims then you have knee-jerk reactions and the entire thesis of what we started here was to be left a boom and be prepared for a more thoughtful planful response if you have advice of either the right all you know sibling organizations like propublica or documentary films or the use of fiction or alternative ways to enhance the yield of this mission whether it's right now or later i do think it's worth you know our own family's access to care and food water shelter safety that this group gets better at it i think everyone's trying really hard but we are
very humbled and open to increasing our yield in alternative and creative ways and would treasure any advice you'd give us any last words from either of you yeah i would just say um same like appreciate all of you and just want to be in touch those were good last words i did that thing that thing yeah okay okay well let's uh thank our journalist friends here um we should invest in them we need them they're a very important independent voice that can tell stories that uh can be in the public interest in public good and i'm very grateful for the two of you [Applause] all right we have it
[Music]
so
[Music]
[Music] [Music]
[Music]
[Music] do [Music] do [Music]
[Music]
[Music]
do
[Music]
[Music] do
do
[Music]
[Music]
[Music]
[Music]
[Music]
panel discussion today is on software bill of materials and we have with us fortunate to have a great panel that we're going to discuss and talk about where we stand today with it and where we plan to go with software bill of materials or s bombs we got to let her speak first
[Laughter] so uh katie would you like to introduce yourself yes sure hi uh i'm katie bratman i work at new york presbyterian hospital as part of the vulnerability management team and also as part of the daggerboard development team we'll be talking a little bit about daggerboard today it's an s bomb analysis tool adam hi i'm adam kojak i'm a developer at new york presbyterian hospital as part of the information security team alan alan friedman i'm from the government i'm here to help i'm from cisa and i'm the guy who doesn't shut up about us bomb and lastly my name's christopher gates i am a medical device developer and an expert in embedded product cyber security and i've worked with alan now
for seven years on various projects something like that yeah so the first question here is why do we need an s-bomb so if you go to the store and you buy a twinkie it comes with a list of ingredients uh to me it's just kind of baffling that the most important software in the world doesn't have the same level of transparency as we expect from a non-biodegradable snack uh the important analogy of sort of that list of ingredients that list of ingredients by itself won't magically save you but one would you buy from someone who couldn't tell you what they were encouraging you to eat or in this case have your life depend on so
the value of doing it even if you can never touch the data yourself you still want someone to know that they have it but the other thing is right you can't do all of the things that we might do with a list of ingredients uh right protect your family from allergies vulnerabilities uh follow a uh a certain religious based restriction right the analogy there might be hey i don't want things from certain types of development environments or countries on my network for certain reasons so it really enables a lot of great data to say or it's the data layer that allows us to sort of think more broadly about different kinds of risk um i'll let you guys jump in about why
it's important for your orgs yeah so we work in healthcare and where s bombs really come in handy for us what we're seeing is with medical devices so oftentimes what happens is like we get a medical device it's thrown on our network we can scan it with a vulnerability scanner but we don't get the full picture of what it's made out of that list of ingredients and so this is where an s-bomb can really help us we can take an s-bomb before we buy the device in our purchasing process and then analyze it and say do we really want to continue with purchasing this if there's a risk that was found how can we mitigate it before implementing it
and so on so i'll join with one more thing there are organizations in the world that only use like six types of software right if you are just a traditional bog standard enterprise then s-bomb is gonna be nice for you but there's going to be public security advisories for your microsoft products your cisco products right those have mature product security teams and data about vulnerabilities and risks in those products is well documented and easy to access what we're talking about in the cavalry track is a very different world right it's the world of things that are literally matters of life and death those tend to be much smaller organizations that produce them and yes some of them are starting to do
vulnerability disclosure and security advisories but often the risk is not going to be this particular widget uh in part because they haven't they don't have security advisories and also because someone may not have developed a targeted attack for a foothold on that product the risk is sort of automated scanning tools that are going after some of the components and so that's why particularly in safety critical world medical infrastructure ics having that understanding is going to be important and the last thing is right it doesn't always feed into hey i need to patch this some of you are familiar with just how hard it is to patch something that if i unplug this people lose power
people lose water people die but what we can do is start thinking about it as more maturely of oh there's a potential risk here so uh my long-term plan is segment my network my long-term plan is work with my threat intel to sort of see if there's anything that is closed so again it sets up the longer term risk especially in our cavalry mission space in a word it's all about transparency and it allows you to assign risk to the end user because somebody manufactured something here you manufactured this box or that particular projector or that wireless access point that's sitting over there we look at that and as engineers we go oh i know what that is and i can do a
physical attack on it tear it apart reverse engineer it all right but as a user what they don't know they're bringing this in and putting it into their environment and potentially affecting their environment their neighbors environment everybody's environment i mean we've seen a lot of denial of service here distributed denial of service attacks so it's actually more than just the 16 critical infrastructures the united states it's everybody we don't want another marybot type of episode again transferring that knowledge to the end user so they can say how much risk do i feel comfortable with right oh this has got a vulnerable version of open ssl in here i can live with that i don't really care and some
people will other people will take it offline and pull it out so it is about disclosing all of that information not keeping it a secret moving it to the people who really are going to be affected by it and that isn't the people who manufactured it so about four years ago uh i was working with alan on another ntia project that had to do with updating firmware and how you patch and do upgrades of firmware and for devices in the field and we rolled off that and alan said hey i'm thinking about working on another project here that's called a software bill of materials i said oh what's that he says it's kind of like a list of
ingredients uh-huh of what goes into your products i went hey you know what are we going to do a month of that i mean that's going to be a short project that was four years later i hadn't a clue what all the edge conditions were that we looked at and in that period of time allen put together multiple working groups that covered different aspects of software built materials but they all consisted of experts in the field and really humbling experts i mean i quickly realized just how ignorant i was of what this environment required and how to handle this and to this day i still find things that's like oh yeah i didn't think about it that way
that somebody else already thinks about great multidisciplinary group nobody nobody heard cats like alan all of these people are experts they're we're all egocentric a-holes all right and we all have in our opinions and this man would manage to keep us going all in the intended direction and everybody working together uh you know if if we spot that uh comet that's about to hit the earth and we need a team to mitigate that i want him leading it so it's been really interesting and very educational over this period of time and now recently allen moved from ntia to csa and where there's new working groups have started up so ellen i've talked too much already
what are your what's your feeling on the work groups both at ntia and csu first of all flattery gets you everywhere thank you um the and and this is actually what i love about what we built for the s bomb community right it's always very clear very much not my idea in fact since i joined government pretty much every idea i've stolen from the cavalry and said okay they start off with a great idea let's actually try to use uh the flag behind us to pull together communities to make it happen so i want to uh doff my hat to uh josh and bo and everyone who's been part of the cavalry uh i think it's it's this is what it's
supposed to do it's supposed to sort of take ideas mature them and then hand them to folks who can implement them so uh the first phase we have to do is actually come up with a shared definition the idea of an s-bomb as chris said is is not that complicated but we need a couple things one we need a shared vision of what it looks like uh and no one had actually sort of articulated this is what an s-bomb is these are the boundary conditions and how to make it happen so the first two years was focusing on the what the why and the how uh and there are the good news is there was at least one data
format that could transfer and to convey s-bomb in a standardized fashion the bad news is there was more than one uh and that's some fun that we still have is sort of managing the data formats uh we're gonna have a meet up in uh right after this if anyone wants to talk more about that so we're now at a stage or last year we would have stage where we have the basics there's no reason why an organization cannot start producing s-bombs asking for s-bombs thinking about how to use them our focus now is on operationalization scale and automation right if we can't do this if we can't integrate this data into all the other things that we care
about in security then we're not going to make the progress we need uh right my vision three four years from now is that we're actually not talking about s-bomb as a unique thing it should just be a standard part of our security ecosystem the same way that right most of us don't spend our days thinking about cves there is a small community that does but it's really we just integrate it into how we think about our security data so the plug for the work that's going forward uh if you'd like to get involved and i'll remind you at the end uh we've got a couple of uh focus points one is uh cloud and sas
a lot of the focus on s-bomb thus far has been on on-prem software which makes sense right i have to defend it it's on my network uh and certainly in our safety critical domain around the cavalry uh that's huge but i would and you you guys can tell me if i'm wrong i think most medical devices now have a cloud management component right frequently yeah like that uh and the same is true in ics the same is true in in a lot of other in automotive so we need to be able to tell a story of what does transparency mean for a sas application that may have a daily or hourly build uh and could be very large so we want to
tell stories of use cases and then ultimately implement it right so does the s-bomb of a container is it just the application logic is the application logic in the os is the application the os and the orchestration we want to sort that out we want to do that with both producers and people who use that data defenders the second piece is going to be focusing on tooling how do we make sure that if two different suppliers or two different open source projects produce an s-bomb they're reasonably similar uh and then we're gonna talk a lot about consumption and the great tool that nyp has developed in a little bit we're also gonna be focusing on um
uh moving this data around so how do we move all this metadata around especially a complex supply chain so it's gotta go from you know an rtos vendor to a medical device manufacturer to a reseller to the hospital and then the last piece uh is going to be continuing some work that started at ntia which we called the awareness and adoption group uh which was very ably run by audient josh uh and that is to sort of help think about one how do we make it easier and cheaper for organizations to uh engage in s-bomb and also how do we coordinate all of the different things that are happening around the world so right now there's
some work that's happening at the international medical device regulatory forum one of chris's favorite organizations uh there is there's great work happening in all sorts of different corners of the ecosystem we want to make sure that there's a common hub to share information absolutely and when we started off i mean really the only person i'd ever heard talk about software bill of materials before allen was josh and josh started this and he was sort of out there alone and we all went okay i don't know what that means and kind of ignored all that but in these working groups he really set the pace with crawl walk run and how we implement that and we have
been crawling for the last four years literally we spent a number of weeks deciding how to spell s-bomb i'm not kidding okay is it a lowercase overall you know uppercase out all right there were things like that and we started at that level and now we have all these tools and techniques of how to apply this more importantly we've got an executive order that came in that now makes it not just little old medical that's involved because they were the first ones into it but the 16 industries of that are critical to the infrastructure united states so now we have vendors and organizations that are looking at this saying oh yeah i need to supply you with
an s bomb we're moving from crawling we're into walking what is run look like and how do we get there that's what we're doing we're moving forward this is both an open source problem and a proprietary you've got tools from both sides and that's where we are right now we're right now and walking and trying to head toward that running so um i keep talking about medical because that's what i do is make medical equipment but there's a lot of other industries that are right behind us in some cases they might be ahead of us i mean automotive comes to mind they're very much into this and certainly it's not just the united states it's across
the entire world countries like japan and stuff are very interested in this and what are you guys seeing are you getting other interesting you know comments from other industries oh i'm going to put it to all three of you uh or even from other hospitals besides the new york press and what does this look like is they're interested in adopting this and getting ahead of this ball uh i do think something that's interesting in the hospital field is we've seen some hospitals start by like when they negotiate new contracts with different vendors saying okay well when you provide us with a medical device we need you to provide us an s-bomb as well otherwise we won't sign
this so that's one way that people are really pushing s-bomb in healthcare money works money always works don't count on somebody's better nature if it's in their financial interest you'll have an s bomb the next day and then i think something interesting that's come from daggerboard is so we publish daggerboard it's open source and it's on github and we're seeing people that are wanting to contribute they're kind of interested and recently we were asked okay so how can the results from daggerboard integrate with an asset management system and that's something that we'll talk more on later but it's kind of an interesting idea that's really pushing s-bomb further so the tools that have come out of this
like daggerboard and we're seeing them break up to it as we first got into this in ntia early on we realized we had to do authoring tools because it's a chicken and egg problem and we've got to start somewhere all right nobody was asking for this but then nobody could create one even if they were asking for it nobody could consume one nobody could distribute one so we're starting to see these tools now come out for the the two major formats spdx and cyclone dx that are used for uh software build materials and the tools are there and we're starting to see more of these pieces of the chain over its life cycle fall into
play and and tools like daggerboard are definitely it and uh i think that's good that we're seeing both commercial ones i hope they hang in there long enough to see success as well as open source ones like daggerboard and uh at this point we've been talking about daggerboard and why all this but uh adam and katie a history of daggerboard where did this come from i i when i think of open source my first thought is not a hospital uh so how did this how did this come about as the fact that you guys created one of the better tools out there yeah i think that we feel really lucky to be on a hospital information security team and
have the opportunity to make this tool actually it was a lot of fun but basically where we started was so arcizo he's a member of the healthcare working group proof of concept for s-bomb and based on some of the meetings that they had he came out with this idea that was hey we have the resources i think we want to make this tool that makes it easy to take an s-bom and analyze it find all the vulnerabilities associated with it and report it so that the user really doesn't have to do that much because there wasn't anything out there that was like super freely available at the time so that's where the whole idea came from
was let's make this easy to use and free and here we are now we actually managed to make this app open source it and it's free something that was kind of interesting along the way was it's just like any other sort of open source project where we started really small we had the ability to do one thing like a script that analyzed s-bombs and now we have this project that's just grown a ton it can do a bunch of things um we'll see in a minute in a little bit like we're gonna add more features have a version too uh and because it's open source we are always looking for people that want to contribute and work with us
yeah and it started out as a beta project originally by the vulnerability team and we linked up as the devops team just to make it revamp the ui make it easier for the analysts to use as a whole see all your vulnerabilities in one place all the packages in one place as well as making an open source which was really one of the main goals for getting this app uh out there some of you folks i may have undersold what an s bomb is you're thinking hey i've used three pieces of libraries and frameworks and operating systems in my product in my device that i've created why do i need tools like daggerboard and the answer is transitive
dependencies um are you all aware of log4j and what happened last december okay yeah i'm getting a lot of head now it's good okay log4j is a very commonly used java library for implementing logging so you're sitting there at your day job and one of the requirements says oh my application has to have logging first thought that comes to your mind is log4j it's used by everybody in fact some 80 000 projects have used log4j so you reach out you include it when you include log4j you go hey that would be one of my elements in my software bill of materials and indeed it would but there are 294 sub dependencies in log4j that you as a
developer have no clue you just brought into your product and with intentional corruption of these public libraries skyrocketing last year it was 650 percent the year before that was 400 and some percent i don't know what it's going to be this year we just heard today about pi pi there were 10 libraries commonly used libraries in pi pi that have been intentionally corrupted to harvest credentials people are poisoning these repositories so if you have 294 of those subdependencies you can't do that manually that's why you need tools like daggerboard and other tools to go over there and look at this and tell you what it is on a daily basis to come up and say oh this one's buried 17 levels deep
and we found it here and it's we now have a known vulnerability that's in this product so it is the only way it's where everything in cyber security is headed you've got to automate it all of this is becoming way too complex way too big for any person or group of people to do with the manual process so tools like daggerboard excellent and with that let me stand by the way and why don't you do a demo of daggerboard
i'll sit up here with ellen thanks for keeping me company yeah by the way uh we're originally underdressed i was like where's your jacket i don't understand we had a whole plan didn't happen all right so let's go over you know what's important when we built this application so s-bombs we need them all in one place but we need to start at the design level so and i don't want to bore you guys with the technical stuff but i'm going to go over pretty much how simple it was just to start the project and what features it had so first and foremost like we use the django framework just to start the application to build the as a web application django
is very friendly with it's very data driven very friendly with using data and you know using sbom data it worked in our favor so and mainly the main tool is to scan the parts the s-bombs and being able to scan based on spd-x or cyclone dx another thing was to be able to integrate it within any application and that's where we had ldap and local authentication so you could implement in your your own and we have fancy charts and graphs all over the place so any analyst could look at all the all the graphs that are needed there and most importantly the open source that we made it for open source purposes so our app doesn't generate s-bombs and
we want to be clear with that um s-bombs should be provided by the manufacturer uh but if an s-bomb needs to be created from a manufacturer so if you're from a manufacturer um it's very easy to build and that we for for us whenever we need to build one we use sift which is a great tool and it's really easy it's just a one-liner um just you could connect it to your docker image or your requirements and produce all the s bomb in any format in cyclone dx or spdx any format that's required so for daggerboard what are the important tags in this case for this this is an example of an spdx document and
the document name we use the document name the creator the organization the package name and the package version those are the four main tags that we use for all the analysis that we perform uh within the application yeah so in a minute we'll show like an actual demo of daggerboard all the data you see in daggerboard it's just coming from these tags mostly in an s-bom file yeah and this is just a quick example of how easy it is to upload an s-bom within our tool pretty much you go to the upload choose the spdx or cyclone dx document submit it and you are prompted and the analysis begins it's all automated that way you
don't have to go through an xml document and pick it out yourself so we we do all that for you yeah and again so this is like not really a live demo um we don't have internet right now but that's why we're showing a pre-recorded s-bomb upload yeah we're really worried about the demo gods we don't know if our internet will work or not so we pre-recorded that and have a local instance for you guys so what's down the line what's on the roadmap um do you want to go over so before we get into the full-on demo um these are just our next steps for daggerboard so the the next thing that we want to integrate is vex support this
will give us the ability to reduce false positives we have in our data maybe for a given product it'll we can more reliably say these different s-bombs contribute um so we should mention what vex is first off vex was a term that came out of the working groups was a term that came out of the working groups that absolutely everybody in the working group hated and still hate we hate this term it's a horrible term but what it stands for is vulnerability exploitability exchange and the way to think about it is it's extra information that is encoded in an s bomb and it's from the developer of the product to say a couple of things i'm investigating
this vulnerability i've decided this product is not affected by this vulnerability or it is affected by this vulnerability and there's some couple sub states under there that lead with things like how it's configured so if you can figure it one way oh you plugged in ethernet into the rj45 jack yes you're vulnerable uh there's things like that that this stone in there very useful because there's estimates from 70 to 90 percent of unknown vulnerabilities can't be exploited as instanced in a product so vexes allow you to give you more clarity into it but it also gets back to a level of trust so for me if i there's a company i think is good uh phillips phillips medical
okay and i know they do good stuff i'm gonna trust their vets but if it's acme medical i almost said it really almost set a real company there so and i don't have that trust in them i'm not going to care what the back says i'm going to unplug that device until i know what it really is so this starts to bring trust into the whole process and that's nothing we can encode in xml or json that's only something that can be had over years of experience with the developers who are doing this so vexes are extremely useful it is a very much in a crawl form as we get into walk and run in the years ahead
it'll be extremely productive and makes s-bombs even more valuable sorry for jumping in kid no no that was really good uh so the next feature that we want to add to daggerboard is an api uh right now you can upload an s-bom and get results vulnerability results but what we need is the ability to have another system pull this data and say hey what's available and this can get us to our goal of integrating with asset management systems maybe ci cd pipelines those are just a few examples we had in mind so our third uh pro or item on our roadmap map is to add the ability to search for cves like chris had said earlier we all know
about the log for j vulnerability from last winter um right now within daggerboard you can find an s-bomb like based on the product's name and like based on what the s-bomb is but we can't globally search through all of our s-bombs that have been uploaded by for cves for packages so that's something that we think would be really helpful for the vulnerability analysis perspective and then finally uh the biggest thing a part of daggerboard was to get it open source get it put on github it currently is on github we have our link up here we encourage you to take a look at it give it a try and if you want to contribute or discuss anything we're
always open and so this brings us to our demo
all right so let's kick off the well let's get the beautiful login page that we we built here um so we need the icon of the so you're probably wondering what is the daggerboard and it's basically used to keep the ship afloat so and to keep your company afloat you know you need to scan and analyze s-bombs you don't want any holes in your ship so that's what the representation of a dagger board is um so this is our login page let's log in and from we need to know from the perspective of an analyst what they want to look at so the two major uh things here are what are the most recent s bombs that were
uploaded in this case you know and all the data here is not real this is all demo data from very old so there's no scrutiny on apache or anything else that you see here just as a disclaimer for you guys um so yeah we have the last two um s-ones that were uploaded and the overall uh grade for that s-bomb we calculate the grade um based on the vulnerability risk that's within um that we we pull from mvd national vulnerability database and along with their severities high low medium high in the middle section we have our highlights you can see how many total vulnerabilities were found within the application how many weekly the average vendor grade
amongst all the all the s-bombs that were uploaded and at the bottom we have a little um we have a of all the we have a table of all the s-bombs that were uploaded moving on and we actually displayed this page during the the recorded demo this is where you would upload an s-bom this takes us to our s-bom analysis page so what we can do is we can search for the s-bombs we've uploaded to go ahead and take a look and again like adam said we intentionally grabbed really old docker containers to show pretty charts and show a lot of vulnerabilities elasticsearch probably is not an f at this point this is very old um
but so once we've uploaded an s-bomb let's look at our results we can see here's the product uh we have a grade here we have the total vulnerabilities that were found an average cvss score and then we've got a beautiful spider chart based on the cvss vector and then a total of the vulnerability severities so scrolling to the bottom of the page we can see the packages that were found from the sbom file and so we can see we were able to find a cpe here and that's how we pull vulnerability data at this point now this is the part that i really like because i'm on the vulnerability management team i can go through here
and find more information about what vulnerabilities are present i could maybe search for something that's high i can maybe search for for a package and then most importantly one way that you may go through prioritizing your results is we want to know if an exploit's available so we can do that and we get an actual reference to exploit db we can sort by severity and then finally if we want to we can export a report of all the vulnerabilities that were found and view it beautiful csv and then moving on so we have a vendor analysis page and this is just a way to group the s bombs together based on the creator tag so we have apache software
foundation here same sort of sorting if we wanted to view something else but we can see here we have five total s bombs from apache they've got low grades um total 800 air 80 vulnerabilities we can see a table here that shows when we uploaded these different s-bombs and then just the general distribution of vulnerabilities we get a snapshot of whatever was most recently uploaded so groovy was our last s-bomb and then uh just an overview these are the different s-bombs that we had from apache so now we'll move to the admin page yeah so if you want to deploy this in within your own environment or for your own organization uh you need to make it
easier for your admins to configure do certain configurations so we gave it the ability to for our back controls for your users you don't want some people you only want some readers to read the data versus people who could upload the data we also have the ldap configuration so you could sign in easier for your org you can manage your users we have a database um we can basically create you can modify the database if you have to and as well as the grading you can modify the grading policy so it's not just fixed so based the criteria for calculating grades is very subjective um you might want to consider like if there is 10 if there's
10 high vulnerabilities what does that mean to your organization and you could definitely configure that within the environment um and and that's about it um that's the admin portal and this is just one example of what we've been moving forward with on the s bomb initiative and we've it's come a long way as like a healthcare uh hospital and we're very like proud of ourselves and you know we're not really a tech giant we're a hospital so we've come a long way with buildings up and we're very happy with it yeah and i guess it just shows as part of the cavalry track um that organizations like hospitals if you have the resources or you're willing to do
this you totally can do it yeah so yeah please visit our github and please contribute anyone and everyone's welcome to contribute to healthcare as a whole thank you [Applause] i i want to flag one really cool thing about this build there are many really cool things about this project but one of them is there are a lot of security tools that just do one security mission but don't think about how it fits into an organization and i think the thing that struck me from this demo is oh yeah you we need admin tools we need to actually understand who's going to do what this organization figure how it fits into the broader mission and so i think that's a
really cool feature uh that as we start thinking about more open source tools that we can build instead of saying don't don't just do the the small thing figure out how you can integrate into the bigger picture yeah yeah uh do you want to go ahead oh so i think that's an uh thank you uh an interesting point too is that so when we started on this the daggerboard idea came from the healthcare espan proof of concept uh and initially in our first phase of daggerboard we decided okay this is only going to be for medical devices let's keep it specific to healthcare but then as we got to our second version of the app we realized okay no this applies to
every industry that works with s-bombs other people are working on it too we want daggerboards to be used by anyone not just hospitals yeah and just the admin configurations alone i mean that was just that took a while to build and you know to understand how to map it and there's a lot of configurations that we did but to come at the end of this like it's basically this is just a starting point for this application um we're really looking to build this uh further and we definitely have a road map ahead for all of it one of the things that i took away from that presentation that's really important is it gives you the quality
idea of your vendors the people you're buying from so maybe you've got uh you know that pdf viewer you put in your program from a large pdf vendor who will remain unnamed but you all know the cut name starts with an a uh and you go you know these folks never give us patches they have nothing but vulnerabilities and all this the next time i'm going to create the next product maybe i'm going to go over and try some other competitors product and put that in there and so you're looking at this and you're doing continual improvement upon your product based upon not just availability and cost and form fit and function but also cyber security how good of a program did
they have are they working with me are they a partner or are they part of the problem so i think a tool like that is really useful to know remember we're all in this there isn't anybody who's outside of this we're somewhere in the supply chain that either we're sourcing it originally or we're building on top of it but i do have one question for katie and adam do you have an s bomb for daggerboard yes we absolutely do of course yeah it's published on our repo forever it's on github yep and that's where it should be and that brings up an interesting topic which is some of the people you're now seeing complain about the fact that oh
well if we give you our s-bomb we have to hide it behind a firewall credentials and the fact that you can't just get access to it the proper place for s-bomb is publicly available in this case in the repo all right if you sit there and think that your intellectual property is going to be exposed by the third party software components that you use then it's not your intellectual property it belongs to the people who created the software components okay so so go ahead i i want to show you the first i i agree that is the the awful place to do it um many of you know sunil yu an early leader in thinking about s-bomb
and transparency uh he moved over to a company called jupiter one one of the first things he did was saying we want a public s bomb and so jupiter one.com s-bomb is their live s-bond right it's a cloud-based product which means there are very frequent builds uh but so you get the live build but i also want to zoom out a little bit and talk about some of the cavalry mission which is it is our job to hold people's hands and walk them down the path for better security we can't just say right do security now and we'll beat you right nerd harder who says that doesn't work we haven't tried that approach but i think one of the there are
companies that just have very natural aversions to sharing anything right there's no okay part of it is there's massive amounts of tech debt so they don't want to show that they have been shipping old stuff uh and part of it is just culture right so my lawyer says i shouldn't do this and i don't know or care enough i haven't talked to my customers enough customers enough to push back so one of the things that we've done is sort of try to push back on why it has to be secret uh there's a great myth busting document on the nti website that sort of pushes back against that but the other thing we can do
is to say you don't want to make it public it's okay why don't you try to figure out what access control at scale looks like for your thousands of customers and and yeah just you just need to make sure that all your customers have access to it uh in a timely fashion and it can integrate into all of your customers security tools and by the time you sort of they start to think about how complex it's going to be to engineer a way to share it safely they'll realize this is just a lot easier to share directly uh so there are ways that we can use some jiu-jitsu to get people uh to be more comfortable sharing
at the end of the day you're moving around xml and json files this isn't exactly rocket science there's lots of ways to do this there i mean obviously you could email them doesn't scale okay nobody wants to see that you could put it on your website probably doesn't scale either it's a good first step you could actually set up your own servers and have those servers they're nothing but repos for s-bombs that's now getting into the place where you want to be there's commercial activities like uh archivist is one of them which one of my open seats whatever c3 whatever some of them that do distribution there's even a an awasp project to establish a open api
so all of these repos will have the same api restful api so you can just reach out and automatically with programs like daggerboard once you indicate i have i want to fetch this one it constantly fetches and updates it brings it in and uh so it's it is definitely moving into that walk run area where this can start to be really really useful so we talked about where we've been we talked about where we are what's next where are we going with s-bombs what do you guys want to see what do you think we're going to do alan what is sisa going to do with s-bombs now that we're here uh well what so as i mentioned the current
status is where we're all able to sort of start moving this direction the longer term vision is really integration and automation and that sort of means hey we've got to have interoperability those are fun challenges uh they're engineering challenges they're also business challenges uh policy challenges uh and and also massive opportunities so you know i i love that dagger board and hopefully you'll talk a little more about how you're going to integrate this into asset management because that's where we need to be going um i've talked about briefly these four work streams that cease is going to be mentioning uh the i'm going to start repeating an email address if you'd like to be involved and you're not on my
mailing lists uh sbom sisa.dhs.gov what was that allen s-bomb at cisa.dhs.gov just send me a note we'll sign you up for these work streams and then of course the other fun thing is let's start building things and that's kind of uh where we're winding down here at this point but i do want to point out that what we need is more open sourcing we need more community involvement uh this is a call to arms all of you need to get involved producing creating tools making s-bombs making them available this is a big project it's certainly not what i thought the first day ellen started this okay it is a giant tidal wave that we all need to get behind and
help what do we do for the cyber poor who are out there how do we get them in a good place okay they need help there is a hospital that's in ohio well as a medical device manufacturer we always refer to hdos health delivery organizations like it's this monolithic thing they're all the same like they're all mayo clinic or new york presbyterian or an organization that's large and doing things correctly they're not they're a spectrum so they go from that far extreme to this hospital in ohio where the same guy who mows their lawn sets up their network i'm not kidding this is the case and everything in between how do we help those people all of them they all matter
especially those little guys they don't have the resources the knowledge the funding to do this it's all incumbent upon us to put these tools out there get the information in front of them lower the bar so they can adopt this stuff think about this it's your responsibility these gentlemen started the ball rolling the rest of us are just rolling along with them and i'm including all of you in that as well too last words alan you were last i'm going to start with you first s-bomb sissa.dhs.gov [Laughter] uh i don't have anything to promote besides check out our github but um just to touch on what chris said yeah i actually used to work for a hospital in
kansas and we would buy a lot of those small hospitals from rural kansas that were tiny and the first things that would happen after you bought these hospitals was they got hit with ransomware man what do we do now because no one can do anything and so um they didn't really have much i.t there and this is where s-bomb really helps us let's let's get s-bomb out there let's find out what's in these environments let's help these smaller hospitals figure this out adam yeah uh i i guess just contribute um contribute uh right now you know we have our day jobs and now that we've deployed something on on github as open source we have outsiders telling us to hey can you
fix this can you fix that but we have our day jobs we also need to you know work for the you know new york presbyterian um so please help us out this is matters this is important we don't have any other way to do this when you look at those black boxes you don't know what's in them nobody does even the experts like myself i'd have to tear it apart and spend weeks on that one version to find out what's really in there a lot of work a lot of effort we need to lower the bar on this we have a huge installed base of really crappy products out there right in the medical device industry we
specialize in that we've got hospitals that have 30 year old components in it all right 30 year old components how secure do you think those really are they're not we're looking at all of you think about this this matters this could save lives this will save lives as we go forward so whatever industry you're in make certain you're looking at this ask your vendors for s-bombs whatever you're writing even if it's a diary app that you're going to release you know out to the app store ask your vendors that you're using where's your s-bomb ask this keep that ball rolling keep it going i want to thank all of you for the gift of your
time here today to listen to us and we hope to see you in our meetings
so obviously this is super valuable especially for the target rich cyber poor we discussed yesterday they may not be able to buy new things or hire new people but they can at least know what they're getting in these obsolete or older technologies and know am i affected where am i affected when the next logarithmic j happen the bad news on some of these hospitals is now seven eight months later some of them still don't have a commitment on what's in the software when are they getting a patch so we're on a journey we're crawl walk running but this wasn't merely about s bomb and the great work here i just want to look at these two developers
and the attitudes we keep hearing from thought leaders in cyber security that s bombs too hard no one can do it no one can make use of it they say screw it let's just build something and part of this was a meta hint that i wonder if part of the next phase of the cavalry is we start teaming up with software developers open source contributors and that this is just one of maybe dozens of projects that could help waste water treatment could help oil and gas could help the target which cyber for in the food supply and i kind of like their attitudes i kind of like the contribution and i kind of like the idea that instead of just
just pointing out problems we might be able to create the tool chains that the private sector is not yet willing to do so this is a hint and a nudge not just to take s-bombs and maybe put the known exploited vulnerabilities list in it so that they can target more specifically but also what tools could you ask for build or contribute to and advocate for so this is a hint maybe for the future of the cavalry we're going to be outside if you want to talk with us after this presentation if you've got any other questions right now if you don't want a voice in public right now it's a q a period questions
we don't have any portable mics i haven't i haven't been on the discussion list so have you been thinking about it and i'm already well familiar with s problems josh may not recognize me but he'll know me in a minute um have you thought about when you're talking about cloud transparency other aspects that aren't just software management tying it into information security policies at the organization that's building the software as a cso type role i get asked by people about the practices in our organization and everybody sends me a one-off questionnaire with 150 questions there's no standard for and i'd love to have a standard form that's s bomb-like to give them to say here's the change management control
practices we follow five terms for hospitals to use in contracting with medical device suppliers and it talks all about the different cyber security expectations the most dysfunctional relationship ever is hospitals to medical device manufacturers always has been i've been doing this 50 years it's been horrible i'm trying to make it better i know a lot of other people are too it's kind of our goal we're starting with cyber security but this goes into a lot of different places so how do you do this so when you get those questionnaires mds squared of course obviously you're going to be filling that out obviously but the custom ones we're trying to minimize if we can get them to align to things
like this mc squared that will be a much quicker way to answer much less burden on the manufacturer in a much better position we'll all be in so on the idea of enhancing small i haven't seen a lot of discussion around specific security features although measurement of security features is one of my key passions which is how do i sort of quickly and cheaply determine security features where we security process where i actually got the source from uh and having each of those artifacts signed along the way the challenge is how do i meet how do we put enough semantic value into that data so that we can enforce it at policy level otherwise it just turns into right
a paperwork exercise what we want to do and there are organizations including solarwinds today which is sort of saying here's the policy and we're not going to let code go to uh go live until it's checked all these boxes and one thing that the two major formats that we've influenced in ntia are cyclone dx and spdx they are both in motion and generally if one does something good the other one does something good and copies it i'd argue that you go out i'm not going to tell you which one's better i have my opinions let you find your own but cyclone dx does have some of the stuff you're talking about in separate artifacts hopefully spdx will answer that and
everybody will keep motion and we'll have ways to uh rosetta stone all that information together because that's where we need to be i think we've just got two minutes i'll be quick i have a really basic kind of s-bomb question i guess within policy is there anything in the executive order or anywhere else that kind of uh mandates uh the existence of s bombs for federally funded software or products or that sort of thing that exists right now where that's going to be coming down the line legacy thank you yeah or well we're going forward either way so the executive order 14028 from 2021 uh mandates that everything the us government buys will ultimately have to
have an s bomb that's going the mechanism is first it's going to be a memo from omb that's going to come out soon and ultimately is going to be part of the federal acquisition rules uh there's also some language in the draft ndaa uh the national defense authorization act uh that is going to apply to a couple different parts including a dod land happy to chat more about this and one last thing in medical device right now in senate is it's already out of house is the patch act i see someone who knows this okay good uh they added a writer on there that says not only is all the cyber security work needs to be done for release of the
product and that includes s-bomb but also all the existing legacy product out in the field one more question do we do one more question one more question since we're taking up lunch great thanks um is it a reasonable step for medical device manufacturers to just put that in the mds2s like the s-bom details kind of as a as a baby step to having more of a live stream there are questions in the mbs-2 about if you have an s-bomb and all that it's not a great place to do it it's not in a format that's easily machine readable so let's say i've got javascript is the basis of my device in some way from i may have
two three four ten thousand references doesn't really work in that you want to stick to one of the markup languages json or xml that can be easily sourced consumed we're out of the days of emailing and texting each other to each other if i have to open up excel to pull something out of it ain't going to happen right not going to be scalable so flag that you've got one even point to the url where it can be fetched but now thank you everybody um you wanna come up we're gonna be on the balcony the balcony overlooking the pool overlooking the uh fire pits uh to have a little meet up and i've got stickers
who doesn't love stickers
[Music]
so
so
[Music]
[Music] me [Music]
[Music]
[Music]
[Music]
[Music] do [Music]
[Music] do [Music]
so
[Music] do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
you
[Music] so
[Music]
[Music]
[Music]
so [Music]
so [Music]
[Music] do
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so
[Music] do [Music] [Music] do
[Music]
[Music]
[Music]
do [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] so
[Music]
[Music]
[Music]
[Music]
[Music]
uh
[Music]
[Music]
[Music]
[Music]
so
[Music] so [Music] [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do
do
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
so
[Music]
so
[Music] do [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do [Music]
[Music]
[Music]
[Music]
do [Music]
[Music]
[Music]
[Music]
do
so
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
uh [Music]
[Music]
[Music]
[Music]
do [Music]
[Music] so [Music]
[Music]
[Music]
[Music]
[Music] do [Music]
so [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so [Music]
[Music]
so [Music]
[Music]
[Music]
do [Music]
[Music]
[Music]
[Music]
[Music]
foreign [Music] do [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music] so [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] so [Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music] do
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so
[Music]
[Music] [Music] do
[Music]
do [Music]
[Music]
[Music]
[Music]
[Music]
do
[Music]
foreign [Music]
[Music]
do
[Music]
do
do
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
so
[Music]
ah
[Music]
[Music]
[Music]
[Music]
[Music] so
[Music] [Music]
[Music]
[Music]
[Music]
[Music]
[Music] so
[Music]
[Music]
[Music]
do
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
do [Music]
do
my
[Music]
this [Music] [Music]
[Music]
[Music]
[Music]
[Music] do [Music]
hello b-sides good afternoon and welcome back this afternoon we're going to have a session ics security assessments 101 and how to fox i test this so a few announcements um we'd like to thank our sponsors our diamond sponsors lastpass and google or sorry palo alto our gold sponsors google intel and blue cat um with their support supportive donors all of our sponsors and volunteers that help make this event happen so thank you to everyone and we're glad to see you back this year so glad for you to be here and for those that are joining online if you have cell phones if you're in the room if you could please silence your cell phones no photography without
permission of the speaker questions will be taken at the end there will be a microphone here so if you do have a question please feel free to come forward with those at the end of the session so with that this is i am the calvary and i'm going to pass it over to yael thank you thank you very much you can hear me well there yeah okay so um welcome thanks for coming i'm jael i'm a security consultant working at bishop fox and today i'll be presenting an introduction to ic security assessments so let's begin with who am i not i am not an ics security expert but before you leave the talk before i begin
let me explain why this talk over the past eight nine years that i have been working on cyber security i have done several penetration testings and other security assessments where clients sometimes come to us as and ask for for us to perform or include ics components into these security assessments so usually our approach is to avoid this because you know it's critical infrastructure and can be broken can be something that failed and that's not good right however i have done some research on this and the time of this talk is not to give a methodology per se because there are a lot of material out there that you can check it out regarding methodology and how to approach this kind of
infrastructure the idea of this presentation is to provide from my consultive experience some examples that fit in each of these steps of the methodology so so you can figure out a yeah what can you do when you face this kind of infrastructure so um i'm not presenting new cool hacks for plc's or something like that but i did include some technical demos just to keep it spicy so first of all what is ics uh according to the nist special publication 837 ics refers to multiple components that can be found in industrial sectors and critical infrastructure this includes sensors actuators software hardware and so on basically it refers to everything that controls physical processes and usually the right there are physical
actuators in there to process data give an output and make a physical change into an industrial component some examples of those probably you have heard of many of these ones we have a supervisory control in that acquisition or scada which usually are software installed on a computer and it allows users to track information that is coming from different kinds of equipment this software also allowed to perform operations or changing the programming of these components we also have distributed control systems or dc s which are a gathering of control components to be managed in a distributed manner instead of individually we have programmable logic controllers or plcs which are most of the commons ics components that you will
hear and this is basically hardware with programmable memory that allows to process an input and doing some internal processing and generate an output with this rtus basically are hardware connectors between scada and dca systems in but we have we may have more control components which are basically distributed in a layer structure from the physical layers until the corporate network that we will be analyzing in a moment but are these systems secure as you may know system is secure at least not all the time ics is not the exception actually they tend to be even more insecure due to different factors and problematics that we can found that um differ a little bit from what we
ordinarily see in it some of this is the inner design these control components generally are not meant to be secured they don't have security built on these devices why well because vendors have other programmatic at the time they focus on operation and make the operation not fall to continue and these systems were never meant to be connected or interconnected in well in the internet that we are living today these systems are really also basically that's why the security is not built in so because of that we have for instance no controls for any authorization or confidentiality or integrity and disallow to perform different attacks that you may see on the internet there are a lot of talks in
youtube videos regarding how you can inject into these protocols how you can intercept and or modify the operation of these devices however we also have implementation flows besides the inherit lack of security design in these systems when we implement those components into our audi network we have also some implementation flaws for instance it and ot may be incorrectly or insufficiently segregated or isolated one of each other maybe more hosts that we intended for are have access to these ote devices we may have also hardening issues such as default credentials lack of credentials you name it and what is more important is that we may think why vendors doesn't implement how are they security into these devices reality is that industries
usually have little ability to include security controls into ics components so because of all that generally when we got access to ot devices that already means compromise but so how can we attack or how can we assess this kind of yeah of components um i have included here three main categories you can find more in in the late territory of plc's and highly controlled environments methodology for the sake of this presentation i i took research which generally is made by an external entity or for vendors where the main goal is to test specific functionalities new releases new components to find nowadays new vulnerabilities all from the point of view of the research that doesn't mean that we as a
security professional can do this if we have a research area we may have contacts with different vendors that could facilitate us to have these components and to test them with this specific goal we also have cyber security assessments which can be done from a consulting perspective and the idea here is not to find new vulnerabilities or old days the idea here is to doing not an intrusive test not an active test but we want to check that the implemented or the deployed components have actually met a best practices that they are hardened that there are policies and procedures not only in paper but they are applied both to it and ot we have access control and many other
things and then we could can move to a more active testing that could be a pinterest or a red team also from the consulting point of view and well the goal is like every other pentest is to gain access to persist to people to find vulnerabilities that can be exploited or we can maybe emulate a unknown threat or to achieve custom goals that the client is requiring so how can we test these components the right way and i mean the right way because the reality is that these systems are prone to fail and we may broke something that's natural on on this kind of environments so the methodology made although logical approach we follow must take this into consideration
and we always should keep in mind that we um we must avoid causing unacceptable impact on operations so does that mean that we can actually cause any damage to operations and well we don't want but as i mentioned it it is a possibility and that can happen so we can we must plan on this so we don't actually disrupt critical components if we have the right conditions of when where and what to test we can avoid causing an acceptable our yeah damage to critical components and finally one thing we should care about as a consultant is that we need to prioritize efficient risk reduction i will dig a little bit more into this later in the reporting section but the
idea is that we want to address the issues we found in a way that actually provides value to our clients so let's dig a little bit into the methodology and dan weber from karaoke security established these steps as part of of their methodology i took this as an input and the idea is not to dig deeper into these steps but again the idea of the presentation is to provide some examples that fit on each of these steps so the first one is security effort prioritations what we want in this step is to understand the security requirements the organization has what policies exist on the plant or industry both for ite and no team is there any segregation isolation do we
have access control assets inventory do we have insulin response and recovery process and procedures we need to start with this because if we don't have these initial steps chances are that we will be able to dig further into more active testing because it won't provide any any real value to to a client if there is no an initial step on their security poster so oftentimes organizations may ask for a penetration testing on a red team in their industrial components but probably we need to work with the client and rescue this assessment so we can find a way we can provide actual value to to the things we are assessing the next step is process familiarization
here we can here we can do architecture and physical reviews inspect processes to through walkthroughs in the plant or in the processes interviews with the many stakeholders is one of the key points of this kind of assessments we need to talk with the people that are actually executing these processes and exec and understanding how they are working what processes are really in place not not only in papers and so if we have the previous step if we know already that there are policies and procedures we can test on them if there is segregation in isolation stated on paper we can test that we can make sure that this segregation is actually applying to ot and it
so how can we do that we can do this by leveraging some published frameworks most of the um yeah one of the most famous is the nes cyber security framework which states some domains categories and subcategories this is used in order to evaluate the security model of the organization that we are assessing in order to understand what what is the maturity of this security program it is not a one-size-fits-all approach however we need to customize the the assessment we are performing for the specific industry we are dealing with also it is worth mentioned that these cyber security assessments can be really complex can be a whole assessment by itself not just a step on the
methodology why because it requires a deeper understanding not only of security we need to understand architecture we need to understand technologies we need to understand the processes and policies that the organization is using or applying to their their systems imagine we we are telling people that has been working with this kind of devices for more than 30 years and we just go there and tell them you have no authentication for your protocol you need to implement security here here here you need to change the way you operate that's probably not going to happen if we don't understand that again as i mentioned before security sometimes is not on the side of the clients or they
have little ability to implement security like this in in their systems so we need to understand the technical and not so technical aspects of their jobs in order to provide real value what we are recommending to fix also people on ics security and ics sorry usually appreciate when security consultants or security experts understand the different approaches that it and ot have the different problems they could face so how does a cyber security assessment look looks like looks like a spreadsheet sometimes it can be a tedious work it is indeed and in this first image i'm showing a requesting uh well yeah it's kind of little but the idea is uh this is an information gallery request for a client
where we want to know what procedures what documentation what they have for different domains based on the needs cyber security framework we want to know if they have asset control policies if they have a security on or audit procedures if they have assets inventory so we are requesting for this information so then we can leverage the cyber security framework using the categories and subcategories to review the documentation and find possible gaps into the implementation of these components here we can leverage also the interviews we have we or the walkthroughs in the plant because it is not a checklist we should not take this as a checklist okay they they have policies for access control they have
asset inventory okay check it's not just that we have to make sure that the inventory is covering not only i.t they are covering the ot components that are actually deployed on the plant and make sure that they are complaining with this is not an auditory it's not a cyber security assessment so we need to find those gaps and see how can they improve their security model but we can not only rely on spreadsheets there are also tools out there that can help us to leverage these kind of assessments one of those is the cyber security evaluation tool it is by the cyber security and information security agency and the awesomeness of this tool is that it
allows to load different frameworks that are published for scada and other process control components we can leverage these frameworks loaded into a matrix as it was an interview and what is more important it allows to customize these interviews or these questions that we may ask to to the organization so we don't always need to perform a whole csa and again this can be really complex really time consuming and an assessment itself by a client that probably they want to buy it or they can afford it but we have to do process familiarization always to better understand the organization from a cyber security perspective why because this will allow us to do very important questions that will
help us to go deeper into more active testing for instance we will know what is the most important part of their plant what happens if one or several critical processes they have a breaks down or fails what happens if it fails for five minutes one hour a week some processes may fail because of their nature but others can cause even dead of people if they fail for even one minute so we need to understand that we need to know if they have a segregated networks if they have a access control for the control components or the critical infrastructure limited to specific holes or specific components we also need to know if they have laboratory environments this is really
important when we want to test critical processes that cannot be tested online so these important questions will help us to go deeper into the methodology so then after process familiarization we can move forward to a passive information gathering here we can include monitor network communications this can be done online or offline we can do test hardening or we can test best practices that the components are deployed are meeting specific best practices for these protocols components and so on and we can also include hardware hacking in iot depending on what is implemented on their critical infrastructure to do those this sorry we have well several tools both automated we can do manual testing we can do or leverage scripting that
many other people have published on github and there are also vendor solutions that implement this kind of threat monitoring by doing a constant review of the network communications winning io within ot sorry so let's talk about a little bit about protocols interfaces and instruments in the ics systems they use different protocols for real-time communications most of them are really old they were first designed for serial communications later on they implemented their tcp versions and for instance let's take modbus and x as an example which is one of the oldest protocols we can find of the on ics components these data is from the 70s and well it meant it was meant to communicate with kelsey's
initially through serial communications later on with tcpip and the idea is to modify the programming of these components we can found these different protocols also in a layer structure depending on what we are seeing if we are seeing actuators plcs scada systems or other systems uh can talk different protocols to communicate with different ics components and well this a little demo i think it can be seen more or less okay the idea here is to analyze a packet capture from modbus communications so we need to know a few things about the protocols we are analyzing in this case modbus lacks of authentication and lacks of encryption so we only need an ip address and a
valid a function code in order to create a modbus session so for instance in serial communications uh [Music] modbus uses broadcast to basically this means that with one simple function called we can cause a denial of service condition because it is broadcasted through the entire network so again this protocol was designed for programming and modifying plcs rtus and other control components and leveraging this we can inject basically a malicious code into into these components or modify the programming of these components in the give here what we are doing is analyzing this up this modbus operation we discriminate by iip and the package is sent between these hosts and if we go back a little bit into the
process familiarization and the information gathering if we have a valid asset inventory we can leverage this and identify if these communications are happening between valid or allowed devices or maybe we are seeing an unidentified device that is talking with our control components and actually modifying their programming that could be interesting to further analyze it could be a yeah a system that is not inventory or maybe it's a real attack we don't know but we can further investigate if we have the previous steps done so in the analysis we have two codes uh code five which is a writing function and an id which is a custom code that can be used for several things however there's an actual vulnerability
in one schneider electric component that uses these 90 codes to overwrite the programming of the of the component so again if we are not using these operations that can be something to analyze for or to investigate further now talking about active testing normally if you can access or scan the ot it's already game over this is a phrase once a friend told me this guy is actually an actual ics security expert and what he means is that even the most basic scanning most basic and non-intrusive scanning on a plc or hardly a controlled network can be pretty dangerous that can because denial of service all around the the network and the components so when we are doing active testing we need
to understand what consequences it can have we can stop processes we can cause some damage to physical equipment and we can also even kill people in some extreme cases depending on what we are testing we what we are assessing so to avoid these potential damages we need to leverage or we can leverage staging or lab environments we can use redundancy if there is redundancy for critical components we don't test what is in production or we we don't test the only point of failure that could break something really important for for the industry so also we can select best times uh of testing maybe there's a window where we can test these processes and nothing critical will happen
if we do the interviews before probably the the main stakeholders will let let us know when we can test better these these components it is important also it is not on the on the slide but we also need to know our tools we need to be very careful with the upside we do with our testing tools we need to know how to customize them to adequately to the environment where we are testing and not just drawing tools that we find on github or or so so within active testing we can do many things we start of course with basic enumeration ascending other security assessment but we don't scan for all ports we don't do an n-map minus minus we can cause a
denial of service sometimes there are some components that have open ports expecting to receive updates or a programming instructions and again we we don't have authentication at all in these protocols so by sending random data using nmap or whatever tool we can cause these devices to interpret that as an instruction it will change configuration and it will crash so we don't want to we don't want to do that there are also several tools and scripts out there in github and other both free and from vendors solutions that can or try to do this basic enumeration the ideal it depends again on where and what we are testing the idea is to follow a more manual approach or to know
really well our tools and how to personalize and to use them in certain environments the high-level methodology is to do a network sniffing follow with an rap scan do a map to specific ports we look for management interfaces we don't look for specific ics control components protocols yet and we do network analysis it can be both online and offline and then we can follow custom attacks and provided we are careful well we are what we are testing where and when other thing we can do with active testing is full feature scans some of the vendors of specific control components will provide a some audit files that could be loaded into specific tools for instance the bandolier project from
digital bonds they created some of these audit files to to be loaded on essos and to analyze a specific i think it was modbus protocol and nesos itself include included a function called ics scale as smart scanning but as rapid 7 states it could break something so even with that features we need to be careful and we need to understand how they are working for this specific case the idea is that when wherever nessus finds an ics component it stops scanning so they don't bring things but in order to identify that there is an ics control component there nasa's already touched it and this torch could cause a damage so we need to be careful with
that another thing we can do is secondary application testing in this case if we found management interfaces in these control components we can perform a basic web application testing we can see if those interfaces are protected by password for instance if they are to analyze if they have weak passwords or default passwords and for instance well we need to remember that the attacker will take this pass but not necessarily they will start attacking ics protocols because that could be um really dangerous even for an attacker and it could go just by testing the web applications that are published and are accessible for anyone from the corporate network for instance in this case we are seeing some uh two interfaces one from
honeywell intellige intellidoc system which is for gas detection and the other one is at a automation portal these devices have default and weak credentials and basically allow access as a domain user and they allow to modify the network configuration of these devices depending on how and where they are deployed this could be really dangerous if they are detecting critical components so and then we can move with ics exploitation again when we do exploitation in ics it's not the same as a normal penetration testing we want to follow an approach of a means to an end we don't want to exploit things that the client or the industry already know they are vulnerable too we don't want to to arrive the organization
and tell them okay i can inject in your modus protocol they already know that i mean there's a lot of information on that we need to be very careful why we are exploiting certain things maybe we want to assess the actual risk of a critical process the the client wants us to share to show the the actual risk to to exploit something in that critical process um and what again we can use different tuning meant for this kind of components and it will be always dangerous to use this tooling and without the proper upset on those so we also have the manual testing as mentioned before there are a lot of protocols components configurations network have some
different things there that we need to be careful of we need to understand the context we need to have a testing methodology and then we can start scanning or attacking as then whoever mentioned we start testing passively and we escalate safely so knowing the industry and understanding what are the context of what is the context and the needs of the clients we are assessing it's necessary for us to perform an efficient risk reduction recommendations and not only ask our clients to fix everything or to include security in everything and we need to remember that rich rich of these systems probably means already game over we don't want to just came to the organization and tell them what they already know we need
to find any different ways to provide actual value to to these clients and if you want more i have some references for you some talks some documents some do it yourself things and some tooling on github and that's it if you have any questions you can pick me up twitter and or ask here [Applause] okay so sometimes i role play and let's start by saying i am not going to be attacking or critiquing his talk one of the things i tell the cavalry folks is that sometimes the opposite of a profound truth is not a lie but another profound truth and together you have a greater truth and sometimes we have cognitive dissonance so to connect this to chris hopps
keynote yesterday about fragile versus resilient versus anti-fragile if even scanning these environments is dangerous and therefore we choose not to prove how dangerous that is because the client wouldn't want us to break things that's a truth and it's dangerous so do you believe an adversary from north korea russia or even an accident from a ransom crew would use kid gloves and just not actually sneeze on these things so one of the things we wrestled with at sissa and we made the sisa.gov bad practices was that the use of unsupported end-to-life software is dangerous in critical infrastructure now that is not to lack empathy for the fact that these are really old devices they are very mission critical if you
break that mind that mind might shut down forever and if it's that fragile are we resilient that's not a rhetorical question if it's that fragile are we resilient and back to his other truth that these things once you get physical access there's no authentication you can just inject all this all the modbus traffic you want we tend to believe in an air gap and they rarely were ever there but now you every single one of your employers has a digital transformation officer who's deliberately adding remote telemetry bi-directional predictive analytics predictive maintenance to be able to avail themselves of data science and machine learning and other good things but the assumptions that were safe because of an air gap are
gone so the world has changed and adversaries are now being more brazen and they are attacking water and food supply and while it is incredibly disruptive to maybe physically break some of this old equipment the only thing harder than planned change management window is unplanned change management window and during the pandemic on the vaccine supply chains we didn't have the luxury of hoping that no one would sneeze so i don't know what to do about it but what i know is when you look at a consultant being hired by an enterprise we're looking at the stakeholders wants and needs and fears and what's off limits but when you add in other players like actual adversaries or the people who
need the water the oil and gas the electricity the timely access to patient care there's a lot more people affected so we're in a very uncomfortable space and part of what i'm asking the cavalry to do is we should not just walk past these fragile brittle unscannable systems and say that's as good as we can get there's no money and congress is increasingly getting appetite so i don't have an answer for it but take everything he shared with us which is valuable true and compare it to now what do we do when anyone can sneeze on these things or just even port scanning them may be too violent an act do you feel safe that the critical
infrastructure is resilient don't know what to do about it but i want us to simmer in that discomfort thank you
thank you
[Music]
[Music]
[Music] [Music]
[Music] do [Music]
[Music]
[Music]
[Music]
[Music]
do [Music]
[Music]
[Music]
do
[Music]
good afternoon b-sides welcome back been a long day so thank you for hanging in there with us um this will be the last session in this room after this session um i am the cavalry we'll move upstairs to higher ground and we'll make an announcement afterwards but for those that are here now that's the plan um so this is the i am the cavalry besides las vegas thanks to our sponsors so our diamond sponsors lastpass and palo alto gold sponsors amazon invisium plex track and all of our other sponsors donors volunteers that made today possible and thank you for coming back and joining us in person we're so happy to see your faces courtesy please make sure that your cell
phones are turned off into silent mode this is being live streamed live if there's questions online you can put them in the chat if you do have questions there will be a microphone here you may come up and ask your questions there will be time at the end as well to ask your questions and then just a friendly reminder out about photos making sure that you have permission before you take photos of anybody that's in that photo so with that i am going to pass it off to ray and audrian and they're just left my screen they're not from the government but they're here to help you help them so thank you thanks for thank you very much
and if anybody for some reason wants to take my picture you have my permission just um i want to speak to the people that are watching online i'm assuming this is going to be recorded okay so this webcast was recorded at 303 pm pacific standard time things may have changed since that time but i will still be giving my talk that's from if you ever listen to npr on the radio they do these timestamps so so we are not from the government we are here to help you uh this is my new best friend uh adrian jay ojay who's going to talk about the cyber peace institute but in the meantime i'm going to tell you
about the michigan cyber civilian corps we're going to talk a little bit about civilian cyber defense in general so i want to tell you a little bit of sort of who i am why i'm interested in cyber security i started my career in cyber security back when al gore invented the internet i was in graduate school at purdue in the early 80s and the undergraduates went away i think it was in 1982 the undergraduates went away in the summer and when they came back all of us graduate students had been playing with the 3270 terminals that had appeared sprouted up all around campus and we ran we read thousands of man thousand pages of man pages and learned
how to use unix so that's how long i've been around um computers i i only use it as a tool i wasn't in cyber security back then because the the kind of cyber security we practiced was making sure to lock your terminal before you went to the bathroom because somebody was going to come back and put horrible ascii art in your dot profile so the next time you logged on it would there would be i mean for as obscene as ascii art can be that that would be on your your terminal um so i got my i got my graduate degree i started working in industry for a company called syntax you probably haven't heard of it but you've heard of
the sexual revolution they were the first um market of oral contraceptives and some of you may know the company that is now at their home address anybody know who is at 3401 hill view avenue in palo alto it's now the home address of vmware so syntax doesn't live there anymore vmware is there and xerox palo alto research center is right across the street so it was an amazing time to be learning about computers but syntax is no more i was a pharmaceutical scientist for about 20 years i worked for pfizer some of you have probably heard of pfizer they've been in the news in the past couple of years so i live in kalamazoo michigan that's
where there's a there's a vaccine plant in kalamazoo so i worked for them for 20 years i saw the value in the dark side and i went over to it i participated in a project to take away admin privileges from 18 000 r d employees of pfizer globally and you might notice i don't work for pfizer anymore that's not why but so i was i was very interested in cyber security as i got as i tried to say started taking sans courses when i assisted my employer with their downsizing activities we had a retraining allowance and i fell in with a group of people at sands and started doing facilitation i got a bunch of certifications that sort of
thing i even got hired at sands helped him spin up a master's degree so i i was around security for for sort of a long time and it was exciting but you know these days things are changing and we have covid we have ransomware we have all kinds of other uncertainty and this really is my my motto these days um i am a dancer or i started dancing again the love like you've never been hurt you know if if you've never been hurt in love you know bless your heart but um the credits up there i i don't like to use uh pictures without credit so poetic boredom i think is is where i got that and i think the
the soundtrack should be life during wartime earned all my notebooks right so things are kind of depressing i mean you heard the thank you for for joining us back in back here in person it's been a very long time i went to shmukhan in march and it was really weird because everybody has masks on and you know you see the same people every year but i'm not used to looking at their eyes and bless i you all have beautiful eyes and i remember you but you know it's really difficult to know who to give a big hug to so just give big hugs to everybody okay that actually is a pretty good advice so what do we how do we
knowing that everything is going to hell in a handbasket what are we going to do and my my person one of my personal heroes is mr rogers i think mr rogers is a badass uh mr rogers figured out there was a need to be met and he petitioned a large organization to make a change in their policy so that he could build a team to address that need some of you know the story his thing was children and he thought that tv was the way to to minister to children or communicate with them but he established a program to help children specifically and he had to get a large organization the presbyterian church of the united states to to change their
policies not i don't want to talk about religion or anything like that but the fact is you know he's a pretty badass guy for being mr rogers i think the the thing that we do in the mic-3 the michigan cyber civilian courts we help the helpers we interoperate with the national guard law enforcement other people cyber defenders we also do some training we train the trade trainers and the the bottom line message here something that i have learned in the past couple of years or you know i knew this but it's been brought home is that feelings drive behavior i would love to think that i have a phd in engineering okay i i think about things very logically
i'm sure all of us in this room think we do but the fact is that feelings drive behavior we're not as rational as we think so we have to we do still have to use rational facts all that sort of thing but remember as we approach people the feelings matter a lot so there's a problem we look around in michigan who needs help if we're going to help public entities how big is the problem in michigan we have 2500 plus public entities and i have been told that about 500 of them have a contract within it with an msp so that leaves 2 000 entities that are all on their own these are counties cities so any municipal division of
government plus we have school districts and we have other districts that have technology in them so we'll have an intermediate school district or an educational district where you have a group of people that are responsible for supporting the technology it's also true that sometimes like an intermediate school district the boundaries will overlap that of a county so trying to organize a response in a situation like that is a real challenge so we have 2500 plus public entities and a lot of overlap we have done and i think this is more than many states have done we recently did a multi-agency statewide exercise to test the response of all of the agencies in the state to see how well we would do
and does anybody want to venture a guess at how well we did anybody ever seen the keystone cops like i was not in the in the emergency ops center but you could tell there were people that probably had some notebooks on their on their uh on their shelf and they were pulling them down and opening them for the first time in a very long time so we had the way that worked was we had our state emergency op center folks working and we we had had some personal change personnel changes in the upper management level but as we got the third entity affected you could just see things falling apart and i i would say
don't tell anybody but you know if there's an estate around in the united states that that wouldn't happen to i i'll be surprised i mean this is just normal kind of stuff so we're trying to figure out how to prioritize where to put our energies and how to organize helping people who are who are in trouble we talk one of the themes that's been talked about here is the cyber poor and this is a a list not of all michigan counties because i can't find them all in in this list uh but if you look along there this is the the shape of that curve is fairly typical we have i think 10 counties that
are over 100 000 and then it drops off very precipitously you see my my county kalamazoo county is number five we have a couple hundred thousand people so there are some folks wayne wayne county on the far left that's detroit so you can see this is detroit flint kalamazoo lansing or the other ones those are those are the folks that you know they have a full i.t staff but the folks you get out past about the tenth one they may have one person and if you get way far out to the right and some of you know the geography of michigan we have you know we have one hand here and then we have this one and up here
that's the upper peninsula and we we all make fun jokes about our our eupers that live up there but there's not a lot of people there's way more mosquitoes than people and so finding cyber help up there is really difficult we have as an example in the mic 3 we are fairly typical typically distributed like the normal population we have three members in the in the u.p in the upper peninsula so we really do have some folks that need help often you'll you'll have the you know the it person there if they have one is the kid that was really good in the math class in high school right and that's like i i say that fondly you know because bless
their hearts they got to change passwords they got to make sure that everything works and at the same time think about being secure it's not going to happen so we had exceptionally uh technically oriented government officials for a while and they came up with an idea to to utilize the cyber defense resources that we had in michigan because back when the michigan cyber civilian corps was organized around 2010 2012 the in the main industry in michigan well i just asked if you think of michigan you probably don't think of computers you think of the auto industry probably and the auto industry has had its ups and downs and so when rick snyder whose twitter handle by the way is one tough nerd so
he thinks of himself as a techy guy governor snyder worked with our chief technology officer our chief information officer who and i get them mixed up i think cio is dan lorman who now writes for in from infosec magazine i think he's a he writes some pretty good stuff um david bien was the cto he now works for la-z-boy i believe so we had we had a cadre of high-level government officials who thought this would be a good idea to organize a cyber defense group so they created it in 2013 i actually found the youtube video where he announced this at we used to have the governor's cyber summit it was the north american international cyber security
summit it was held in detroit it was international because you could see canada across the it wasn't like seeing russia from alaska but we could at least see canada so they he announced that we would have we would form this this group of volunteers and it was in collaboration with something called the merit networks merit is a non-profit in the united states that were one of the first uh participants in creating the internet and distributing the internet across the united states and i'm pretty sure when i went to purdue in graduate school merit was providing our our internet service i don't know that for sure but they've been around for forever and they had they had hired a guy named
joe adams from the national defense university to come to michigan move back to michigan and start their cyber range and this you know there are people that have cyber ranges these days and the idea is to use the cyber range cyber ranges for training people so they had the idea that they would take the the cyber defense level of expertise that we have in michigan and use the cyber range to provide training to those folks and then they'd be able to use those those people to respond in the event of a governor declared cyber emergency governor declared cyber emergency in michigan we have a cyber cyber disruption response plan that lists five levels of activation the last
level is when the governor declares a state of cyber emergency and the conditions for that are that life and limb are at stake like people might die and around this time we're talking about 2013 it took them actually a few years to get that started so i don't know if you remember around 2007 i think the department of energy got an electro an electric uh generator took it to idaho and blew it up using using packets and that caused a lot of uh consternation got a lot of publicity and people it was not that long i don't remember which happened first but there was this the squirrel that took out the northeastern seaboard so we knew about
the the fragility of the electrical grid so the idea was the the fear that we articulated as members was okay so nation state takes out two power plants in the upper peninsula that's that's the one up here where it gets really cold um and people are going to die okay cool we are the michigan cyber civilian corps we're going to respond to this now imagine so you're you're a normal cyber defense person and you get a call that a power plant has gone down and you have to go help like what are you gonna do i it's you know i don't know this for sure we never really went through a tabletop exercise on this but it seemed to me
that the what we would mostly be doing to be handing out water bottles and you know making sure medical care got to where it needed to be and that sort of thing wouldn't do the thing you really need at that point is not people looking at packets and anyway you're probably not going to have a bunch of people you've never seen before come look on your network as a member of the the mic 3 i i joined in 2015 i think 2014 2015 was after i left sands they got accreditation for their master's degree and they didn't need as many people as they had and so i assisted them with their downsizing activities and volunteered to work for
this group we were the members were fairly frustrated because we didn't know what what we were supposed to be doing and we agitated we got really lucky one of our members got seconded to the state to be the chief information security officer and he got a budget so the next year we had sans training so if you want a hint on how you can make this happen it's offer something valuable and somehow that rfp never got out of the procurement system i have no idea how procurement works at the state but somebody did something we got some sans training so in 2015 we negotiated with merritt to take over responsibility because the original vision we had people
already in the core that were more qualified than the training that mayor was offering they were doing sort of beginner security which is totally great very happy that they are doing it but for our members it didn't didn't work so the state assumed responsibility we gave for those of you that are familiar with sans training we did security 504 which is sort of the entry level ethical hacking course it also teaches you an incident response framework they use prevent identify contain eradicate pickerel uh restore and lessons learned i know nist has it has different words but it's the same same thing so um so we gave 504 the next year we did monitoring network monitoring which we
figured would teach people how to understand what a normal network looks like the year after that we did network forensics because i this is personal prejudice we have members that would love to do memory forensics we'd love to go take a disc image and see what the malware looks like but in terms of our job is to get public services back online when a local government goes down people may not be able to get their child support checks they may not be able to file their marriage certificates they can't they sell their house they can't file the title that sort of thing you know this is what we're trying to i um i don't want to say i don't care
what the malware looks like but i want to get people back that i want to get people back online so that they can live their lives so um so we did network forensics which is the way that you can determine where the in fact infection is and get rid of it not so much about endpoint forensics though i do have some members that would like to do that and wisconsin has a program that i'm hoping we can collaborate with and help some of our members get trained talk about that more in 2017 the cso of the state of michigan was a guy named christa russia chris is now the federal cso it's in the white house
and he had come from the the white house of the 44th president so he had some contacts in the federal government to get some language around legislation and in 2017 we got legislation passed called the michigan cyber civilian act and you should look it up if you're if you're interested in doing something like this you should look it up i never thought i would be thankful to legislators but they wrote it in a in a very flexible way we have to have we have to have technical standards for our members we have to have them background checked but that's pretty much and we have to have an advisory board that sets our processes and of course we have to
follow the processes that our advisory board sets but our advisory board is people that are friendly to us and i i don't want to give anybody an impression that we're trying to manipulate anything but it is it is it is just a really well-crafted piece of legislation and to support that i will say that texas stole or borrowed copy pasted some of our language they have a group called a volunteer incident response team and they use some of the language in their legislation that created that so it's pretty good legislation um it was amended in 2019 so if you look at it online you'll see two dates there's a 2018 effective date 2020 effective date
the original legislation required that we do federal background checks including fbi criminal background checks and just as a heads up an fbi criminal background check will flag youthful offender stuff so if you were 13 and you pled guilty because and were assured that um the record would be expunged expunged doesn't mean it goes away for forever so we have it the way our situation works we have the state police review whatever comes up and they get to say whether somebody passes their background check or not i don't know what would flag somebody fail in their background check and i don't want to know because i don't you know i don't want to have that conversation but it
happens that when the legislation when the first legislation got signed you know when the the chief executive signs a bill they like to have a photo op so we wanted to have some of our members come sit behind governor snyder while he signed the the legislation and it was it was short notice and the only person we could get was one guy who had been with the organization since merit so he was there and there were a bunch of secretaries if you look at the photo op um it's not really people with the mic3 don't tell anybody so we had one member there and you can see what's coming this is a member who when he was young was
involved in something he keeps trying to tell me about it i don't want to know but he got he couldn't be a member so it's the the thing that happened in 2019 is brett's amendment so brett can now be an advisor he can't go out and type in the keyboard but he can give advice he can know things so i'm very proud that we were able to make that happen for him and you know we try and take care of our members i talked to my i talked to our members for about an hour a couple of times a year i'll make a point to to do that to help understand their needs so we we went through all the uh the
logistics to make that happen the benefits we are now the mic-3 is now included officially in the state cyber response plan and exercises of the plan that multi-agency tabletop i was a participant we didn't have members participate because we kind of knew we wouldn't be doing much response but we were officially included now the members receive regular training and professional development i sometimes say that the reason you want to form a community like this is so that you'll have somebody to laugh at your jokes you know if you like who else are you going to tell that you've got a udp joke but they might not get it right people in this room about yeah okay
maybe we won't tell that one again but it i mean like i said feeling strived behavior and if you can make people feel comfortable they know they can pick up the phone and and talk to somebody and check something out it makes them feel a lot more comfortable we also have had i told you i talked to my members over 80 percent of my members have changed jobs in the past year and a lot of that is because they know people right they get they get a clue about a job that's getting being posted we're trying to get involved in k through 12. it is more difficult to corrupt the youth than it should be but
i'm working on it uh my my 12 year old neighbor is watching my dog at this time and i left all my lock picking stuff out and deviant's book and a link to a youtube video about picking locks so you know we're trying to do this we're also going to go out we have a grant from dhs to to work with a marketing company in michigan to put together some stuff for election security and i'm hoping some of my members are going to be able to go out and sit with clerks as they present this information so basically to provide a some um competence and be another person to stand up like they know something there's a fair amount of
value you can provide just by doing that i think so we have members have to have two years of experience they have to have a certificate uh a certification and i'm not going to argue about certifications okay i have i have many so i know what they're worth you don't have to convince me um we do ask them to go to their employer to get support for 10 days of participation that counts training and we're going to see how well that works we have a face-to-face exercise coming up on august 26th which is our first face to face in a couple of years so i'm hoping people will wake up um we use we have not done sans training
in a while we're using a company called rangeforce which i like a lot i'll be happy to talk to you i don't get anything from them i don't get anything from anybody i'm a public servant um the application process i'll go this through this fairly quickly the member applies at our website you have to give us your name and address and now you have to register with the statewide identification system which is called my login everything in michigan starts with mi because michigan and it's pronounced my so you may think in your state you have stuff it's all my stuff so you fill that you give us your name address you get a my login um
and you you give us your personal information no credit cards uh then you take tests we have four tests which i'll admit need to be changed they've they haven't changed um in too long we have beginning and advanced incident response and forensics there's an initial test that's pretty much ports and protocols uh basic networking so if you pass that one you get access to the basic ir in basic forensics if you pass the basic one of either of those you get access to the advanced one okay five tests total you have to pass four and um i have i have enough history that i know which one people are going to fail we we know which questions are bad i just
don't have a test writer to to change the test i'd love to go to something like maybe security plus and have somebody else administer this but we are where we are i i should say that the michigan cyber civilian corps is at a point of minimum viable product for those of you that are familiar with entrepreneurial stuff it's it's good enough to show people it's not perfect totally got problems and you're welcome to tell me what the problems are but it's good enough to to throw out there and let people get ideas from it so you have to pass four of the five tests if you do that like i don't care about your documentation if you can't
pass the test with all due respect if you pass the test then you send me a resume you have to fill out a volunteer agreement which includes non-disclosure um the employer agreement which is not a legally binding document it's a piece of paper that you take into your manager and say hey i want to do this the reason it's not legally binding is well that would be a pain but general motors is look as as an example is located in michigan and i have people i have members that work for general motors general motors legal counsel is never going to let them their manager sign a document like this so we just blow it off
uh if if people don't show up for activities for a period of time we can declare them inactive and they won't get access to training that sort of thing so it's it's not a perfect solution but it works once once we have all the mandatory documentation then we do the background check because we don't want to for obvious reasons costs money that kind of thing when we operate this is how we operate if somebody gets whacked they call the state police we are not um law enforcement so they call the state police and the state police vets the the call maybe there's no criminal element or maybe there is they they do the evidence thing but law
enforcement as you know doesn't help people recover so we have a very good work in relation michigan has i don't know all the other states michigan has a really good cyber command um department the guy that runs it lieutenant first lieutenant jim ellis is always out of town going to other states talking about it i love working with these guys because they're the ones that actually do their memory forensics and they run ida pro and things like that and they will come to our meetings and talk to our members occasionally so that to whet their appetite you know because we are going to do that one day so the the michigan cypress command center vets
that's the the incident if the affected victim wants help there are a series of legally legally um significant documents that go go by the person that the victim has to ask for help we have to say yes we can provide help if you are authorized to accept the help reply in the affirmative and if they do then they've said they're the one to you know because we we need to have a throat to choke at the at the victim organization um as it were so that's how we operate i hope that i have given you a lot of information that will prompt questions because i'm going to turn it over to to adrian in just a
second but the the bottom line this is like when when covet started everything shut down and we had been doing cyber defense for a long time and we couldn't do that like we had done it and so a lot of us were looking around for ways to be to learn about serving the public we in the executive branch of the state of michigan we all took a 10 pay cut temporarily well it was going to be temporarily it hasn't come back yet but you know i i'm not working for the government to get rich i went like i said i live in kalamazoo michigan there's a pfizer plant there we produced the vaccines and when the
trucks started rolling from the plant it was people in kalamazoo were like in tears because it's our family you know we can save we can help save the world so after a while they started doing vaccine clinics and the kalamazoo health department called for volunteers and this there's a lot of other stuff going on and i needed to do something so i volunteered to help with the with giving people vaccines and i ended up working as you see this is a the front page the incident response plan for for one week of doing uh vaccine clinics so i ended up working like a ten tenth of a year for the health department but it was it
was hugely valuable i it was very difficult for me to describe how it was valuable but to go down there and stand for three or four hours with regular human beings who were coming in afraid they were going to die i mean this is cyber this is defense you know it's not cyber defense but working with people who were really concerned about the pretty bottom line you know it's an existential issue so this this is meeting people where they are i got to meet a lot of regular people i also got some contacts with the local emergency operations folks kalamazoo has an international airport the faa requires as you might expect people to test their incident response
plan in an airport incident response is a mass casualty event so during the course of this i also participated with the the faa um test of the kalamazoo battle creek international airport there were like 50 of us the people that got there early got to get makeup and crawl up on top of the school buses that they were using as the airplane fuselage so this gave i mean it was a lot of fun certainly but it gave me an end with another emergency operations group we said somebody said earlier that the reason people are not careful is because they don't think cyber is part of life right well cyber is part of life and life is part of cyber and the well you
know that sounds stupid right but but still there is tons of overlap we have to get people thinking um about it like brushing their teeth right so so getting involved in this way is a way to get your get your hands dirty get your brain working um so you can find me on twitter and linkedin you go looking me up i'm not the fashion photographer that's eric ray davidson i've offered to do a job switch with him for a day but he hasn't responded you can write to me at davidsonr5 michigan.gov that's our website and if you are looking for something to do locally your state doesn't have a program and some states are are a little slow
there are about a dozen states that are doing a program like this but if you want to do something and there's not something in your state you can do something globally and adriana oger is going to parlay with you about it [Applause] hi everyone um adrian here agent and call me agent works too um i'm the ceo of the cyberpeace institute an ngo based out of geneva in switzerland it's an honor for me to be here so thanks for coming thanks for watching yeah all right better yeah okay thanks sorry did you hear that though okay all right um so i'm going to talk to you about a similar program that we've launched about two years ago um that's
operating globally it's open to um to you i guess as well um called the cyber peace builders program run by the cyberpiece instead private sector is not really as active as it should be then you've got a lot of people that depend critically on on these services and actually if you go outside in the streets right here in vegas it took a walk yesterday it's a lot of people on the streets that depend critically on ngos in developed economies as well um just and i was attacked by cyber criminals and we helped them investigate the the incident they lost over a million funding that was destined to afghan farmers so that is money that was
not provided to vulnerable a vulnerable community because some criminals thought it was a good idea to steal it we're not able to retrieve the money maybe you've heard of the international committee of the red cross that got hacked earlier this year half a million refugees and all the people at risk that were whose personal data was was stolen and and potentially monetized um and also just uh another attack that maybe you haven't heard of are actually right here in philadelphia ransomware um that led to at the end of the day um two million dollars being taken by the by criminals um ngos still mean high payout right when you can make out with over a million dollars
it's it's four criminals it's enough it's i don't wanna say it's enough money i don't know what you know dependencies are but or what their objectives are but but it's certainly comparable to some of the amount that they can steal from much more much more um protected targets uh in particular critical infrastructure organizations like caesar or ansi were french cyber security agency where i used to walk um other such organizations right ngos do not enjoy any of the benefits of of these organizations and for for criminals they don't need to use complex zero days they don't need to burn their zero days on end use they can use a password that like 10 years ago
which is what happened in some of the incidents that we've we've investigated and the big problem is that most of the ngos have started to operate their digital transformation only recently in the last decade more or less but they do not invest in cyber security right it takes cyber security professionals and presents them with a very short-term engagement opportunity so they can go and help our ngos so it's literally as simple as this job board where you as a cyber security professional can connect you can see opportunities that we list with your activities on the left and see all right this ngo needs advice on 2f8 actually up with the ngo and then you can deliver
that help so remote help right it's short term um and like i guess what you guys did i am ic3 where it's a long term engagement um obviously it falls outside it's curve of incident it's pre and post incident um stuff right um just briefly we have over 70 ngos right now that are our internet walk most of them operate in the humanitarian context peace development things like that um we have over 200 um volunteer-led missions in the platform after a little bit more than a year year of launch but that's that's just where we are today by the end of the year i want us to get to about a hundred ngos by 2025
to a thousand ngos so we can get closer to that one billion people than all of these ngos support um just a couple of highlights in terms of missions maybe to make it more concrete uh for you guys and i'm happy to show you the ball after it's it's live you know on the internet so if you want to pass by but just a couple of examples of what our volunteers did so one volunteer for instance spent well we had you know scoped it to four hours ended up spending 16 hours but not all jobs are like that most jobs are like one or two hours but doing a code review for an app developed by a mexican ngo to
help pregnant women in empower rich villages to tell them you know if you have this symptom that symptom this is what it means when they cannot consult doctors so they're looking to secure that code before launching it and so volunteers are able to do that um volunteers can off websites of fsd which is a lesson studies the diminished swiss demanding ngo we had um various one-hour security assessments that basically help ngos understand where they're at so it gives them a one-page report with a three by three matrix uh with red you know orange and green tends to be red uh but is really visual and helps them convince their leadership that their cyber security is not at the
right level and they should invest you know in those basically nine pillars like authentication disaster recovery planning things like that um we supported uh an ngo implementing um pgp uh another one some of them come without requests like we were not expecting that but they came asking for insurance advice which is good you know in terms of maturity maybe not so great because maybe they're trying to just outsource all their risk but still you know they're asking themselves some interesting questions this is an example as well to tell you that our volunteers are not all technical folks so we have people that are yes forensic investigators that are incident responders that are you know people working in in red teams and stuff
but we also have people that are have a background in low background data protection even a background in communication like we have people who help ngos with their christ comes sometimes or maybe can even design a poster to raise awareness on two-factor authentication such things so it's not just technical advice um yeah we run an um wellness raising session in arabic so one of our volunteers um that is actually um i don't know if he's here today but he's coming at um def con working for linkedin translated the presentation in arabic and delivered that to to an ngo and voila and that basically helped us to surface some of the most record needs that ngos have and temp templatize i
think same english uh the the jobs the missions so that other ngos that start you know from the bottom and really don't even know what to do couldn't have an idea in terms of all right maybe you know i can get a training for myself maybe i could do a pen test all right what is it pentastar let me let me do that with one of the volunteers etc so it helps them you know advance in the journey and our hopes is to standardize that all the way up to trying to bring them to iso 27.1 and other you know nist frameworks for instance in the us or the frameworks in other parts of the world
um to to help them have a more structured approach just in terms of setting up that volunteer initiative um a couple of the you know the benefits and the incentives i think ray talked a little bit about that so let me do the same for the builders um what we try and do is foster human-centric approach you talked about the importance of feelings and i talk about the importance of face to face you know seeing people and through the through the cypress builders you actually get to put a smile on people's faces you know no one helps these ngos so they're really happy when somebody comes and takes the time and so it's really that human connection right
seeing who you're helping having the ability for them to speak to someone rather than you know read a guideline or download the document or go to a conference but not engaged directly right having that personalized approach it's flexible for for the ngos as well because they can come up with whatever needs they have they can really ask us anything they want of course if you know it's incident response we would say it's out of scope if it takes you know 100 hours we tell them we cannot do that but but it's still flexible i cannot they can ask what they what they want gives them access to industry industry-grade expertise right all of our volunteers
have jobs right so they they they they come with that without experience big incentive right free of course um and back to some of the points you mentioned right it's also the community approach allows cyber security staff or infosec staff in ngos to know each other and to start helping each other and building a sense of a cyber security community within that within that humanitarian sector i dropped some of the some of the comments that folks have left in the surveys we do after every engagement in terms of volunteer incentives so for people you know like you who may want to join such a thing um of course you know it's a purpose-driven activity so
it's something that sometimes you don't get to do in your day job and i think that most and i hope all of us in the cyber security industry have this kind of innate sense of helping of protecting and sometimes cyber security jobs can can you know stray a little bit away from that initial mission and so volunteering whether it's with a michigan uh with your state at the international level or through other initiatives right we're not the only one um kind of appeals to that kind of sense of purpose that many of us have it's beneficiary facing so you actually get to see the impact of what you what you do what you provide in terms of
advice it's flexible as well right it adapts to your schedule you don't have to do things when you cannot gives you mentoring opportunities and shadowing opportunities because you can partner up with other volunteers and learn from them learn their techniques uh it's a good way as well to up-skill right we provide a training actually i think you have it here on the right where we give you some of the skills that you may need to enter into that kind of humanitarian context so what is international you know humanitarian action and humanitarian law um what is you know give you a sense of victimology and trauma communication a bit of diversity and inclusion in cyber security so it's
a course that we made specifically for cyber security professionals but to introduce or develop some of the soft skills they may need for that that um volunteering opportunity basically and and yeah it's it's a good way to train us um train um more junior staff which is something that companies in general are interested in we walk a lot with companies so we try and do outreach directly at companies so that we can get access to volunteers um from their cyber security workforce which which helps with uh recruiting rather than you know going one by one we do both but uh but it helps for companies so in terms of incentives um it's obviously corporate social
responsibility the pledge one percent and all the such movements take all the risk etc uh it helps with talent retention back to the point i was making earlier on purpose right it helps because it helps align the values of the companies with that of the of the employees it helps train the junior staff right through shattering opportunities um it helps with soft skills developments as i mentioned obviously there's a visibility component associating um the company's um vis brand and you know name with that of the cyberpeace um institute and there's because we we do run on donations so if companies are willing to help there's also of course fiscal advantages um just a small thing that i didn't talk
about but i'm maybe some something to come up in the questions it costs to rent these initiatives i have a full team that's that's behind me and there's actually a bigger institute so we're 30 people out of geneva um walking on three different pillars this is one of our three pillars right providing assistance um we then analyze all of the um information that we gather so that we are able to influence these issues go back to decision makers at united nations level in the different capitals around the world and tell them look this is how people are suffering online this is what we need to change right um but this is to say that there's a
cost a cost to running all this and and it's not that simple i think there's more information online about you know what it costs for other states but it's roughly between half a million and a million us dollars i think a year to run these initiatives um yeah so very briefly if you if anyone is interested you come and talk to me um just to to tell you that um if you would want to do something something like this and you'd want to do it during your walking hours you're going gonna talk with your employer so we actually have an agreement with employers we do struggle with legal counsels but we have been able to get some to join
um and so once we get a corporate agreement sign it's much easier for us to be able to get employees um and then what we're finding out is that it takes time to go through legal counsel so we're also opening a route to um for volunteers who want to join without necessarily having a we're leveraging background went through such a background some questions yeah thank you thank you very much you can do sans training because she has a budget of away definite numbers but that's for michigan assessing skills are you in any way determining of engagements where you help people like clean perceive that there was a need they prioritized the response part as we started doing response we have a
program called cso as a service which is a community of a lot of the people that work for the state and do cyber and it's similar to the mic-3 only on the proactive component yes we are doing that one hour from now it's moving upstairs to the higher ground track but we one of the open-ended questions is as the cavalry expands and evolves its mission should we go down and we don't want to reinvent the wheel so in some cases we might want to partner experimentation with variants to comprises left a boom because uh to quote the late great dan kavinsky of the many things that hackers smash a crisis than a real one but we don't want to
reinvent the wheel and we'd love to hear what would be the best compliment that that can be fun and must be fun i bring you up and um interest from people everyone was i know they are popping up in different parts of the world as well and i i join you know with b-sides here but there's also right if you walk with sltts i don't think that you can get ngos there right so there will be differences and there isn't one who are able to make connections that make sense it's like this global it's global and local at the same time so
Related talks
51:51
32:48
33:48
54:33
31:06
20:51