I Am The Cavalry Track: A Decade of Critical Infrastructure Security

[Music]

maybe you'll give me five years I'm gonna butterflies [Music]

fly [Music] baby

[Music]

[Music] oh oh oh [Music]

[Music]

[Music] foreign [Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music]

thank you [Music] foreign [Music]

[Music] [Music]

[Music] foreign

[Music]

[Music]

[Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music] thank you [Music]

[Music] foreign [Music] foreign [Music]

[Music] foreign [Music] foreign [Music] foreign [Music]

thank you [Music] foreign [Music]

[Music] thank you [Music]

[Music] foreign [Music] questions [Music]

thank you [Music] foreign [Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music]

[Music]

[Music] foreign [Music] foreign [Music] [Music] foreign [Music] thank you [Music]

[Music] foreign

[Music] foreign [Music] foreign foreign

[Music] thank you [Music] foreign [Music]

[Music] [Applause]

[Music] foreign [Music] thank you [Music] [Applause]

[Music]

[Music] thank you [Music]

baby [Music]

[Music] don't leave me alone [Music]

don't wanna overthink it baby [Music]

giving me Wind and Rain some kind of butterfly baby [Music] [Music] but I don't wanna miss you baby [Music]

[Music]

maybe you'll get me [Music] away [Music]

[Music] some kind of butterfly foreign

[Music]

[Music] oh [Music] my God

[Music]

[Music] all right [Music]

[Music]

[Music] okay thank you

[Music]

[Music] moving up

[Music] thank you [Music] foreign [Music]

[Music]

[Music]

[Music]

[Music] thank you [Music] foreign

[Music]

really

so um

community and Government after thoughtcon and 20. okay hi I'm Andrea matrician I'm a professor at Penn State in the law school and in the engineering okay

testing testing all right welcome to the I am the Cavalry track 10th edition say happy birthday for a decade [Applause] um I am not Jen Ellis so you might have seen when you were deciding to come here that Jen Ellis is going to be the lead speaker but sadly Jen got stuck in the UK and we're all sending her hugs she has recorded a little bit of an address for us and uh Beau and I will do our best to augment the things she did not say um but uh this track uh is gonna we're gonna basically do a little bit of reflection on where we were 10 years ago what's happened over the last 10 years and

maybe what could be the next the future here in this first opening orientation but also we're going to um try to give a little bit of a preview of what you would see if you come here and if this is your first time in the Cavalry track some people come and go a la carte some people really benefit from the contiguous experience because several of the talks were chosen and sequenced to build upon each other and cross-reference each other so um I will probably go last um but I'm Josh Corman I'm Beau Woods and Jen Ellis will introduce herself um via video but um yeah 10 years ago here we gave birth to I am the Cavalry the idea that

Cavalry isn't coming the co-founder was Nick broco at the same time Beau was given his own talk and we met in a lot a green room afterwards but the ideas are dependence on connected technology was growing a lot faster than our ability to secure it in areas affecting Public Safety human life that was my raison d'etra Nick was pretty concerned about the increased criminalization of research and snow Ninja's happened so you know it was uncertain times where I both felt that we were powerless and at risk of being further marginalized but also that the world needed us more than they ever did and we asked if people would work together and try some radically uncomfortable experiments to use empathy

and a Long View and meet people where they are in safety critical Industries to try to make the world safer sooner if we work together we had a slightly bigger scope than we ended up in deploying but we said if you like this idea meet us at Derby Con in eight weeks and we'll do a uh constitutional Congress to identify our Mission Vision goals so we're not going to do the entire history here I just went through quite a bit on the upstairs keynote um but we do want to make sure you understand what why we're here today uh and what we're going to be doing today and tomorrow and to do so um if I can get this thing working again

um Jen couldn't be here in person but she does want to share her thoughts this is not Jen this is Andrea

and Bose seeing this for the first time so he may disagree with everything that Jen has to say

okay fingers crossed High B ciders I'm so sorry that I'm not able to be there with you but I hope you guys have an amazing time out in Vegas um not sure that I'm envious given the current Heat Wave and everything but I hope it's a good time for you guys um I just want to really congratulate everybody uh 10 years on the Cavalry is amazing um when I think about some of the Milestones that you guys have had or that the cavalry's hair in that time and some of the impact that's been created I just I think it's really phenomenal and I think when you look past to 10 years and like what's changed in

that time I do think you can see the sort of sticky Fingerprints of and the Cavalry members on um on some of that Evolution that is so important and impactful for how we think about um security risk across Society um so for those who don't know I'm Jen Ellis hi um I work really closely with governments around the world and non-profits around the world to um try to think about how we create behavioral change the reducer security risk on a societal level um and like weirdly I've been sort of in the background involved in the Cavalry since the very very beginning but it was all a bit of a coincidence on timing really so like I'm with me as we go back

back through time um so 10 years ago I was working for security vendor and I was um supporting a security research team and working really closely with them and one of our research leads was threatened with legal action for a research projects that I've been really closely involved in with him and it was all a bit stressful as I've imagined that you can relate um and it didn't sort of lead to anything terrible but the whole process was so stressful that he sort of said I don't know if I want to do research anymore and I was kind of like yep I get that but at the same time there's a sort of consumer rights issue

here because you know research is the thing that arms uh consumers to make informed decisions to manage their own risk right and and I was like what are we gonna do if researchers feel like they can't do research and so I started to look into it more I learned about the Computer Fraud and Abuse Act I learned about the Digital Millennium Copyright Act I learned about State hacking laws and what impact all of this was creating um for researchers in the US and then sort of anti-actionals around the world that are very similar that are also creating a chilling effect on researchers and I was like something's got to be done and even though um I have no party background no law

background and in fact I'm not as you probably gathered from my accent actually American I decided that I was gonna be the one to go and change it um and so I started to talk to some other people who kind of maybe felt strongly about this too and tried to sort of think about how we could go and engage policy makers and that all started 10 years ago um and weirdly at the same time Josh was going through his own sort of um process of like realization and Awakening around the risk that existed in society where um physical and virtual meet and have the potential to create harm I'm sure we've all heard Josh say uh where bits

and bites meet Flesh and Blood um a thousand times if you're playing Josh Coleman bingo I am here for you um so we were at a conference and Josh and I were just catching up and he was talking about what he was doing and I was like oh that's kind of weird because I've also started doing government engagement recently on this thing and what we realized was that the two initiatives while separate and and had come from separate places were going to be super super complementary to each other um you know every time I went and I met a policy maker and I said hey um security research is being chilled by the current legislative landscape they

would look at me and it was a little bit like I'd said um we need to build a rocket ship to the moon and they were like what's the Moon um and my ability to answer that question what is security research and why does it matter was really aided by the Cavalry right I was able to go well for example when you look at cars uh let's think about how much software is in cosm and and most people don't know these things right they don't think about things in these terms and in fact at the time when we started automakers weren't thinking of themselves as technology manufacturers they told me directly they weren't and so you know

having the conversation around cyber security with people who don't think of themselves as software companies is is really hard and and talking to legislators about that kind of stuff is really hard but every time I met a researcher through the Cavalry who was working in one of these areas of you know potential harm and like the thing that's cool is you know obviously it's terrible the level of harm but they're very relatable areas to talk about if you talk about a medical device or you talk about a car or you talk about a plane everybody knows what that is everybody's interacted with them in some way shape or form they've seen them on TV they're very tangible and they can

understand the potential for harm in a way that when you're talking about for example the confidentiality part of the CIA Triad it doesn't engage people outside of security in the same way it doesn't get them sort of like going oh I see what the impact of society is going to be so being able to to talk to researchers in the Cavalry and and like learn from some of the work that they were doing and the conversations they were having in the sectors and then pulling that over onto the policy side to talk about the impact of research the importance of research why research was actually hugely valuable to society was really really helpful and I and I hope and I

think that the conversations that we were having there about that role of researchers and the importance of research has also sort of fed the cavalry's ability to engage with them we open like lots of doors and sort of try to say hey you should really meet these people I took a number of people who were involved with the Cavalry in to meet with policy makers um and then over time there have been many many many many more touch points that have been created which is amazing and I mean if you look where we're sitting today besides has now for a long time had policy makers coming and participating I think I was one of the

early people all to drag policy makers with me to come speak with me um but now it's like quite a habitual thing and I know that you've got sort of members of the FDA who'll be speaking the Cavalry track I imagine they're probably there's some people coming over from the UK government who are talking about related topics um I think you'll have many others from the government sitting in the audience if not speaking as as part of the the talks and Common Ground often has um policy content but that's not all right black hat has a dedicated policy track Defcon has policy at Defcon which is huge this year really huge like two separate tracks and then a round tables

room like it's uh it's loads and we're seeing it now spill out into other areas one of the things that Cavalry has always been a part of and as fact has been hackers on the hill an initiative to help security people get experience of going and briefing um policy makers and those touch points are critical because you know at the end of the day cyber crime is estimated to cost the the global uh Global economies around seven trillion dollars a year right which is like almost three times the GDP of the UK just to give you an idea yes the UK is much smaller than the us but still um it's quite a big number and so like

you know the governments are going to be paying attention you know the government's going to be trying to do stuff on cyber security and cyber crime and cyber risk and so if we can put the people who live and breathe this every day the people who are doing the research who were doing the jobs who are working on the front lines of this in touch with those policy makers and then we can try and avoid unintended harms or outcomes we can try and make the policy fit for purpose as much as possible so I think it's really critical that there is this relationship and that the Cavalry has engaged with uh governments in the way that it has and the Cavalry has

taken a position of leadership around so much I mean you know things like the patch Act um the some of the things that have been done that weren't direct policy engagement but they helped influence policy engagement things like um the the um uh the the five-star Auto for example you know those kinds of things have then kind of been fed into sector-specific people I mean the work that the Cavalry has done with the FDA has been phenomenal and huge props to the FDA team on the other side of that who have just shown astonishing leadership over the past 10 years on these on these topics um so you know I don't I don't want to

sort of you know believe the point but I do want to just say I think when I look now you know 10 years ago people like policy makers don't want to hear from us and I could see why people felt that way right and now I look at things I look at the landscape and I'm like yeah there are still problems it's still hard to know who to talk to and when to talk to them but no one can say policymakers don't want to talk to the security Community anymore there is demonstrable evidence that they do because they're coming to us like coming to our conferences and they're doing that so they can meet more people and talk to

more people and I think that engagement and that potential to create impact is incredible and it really speaks to some of the work that's been done and the and the trust that's been built and the solid foundation the groundwork that's been established to show how productive those relationships can be and also to show that we can be trusted Partners right we can collaborate we don't have to be just you know trolls on the internet that criticize and have no interest in working on making things better and I think that is all really really positive one of the things I love about the Cavalry so much is it's not just about pointing out problems it's about finding Solutions it's about that

collaborative effort working forward together so I just want to say again a huge congratulations to everyone who's worked on that the work is not done there is more to do but there's also so much more opportunity now to engage and to create influence and I think that's amazing so I hope that all of you will feel really inspired to go on and to continue and to think about how you can play a role and um and I say good luck and I look forward to it and let me know how I can help enjoy the rest of peace signs thanks very much bye

okay so that's Jen we'll take feedback to her she can't hear you right now even though it felt like she was in the room um and fun fact as we pivot to Bose Reflections uh Jen did not see the call to action even though she knew what we were building and kind of helped advise on what we were building she was across town at black hat doing a super secret meeting that never happened uh establishing trust with people about hacker lawfulness um so she never heard the call to action and has been one of the top contributors for a decade now that takes some stamina uh and uh we treasure her so someone else who did not see the call to action but

has also been in the top Trio of making the worldly safer place is bow why didn't you see the song yeah so uh I at the same time I was in a different room giving a talk uh about how to let go of responsibility and like go travel around the world um which was fun for a little while but this is a lot more fun and more meaningful uh so uh I guess I let Josh convince me to uh quit doing that and and to come in and out of curiosity how many people in here uh we're also not in that talk that first year at besides it's a majority of people I think the lesson we can take

from this is that there's a lot more people who were not in that room who were helping to change the world yes uh which is awesome I think and it's a testament to how far we have come um and I don't know we didn't plan this by the way uh because you know we kind of had some last minute stuff but do you uh you pulled something no no you know you have something okay you you should take about the same time she did so okay um so as I've been thinking back in the last 10 years like 10 years ago what did we tend to hear about when you had you know the so-called adults in the

room talking about stuff it was Banks and uh the electric grid and that was basically it films I think uh Sony had just gotten hit because of the Seth uh the the film about North Korea um and that was that was a lot of it like if you if you talk to policy makers or anybody about uh oh I see you're doing a montage you can ignore it um look how young he looks oh camels um if you talk to policy makers uh or others like they hadn't really thought about security implications of Internet of things that really thought about they didn't realize that computers were in cars um so a lot of the things that we had to do

back then uh to make things relatable today that situation has totally changed um so it was was it when did uh cybermed Summit happen in the hotel room the year no harm the first Do no harm yeah um I think it was your two or three of our birthday yeah yeah so this would have been like 2014 2015 um we have some very awesome doctor friends of ours who are also hackers uh they given toxic Defcon and also save people's life in the emergency room and uh they're like hey we should just like have a conversation about medical device security about Healthcare security like there's some really serious issues there it's going to get people killed uh and we're like all

right cool what do we want to do like well let's get some people together I was like all right that's cool you know Tuscany has great big hotel rooms we could just do it in my room next thing I know I see a tweet go out like hey Tuscany this room everybody just show up and I'm like oh so I'm like okay I guess we gotta prepare for this um so I like uh emptied out my bathtub I hadn't showered in it and put a bunch of ice in there went and made like 10 trips to the ice machine getting a bunch of beers and threw in there and we had probably 20 people sitting around in a

hotel room just having some really really uh interesting conversations about medical devices Healthcare security some really really good people in there who uh still to this day are doing amazing things in medical devices and we um formed an idea uh to do what we call the cybermed southern I'm just going to wander and ramble on this um and uh we we came up with this idea to do clinical simulations where you put doctors in a room to treat a patient the doctor doesn't know what's going to happen to them they don't know it's going to be something cyber security related but they've got to support staff that has a script you've got a patient who's a professional actor who does

these things all the time um and so we we did this really amazing thing where we killed someone on stage we killed someone in an operating room but not for real but we simulated it um and it was incredibly powerful uh if you've if you've never seen what happens in an actual emergency room it's worth going and finding some of these videos online uh we had that first year we had Nightline did a story about it um so like you know major Prime Time news does an eight-minute segment on how we killed people and and uh made sure that that didn't happen in the real world the next year I think somebody else came out Hearst came out and did a story on

us Hearst is one of the biggest um uh print media they have Print TV and radio media a lot of your local news stations are owned by Hearst uh we've had a lot of others pick it up um but anyways it was it was super cool super powerful we tweeted about it and then I think I got on a plane from there because we did in Arizona I got on a plane from there and flew over to Germany for an automotive conference um speaking of you know Jen mentioned uh Jen mentioned the automotive side of it and uh so like I landed I went to The Venue it was the night before so I had

like the speakers uh party or whatever and um I was just going around and talking to people and this one person was like Hey you know I heard about this really cool thing that somebody just did in the U.S where uh they they had a simulated clinical thing I'm I'm looking at this person I'm like clearly trolling me because you know that I was just there and then he's in the room and uh it's like so just let him go on and describe it I was like this is really cool I was like are you are you saying that because you know I was there it's like wait you were there but now you're here and uh that's gonna be one

of our speakers later on today I believe or tomorrow tomorrow and the British are coming um that person David Rogers who's sitting right there um uh has also been one of the loudest maybe not loudest voices but certainly one of the most impactful people at getting uh iot security governance put into public policy around the world um I won't steal his Thunder for tomorrow uh you should come and listen to it but it's it's really really cool uh and a lot of the Global public policy around iot security is because uh you know somebody from the hacking Community was like hey you know what I think we can do better I'm going to see if I can make that happen and

brought us all into that process to be able to to put things together in a way that's compatible with uh with public policy and I think there's a lot of stories like that a lot of threads like that that have happened over the last decade where one thing led to another thing it cascaded it snowballed um people uh heard about this idea uh started believing that it could happen uh eventually changed internal roles in their organization so that they could be on the product side rather than on the internal security side and that happened at least a couple of times Kyle uh did it at Tesla um this guy and yeah that guy um and then um Colin at J J Mike Murray

Mike Murray yeah um and like those are seemingly small transitions but when you think about what it takes to get a multi-billion dollar company to change what they're doing um that's actually massive uh and I think it's it's been understated you know they're not up giving talks about those types of things because largely you can't talk about what goes on inside companies like that but they've been some of the most influential and Powerful change agents uh and while Josh and I are up on stage blabbing about all this stuff they're actually going out and doing it and making it happen um and so there's a lot of things where a sequence events comes together that

couldn't have happened otherwise um and it's amazing to just see all the progress that's happened over the last decade uh both from people in our community as well as on the public policy side uh some of the policy makers who have changed um we had uh so Josh and I in 2016 um went into an organization I think tank in DC called the Atlantic Council and at a certain point we got this intern oh yes yes and this intern was kind of incredible like on paper it's like wait a minute you've got a law degree and like a couple of master's degrees why are you interning like it's like well you know I'm waiting for my

clearance to go into to government it's like okay uh cool well you already know a ton so we're just gonna like put you to work and like we're not gonna have you running and getting coffee like a lot of interns we're gonna have you like you know you want to figure out how to get members of Congress out to Defcon um and so uh she did did a great job of it um went and put together all the paperwork managed to figure out how to navigate the labyrinthine Congressional ethics rules and so we brought a couple of members of Congress out to Defcon will Hurd uh and Jim langevin um and they were two of the loudest

voices in Congress on cyber security issues uh after that three-month period or whatever I think she eventually got her ability to go in uh to government and ended up trying to remember the sequence of events Shin it up uh was it the newly formed then newly formed sizza for a little while and did that and did just phenomenal work she later dragged Us in uh to sizza or helped drag us into sizza to work on the coveted task force but now she's at the White House helping to write strategies and implementation plans um and she's not alone on the policy side there's a number of folks that we started working with just a long time ago when they were

Junior staffers but they were super interested really passionate wanted to learn because they wanted to do better um and now they're also in just amazing roles like legislative directors for members of Congress or taking roles in the White House the The New National cyber director role that has come about many of you know about it if you don't know about it it's basically somebody who reports to the president and their only job is to think about cyber security issues and they have a large team I think they've got 80 to 90 people on staff now um and they rated some of the best people from Congress uh to go and do that so the best staffers as well as

other people people who had been you know formerly in the White House um so a lot of those policy makers have made that Journey as well both the elected and appointed individuals as well as their staffers and you just see these folks level up continually from being fairly Junior but curious passionate about something they look a lot like us actually um not in the you know Mohawk sense or in the Kilt wearing sense but in the sense that they're young they're energetic they're curious they're smart they're creative they're talented and they go really really far in their careers um that wasn't the case a decade ago I think 10 years ago when we went and

started talking to folks in Congress there were actually more members of congress with a degree in computer science than there were staffers and there's like five times to ten times as many staffers as there are members um and it was just accidental that those members were had computer science degrees because they were you know business owners they ran businesses and got into politics that way uh but today I think that um almost all members of Congress have somebody to part of their portfolio is cyber security I can't think of anybody who doesn't and many of those have computer science degrees or they have a technical background they know about technology and they're serving their their version of public good uh

they're doing Public Service in that way uh and so from all sides the world has changed in a decade and changed in a lot of ways for the better um but the faster we get better uh the amount of things that have been connected that have been had software added to them has also grown and probably grown faster than we could possibly hope to to do better with just looking at a single Community or a single approach and so part of the original thesis of the Cavalry was you have to to fuzz the chain of influence figure out what works repeat that and then avoid the stuff that doesn't work or you know tweak it

and try again but to be kind of relentless on pursuing this um and so we found a lot of stuff that's really cool and that works after a decade and we're going to be talking a little bit about that tomorrow afternoon um we found some stuff that doesn't work so great and so when others want to go in and do things we want to help them avoid those things there's some things that others have tried that uh have been have worked really really well uh Ray and the audience here um worked on uh at state level how do you form teams of people who can go in and help during a crisis or can be the go-to when you need to talk to

people in policy at the state level so there's a lot of stuff that has happened in the last decade there's a lot of stuff still to come all right um I was doing a little bit of visual support um so it turns out there was a lot of confusion about the keynote so I didn't want to repeat much between what I did but very few you saw it so just a couple looking back highlights I think um I'm going to read a very short comment from our day one co-founder who is not here in Vegas this year um so I did not read that upstairs but I will read it now so this is Nick broco c75 used to be at

uh well he he runs well he started thoughtcon I don't know I think he's still involved but um he was at spider Labs at the time and of all the people that I was socializing my crazy idea to before I chickened out um he really stepped up and thought that we could not just use our skills for our employers or for our hobby but maybe for the greater good um so but for his teaming with me early I don't think I would have had the nerve to do these talks and launch this here August 1st 10 years ago um and Defcon the following Sunday we got the main stage after being rejected uh Jeff ma said we gotta put you guys on

stage so uh we got the keynote stage on Sunday morning again so we did this talk twice but um as we grew and changed and focused more on Public Safety human life and a little bit less on his stuff he was less involved but he's always been a good Touchstone um if you're at the launch or and or the uh the original call to action we had talked about our general concerns or the three planks of a platform were um essentially body mind and soul like I was deeply concerned about Public Safety human life and the increasingly connected Technologies uh he was deeply concerned about the preservation of lawful research in our profession and so

was Jen as you heard from her comments and uh and I was also concerned about President the intersection between technology and civil liberties or human um Soul basically body mind soul um as Bo and I really decided to make the front line tip of the Arrow Public Safety human life where bits and bites meet flesh and blood I never wanted to let go of any but we kind of did a turducken where by saying we can help the public safety public good they're like oh my God how can we help and Congress really cared and they would lean in and we said well for one there's a chilling effect on good faith research and they said oh yeah yeah we can help

with that so we took the side door while Jen and others took the front door so I think we never really let go number two for Nick but um here's his remark and I can't speak like he does but it says in the past decade since the creationist movement the world has changed in ways we could not have fully predicted we knew that information security Community wasn't headed to the place we thought they should be in five years ten years or even 20 years by mostly ignoring the side of technology that mattered the most human lives the goal of changing the trajectory of where information security was headed was certainly achieved by the hard work

and dedication of Josh Beau and hundreds of others recent events have shown us that the Cavalry is needed now more than ever but all of us should be proud of the work the uh that the world is certainly a safer place because we exist so I told him I would read that um time permitting I may play one short thing from Andrea matuition as well who helped kind of architect this before we even launched it La professor um if you ever heard me talk about the Cuyahoga River that is hers but basically she's been working on similar problems from the legal and Academia perspective for a long time um a couple accomplishments for people that are brand new because there's

always somebody brand new um one of the things I said day one is I don't want to find and fix a single flaw and a single medical device from a single manufacturer I want to chain hack the rules for all medical devices and when I watched that video yesterday I got very emotional because we did it um I'll start at the end one of the most important achievements we had is we passed the patch act into law worked on it for almost nine years to the day you're gonna hear a lot more about that tomorrow from the FDA session with Bo and Suzanne shorts and her team her incredibly courageous team but that you can't get one of the problems with

cyber security is we always say let's focus on a low-hanging fruit but when you pick all the low-hanging fruit you know what you're left with really really hard problems that take a decade to fix but this crazy idea was let's let's have a Long View let's do the work let's have a campaign let's not focus on activity that make us feel good but results that make us actually safer so the patch Act is essentially is an acronym but um you'll hear more tomorrow but at this point is law of the land that every medical device that gets approved going forward has to have be patchable has to have a coordinated vulnerability disclosure program to work with good

faith researchers without fear of legal reprisal has to have a software bill of materials has to have threat models a bunch of other stuff she actually has specific authorities and now budget to regulate based on the cyber security of the device not just the clinical effectiveness of the device and uh when I first pitched it to one of the staffers that you'll meet tomorrow Jessica Wilkerson she was in house Energy and Commerce and Young and the other uh computer science major she she laughed at me when I pitched that idea as my second briefing ever on this on the first day ever she laughed at me and I I couldn't tell if it didn't feel

cruel but I asked I'm like is it funny and she's like oh no I think the idea is elegant it's just it's gonna take a decade so during my uh belated honeymoon with ADI last December and when I I come back to the resort in my phone connects to the internet and I got like 20 messages saying congratulations congratulations I'm like what what did I do and uh the patch act got squeezed in fought for to the death against a lot of lobbying money to be in the Appropriations Bill uh a senator from Louisiana fought took it to the mat and it is now law of the land so that is uh just one of the crowning achievements

is that even though it's gonna take a long time to drain the swamp of unsupported into life operating systems that are not patchable it's probably gonna take 15 years um all hospitals large medium small rural will benefit from a more defensible resilient future and I just the Symmetry between you know a moonshot kind of idea 10 years ago upstairs and we've actually done it so I called Jessica and I said I beat you by a year because it was only nine years but uh she'll tell you that tomorrow um I'm almost uh done I wanted to give a little bit of a a rundown of some of the greatest hits but um and then 30 seconds

on each talk you're gonna see today and tomorrow so I think um most of our on the one year one we published a five-star Automotive cyber safety framework saying how all systems fail this is a really simple bare minimum way to look at Cyber physical selected cars a year later Beau um had translated that into the Hippocratic Oath for connected medical devices using medically familiar language we built trust with Suzanne caused the first recall of a medical device in history with no loss of life for cyber reasons we convinced her that an unmitigated Pathway to harm was enough she started weaving in the Hippocratic Oath into the pre-market guidance around the same time to bring a device to

Market add a coordinated disclosure to her post-market guidance trust we built got me on a congressional task force for health care in 2016 and 17. which put things like s-bomb into the into the high gear and then into the hands of the shiny one Alan Friedman um when the pandemic hit actually before that um the Mirai botnet taking out the internet for a day scared Congressman so Senator Warner spent a couple hours of bow and I and out of that conversation came these iot cyber security Improvement Act of 2017. um later testified to will herd's committee on that it failed but during the pandemic it was reintroduced and passed in the law so hackers caused a

federal law uh and then when I thought uh in parallel with that when the pandemic hit 10 years ago there was no sisa there was no defensive critical infrastructure Wing in the federal government there were pieces of it in nppd and DHS but sisa has a dedicated agency was fledgling and then director Krebs when the pandemic hit the planet uh called me and said would you like to serve your country for a year so I got to design and Implement what became the ciscovered task force as the chief strategist and work with many brave people in this room that helped keep hospitals as safe as possible vaccine supplies and other pandemic induced strains so that was

traumatizing but also you know proof that people Trust the cavalry's mission as not public sector not private sector but public good and during that time we did document the first statistical proof of loss of life that Ransom attacks can have lethal byproducts and consequences and as Congress digested this and the White House digested this the ensuing actions were that despite significant lobbying that patch Act passed into law our second law uh and uh the White House consulted dozens of hackers intensely to write their White House National cyber security strategy and probably the most chills I get even though I think we're going to blow this Overton's window and fail to act on many of the things that

are queued up just because they're uncomfortable for the private sector is that there was a day when Beau brought us all into the white house uh for hackers in the White House 1.0 with Chris English and oncd and NCS and all the different offices and I realized that there were about a about a dozen people working at the White House that were in and out of the Cavalry influence and they were I don't really say it this this way often but I felt like the Cavalry was a an antidote to gatekeeping and Rockstar culture here and who who had permission to do things and we met anyone and everyone where they were and invested in them

and some of them are now driving the newest part of the White House and writing laws and crafting strategies and you're going to hear from at least a couple of them today and tomorrow um and it's not just us I think the iot idea is turned into UK law and you got a countdown clock that you'll hear about tomorrow we're influencing emea we have Klaus in Europe doing standards work we have the Netherlands adopting these things we have Japan cert all in on s-bombs we have um uh the British are coming tomorrow uh so they want more influence on some more actions that they're thinking of for regulation so what we started is great and it's not

nearly enough we're not going to scale if it took us 10 years to do the things we have in the meantime we're getting worse faster they've been serious questions about do we end the Cavalry transform it into something new uh or combine it with other initiatives to get the critical mass and I'm going to spend at least the next one to three months I ended my private sector work on Friday I'm jumping all in and those that want to help craft this just like we did last time we said let's have a constitutional Congress so let's figure out what the next phase next decade should look like as for today real brief rundown you're going to hear from Emma a lot of

the common theme today is Target rich but cyber poor in Lifeline critical infrastructure so Emma's going to talk about small medium rural facilities for electrical with David Bots which is not just about electrical you're gonna it's gonna lay some framework for the following two socks after lunch which is about food so hungry hungry hackers with sick codes and Casey John Ellis in some manner fashion there's been some complications there and then Paul Roberts and those two and a recently former from the White House Steve Kelly just left a federal service and entered the private sector yesterday and he'll be here to talk about food supply after that next break water water everywhere with our brand new friend

from last year talking giving us an education on the water sector and exposures that water has and how critically important it is to hospitals to food to everything to electrical we'll and then let me wrap the day up with uh Spanky and ion uh crossing paths in and out of federal service towards towards and from hackers uh to show that anybody and everyone can make the world safer irrespective of their origin we'll give you a rundown tomorrow later today but thank you for joining us for the intro and uh shortly you'll get to hear about some Target rich side Rapport in areas like food Water emergency care and electrical which desperately need our help thank

you [Applause] [Music] foreign [Music] foreign [Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music]

foreign foreign [Music]

foreign [Music] foreign [Music] foreign [Music]

[Music]

[Music] thank you [Music] foreign [Music] thank you [Music]

[Music] thank you [Music] [Applause]

[Music] thank you [Music]

[Music] thank you [Music] foreign

[Music]

[Music] baby [Music]

[Music] don't wanna overthink it baby [Music]

some kind of butterfly baby [Music] whip up my appetite don't leave me [Music] so [Music] let's begin oh [Music] baby [Music] baby you'll get me foreign [Music]

[Music]

[Music]

oh [Music]

[Music] thank you [Music]

[Music] thank you foreign [Music]

[Music]

sorry cavalry welcome to this track uh this is electric grid plus plus given by uh Emma and David a few announcements before we begin sponsors we'd like to thank our sponsors especially our Diamond sponsor Adobe and our gold sponsor uh gold sponsors prismaclad some grab blue cat and some others it's their support along with our other sponsors donors and volunteers that make this event possible these talks are being streamed live and as a courtesy to our speakers and audience we ask that you check to make sure your cell phones are set to silent as a reminder the besides LV photo policy prohibits taking pictures without the explicit permission of everyone in frame these talks are all being recorded

and will be available on YouTube in the future all right I'm gonna pass this off now one more word all right I'm gonna since David's gonna be our MC I'm gonna slightly emcee this one at least um plus I'm just a loud mouth so um these are two of our newest and favorite new collaborators um so no pressure uh but not only is am I going to outline some energy specific things but try to remember the things she's saying because they're going to touch on water on food supply on anything small medium rural Co-op or Target cyberpore and thank David as well who's co-presenting because he helped make sure this track happen again for the second year and

will help emcee today and tomorrow so welcome some of the newest change agents [Applause] hi there and that was quite the introduction thank you um today I'm actually going to talk about some of the work I've been doing with electric co-ops for the last few years but I was going to start with an introduction to me that explains why I actually care so much about the electric sector because usually people look at you like why do you care about electrons so much so this is me um the picture on the bottom is my first ever job working for an electric utility it's me wearing PPE that's far too big for me because I was 18 years old and it

was in Scotland and they didn't actually have any women ever on the field crew so I turned up with uh gloves that were too big a giant yellow jacket that my dad probably could have worn and nothing fit me I did that for a little bit decided it was too cold and wet in Scotland for me to continue doing this I just didn't want to went to University worked in electrical and mechanical and electrochemical engineering so I like to spread between different topics and then I decided I was kinda done with Scotland not gonna lie it was cold again I moved to California worked for four of the National Labs currently working at a National Lab moved around quite a bit

because every time I wanted to learn something new I ended up going to a different National Lab to do something different so Sandia worked in hydrogen Berkeley worked in renewable energy and distribution grid Livermore worked in other stuff and I know Idaho where I'm the chief peregrid scientist for various things my more recent past before that was also working for National Rural Electric Cooperative Association which is one of the big trade associations in DC that essentially uh looks after the 900 electric co-ops in the country so my entire career has been in the electric industry since I was probably 18 years old at this point which is quite long ago I'm not going to lie

um so I'm deeply passionate about how our electricity is delivered in multiple countries and that's somewhat how we got to hear and talking about all of this but I also have David here good morning so my name is David Botts I work for the Edison Electric Institute I started in computer science uh my first I say my first real job after being an intern was for a utility company so I work for a utility company in the Midwest of the United States for like 20 years uh before having an opportunity to move to atis Electric Institute if you're not familiar with the electric industry you might say what is Edison Electric Institute and why does it matter why do I care so Edison

Electric Institute is the trade Association for investor owned electric utilities in the United States and a holding company level we've got approximately 63 members at a holding company level those 63 members are responsible for a little over 70 percent of the electricity that flows in the United States of America generation transmission and distribution the energy Market the electricity Market in the United States is wild why because there are 3 000 Electric utilities in the United States Emma just mentioned 900 of them that are that are called co-ops and then in addition there are about uh two almost 2 000 Public Power Municipal power companies most of them are teeny tiny um most of them are distribution only

some of them are huge so let's talk about a huge one ladwp they have uh on the order of 5 million customers so they're super big but most of them are not big most of them are small and we're going to talk about uh bigs and Smalls resourced and under-resourced electric companies for the next 42 minutes Dr Emma so the other thing I wanted to add in here was a little bit of hit here as well because in a ridiculous number we did form there is a long history of how we actually got to having that starting in the 30s but I'll talk about that in a minute but the big thing that's going on just now with

all of these Electric utilities as we are going through an energy transition also at the same time if you've missed them talking about the energy transition the renewable energy climate change that is all happening just now at exactly the same time so what we're really been faced with just now in the electric sectors how do we do everything absolutely all together all at once we've got to make it clean reliable Equitable resilient and secure all at the same time when we're not really there yet so interesting challenge I like to tell stories um this actually happened summer 2020 again it's not really a cyber angle but it's an interesting piece for how do we look at interdependent infrastructure

and why everything actually matters um summer 2020 great time of the year went out for a run uh there was this giant lightning storm kicked off in Livermore California where I lived um huge dry lightning storm that eventually became known eventually caused what was known as the lightning complex fires I was out running went back to my house I was like I am putting my stuff in the car this is how it ends so put everything in my car was ready to go it was also 100 degrees in Livermore at the time Cal ISO the independent system operator had called rolling blackouts for the area because this system was overloaded the grid was in trouble at that point also along with

three giant fires that covered the majority of the bay area which is kind of insane um Livermore was having problems we were struggling the lightning also hit a massive power transformer there was two big Power Transformers sitting in Livermore it hit one of them as it turns out well if you want to talk about cyber protections most of Northern California doesn't have lightning protection on their substations because we didn't have lightning until you know within the last 10 years lightning suddenly became more common thanks to climate change so they didn't have lightning protection on this giant large power transformer worth millions of dollars the Transformer died half of Livermore didn't have power because they were trying to roll Powers

through Livermore as well um at that point the Wastewater plan also failed because they hadn't tested their failover which we told them they needed to do and they didn't do it Wastewater goes down as well at the same time at that point in this series of things happening the fire had turned around the one on the bottom right of the red blobs essentially turned around and started heading for the city of Livermore rather than the rural area so city of Livermore if you don't know already also has a massive National Lab in it which does a lot for confusion research for example the National Lab basically called that we were going to have to evacuate at

that point most of us weren't even there at that point so we're trying to work out what we're going to evacuate from the lab at the same time but evacuation orders all happened at the same time right along that time talking about politics and all of this intersecting Gavin Newsom decided to announce that all of California was going to go for uh phase out gasoline-powered cars by 2035. just announced this in the middle of there being a massive emergency taking up half of California we hadn't had power for three days at this point in various places and we're like so we're not gonna have gas vehicles but we're evacuating a massive City just now that hasn't had

electricity for three days what do you think would have happened so it's a really good example to me of how things aren't connected just now people are saying we need more electric vehicles on our network but they're not talking about the fact we need electrons these small utilities are generally burdened with trying to cope with doing these things like electric vehicles renewable energy without us saying oh by the way it turned out you actually needed bigger wires or you needed more cyber protections or you needed more Engineers even working in your system they get all of this put on them and that's what we're really talking about today so so a little bit of history repeating and

a rather threatening picture it's coming electricity for you this was actually an advert in the 30s that came out um I actually likenessed a lot of the threats we hear about cyber as well like cyber's coming for you this is the same thing they did telling everyone on electricity was coming but it was mostly to the rural areas at that point so in the 1930s I believe only one in ten rural Americans actually had power we were going through a revolution at that point electricity was coming to all the big cities people were changing their lives based on this but we still hadn't actually got power out to the rural areas in America which cover the

majority of the landmass of the country um in 1936 uh they brought in the rural electrification act which essentially allowed low-cost loans to go to Pharma low-cost Federal loans to go to Farmers so that they could Farm electrical co-ops and work with larger utilities to actually get the lines run through contracts essentially to say hey we will serve everyone in the U.S power at this point so these social contracts came into play for electric power these sorry electric electric co-ops were formed they were all not for profit but they were essentially the local community where all members of the Electric Co-op and that's gone on to this day so this happened in 1936 and since then this has

continued we ended up with the rural electrification Administration which later I think evolved into USDA and the rural utility service again when we talk about electric zeros we all think of Department of energy turns out USDA actually has a huge play in how the rural electric co-ops work as well most of their loans and financing actually come from the Rus program instead of doe so when we're talking about these small rural utilities that's who we're talking about these electric co-ops not that it wasn't controversy about this this I'm looking at David just now with the holding companies um there was seven major holding companies in the US at this point um as part of this deal they were trying

to break those up because they held monopolies over most of the electric service in the country um they created these fancy cartoons they were calling it the death sentence uh to the electric service and So eventually this is where the Real electrification ACT came from okay so how did the grid actually evolve from my eyes like how did we get into this mess it's a very simple picture so talk about 1936 we've got this nice dumb grid it works there's no fancy Electronics there it's working the electrons flow in a certain way it was remarkably reliable for what it was um so we created the dumb grid fantastic roll on 2008 we've decided to do the American

reinvestment act the smart grid program comes into play this is where we started getting all those devices that could talk onto the grid primarily Smart Meters ended up being one of the big things and solar energy those Smart Meters actually weren't as smart as everyone thought as it turns out because most of the customers actually rejected their being mostly Smart Meters connected to their house so we ended up with very fancy smart readers that couldn't do anything which was great then we had the clean grid that this was around 2008 as well the Sun Shop program started for doe which had a goal of a dollar per watt essentially we already met a dollar per watt of solar they

actually achieved that goal in 2015 but side note what that also meant was a lot of the solar industry moved their manufacturer offshore at this point and roll on to 2023 where we have certain problems with supply chain so decisions we make now generally for this sector roll on for the next 20 years then we had the physical and cyber secure grid we're around 2012 2013 at this point where we're starting to worry about physical attacks we're starting to worry about things that happened in the Ukraine now everything has to be physical and cyber secure we change direction every few years and what we're worrying about on the electric system apart from that we're worrying about

everything then we had the smartest grid we decided we've done it everything's super smart we're doing really well with the physical uh the physical and cyber secure grid you mentioned 2013 a big event happened in 2013. Metcalf Metcalf happened uh I'm sorry we've got a mandatory tie policy here in the uh there's your chocolate chip cookie [Applause] so far be it for me to defy dress code Mr Damon thank you um 2013 the Metcalf substation attack occurred where unknown and they are as as to public records still unknown adversaries um they first physically attacked two underground communication vaults which which they're little literally underground the the covers are covered with dirt and leaves and branches and garbage

they opened up the communication vaults they cut the fiber optics communication cables not in one volt but in two volts one one right after another and then an unknown number of parties started to shoot um at Transformers that were in the Metcalf substation so that is in 20 13. and we're going to talk a little bit about how old is New Again so moving on a little bit we got to this interdependent grid which I'm moving pretty fast at this point all the blocks after physical and cyber secure essentially happened in the last five years we got to interdependent which is where we get to the Colonial Pipeline and various other things that happened there fun fact is when Colonial happened

it wasn't just that we depended on gas and that was the biggest worry when they had to restart that pipeline we had to pull out plans that happened during Hurricane Katrina on how to restart that pipeline because all of the pumps had gone offline the pumps are load starting a pump requires a whole bunch of power and affects your voltage pretty badly at the time of Hurricane Katrina the daisy chained diesel generators together to actually restart that pipeline because they didn't have the utilities working so to get the pipeline to start to get the utilities to work they were in a giant cycle so they Daisy chained these diesel generators together to get enough power and support the

voltage to get the pipeline back on we were back there except this time at least we had electric power um so they got everything back online but again this was the circle of life that we had on the grid is when some big thing goes down we end up stuck in a circle of problems but daisy chaining diesel generators worked out um then so then we have the uh I spoke too soon it turns out everything's interdependent everything's not that secure and we have a big problem with physical security and as it turns out when we try to make everything super smart we also left ourselves open to a whole bunch of problems uh moving on really quickly we got to

Shields up uh 20 21 22. I'm getting dirty looks but it's okay we got to Shields up which is when we just started yelling at small co-ops that they needed to put their Shields up but not necessarily telling them how or why or how to do it we did tell people there were certain instructions they could give that those were coming from DHS not necessarily from doe and everyone was told this repeatedly and they got exhausted I'm not gonna lie there was a number of small utilities calling me in my last possession saying how romantic are Shields up when we can't hold up our polls most of the time right now so then we got into climate impact

um I think 2023 we had the most number ever of billion dollar weather events um causing damage to the Electric System including one that just happened about 12 hours ago in the Northeast where there was essentially a huge storm if you look at the news just now there's pictures of entire distribution lines just laying flat on the ground because a tornado went through them um then we get to more physical security also again people were shooting up Transformers in November that was that was a wonderful terrible event that was going on again we've directed everyone away from Shields up now to looking at physical security so everyone got distracted Again by the squirrel and then now we have money we have 1.2

trillion dollars everyone's trying to push into the electric grid for this energy transition but what we're going to do with it is a big question so where we are now um this is the picture of the three big types of Utilities in this country there's investor Owens that's his fault um there's publicly owned and then there's the Cooperative utilities but if you look at this map the large majority of the landmass is actually served by the smallest utilities where that's an issue the grid doesn't enter depend the degree is an independent little pieces of utility that all operate by themselves if you want to get from point A to point B most of the time you're

going through five different utilities to get there and they could be three different types they could have three different regulations and three different styles of security that they're working on it's a large pathway of systems and we're having issues with how that's defined so nice where everyone gets a slightly sad face and feels bad that they're in Vegas there's around 21 million people in the US that ow over 700 on their energy bills that's one in four houses that don't actually aren't able to pay for their power with around 12 percent of those reported keeping their houses at an unsafe temperature that's pretty bad when hurricane Yuri happened in a hurricane Yuri star Murray happened around 200 people died because

they didn't have power so they didn't have heating at the time it was freezing so they didn't have heating they died for various reasons because they didn't have power some of those people now 12 of people are keeping their houses too hard or too cold because they just can't afford the power why this is important our electric bills and what we do with security uh you pay for it we pay for everything that happens that the electric utility has to do there's no magic pot of money coming it's your rates that pay for it so whereas we might be okay there's around 25 percent of people in the country are not okay for paying for

their power so how do we make this work how are we reached a point of no return um how do we actually make this work for everyone without burdening those that can't afford at least other interesting part of this picture um the co-ops serve 92 persistent poverty counties so those land masses in rural areas usually are where some of the persistent poverty counties really are if you look at this map also it's also where a lot of our military bases are so about 110 co-ops serve critical military installations I believe there's a lot more ious and Unis as well but we have this group of things coming together that makes our issues with criticality pretty interesting

one of the really interesting public policy issues is the following is it right is it appropriate to demand of an under-resourced rural Cooperative that they themselves are responsible to defend themselves against nation state and state-sponsored actors are they financed for that is it going to show up in their rate base but it and it's just it's not just the co-ops it's also true I mean I think the question is a reasonable question to ask for utilities of any business model so both Public Public Utilities you know the city of Manassas public utility that is that they're responsible for the local distribution in that area or invest your own Electric utilities is it the right thing to demand of

electric companies to be responsible for defending themselves against nation state and state-sponsored actors so another intersect of who delivers your power I guess it could be a game show quiz or something and are they considered critical in the sector because Josh show his picture earlier of all the different sectors to like how they're split up the electric sector to me is one of the most complicated ones in that um you had a line Josh had a line that goes from doe to again to the electric sector again there is also USDA and doe and USD and EPA that currently have 10 billion dollars for clean energy for utilities to apply for to put on their

system that's a lot of money but but it's not ran by Doe and the security requirements aren't necessarily being defined by Doe so clean energy is electrons going on to your system it's an interconnected system to your operational Network do we have requirements for that yet or are we just running in into the clean energy world without actually working on that so I the little baby picture is basically showing I think if anyone has kids they probably remember that there was a baby formula shortage in the last year it was pretty bad um that baby formula Factory was actually in Sturgis um I was doing a little math of where that was and what it was served by and

who serves it and to try and work this out believe it or not with 20 years of electrical engineering I still had to sit and draw this out on a map because it's not that easy so essentially that baby formula Factory is served by a municipal electric company which Wheels power through a co-op which is served by an IOU for transmission so there was three different Utilities in play for what that was it was water that was their primary problem but if I let any one of those electric entities was down they'd have had exactly the same problem and probably not water as well so the criticality piece for me it becomes defined by The Entity by who they're

owned by by what their boundary is on their system I'm not sure that's working anymore for the electric sector there's too many PCs that are connected together that is relatively unique to the electric sector those critical functions that we've looked at make sense in the Metro areas that's great but not necessarily in rural because again we have a rural hospitals we have our other things that are out there we have mysterious baby formula factories that happen to sit you've never heard of until you didn't have baby formula these are all sitting in these rural areas served by multiple different utilities who are all struggling essentially to meet requirements for security so David already mentioned the military

functions again private defense from the baddies I have worked with a number of the locations that serif militia bases or have contracts that keep them alive to serve those bases to meet the requirements they have and they all Wonder like what's going to happen when the next requirement come and they can't change their contract with the military base to increase that rate as well so people aren't necessarily working together to fix this problem which is something I would really like to work on solving so this is the other scary slide that makes people have a sad face there's over 900 facilities in this country that don't have a single person working on it not even not once again 900 over 900

thank you less than one person and that person might not even be the person focused on it um I was doing instant response with a 7 000 person utility as in the serve seven thousand customers they had a three million dollar ransomware event happen where that Ransom was actually on their outage management system they were in Texas there was another storm heading for them this was right after storm Yuri and I get a phone call from their singular person to say they think they're about to be fired they're crying and I'm instead of on instant response immediately I was on Mental Health response so I was more worried about this person than I was about what was

going on because they were going to be okay but she was distraught and she was told she was gonna be fired by her bored that didn't understand what had happened even though months ago she'd said hey I think we need to spend some money to fix this and the board was like nah let's not do that um but she was working with the outage managers it turned out she was also the storm responder funnily enough and the communicator for this utility in particular so she was going to have to first of all tell the customers they had a ransomware event while also responding to it while coordinating their insurance which turned up to try and help while

also being threatened to be fired while being paid seventy thousand dollars a year for doing this um but she was dedicated to her community and that was part of this this is something I think we should work on um her dedication to her community meant she was taking it from all sides at that point but also standing there trying to stand up this utility and keep things going while everything was falling apart in both her life and the utility itself so this community angle is something I'm very interested in like how do we look at these areas differently instead of them being electric Hospital gas food is there a way that we look at them as

actual units of importance that can work together on fixing this um again I've covered most of this but salaries are lower in rural areas we can't they can't compete like there's in most utilities they actually can't compete if you've ever been paid by utility or a trade Association um but yeah most of them are using managed Services I know some that are using the Geek Squad for for their utility um and again we've got infrastructure dollars flowing there's OT providers like never before some of those are actually shadow oit as in people have got this distributed resource management system installed it's sitting on the cloud yay look at us we've got a cloud managed distributed resource system they

don't have an OT Network they never had one they think it's been managed by the cloud provider they have a generation of transmission entity that runs their scada but they never actually had a cloud entity before doing this so they don't have the people to actually support it so the other pieces we've got mixed messaging from federal funding I mentioned the 20 or yeah 20 billion dollars just now from EPA doe and USDA um what their Federal funding went towards from cyber security in the same program for the electric sector in particular was 50 million a year so we've got 20 billion going into clean energy but then they also put 50 million towards cyber security so if you can do

the math and the percentage that's what we're telling them is most important like that's the message they get is that's how much money your provider should put into cyber security for that system we do have all of these low to moderate income programs also that are saying hey we're going to give everyone that can't afford their electric bills some solar on their house great we did that really well when we built a load of Housing and flood planes um so now we're at the point where we're putting solar in people's roofs and basically creating another cyber flood because we're making them responsible for their own cyber security as well so again you'll never be able to read this

but the whole point is who's responsible for keeping the lights on just as a factor um distribution system operators Independence system operators transmission system those are the only people actually legally responsible for keeping the lights on who isn't responsible for it is the energy Market providers the distributed resource operators your cloud provider the manufacturers and implementers the installers of the solar in your house have zero responsibility for this and they definitely do not want it but they're becoming response they're becoming what's the word responsible for installing the devices not responsible for keeping the lights on if that manages is somehow spread into a utility system so the number of providers is impossible when we talk

about regulation for distribution this is why it's not because I don't think distribution should be doing better with their cyber security is because I don't know who is actually responsible for keeping the lights on at this point when we look at this system so good example that recently happened um who's actually responsible for securing the grid um when when it's behind the meter essentially your solar on your house I worked in Hawaii um at some point they yeah I know I'm super lucky um at some point they actually installed a whole bunch of uh inverters on the system the part that connects the solar to the grid um because one particular manufacturer was super engaged with the particular

with Oahu essentially everyone ended up with this one particular manufacturer inverter how many how many different manufacturers one one just one one um everyone ended up with these inverters across the island um again that was great until there was a firmware push that went awry and everybody's inverter started misbehaving and the whole system had an issue but again this recently had an impact from the cyber's perspective um again we're asking people to install solar on their houses about a few weeks ago and phase themselves had a vulnerability in their system their the alert that came out one of the guidance PCS was make sure your firmware or your firewalls are up to date for the system

and I'm like in your house like you think I have firewalled my solar like I I might have I don't know anyone else that has so the guidance was for the customer behind the meter to somehow secure their devices on the system and that's that's not going to work especially your grandpa your grandpa needs to update his face inverter and his firewall yes okay that that should be pretty straightforward that should be easy you know you're getting that phone call so last sort of example of where I think we have challenges again I do have some solutions I just like to rant about the problems initially um you might have gathered my accent in Scottish so bear with me on this story

um this is a load curve from uh the UK um a few years ago Andy Murray the tennis player was winning Wimbledon and Scotland's pretty pathetic at sports so when when Andy Murray was winning the whole country was super excited like everyone's like he's winning he's a local hero we're doing well so how why this is important um everyone in Scotland a couple points and that day sat down at the same time and also turned on their tea kettle exactly the same time so I'm not joking it really happened like what they did when they all turned on like there was two different sets where Andy Murray was winning they broke for the set the tea kettles

went on they caused a frequency swing because the whole of Scotland did this like one half of the country and so this happened twice during the day and then if the red line is essential what was happening the blue light is a normal day the red line was what was happening during Wimbledon the blue line when the red line dips dramatically that was the very last set where they thought he was going to win and everyone just sat down and so the country lost 800 megawatts of load at the same time because all of Scotland stopped so the UK isn't that big a country compared to the US but what I'm saying here is our behind the meats are load is really

important it can have massive impact the humans actually control it and we don't really secure it that well at all like I know who you all are I'm fairly sure you can break into a tea kettle when it has a Twitter feed on it so we're causing some issues here by making everything smart but making the customers responsible for it and the small utilities so are we creating a cyber tsunami is my my question by pissing lots of federal funding into our smart grid into our clean energy into our smart kettles for some unknown reason I just bought a house and my Kettle is actually smart and I don't know why Emma I'm sorry I didn't know what to do the guy had it

in the house but can we fix it is the question and I really want to as you can gather I deeply care about the electric grid and all the things it serves but can we fix it it's what I've been asking myself for the last few years so what I really think to be honest is we need to go towards an old Disney approach where there was this contract to try and serve those small Utilities in a way that made sense or those customers in the rural area we need to do the same thing from a security perspective it's not really about what products or what profit anymore it's about making sure they aren't the ones

that are affected by this and working out who's actually responsible for it so again I also believe in climate change and I believe we should be securing the path to Net Zero and we should be going towards our renewable energy future I think we just should be doing it right um so securing the pathways to get to that Net Zero future or something I think is really important also securing the people that do it again all these renewable energy providers I worked for one I don't care about security when I did that I wasn't in this field we didn't have requirements to do it we were just evaluating if that system was bankable for the next 20 years for a

particular developer not once was I asked a question if it was Secure not once um then secure the technology again we can't be making customers responsible for updating their firewalls to stop the energy system from crashing so secure the technology is really important but again we need to look at building digital resilience through these Partnerships from the ground up I really want to look at how we redefine critical I know some of this is going on but for these particular regions by region or utility class isn't working anymore or Sorry by utility class isn't working anymore I think Regional or boundaries around locations would be more valid for what they're serving I think we need to look at how we design

the future grid like okay we've got a grid it relatively works sometimes most of the time um we need to focus on getting that future grid secure because it's going faster than us um consider the defense communities and also well again build from the ground up so decision support has been one of the biggest challenges I helped a number of utilities work on what they were meant to deploy for for their security posture I couldn't make Hayden or hair of a lot of what was available on the market I yeah I I'm a smart person I there'd be 10 different products and it would be really hard to work out which one they needed for what because they were all

pushed with different words decision support is something we absolutely could and need to provide to the smaller locations like what do you at the ground actually need to make your system secure one of the things I did was um I basically took the DHS core performance goals I had worked with a number of co-ops like which ones of these do you think are the basic 10 that your Co-op should be doing and we created a program for them that essentially was hey we're going to give you a shiny coin if you do all 10 of these and it worked like people were excited by it they got a reward for doing these 10 things and it

gave them an actual Baseline to even get to the next level which was these goals we need to set the bar in a way that makes sense the bar is a spinning plate from my perspective right now we don't have a consistent discussion on what is the bar for security in this country it's I know it's constantly moving but we need a bar for people to actually get to that doesn't move 10 minutes later I do think the public-private Partnerships we need to consider taking those that profit the most out the equation for what decisions are being made for what's necessary on the system um again that's not popular I'll probably get yelled at by 20 people for

saying this but I don't necessarily think profit should be in that partnership as uh for these not-for-profits themselves and again holistic support approaches for local communities um that one person that works at the utility who's the the singular I.T person or the 0.8 of a utility I.T person that's there they're probably also working for the hospital they're probably also helping the small businesses in the community if there's water they're probably doing that too because they're doing everything so how do we look at backing up those people in a better way that accounts to the fact that they're not going to have independent people at every location I'm gonna move on to this one I despite I work at Idaho I'm I'm very

excited about cyber-informed engineering I had to say it but I was excited about it beforehand because I think it did give people an opportunity to improve what they're doing now without worrying about different pieces one of the things I was thinking about was how to design better principles for the interconnection of Renewables that does include security but isn't necessarily saying again the customer needs to secure or the utility needs to secure it I mentioned the end phase problem again that's local that goes everywhere now everyone is keeping up with the Joneses and solar in particular your those installers that walk around the neighborhood that I terrify every time they come near my house um they basically they'll sell a single

inverter to the whole neighborhood I think one of the analyzes I've actually done is if you just change the inverters out in one feeder and made a rule saying you can only have 10 of one particular brand we'd actually be in a much better position if all of those trips off all at the same time it it there could be a massive Cyber attack on those inverters we'd still have power that would be I did this analysis for one large state that I won't talk about but we'd still have power we wouldn't cause a voltage event everything would recover the system would be okay even at very high penetrations of Renewables that's a basic principle to apply that is very

difficult to get through just now so even just looking at more heterogeneousy on the network would be helpful as a whole and that's where the cyber-informed engineering and secure by Design principles can come from so again the other piece that people have talked about we're stuck in the middle of this right now one of the most reliable and resilient things we could do to the electric grid with all of these devices is connect everything as in if you connect every single device and make them all work in concert we actually could have a fully renewable operational system electrically electrically yes if you connect everything and make it all work in concert we could have a very

insecure system that falls apart tomorrow so we're it depends who you're talking to what we do I'm joking with my Meme here about uh from Forgetting Sarah Marshall where there was a surf instructor who said do more do more no do less that's where we're at with this like watch how what is the optimal optimal amount of connectivity that we need again cyber formed engineering I won't go into it too much but I do think we need to set future data standards for cyber security monitoring in particular um data quality and quantity I worked in sensors I developed a sensor it before we even got as far as it being installed anywhere we wrote a taxonomy that's not

able to see but it was what grid state are we actually trying to measure with this sensor like what's our expected outcome before we actually develop a new device to measure it um I've noticed in security we tend to see let's just monitor everything and we'll work it out afterwards we've never done that in the grid before ever like that's not a thing we don't monitor everything we don't monitor distribution Transformers the recent GAO report that said all distribution Transformers are insecure have never looked at a distribution transformer which is concerning um they're not they are secure because they have no communication and no Management on them at all they're just these devices sitting on the top of

poles you can shoot them but you can't necessarily attack them from a cyber perspective unless you go through an interconnected system so data fusion and data taxonomy to me are Big Technical PCS that would actually help us Define what's needed a last point on data science um one of the bits of research I did in the past which is completely not cyber security but is relevant here um was looking at Healthcare data I've spent an unfortunate amount of time in the hospital um I was looking at Healthcare data if you can fuse together MRI data measure data voltage data infrared data together you can actually help transform Healthcare outcomes this was a big thing we did with multimodal multivariate data

transforming those outcomes at that time was to prevent deaths it was to prevent the death of actually in this case veterans with traumatic brain injury satin was doing some of this work and I was like humans kind of look like Transformers in this regard you're like you're looking at me like oh God but humans in this case actually looked like Transformers and we used that to create a new system for evaluating failure on a Transformer that you can't see with measured data so it's called incipient failure from my perspective a Cyber attack to a grid person looks like incipient failure we can't see it the lights are still on we actually wait for Transformers to fail believe it or not

the smoke signal is usually the first sign that it's failed so data fusion and breaking data silos is really important for our future again Community Workforce engagement something that's working really well and I think needs to continue is they've created these centers for regional engagement I think supporting those is really important they're a few different states Arkansas is one of the big ones just now who created a center for regional engagement it's bringing in students people that actually don't need to be paid that much sorry but the students are coming in to help be the sock they're helping being analysts they're helping learn their trade from the bottom level but they're also helping support this has got

multiple cooperative and Municipal Utilities that are bought into being part of this Regional engagement and that means they're getting support from this Center that's also federally funded which is helpful from the state funding that is more than the federal funding for cyber security so they're being helped to do that with these centers I think supporting those has been a really big success s and something we can continue to do in the future so I've probably gone slightly over time but I'm excited about opportunities and challenges and the big questions and how to improve the electric good I do have hope it can get better I do have hope that there is a way to fix it but it's

going to take more than just people that work in the electric sector to do it so that's it thank you [Applause] so we've got a couple of seconds um for questions uh yes and so uh if you got a question come on up to this mic and uh or the mic will maybe come to you I might have to turn up the mic because I turned it down thanks

it's multi-talented I will start singing until okay I'll stop so first of all thanks a lot awesome the question is you you said you know you wanna restrict the number of of vendored devices in in certain areas it make me think about something that happened in Germany a few years back where it was more on the other side there was a free scada system in the cloud and everyone started connecting their solar systems so that's free skater system causing the same type of issue so I think we need as an industry to start thinking you know how are we managing those kind of things yeah and also who's responsible for it because the free cloud system and the

solar industry like they're responsible for it but they're not responsible for security so so it wasn't uh one more thing a great presentation and uh in terms of the point around cyber informed engineering uh what we also may need to be thinking about because we talked about criticality at the start of the presentation is more around consequence driven engineering as well because that's going to help shape up how all these individual sectors and the impact it has on global societies to kind of Drive the risk impact to drive how consequence level engineering could be reinforced in small and medium scale organizations so that could be one of the critical factors going into the future what's really cool about the electric

grid is we actually do consequence before we actually connect anything we just don't do it from a cyber perspective we look at consequence of interconnecting any renewable device we know what will physically happen if it just disconnect from the system but the rest of it is still a question so hey there great great presentation thank you so much um you mentioned on the solar side um you know that we were successful in expanding solar capacity but a lot of that was as a result of offshoring uh production of the critical pieces the panels and the supporting infrastructure as I understand it talking to some of my cyber security friends a lot of that sporting infrastructure is talking back

to the country of origin for many of that for a lot of that infrastructure uh presenting some um challenges in terms of cyber risk um is that from what you know is that something that's on the radar of folks who are looking at you know overall risk of uh the system uh and what ideas are floating around on how to deal with that so it's absolutely something everyone should actually be talking together about um it was not well known by a lot of the utilities for a while that their electric inverters would talk to their country of origin some of us knew it some of us have played with them knew found well there was a radio sitting

there that was talking back to wherever um interesting fact is I also bricked one of those devices and then made a support call and they managed to unbrick it somehow with that radio device so they have control features in some of them also which is concerning um yeah I mean it's a the thing is we have to balance how much it costs versus what it does if that makes sense so it's a difficult challenge but people need to actually talk about it at this point and accept our supply chain is interesting at best for this and part of Emma said something that's really important and that is as a general matter we have not priced resilience

into the equation of items that are purchased maybe there's an opportunity to change that and to think about resilience and think about um I'll say near Shoring some of these things for a lot of reasons the United States has gotten out of the business of manufacturing I think there's a great opportunity to um if not if not bring back more manufacturing to the United States which I think we should do but there are there are allies that are close that can also do manufacturing so it doesn't all have to come back to the United States but it can definitely go to friendly countries as opposed to countries that are not Scotland like Scotland so the question just how can we push

those changes like for example data standards in the government can take a really long time to be pushed through how can we as a community help push that to happen also how can we get smart people like yourself to be helping to make those decisions because it just seems like there's a lot it's hard for us all to make this happen I'll very quickly answer that before we say we need to measure something everyone needs to actually tell others what they need to measure from my perspective if you're an expert in analyzing data work at what you need to measure and tell people that is what you needed like because not everyone understands everybody else's like

expertise I can give you a physics presentation on the electric good and you'd all want to kill me but I couldn't tell you exactly what data to measure in some of those cases so that's my biggest comment would be actually people just telling people what they need as opposed to trying to sell giant data sets to everyone so I'm going to try to squeeze in a combo of three questions because we might get the hook um number one are any countries doing it well like differently than us but could be a model for us to repeat number two the 900 number felt small uh was that just electrical municipalities or does that include the water and the other things and number

three of three um isn't it often the same person doing the electrical the water Wastewater the healthcare the fire and what if they get sick because there's no Municipal hospitals because all the small medium hospitals go away 900 was electric co-ops so I'll go with part one um so it's 2 000 for municipals yeah um 900 is also the number it was actually mixed for the number of co-ops immunities that don't have a person that was the the number um yes if that one person gets sick it's a disaster there was actually a co-op or a utility it was a co-op that had a person go out having a heart attack um and that heart attack their father

was having a heart attack they were leaving there was an email sent to say hey I need to leave my dad's had a heart attack and they were hit with ransomware at that moment um that was lovely it was horrible to hear but yeah that when that one person goes out sick and I hope he got the hospital in 4.4 minutes or whatever the number was see I took it um but yeah he was okay um but there was the repercussions of that were terrible so there's a lot of opportunity to look at resilience across that whole Space what's entrepreneurial things have been happening that that one person ends up forming their own little managed service provider that then

manages to hire more people in the community and builds up from there but they need support to do that right because once they do that they're outside the boundary of uh reliability for the utility and then we're in a circle so they need support right and and if a city operates literally both electricity and water then in fact it it can actually be the same human and they're super busy

which country is doing it well from a distributed resource perspective Australia's started to do really well with actually considering security requirements for that so if anyone's Australian good job

it was told we had a room announcement from someone in purple is that you okay

um

I know

affirmative consent

okay let's thank our speakers [Applause] I hope you can see a small glimpse of why we love them so much um after lunch we have hungry hungry hackers and food supply for Farm to Fork I think we're calling it so come back hungry uh and uh lastly there's an s-bomb lunch see the shiny one follow the shiny one if you want to talk about software build materials

in the afternoon as well

all right wants needs and fears about s-bomb Becks and supply chain it's a piazzo lounge yeah um I think it was there [Music] [Music] thank you foreign [Music] foreign

foreign [Music]

[Music]

foreign [Music]

[Music] thank you [Music] [Music] foreign [Music] thank you [Music]

[Music] thank you

[Music] foreign [Music] foreign [Music] foreign

[Music] thank you foreign [Music]

[Music] [Applause]

[Music] thank you [Music] today [Music] [Applause]

[Music] foreign [Music]

[Music] appetite don't leave me alone [Music]

[Music] baby

giving me the rain some kind of butterfly baby [Music] [Music] but I don't wanna miss you baby [Music]

[Music] foreign [Music] don't leave me alone baby

[Music]

[Music]

oh [Music] oh

oh oh [Music] [Music]

[Music] thank you [Music] foreign [Music]

[Music]

[Music] foreign [Music]

[Music]

[Music] foreign [Music]

[Music] [Music]

[Music]

moving up

[Music]

[Music]

[Music] thank you

[Music]

foreign [Music]

[Music]

[Music] thank you [Music]

[Music] Hallelujah [Music] oh yeah [Music] thank you [Music]

foreign [Music] foreign [Music] foreign

[Music] thank you

[Music] thank you [Music] foreign [Music] foreign [Music]

foreign

[Music] foreign [Music] foreign [Music] thank you [Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music] laughs foreign [Music] [Music] thank you [Music] all right [Music]

[Music] foreign [Music]

[Music] foreign [Music] foreign [Music] thank you [Music]

[Music] thank you [Music] [Applause] [Music]

[Music] foreign [Music] [Applause] thank you [Music] foreign [Music] foreign

[Music]

[Music] country

don't wanna overthink it baby [Music]

some kind of butterfly baby you look at me [Music] don't leave me [Music] but I don't wanna jinx it baby [Music]

[Music]

[Music] thank you [Music] baby [Music] don't leave me alone baby you'll get me in the rain some kind of butterfly baby

[Music]

[Music] oh oh [Music] [Music]

[Music] foreign [Music] foreign [Music]

[Music]

[Music]

[Music] foreign [Music]

move it up

[Music]

[Music] thank you [Music] Hallelujah [Music] [Music]

[Music]

[Music]

[Music]

foreign [Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music] foreign [Music] thank you [Music]

[Music] oh yeah [Music]

thank you [Music] thank you [Music] foreign [Music] foreign [Music] foreign [Music] awesome [Music]

[Music]

thank you [Music]

[Music] foreign [Music] [Music] foreign [Music] thank you

foreign [Music] [Music] thank you foreign [Music] [Music] thank you [Music]

foreign [Music]

[Music]

[Music] foreign [Music] [Music] foreign [Music] all right [Music]

[Music] thank you

[Music] foreign [Music] foreign [Music] foreign [Music]

[Music] thank you [Music] foreign [Music] [Applause]

[Music] foreign [Music] [Applause]

[Music] thank you

[Music] thank you [Music] baby [Music] appetite don't leave me alone [Music]

[Music] baby

giving me Wind and Rain some kind of butterfly baby [Music] [Music] oh but I don't wanna miss you baby [Music]

[Music]

maybe you'll give me [Music] away guess I'm gonna butterflies [Music] don't leave me alone [Music]

[Music]

[Music]

oh [Music] oh [Music] oh oh [Music] [Music]

[Music] thank you [Music] foreign [Music]

[Music]

[Music]

[Music] foreign [Music]

[Music]

[Music] foreign [Music]

[Music] [Music]

[Music]

[Music] moving up alone

[Music]

foreign

[Music] foreign [Music]

[Music]

[Music] foreign [Music] foreign [Music] oh yeah [Music] thank you [Music] foreign [Music] foreign [Music] foreign

[Music] foreign

[Music]

foreign [Music]

[Music] [Music] thank you [Music]

foreign [Music] foreign [Music] foreign [Music] foreign [Music] foreign

[Music] foreign [Music]

[Music]

[Music] thank you

[Music] no no no no no no no no no no no no no no no no no no no no no no no no no thank you

[Music] all right [Music]

[Music]

[Music] thank you [Music] foreign [Music] thank you [Music]

[Music] foreign [Music] thank you [Music]

[Music] [Applause]

[Music] please [Music] stop [Music] [Applause]

[Music] thank you [Music] foreign

[Music]

[Music] baby [Music]

[Music] don't wanna overthink it baby [Music]

[Music] baby you'll get me everything don't leave me [Music]

oh

[Music] thank you [Music] baby [Music] foreign [Music]

[Music]

[Music]

oh [Music] oh [Music] my God [Music] thank you [Music]

foreign

[Music]

foreign [Music]

[Music]

foreign

[Music]

[Music]

moving up [Music] foreign [Music] [Music]

[Music]

[Music]

Move Along

[Music] thank you [Music]

[Music]

foreign [Music] [Music] foreign [Music]

[Music]

[Music] thank you [Music]

[Music] happy birthday [Music] thank you [Music]

foreign [Music] foreign

[Music] foreign

[Music]

track hungry hungry hackers we were delighted to welcome and invite sick codes and due to reasons travel and otherwise Mr codes was not able to join us today however we do have Mr Casey J Ellis to to be present and help us walk through some of the issues related to challenges in Security in our food uh ecosystem Casey is the chairperson founder and chief technology officer of bug crowd as well as co-founder of the disclose.io project Casey has been in the business for over 20 years has done amazing things and we look forward to learning more about what is going on here this will take about 20 seconds to light up so but it will light up

I'm sure I will all right Casey who is not sick tell us where we're gonna go in our next 45 minutes thank you thank you so thanks everyone for coming um has anyone seen the we are all sick codes meme um of the sub meme on on the twitters and if you haven't it's all good um so basically uh you know I think the idea is that um when we learned that sequel is going to have some trouble uh getting into the country Josh and I decided over a couple of beers at a cabana yesterday that um you know all tall redheaded Australians are basically the same if they work in this industry and it's it's actually not just that um

when sick did his first presentation at Defcon last year and actually started presenting security research um it's the first time his face had actually been on the internet and people thought he was me um so we ended up swapping out it just turned into this whole fun little meme um and uh you know the interesting part and why I thought you know what I could actually probably have a go at just running through his content and and getting it out there is because I was actually involved in a bunch of the stuff that um that he's going to talk through uh in my capacity is you know bug crowd disclose like all of all those

sorts of things so I am sick codes for the purpose of this presentation um what gets really funny so yeah if if this is like you know hopefully that uh explains any jankiness and in how this all kind of plays out but it's a cool story and like to me what it is um obviously the the title is is um food specific right but to me what this is is really a story about security research like fundamentally changed the perception of safety criticality in Industry um which is something that's very near and dear to my heart so we're good to go is everyone confused sufficiently at this point in time I know I am all right

let's rock and roll and let's honestly let's have some fun with this because it's going to be a bit strange but I'll get through it and and you guys can ask questions the other thing as well is that um Paul Roberts gets a shout out there's actually a panel following this and that's going to be more of a conversation around kind of my my personal opinions when it comes to this stuff so this is partly representing you know sick and his point of view on things so all of those and now I'll get into his disclaimers told you be fun um this is independent research all security vulnerabilities reporters to vendors nothing represents employer partner Association neither

personal present other than description in the presentation nor does it represent necessarily Casey um that's a good call out um slides of CCO Etc all right so basically you know stickers are like a pretty prolific and fairly recent uh security researcher and in terms of you know when he popped onto the scene I actually met him for the first time when he docks me through odesk um in in 2020 that's a fun story that we can tell some other time um but since then he's actually had a pretty prolific vulnerability disclosure and security research career and a lot of his work is actually you know become it's I don't think that these things are complete in terms of their impact but

what it did was it precipitated and capitalized a bunch of changes in thinking um and that's kind of what we're going to go into here and this is where you can find me slash him should be some sort of drinking game for whenever I get that confused right is it but really yeah as I said this talk is is about you know how security research can actually change um cause change and and his views on that right so the birth of of his security research the birth of my all right is it his or Maya what do you reckon vote both all right you guys are tracking it's all good I'm twisting myself in knots up here but you guys are fine so

the birth of the idea um for getting into agricultural research was actually Paul um uh seeing I think a comment that Paul made uh does John Deere have any cves not sure what precipitated that exactly and I'm sure we can get into that later but that was the origin of it so Paul is um very much focused on the right to repair angle of of all this stuff I think what happened in in six mind at that point in time was the idea that like writer repair is actually inherently less friendly from a security standpoint and if you add if you combine that with the safety critical or a nation like a national security um critical industry you've got a real

problem at that point potentially so sick goes off and does some security research finds a vulnerability where basically if you submit an event from a free developer account you could get back all the customer details in the response and I believe that this was innumerable so you could basically go through and get all those those details so you know in the hands of um a cyber criminal or a nation that wanted to understand the mechanics of another country's food supply that's pretty handy right um I'm not quite sure what this slide was for uh why companies need to be ready to receive yeah so so really what he's talking about there is the fact that

like this is a thing that happens you know I I think we're at a point now where Vol disclosure and Bug Bounty it's been talked about a lot a lot of people are adopting it obviously you know that's near and dear to my heart um but this is really one of the reasons why um that needs to be in place as a standard thing because this this happens like humans write code humans make mistakes bad things happen hopefully you find someone who's actually friendly like sick that can you know pick it up and disclose it and try to take it through but as a as a vendor um you just kind of need to expect this

stuff right so yep off you went and uh and told the story it got picked up and it went from there um John Deere's response was was basically not quite in line with uh with with six kind of perception of um the actual technical problem uh and by the way this research um he goes into some other stuff which I'll get to in a sec but this is researchers the in the closing keynote of Defcon last year I highly recommend it on the technical side if you're into that type of thing um but they basically said we immediately investigated and and fixed things um you know nothing enabled access to customer accounts Etc so yeah

who's who's a hacker in the room here what's the first thing you do when a vendor says that like nah bet you missed something and they did um so basically what happened he did some additional research to be able to get you know details in a different way um I believe he continued to kind of pivot and as you know all those stories are out there online as well um so okay cool um this is actually yeah the point of this is that some of the information that he was able to get out in his subsequent research was a lot more sensitive from like a pii and a user standpoint so basically John Deere responded to this by saying

that we meant sensitive data um so that was kind of part of Ethics right and again this is his opinions I'm just trying to render them as anyway this is fun um yeah this is not I mean it was speaking as myself this is not an uncommon initial interaction if you've got a vendor that goes from never having experienced this to experiencing it for the first time they freak out and do stuff like this it's not uncommon um it's all the road to doing it better ultimately uh but yeah John Deere established a private bug Bounty with no Bounty and no disclosure um sick found this very confusing and you know he he reached out to me um we'd

already been talking on some other stuff uh but we talked about the general mechanics of you know Bounty incentives final disclosure like what does it look like as an organization that well I was just saying before it's like this is actually it's not ideal but it's also not uncommon um so what's the pass from not doing this at all to doing it well and what are the bumps in that path so we had a good chat about it and and basically you know I kind of laid out some of the different things that from a pure security researcher standpoint can seem quite unreasonable and seem quite counterproductive to to the outcome that you think is the right

thing um but when you think about it through the lens of the recipient and the and the journey they have to go through like the you know five stages of vulnerability grief like all these different things that happen they're pretty common and and it's all about the next step right so that was the uh the um this is where I get to introduce myself as me literally got these slides 24 hours ago too by the way so we're jamming on who says what yeah whatever y'all get it so this was one of the things I said um basically like like the idea that like full disclosure is not an ideal outcome right I actually um did a speed debate in favor of full

disclosure at a at an MBT con one time and one because to me full disclosure is like you know people don't like it but it's kind of like not liking death and taxes like it's the default failure state if you're a security security researcher who doesn't feel like they can get um people to pay attention to what you're doing right so yeah there's messing around then there's a higher likelihood that you're going to find out so what does sick do he goes off and does additional security research on on John Deere and this is the tractor hack stuff that uh that popped out this is yeah that's what I was referring to before like as a iot

exploitation primer this is a fantastic video he's super funny uh to watch as well um and uh you know it kind of gets the point across but kind of went through this whole process of trying to figure out how to jailbreak a head unit um keep in mind that this is a product that's meant to be assass offering which is where the whole right to repair thing comes into it but he was looking at it through the security lens right found a whole bunch of stuff watched that talk because I'll butcher it if I try to resell it and that was the outcome this is where everyone collapsed

and this was actually like I said it was seeing this um presented but like this was him going deeper to make a bigger Point like what he was trying to do was to get the vendor to understand it like no this is you guys don't seem to get the risk that's associated with what I'm finding um which is always a perilous thing because security researchers don't necessarily have the context of the organization but sometimes they're just not getting it and you have to do stuff like this um this is at a Defcon so really this is what he was trying to get across through through the story arc like should more bugs be dropped should

more things be disclosed is that the right thing to do is the wrong thing to do I obviously have a lot of opinions on this one um but from his his perspective you know does it devalue debug and and make it less likely to be used by cyber criminals trying to do their things stealthily for example because all of a sudden it's public knowledge and you've got iocs you can defend different things like that you're actually devaluing utility is about at that point on the flip side is it newsworthy is it actually going to highlight the fact that like bad cyber things happen to a particular domain um like it did in this case right

um the other thing I would say about sick is that his his um a sense of telling a story and actually getting people to understand like really complicated technical stuff is phenomenal and he actually used that to to kind of get the message out um you know is there an NDA is there a VDP what the are we doing here like all those things come into it um but ultimately like this as a question that is good to ask yourself I think is is very very valid and a lot of what he was trying to think through at the time so I'm not entirely sure what he was trying not bad I think the point of this slide really

was the idea that um the financial because bugs are bugs are worth something right if you've got a vulnerability you can do bad things with it you can sell it to a third party you can sell it through a bug Bounty program you can use it for clout and drop it on the internet um they all have in terms of you know thinking about like putting food on the table a different kind of return and that's just the nature of a bug um I think what he's saying here is that's not a linear thing it's going to be an exercise for the reader and and really depend on what's actually happening at the time um

and of course this one which I think is a little bit more clear the idea that like if you're thinking about as a security researcher that's focusing on on Public Safety impact which is you know the whole themes the Cavalry track the bigger the bug gets the more you know um whatever you want to say that's not balls in in this case because he can pull that off and I can't um it takes more nerve to get that stuff out there because the consequences increase with the amount of impact that you're having right so there is that Dynamic to it as well so coming back to you know where I started with this like the way that I'm my bias and how I'm

telling this story is really to encourage people to think about the security research as being able to have this type of impact right um these are some of the things that you need to factor in as examples um going back to Paul's original question are there any CVS on underwater sea vessels this is I debated removing the slide because it's like um but he's not wrong yeah if there was transparency in the if there was proper you know the proper ability to call out some of the safety issues that existed in that system would the things that happened have happened in the same way maybe not and he said he's on high Mars Rockets um

you think about the ability for you know the kind of malicious actors or adversarial actors that are looking for the same information that's a good one answer um so he's kind of trying to paint that picture there like ultimately like is dropping over day and roasting companies helpful right ever um again back to my opinion I think it's not ideal and it happens as an option of Last Resort is it helpful it was here um does it mean it's always helpful no I don't think it's the thing that you need to do necessarily all the time but thinking about it through this lens it's not just a default bad thing right um all good because of the mic

it's the Australian accent and the fact that I've just come off a flight from Thailand right he lives in Thailand um so this is actually a a issue this was actually the first time we he and I worked on a on a submission or a disclosure together um and it got pretty interesting pretty quickly um basically he was doing research on TCL televisions uh they have a Android subsystem he went ferreading through that and found a bunch of stuff it's like that's weird that's obviously vulnerable but it actually looks like it was put there on purpose um what should I do with this case he reached out to me um and yeah I immediately got a little

bit nervous about the whole thing because it's like okay this is this is actually going to have some impact and sure enough it did um basically it was determined we got it into the right hands it was determined it was a deliberately inserted back door in in TCL um and the thing about this television is it's it's everywhere if you look for that brand it's right across you know in all sorts of sensitive places in healthcare inside government airports like it's all over the shop right this is 2020 before some of the changes happened that made that less of a thing um but you know this actually I think potentially helped precipitate some of those changes

yeah and you know ultimately what we did um aside from you know trying to work out any kind of safety concerns is that he might have with with this kind of information was to get into the hands of the right people um make sure that it was taken care of properly on on that end and it ended up escalating from DOD originally into DHS and uh there was a you know very concerted response to that thank you I feel like I'm leaning right down on this thing but it's all good we good so far yeah all right we're all set codes it's awesome so yeah what what happens when you push a bug's news Beyond its limits because

like literally when this stuff dropped um basically we got the information put it all together it went off into the high side and disappeared through a period and then this stuff started coming out of DHS and we realized it was the same thing um and what you know sick wanted to do this was independent as a decision of his but he basically wanted to push this news further same same idea it's like how do we how do we tell this story and make sure that the impact of this is is known because this has consequences um and it's probably not the only code of its nature that's out there in the wild right um one of the this is when we had the

safety conversation there was a uh pretty dramatic stock hit um as a result of this um and yeah basically the DHS were incredibly um responsive to this particular piece of intelligence and actually used it as a way to tell the story of you know the potential risk of foreign consumer equipment and just the fact that it's possible so it's not necessarily targeting you know China or a particular country it's like this is just a thing that we need to be aware of um in this case you know obviously they're calling China out but that was a big part of the the broader narrative as well so yeah pretty much um you know he found a bug in CCP owned

stuff Homeland Security takes it runs with TCL as a suspected backdoor um what he's calling out here is like what if that same type of security research had gone into some of the things that were ultimately exploited to create this outcome right interesting thought same thing with JBS so we're talking about food this is the this is you know the the food happy hackers kind of um theme but um you know personally I see this as a safety credit critical research issue and these kind of impacts these kind of consequences these kind of questions apply to how you focus your research if you're wanting to have this kind of Mark right grain Co-op you know there's a lot of

stuff in here and we're actually going to go into this more on the next panel so I'll skip for a bit um yeah so ultimately the result of of uh some of the research that he did on John Deere um and the conversations that happened off the back that actually created and spawned really a task force of um you know around around agricultural connectivity um I think it actually got classed as a critical infrastructure domain at this at the same time so yeah this sort of precipitated that um work that people already thinking about that type of thing probably um was this a massive Carlos to it actually happening yes 100 um and you know what was observed at

that point in time is that those tasks those um task forces had a lot of attention uh you know the combination of the narrative that was around some of the stuff that he'd done and the fact that there is growing interest in this particular area Drew attention to the right things now he's going back to the Colonial stuff like I said 24 hours ago but

so you see the impact okay so what he's talking about here is how an incident actually creates the same effect like if you're talking about security research that's a friendly person you know ultimately in terms of their intent maybe as a recipient you're not comfortable talking to them yet but but they don't have the goal of tanking your pipeline or doing anything like that the other version of that is when this happens so so Colonial pipeline does its thing all of a sudden you know everyone's pulled into that cost them five million from from a ahead of when people call extortion payments bounties but Bounty paid um and you know ultimately that that precipitated a lot of activity around

critical infrastructure itself and ransomware as well JBS Foods similar sort of thing um 11 million paid out there yep um Trade Secrets stolen AGCO

yeah so this is tying into the fact that you know this is like we this is the whole I am the Cavalry Titan um and it's honestly a big part of the bug crowd story as well and what we do with disclosure like there's not enough Talent that's directly accessible for organizations to proactively get ahead of these problems like we are going to be a catch-all um in a lot of ways and that's ultimately an opportunity that we've gotten to some extent I actually think it's a responsibility that we have as well this is an example of it actually working and causing change um so John Deere's response to this eventually was to spin up and get very

proactive on on the security side um they did a bunch of stuff you know was the right things wrong things won't go into that I know he has opinions on that one but they responded and they spun up you know their own teams um they created challenges so they actually moved like they went full 360 from trying to like just push the thing into the dock to actually encouraging events encouraging people to come in and do this type of work they spun up a program on a platform that I've never heard of hey no offense it's I mean this is awesome like I don't care honestly who I do care from the entrepreneur hat on who people

work with but it's more important that they just do it it's actually one of the reasons why disclosure exists it's like you know what you don't need to be a bug crowd customer to do this you just need to do it so that's actually a good thing and shout out to foreign [Music] so this is really kind of the the sum of this very interesting talk um I did consider as a drum kit back there I I figured flipping the switch and saying this is going to be like a Led Zeppelin concert I'm just going to solo 45 hopefully this has been useful in terms of some of the stuff that we've gone through you know asking yourself as

researchers in the room and even you know folks that are adjacent folks that are in policy it's not just Hardware or web vulnerability research that's actually factoring into the stuff now like data is becoming more of a thing um you know as as everyone starts to turn their attention towards Ai and ml that's becoming a thing there's all of these different technical domains where regardless of what you do on that side you've actually got the ability to focus your efforts towards these kind of outcomes and you know do it the way he did it do it your own way it doesn't really matter think about the kind of outpat output that you can have in terms

of you know making things safer not just more secure so that's the question you know how do I use research to make positive and meaningful change in the world we are all sick codes thank you foreign [Applause] we should not let sex replacement off so easily as this that was easy what so now is your opportunity to stump the champion if you have a specific question regarding this matter response coordinated vulnerability Etc this is your opportunity to talk to a rock star so if you raise your hand I will run this mic over to you and then you can ask your thoughtful question or I'll play the drums and Main actual Rockstar either way all right you have to be on the

drumstances um okay so I'm curious your opinion on this obviously the juicier the target the bigger of an entity they are let's say you're a security researcher and you find a problem yep you kind of have to take a deep breath and ask yourself okay am I going to open this can of worms by disclosing it and then there's the whole pros and cons between private disclosure versus public disclosure obviously but I feel like by nature of public disclosure they're going to receive that information in the tone that they want to receive that in be it confrontational and I think more likely confrontational than friendly so if by default it's seen as an escalating action more often than

not what would your thoughts be on the right way to communicate that stuff just so that you have the best chance possible at it being received in a way that doesn't involve you know Federal authorities knocking on your door which is a real concern at least in the United States yep even to this day even with advancements made in the public disclosure Place yeah for sure I do think on that last part it's less of a concern now with you know the doj charging rule changes um with cfia and and frankly a lot of the work that a lot of people in this room have done to make hacking safer for people who are operating in good face but yeah it's

still a risk right um and you know probably the better example there is the the version of um of that that happened you know sick wasn't in the U.S when he found that TCL bug so there was like an actual door getting kicked down risk factor associated with that disclosure so you know can't really fix that one going back to your question I think private is always the best initial approach um yeah for sure um I think as a as a kind of a meta um kind of thought to that um just applying empathy like literally just put yourself in the shoes of the person receiving this issue like it's scary you have someone come in from the outside

world and tell you that your baby is ugly right if that's happening to you for the first time like how are you going to react it's going to be immediately defensive and that's that's human nature I actually don't think that that's necessarily a bad thing it's it's to me more a conversation about anticipating that type of reaction and trying to figure out to your question how to get it done anyway right um so I think beyond that really it comes down to whether or not you feel like they're operating in good phase you know some of the stuff some of the stuff that um sick went through in this talk around you know the initial interactions

that he had to him indicated the fact that this might get brushed under the carpet so at that at that point he chose to escalate um do more research you know the hacker challenge piece came into it as well but in terms of how he's telling that story it's going to be your mileage may vary I think for the better part which is a terrible answer but like every bug's a snowflake you know every researcher is unique every disclosure is unique every company is unique as well um so yeah that would be my my probably main two answers to that like just put yourself in issues before you do anything if you're pissed off stop and

wait until you're not pissed off because this could get irritating sometimes if you're trying to get the thing across and it feels like you're shouting down a well um I've done like I've seen that and and Bug crowd has exposed me to thousands of people that have experienced that that's a very predictable thing as well so just yeah think about you know all that um and try to get it done try to get it fixed because the the the other downside with full disclosure is that you're exposing that information to people that might not be as well intended as you are um and there's a there's an equities conversation that comes up at that point

right you know is this going to create more public impact if everyone knows about it um then it would if it was kept private that's a horrible this is the whole reason why things like BEP inside the government are so complicated and convoluted because that's a hard question to answer right but that's another thing to factor in well yeah thank you um firstly thank you for the talk I really appreciated it um I was just wondering I was really interested in how John Deere's responses now obviously changed and they've got the challenge up and are they doing any kind of Public Announcement of the lessons they've learned in terms of their response to the initial approach

from um sick code only because I'm wondering if we're kind of preaching to the converted here right like how do we get how do we communicate with companies who take the same approach that they took initially and sort of say like oh please don't talk to us and if you feel like you're under attack like are they going out to say this is a better way to do it and we should have done this sooner and if not how do we get them to do that yeah for sure like to me the the fact that they've gone out and proactively offered a vulnerable disclosure program like that's that is to me an admission of the fact that yet

this happens like we know we're not perfect and we know sometimes things are going to get found outside we need to have a way to receive that and here's the behavior that you can expect from us if you do that um yeah one of the things that we did with disclosure was to make sure the language Embrace like that actually created a sense of safer Harbor for the security researchers so that the uh the recipient couldn't suddenly change their mind or Cloud up and running all those different things but that to me is an example of them actually acknowledging this and becoming proactive um I believe they've done uh like joint talks and and different

things like like sick and John Deere made friends eventually um and I think there's been some stuff that they've actually done together to tell this story you know particularly in places like Iowa um and you know areas where this is top of mind right um yeah maybe look into that because I'm I'm like 90 confident on that answer but yeah to me going back from that just the VDP in and of itself is a proactive measure oh uh thank you for for tape for doing a presentation you did a great job um I know with with John Deere one of the reasons that that sick did not join their vulnerability disclosure program and he was the first person invited into

it for obvious reasons um was that he felt like it was really just a way for them to get him to sign an NDA that would then basically muzzle him and he didn't want to do that and so he kind of joined it and then immediately left it um I guess I'd ask you and I should point out I you're right I was the first person who said hey dear doesn't have any cbes it wasn't actually my idea I got it from I think simple Nomad I don't know I can't remember yeah it was simple um but um it wasn't my idea but uh I think I'll note uh John Deere still does not have

any publicly disclosed cves on the nist um vulnerability database nvd and um don't know of any plans for them to have them so I guess the question is um as the operator of one of the largest you know bug Bounty platforms um how do we sort of thinking crawl walk run right how do we get vendors to a not just look at vulnerability disclosure programs as a way to muzzle researchers by getting them into ndas B um kind of realize that you know program platforms uh like bug crowd um great way to run both private Bounty programs and access that talent pool but also that they should probably have this larger you know kind of wisdom of the

crowd approach right where you know the the downside is you are going to have some public cves the good side is you're gonna have a lot more people looking at yourself so like how do you kind of get them to engage in that I'm trying to enumerate that question back first things first if it's private it's not available right full stop okay um I think that's a definitional like that was a big part of the initial conversations and sick and I were having around this um you know companies that do that um and companies that let other companies do that like ultimately are doing a disservice to disclosure as a baseline operating principle of the

internet right and that's the second problem is that you know through the last 10 years of actually intermediating this and you know telling the crowdsourcing story as bug crowd like we kind of we're not definitely not the only player in the category at this point in time but we were the first to actually go out and do that um and it was really trying to solve both problems like the problem of being able to receive input from the outside world regardless of how it came in because happens like sometimes you need a lightning rod do you know what I mean like lightning happens yeah um it's better for it to hit the rod than to hit

your ass and that's the lightning that's determining that not you you so like that to me is the reactive thing that it just needs to be ubiquitous the crowdsourcing piece to your point is about actually being able to engage your board of talent pool and the problem that we have is that people confuse those Concepts um because they you get the same from both in a lot of ways um so you know I think telling like that's a marketing and education it's a policy issue you know we've had like hard conversations inside bug crowd where we've had to basically deny ourselves Revenue because we wouldn't do things that customers wanted us to do for this reason

um and I think you know things like that are ultimately what need to happen in order to establish norms and best practice and all those different things like discloso played a big role in that as well satisfactory answer yes yeah cool I kind of have a two-part question sure so for organizations in your experience for getting into an integrated vulnerability management approach where trying to build out a bug exploration or a zero day kind of a program first of all what do you consider as like building blocks in terms of maturity that an organization needs to have and second part of that question is in terms of Return of investment how deep is too deep in terms of going into the rabbit

hole in terms of yeah budget and time and organization should put into considering if they're getting into this maturity at an early stage right um so my personal and very strong point of view on this is that every organization should have a vulnerability disclosure Program full stop um for the reason that I just said before it's like lightning will eventually hit your house so put a rod up and that's something that everyone should do um that's a part of the reason you know a part of the driver behind all the not frankly kind of bug crowd work I've done to affect policy in that direction because people need to know that the good thing about it is that you know

transparency actually breeds maturity or a perception of maturity which is trusted in the market so we're at a point now where companies actually are starting to want to do it because the consumer gets it it's like neighborhood watch for the internet I understand that right so there's positive things that actually drive drive that that whole movement um probably the counter intuitive opinion that I have is if you talk about a bug Bounty program as a vault disclosure program with Awards so like then there's 853 R5 definition I don't think most companies should do so if you're talking about going out to the open internet and saying hey we'll pay you if you can park us and tell us

what you found the problem that that creates if there isn't enough maturity to your point is the inability to actually deal with that right like those bugs are still there those risks are still there there's still a problem but if you add another problem on top of that then the original ones probably less likely to get solved at that point in time right so I think you know a robust Downstream remediation process um you know vulnerability management like in the true sense inside the organization looking at things like you know ISO uh 33 what is it 30 triple one I think um there's the two ISO standards 24197 which is intake outside in and there's

study strip one which is what you do once you get the bug um you do need to have both right um where I disagree with some opinions that are out there on this is that on the VDP side like you don't need like you're out of time right like if someone's found a thing you need to have a way to receive that so that's my kind of point of view on that hopefully I've answered second part maybe in terms of kind of material what would you consider not going to sleep until after fall in terms of providing closure or in terms of research in terms of zero days as a hacker or as a recipient

um sorry so I'm just trying to I'm actually trying to pass the question a little bit so in terms of it return on security investment how deep is too deep in terms of time and budgets for an organization to invest in yeah how deep is too deep in terms of return on investment um with the investment an organization makes in this type of thing you've got to be able to determine return on investment to answer that question in the first place and I think a lot of orgs struggle with that um you know if the entire focus of a security team is just on finding bugs then you're probably doing it wrong and there's going to be a balancing act

between defensive measures you know like what are you doing about like helping your engineers be better at not introducing the stuff in the future like there's all of these different things that go into that so yeah again it's a bit of a how long is a piece of string answer um but those are the things to consider and I think for every organization it's going to be different because they've all got different gaps and different needs right Cloud native are going to find this easier because they can fix faster a 40 year old waterfall company is going to have a hard time with the stuff so the investment is going to be different yeah good afternoon thanks a really

interesting talk um had a quick question for you um so the last like five years or so I've seen dhs's sisa grow from being a really immature organization that could barely spell vulnerability to a credible one that probably has relationships with much of the Fortune 500 so in cases where a security researcher faces uh adversarial relationship with the company they're you know they found a vulnerability and don't have a good way to to bring it to them what are your thoughts on I know this is sacrilege for many hackers here using using government uh to be that broker uh to approach The Firm initially you can definitely do that um the challenge with that is that

they're pretty busy uh and you've got to have you know I think those processes and like shout out in all power to the people that I know quite well who run them um they'll probably thank me for saying this um you've got to have something that's impactful in a way that will actually prioritize all of the other stuff they're getting so if you're sending them like 50 xss phones in some nothing sight right you've just basically created load that they're not going to be able to help you with so again it's an equities thing you know I do think uh you know there's other ways to do that um you know Community dot disclose IO as

a way to basically get help from people you know there was a period in time and shout out to everyone in the room who's in this bucket where if you weren't able to get information into the right hands you'd tweet about it and then someone would eventually tap me on the shoulder and I'd find someone I knew and it would get done that way um that's basically what I just talked about it's basically a forum that helps with that and and kind of scales that idea out there's a lot of different things happening there but yeah I think I think centralization of intake it's risky like I mean you think about what we do we've got thousands of

customers we sit we literally do that um and it's hard it's one of the reasons that what we do is you know people pay for it it's because it's really difficult so to apply that kind of solution across the entire internet maybe not so much I I think it needs to be distributed more than that Josh hey Josh hi this is sort of for both of your personalities um hungry hungry hackers thing we care about food targets six been one of the few that has looked at and found vulnerabilities in the food supply not just heavy equipment but ice cream machines you know solar in the healthcare space Beau had a great idea of We Heart hackers so we initially

started just listing all the medical device makers that had a disclosure program as like a wall of fame right then we had a We Heart hackers Challenge from the regulator to say will you bring your stuff to Defcon The biohaki Village now into the patch act it's like to make a new medical device come to Market you have to have according to vulnerability exposure program yep has any discussion between you and sick or Paul maybe this can lead to the next panel like can we replicate that recipe to accelerate the number of participants from the vendor Community but also the number of participants from this community so we target the right equipment that has the

highest impact and maybe short circuit what was a nine year Journey until maybe like yeah an established signed baselines from a hygiene standpoint all that kind of stuff well there's some private events and like corn con has some of these things but is there a discussion about getting to critical mass yeah so so we've uh he and I have definitely talked about that he's a hardware guy um and and you know these kind of domains tend to have a lot of Hardware in them so it's a topic of conversation that we have a lot um I think one of the areas uh that validates what you're talking about um we got pulled into election security

um in a pretty heavy way in in 2018 um and that's a five-year cycle ultimately in terms of some of the stuff that's popping out it followed the exact same story out right um and I think you know lather rinse repeat like the more the more scalable like the more repeatable this journey of like ah what the hell are you talking about like are you you know trying to take my company through to doing the right thing and the best thing um and then the more you know that can be made appealing to organizations I think the better and at the same time actually regulating it and having stick is is pretty important too okay we've got one last question before

we're going to wrap up six presentation because we we have to assemble okay I'll go put a suit and tie on a wonderful a wonderful panel presentation you're not going to want to miss so here is our last question it's already come back around but I was just Googling something and I thought this was pretty interesting John Deere having no cves is 84th on the Fortune 500 list so thinking more about what you just said about how we take something repeatable how many other Fortune 500 companies have zero cves and I think that's an interesting thought for everyone here to kind of go home with them see what you can make happen from that yeah I mean to

me it goes back to some of the stuff we're talking about over here before around like transparency is ultimately anti-fragile so like you put the computer science hat on and think about it at a system thinking level that's just true so the closer you can get to that the more resilient you're going to be but it converts to Consumer trust as well at this point in time it is starting to I think for us like helping you know the marketers in the room or the the folks that straddle this kind of work and communication or policy helping people tell that story and actually getting more interest in in seeing organizations do this well I think that

creates a virtual salute that we're at the beginnings of but I think that could be a lot more effective as the way we get solved okay please join me in thanking Casey and sick her fabulous joint presentation so you wanna you wanna find your seat in 15 minutes because you might not have a seat if you're not sitting down in 15 minutes so come on back you go get a break come on back and we are starting up at the top of the hour thanks [Music] oh [Music] foreign [Music] thank you

[Music]

[Music] foreign [Music] foreign [Music]

[Music] foreign [Music]

foreign [Music] foreign [Music] thank you [Music]

[Music]

[Music] thank you [Music] foreign [Music] thank you [Music]

foreign [Music] foreign [Music] [Applause]

[Music] thank you thank you [Music] thank you [Music] [Applause]

[Music]

[Music] thank you [Music]

baby [Music] some kind of butterfly baby

[Music] don't leave me alone [Music]

[Music]

giving me Wind and Rain some kind of butterfly baby [Music] [Music] oh but I don't wanna miss you baby [Music]

[Music]

maybe you'll give me [Music] your Channel [Music] don't leave me alone baby you give me the rain [Music] maybe you'll get me [Music] don't leave me behind

[Music] oh [Music] my God [Music]

[Music] foreign [Music]

[Music]

[Music] okay [Music]

[Music] thank you I'll move it up

all right we are at the top of the hour and I am so pleased to introduce this fabulous panel that's going to talk about Farm to Fork which I'm sure that's what that means exactly and um facilitating this panel is Mr Paul F Roberts Paul is the founder and editor and chief of the security Ledger in independent security news and Analysis publication that explores the intersection of cyber security with the internet of things and other important items so um Paul's going to introduce the rest of the folks including not six and away we go over to you thank you very much um yep my name is Paul Roberts and uh editor scary Ledger I'm the Cyber

content lead at uh reversing labs and I'm founder of a group called secure repairs uh which is a group of I.T and infosec professionals who support the right to repair um and so if there are folks in the audience who are right to repair supporters and you want to kind of join the movement come up afterwards and talk to me I can tell you more about over I'm really thrilled to be here it's funny last year I did a panel at Defcon called Brazil Redux which was also immediately following a sick codes presentation of his or maybe it wasn't yeah no I think it was always before but basically the the audience thing is basically the same

thing it's a bigger draw than I uh okay we got an amazing panel um I we're going to start out talking at a high level about the AG cyber problem um threats and attacks we're going to talk about some of the progress that's been made and there definitely has been progress um and also some of the things that we need to work on and we've got some really great folks here to talk about it um and I'm going to introduce them so I'm Paul uh you know about me uh immediately to my left is this strange guy who you uh may or may not have met uh Casey Ellis Casey say hi to the group hello

crazy Casey as you know CTO and uh founder of bug crowd um and um also really involved in a lot of these issues as well obviously just by the nature of the work you do uh immediately to Casey's left is Steve Kelly say hi Steve hello and tell the audience a little bit about yourself well I'm uh recently retired Fed so uh you've lost on side of uh spot the FED uh my first this is my first day not wearing a suit yeah but I'm the chief trust officer at The Institute for security and technology that announcement was made yesterday so I'm a retired FBI special agent and thank you that little thing on your LinkedIn with

the ice cream the thing falling off your ice cream um I don't actually I don't like those it's like I do you have a choice I don't think you have a choice you have a way of doing it yeah um and I just finished up a tour at the National Security Council staff at the White House as senior director so some of these issues critical infrastructure cyber security is something that we've been working on so happy to chat and I do not represent the administration today hey yes uh yeah great to have you Stephen I think you really bring an important perspective and obviously a lot of a lot of knowledge we also um have a special mystery guest um who

we're calling AG cyber guy um but we'll just call him AC for uh for the future going forward and uh AG cyber guy do you want to just uh say hello to the audience hey how's it going um and do you want to tell us as much as you're willing to tell us about uh your own background um yeah so I've done about 11 years in ag I have a deep comfy background I've been doing AG cyber intersection for five or six years okay awesome so we've got some down the weeds uh technical uh experience in the trenches which is great so AC thank you um okay so I thought we'd start off just with a

you know kind of a couple slides just to frame up the problem uh obviously agriculture is a huge industry in this country um estimated at 1.2 trillion but actually that's probably an underestimate if you look at agriculture food and all the kind of related industries by some estimates it is about uh 20 of GDP or about 8.6 trillion dollar industry um and responsible for 23 million jobs or about 15 percent of U.S employment um that includes 2.1 million farms uh with about 164 billion dollars uh in Revenue about point seven percent of GDP also nearly a million restaurants about 600 000 of them here in Las Vegas and and 200 000 registered food manufacturing processing it is a huge

huge industry um but one that is often kind of I think has been sort of overlooked in the cyber security conversation I think largely by virtue of the fact that up until fairly recently a lot of this was kind of just disconnected infrastructure right not not wired infrastructure um one of what we've got going on in agriculture um is a I'm going to use the term Perfect Storm say whatever no I hate I hate using cliches like that but it was kind of a perfect storm and it includes digital transformation which in agricultural food and agriculture section is is characterized by you know kind of precision agriculture um connected equipment connected Machinery taught in even autonomous uh

tractors and other things but really throughout the entire food supply chain uh just greater Reliance on as with everything else on technology always on internet connections um you've obviously seen a transition in the last 20 to 30 years from really basically mechanical air gapped equipment on farms and in food processing facilities to software driven Internet connected uh smart equipment remotely managed huge amounts of data being generated and and in many cases collected and monetized um so that's part of the what contributes here and as we've heard kind of as we're going to talk about um a lack of attention to cyber risk this has been a conversation that has cropped up occasionally pun intended but has not really had sustained

attention in the way that cyber security for financial services and banking have cyber security for health care has cyber security for even Automotive has not had that level of attention um and yet as we've seen increased attention from malicious actors right so um we had the JBS uh ransomware attack in 2021 um and we thought we'll go over some of the other ones but there's you know malicious actors both cyber criminal and nation state are definitely interested in this um this is a quote from a crowdstrike crowdstrike did a good pretty good kind of little mini report in 2020 hacking hacking Farm to Table threat Hunters uncover the rise of in attacks um in against agriculture

more extreme weather patterns have encouraged Innovation and agricultural processes tools and crop varieties this intellectual property is valuable Target for countries seeking to grow their agricultural outputs and boost their competitive Advantage relative to Regional neighbors so um this is part of a much larger intellect transfer right of intellectual property um and it's it's of course defense but it extends to things like agriculture as well and here's a kind of timeline of some of the more prominent although this is obviously not comprehensive but uh you know June 2021 you know 2021 and 2022 were just as I call it farmageddon um GM uh you know um uh just a a real explosion of attacks that touched on the food industry JBS

there were a whole bunch of agricultural co-ops the ones we read about were like new co-op and Crystal Valley Co-op uh in September and October 2021 but there are actually at least four or five others that some of them did not make the media um Schreiber Foods a big cheese processor makes cream cheese um there was according to the FBI an Ag Services firm and multi-state grain processor that was hit by ransomware um and of course even recently in February 2023 uh ransomware attack affected Dole uh and there are vegetables in Chilean business uh cost them around 10.5 million dollars we know JBS paid I think 11 million dollars to get their Network unlocked um

so one of the things that's come up um is that food and agriculture sector is really kind of uniquely dependent on other critical infrastructure sectors um manufacturing water treatment Transportation energy right so it's a Nexus of different sectors um and of course there's the Precision AG compounding all of this and this is what bringing us back to sick codes um right no I'm not actually a Citgo yes or a stand-in um uh this is compounding the Cyber risk right always on Internet connected stuff data controlled via Central you know in the case of John Deere John Deere operations center um that as we saw in the Ukraine war the Russians stole a bunch of JD John Deere

equipment John Deere was actually able to remotely brick it which was we all cheered for that right and yet well if the manufacturer can do it who else could do it right so these are all kind of questions anyway so I just thought um um I thought I'd kind of get the conversation going um with uh you know a question I'm going to start with a question for you Casey which is um you actually have a kind of bird's eye view on this whole on you know the Cyber majority of a lot of organizations um and what is your sense about what how involved food and agriculture you know companies vendors could be manufacturers what have you

um how evolved they are on on cyber security with things like vulnerability disclosure programs private bug Bounty programs those types of things is this something you see yeah for sure um so probably the first thing there I would say is that this isn't unusual yeah I think you you summed it up well in terms of an industry and the technology within that industry being built in a time that wasn't Cloud connected by default um and then all of a sudden you slap some sort of connectivity into it in a way that completely violates um the intended security models like we've seen that with with cars we've seen it with medical devices scada um it's all basically a different

version of the same problem um and the thing that drives that to your point earlier before as well is that you know it's it's efficient um I think a lot of the times it's it's Better Business for the vendor so they're encouraged to do that type of thing creating features for customers all that kind of stuff but it does make the Farmer's Life you know ostensibly more efficient if it's working in the way that the vendor intended which is a whole other conversation in terms of whether or not that's a bad thing but security wise um that's a big part of the root cause so yeah I I do think um you know the combination of the

attacks you know the bad guys kind of pivoted from Healthcare towards didn't necessarily pivot but kind of exciting on both and extended the things that were working already in healthcare when they started their ransomware runs in in 2020 around covert and directed it towards agriculture in your farmageddon period I think that's a pretty accurate way of describing it and you know basically never let a trash fire go wasted in a sense I think um yeah between some of the good faith security research that people like sick were doing and the fact that things were literally burning down um it definitely highlighted the problem um you know what we've seen is that there has been an increase in interest

and an uptake of getting security feedback from researchers in all different ways probably less Vol disclosure and less transparency than I'd like to see it'd be nice if they moved a little bit faster but you know in general they're awake at this point in time it's just a question of you know how to how to address it as a system level issue because the other thing about agriculture is incredibly distributed um but then incredibly Consolidated um from a from a system risk standpoint at the same time so you have to find ways to try to address all of that at once and that's hard I don't think they've cracked up out yet so if I could turn to AC you know you

were on kind of the flip side of this you're working in-house at a major OEM um uh but what um what can you tell us about kind of what was going on within the within within the industry within the agriculture industry on you know some of uh these issues yes Precision agriculture is the future but that brings with it a new kind of risk profile and and what do we do about that yeah it's it's an interesting question right um I think some of the the large oems were aware of some of the issues inherent systemically in their own equipment and it's it's like the same thing you guys have seen over and over right how do you get the budget how do

you get someone to believe without the big nasty thing happening like yeah go and do this um I think the reaction to the last three years in the industry has been positive at a large scale right um in another thing kind of to you know to add to Casey's points is like yes AG is distributed it's also very Regional it's driven by a lot of standards and vehicle regular regulations right so there's like the European group and aipac in South and North America a lot of those groups have kind of built committees and forums really security related things to understand um how do we understand this and I think that's happened rapidly at a large scale

in concert though right like a lot of some regions for instance in the EU their past two cyber security laws about vehicles that specifically do cover agricultural Vehicles right so you're kind of in this in this dual path of like the regulation is here we know right it will spread from Europe to other regions I think Cali OSHA has a regulation about cyber security and agricultural vehicles that they're pushing uh and like that that's a state yeah yeah state right OSHA in California yeah and so it's like how do you meet those things while you're trying to transform yourself as an industry and internally there's kind of three big pillars there right that you're dealing

with at once and five years ago maybe you you didn't have to think about it at all um so that's kind of current state I would say yeah um and if I could ask I mean when you're looking when they're looking at the regulatory standard standpoint is it is this um is it still mostly about kind of you know air quality environmental regulations or is is this yes it was cyber security specifically um so like ISO 21434 right same as cars applies to yeah specifically T category Vehicles which is tractors right I mean see one of the one of the questions is really and you're kind of uniquely positioned to answer this you know what what role should the federal

government um play or I I guess is as AC said even state government sometime in setting standards enforcing best practices holding uh manufacturers and other supply chain Partners to account for some of this and um I guess if you could characterize it kind of what is what has been the position of the federal government on on how involved to get in these types of of debates yeah so maybe I'll even start a little bit with since we're having a conversation around uh food and agriculture what's so special about that and some of the incidents that you highlighted on the screen were ones that came kind of a little batch I remember when they were happening and we were wondering well is

there something in particular about the agricultural sector that's attracting these incidents is there a trend is there a vulnerability within within that sector that we need to focus on there was also a period of time where we had a number of water sector issues that kind of came at a clump and it caused us to wait what's going on and run some policy processes and figure out what needs to happen there um so there's all these different critical infrastructure sectors 16 plus the number of sub-sectors and so I don't know what the total is um where we're struggling to figure out what right looks like in that sector and driving that in a way that makes the

most sense internationally that that we are harmonizing towards kind of a common approach so that product manufacturer can build something that's going to work here and meet our requirements and it's going to work as well in Singapore and the EU Etc we're in a discovery phase right now as to what that looks like and international standards bodies are going to help us get there um you know one example um not relating to agriculture in particular but gets to kind of a theme of an ecosystem risk that needs to be tamed and I'd put kind of dealing with ecosystem-wide issues at the top is uh the the White House and the FCC just a couple a few weeks ago announced that

iot security labeling initiative kind of like the Good Housekeeping seal of approval for for iot products um energy star for cyber uh and and then also Department of energy announced that they're going to be doing some work around some more industrial contexts so uh the first two things inverters and smart meters uh and so you're going to begin to see the federal government getting more and more involved in shaping what secure products look like and then through voluntary approaches like this this uh iot history labeling program for Consumer iot where industry wants it because they want to build a product that's going to be marketable across the world and and differentiate their product but it kind of addresses

an ecosystem wide issue there's other things that are in the National cyber strategy they kind of get at ecosystem-wide issues like uh you know liability and and best practices and Cloud security software uh uh secure development for software is it was it was certainly a focus uh you know at the beginning Administration and moving towards now vulnerability disclosure all of these are pieces to the puzzle so we get to what is unique about AG yeah um I think we get towards the bottom of a cone of of what is the stuff that's so very unique about a sector that you need real experts to understand that and and to build controls around that where almost everything else is just

the regular best approach MFA Network segmentation um you know all of the things that we've been talking at nauseam and we've been hearing a lot about you know that that the AG sector water sector especially for for small uh providers these are Target Rich cyber poor entities and they don't know what to do they're not resourced to do it no one's telling to do it but we're we're working I mean if we I need to unlearn that day the federal government is working sector by sector to figure out where there are authorities to to require and Implement minimum cyber security requirements but I'm kind of as we figure out you know for the AG sector what are the pieces

that need to be there for a sector that would be maturing into understanding cyber risk where normally you'd have a sector of risk management agency you have The Regulators you have an ISAC you've got a sector coordinating Council a government coordinating Council you've got this Universe of things and some of those things kind of duplicate what's happening more broadly in terms of cyber information sharing and such I think we need to get down to solve the ecosystem problems that we can solve be better at blocking and tackling in the very generic use case and then Focus sector expertise on what are the things that are truly unique about a sector that requires special attention special plans special playbooks to be

able to manage that risk okay first of all I wanted to chirp in and say I don't need to we don't need to wait till the end for question so if you've got if if one of these amazing folks says something that Spurs and thoughts and you want to throw your hand up go right ahead and we'll we'll take it there we go um hold on one second uh the microphone will have to chase you Psy um no uh uh no uh so I wanted to ask you if you could answer that question and I'm actually going to ask AC day to take a swing at her too what is unique about agriculture as we look at you know you

said whatever 15 16 different critical infrastructure sectors what what in your mind is unique about agriculture food and food and agriculture actually it off the top of my head it it doesn't seem cows cows it doesn't seem that that you know that much different than any other scenario now now the the scenario where you have a million dollar combine yep that has to work in the week that it has to be harvested yeah and if you wait a week later thunderstorm happens you've lost a crop you know that is that that that is a risk that is unique and if an adversary that wants to hold us at risk wants to pull the trigger at that moment

uh that's a problem and and actually that's something worth pulling on just a little bit I want to drill down CIA director and Jenny easterly has said pretty publicly and I have as well that uh that that is our our are the risk that we're under it's no longer just an Espionage issue yeah um the adversary wants to hold our critical infrastructure at risk for contingencies and and this kind of a scenario is one that I would be concerned about and maybe that should go back to our AC what are you in your mind what's unique about the agriculture industry as opposed to even Automotive or you know something like that yeah so definitely um the the blend of such a a deep and

wide section of Technology goes into the AG stack right if you think about putting a seat in the ground all the way to an agronomist having a huge amount of data about every farmer in a state or a region right you're spanning like like the most mechanical thing all the way to like the you know the most recent cool Cloud hosted stuff another one is but because manufacturing is so distributed because it isn't as regulated as Automotive there's a huge blend of Technologies right you can look at a Ford and you can look at a Chevy and inside they're pretty much the same right when when you go into AG oems that can be quite different

um of course all right stuff like you know the miter attack chain you're following all the same steps but the nuts and bolts on the ground like the energy it takes to investigate a thing right now is not very commonized um in one sense that's great and in another it's really difficult um depends if you're the good good guy or the bad guy right yeah also a lot of Market concentration in a way that you don't see other other places question yes perfect thank you um so this may be a better question as we get later but ACU had noted that it's hard to get budget and buy-in from the people that need to approve these things

without an event happening um One Way in like the tech startup world that we see pressure applied is when venture capital or private Equity is involved and you're able to apply pressure via the private Equity or the Venture Capital firm where the money ultimately comes from they're gonna say what you do do you see more success across the AG world where there's private equity and Venture Capital involved or is that still that's oh that's an intriguing question um I have to be careful I I would say right now actually no I don't think it's any better from what I've seen um because I think the realization even from some of the VCS is isn't they don't

connect Ag and Security even though you know the hardware sense like the iot like stuff is really similar to other things they funded um yeah so I would say right now it's like in general that's pretty weak but I I agree that VCS for you know there's an explosion of the AG Tech startups right and that would be absolutely great if VCS would clock onto that you you think in kind of injecting sort of new thinking and new new ideas into you know kind of these older older Industries and Industry players yeah

right right yes um yeah just an observation about uh sort of the difference in agriculture so I grow grapevines by the way and uh I've done some stuff on viticulture and what I've noticed is they're very keen to embrace iot for example um but they just want to get on and grow grapevines they expect a bit like consumer iot they expect that the service that's been delivered is is secure and they also maybe like the medical situation with the Cyber Hospital stuff is they don't um attribute failure to the iot so in the examples that people have given on irrigation you know cutting the irrigation might be obvious and then killing almond trees that take seven years to grow back and

but over irrigating uh say grapevines for example can have really really adverse effects and these are like 20 25 year old Vines so I think that's what makes agriculture very unique in that yeah yeah and maybe the way that I frame that is that um typically in security we're you know very focused on confidentiality uh if you're coming from you know the traditional kind of thin surf driven background um I think you know critical infrastructure in general places to focus on availability which is foreign to a lot of practitioners and I think that's why it's a discreet thing oftentimes um but without Tech um Integrity really matters so the idea of you know there being a closed feedback loop

um that relies on the Integrity of the output of the sensor like that's pretty easy to mess with yeah you're getting something to do something that's not meant to do and just be off by 10 for long enough like you could impact an entire prop you know there was uh well then there's there's a the chicken example um you know changing the the thermostat um inside uh I'm going to retell the strong but basically there was an attack where a whole flock of chickens in a massive provider was killed because they tweaked the thermostat all the way up and off it went like that so there are you know there are the the prevalence of these closed loop systems

that do have like a binary impact um that you know oftentimes are a lot more easy to tamper with and I think probably the biggest thing is that it's not as native to us as security practitioners to think about securing things in that way right we're thinking about how to keep it secret or keep it on right yeah I think that time and and sort of impact recovery is is a really important distinction right I mean you think about a you know an Equifax or something that gets hacked like you know whatever we lost a couple hundred million people's information you know it's fine you know we got the we got the hackers off our Network we patched our

systems we changed some internal processes and you know we're back to being Equifax and you know but like you said with a harvest right you have this window of time to either get it planted or get it out of the ground and once that window has closed you're ruined or you're you're a Vintner and you've spent 20 years growing these once they're dead they're dead and you can't just say well I'm just going to reboot my you know Vineyard and be back in business in in a couple months you know um and I I do think that that's incredibly significant I think that's that's really true yeah hi I wanted to ask about um some some of these more systemic uh

more serious risks with ag um on the timeline you had earlier all of those attacks described financially motivated attacks the kinds that you could like pay money and make it go away and a lot of the stuff we're discussing now is the more critical defense security yes do we have specific examples where that kind of risk has materialized have that kind of attack taken place uh and if not do we have specific threat into Intel that suggests that like apts are trying that I think the only example that I'm aware of right now that we can talk about here is uh USA herds which is a a databases system that I think it was 12 States 18 states 18 18 are we using

to monitor heard health issues basically the cat was yes apt-41 was called was was was collecting on that and sucking data out so that's an example of a state actor going after that kind of information and then one might imagine why are they doing that um well you know there was also the DJI stuff there were the warnings that the government put out about using DJI drones there seemed to be some evidence that you know a nation-state actor was leveraging that data which was all phoning you know all those drones were phoning home um potentially to inform uh state-backed uh acquirers of farmland and other Assets in the states yeah I would chime in on that and say without giving

specific examples um as you can probably hear from the funny accent I'm not originally from around these parts uh but going back to Australia you think about our position in the world um our population and our our Sovereign depend food yeah it's definitely something that's become like it's accelerated in terms of How It's become top of mind um as a you know defense priority Josh um so I did have a question for you but I love what you brought up so to Pivot from what you said to my actual question um but Woods has a great line that was really effective with policymakers which is malicious intent is not a prerequisite to harm so while many of these adversaries in

food are there to make money they could inadvertently destroy the feedstock for cows for dairy for with a pretty pronounced impact um so the to Pivot to what I was going to ask I think a lot of people think food and they think um tractors or they think um manufacturing like the factory um could I ask each of you to State like one of the less obvious weak links in the supply chain that could be terrifying so for me for example when we were in the cisacova task force we were desperately afraid to keep we had to keep vaccines cold so we'd keep cold things cold throughout the entire shipment process and weeks before the

first shipment of Pfizer Americold a major concentration of risk across the U.S was ransomed now we weren't going to use those particular facilities and they weren't cold enough and they mostly had food and dairy as weigh stations cold chain provider but cold chain yeah the duration of that outage affects how much spoilage and how much backup and how much lost food not just food prices but lost food and they were hit again with Ransom very recently so I don't think people think Cold Storage when they think food and Ag Supply chains so can each of you kind of add a weak spot that researchers could help with that isn't an obvious Factory or an obvious track

one one thing that I continue to think back to which is we've we've uh automated as much as we can in our society for the efficiencies but imagine the grocery store down the street if the if the point of sale systems don't work you cannot sell one grocery item because you take the item to the cash register and they don't know how much to charge you there's no way to do it uh and and you know I don't this is probably one of those things that spans you know retail and and food and AG but uh I think when certain things break uh folks can't shop and the average family probably has 48 to 72 hours worth of food in their house

and uh things would get pretty zesting pretty quick um I would say uh mobile networks uh mobile networks so yeah networks so 5G um yeah the the fact that um you know you kind of low energy uh industrial iot sensors and the way that they're connected and going back to the decent the nature of this industry that's a backbone for all of it and you know fortunately I think that's a that's a domain that people want to secure for a lot of other reasons but this is one necessarily go through so you know denial of service in that sense is a good place for research you see um yeah probably anywhere that there's an aggregation right

um grain storage feedlots you know three days before Market um a physical product but also data like there's a lot of data aggregation happening um but I and I don't hear a lot of talk about at a nation-state level what value does that have for someone long term to plan both um strategically outperform a country um but also you know uh destabilization of Futures markets right for for crops that are consumed globally I think those kind of things are the big ones for me I mean I'm going to say software right I mean it's yeah it's uh well it's it's sort of log4j except you know there's something even worse than log4j some you know

Library maintained by somebody some sole contributor that's you know gonna just take out all of these systems you know so I mean yeah I'd say software is uh linchpin and AG in the same way it's a Lynch pin another yeah go ahead the other recent example was Petro Canada just got hit up north and uh their entire payment system like literally gas stations were having to take cash for gasoline yeah and I just got the breach notification email from their like point system right and it didn't even say the word breach in it sorry about the outage right and and who carries cash yeah right doesn't last long right yes hi uh I just recently graduated from

Agriculture and Diesel program in New York and there's been a seems like a struggle for a lot of these companies to find technicians I wonder is that coming into play too yeah absolutely uh you know the the availability of talent particularly um and I might even Turf this to Josh to uh to chip in on um particularly in the areas where agricultural activity is concentrated which generally again are you know kind of middle at least in the United States governmental part of the country right it's not Silicon Valley it's not Route 128 or New York City um really need that those skills and and folks boots on the ground in those States and it's a it's a Josh I would

kind of turf to you as to what the what the fix is there I don't think there's a single fix but here's a couple things some Coalition when he's been trying this year um up until a couple months ago there was no food ice hack so when we say complete public pirate partnership the itisac had a special interest group for some Factory food and big big equipment and so they've kind of declared themselves the food ice hack now but in parallel Auburn university has a center of excellence and they wanted to like work with land grant uh land grant and agricultural schools to establish centers of excellence train up the workforce similarly Congress has the every five-year food bill right now

or the AG Bill and people were kind of pushing a program to say can we have stimulus and free monies for certificates two-year degrees four-year degrees for uh cyber security training in heartlands or near farming uh like scholarship or service where they could have free school in exchange for time served in sector and there's also been some veteran job training programs there's a bunch of talk about it but the issue is not just the workforce shortage for cyber security it's not just a Workforce shortage for I for critical infrastructure cyber security it's they need to live where the critical infrastructure exists so that's where it gets even harder so that it's not settled but there's a lot of decent

ideas being kicked around that's a that's a really difficult thing because like let's say you plant a crop right of a different seed that you you haven't used that one before and it's not doing well you just call the local extension office right County Extension Office there's a farmer and you get help understanding what's going wrong what you can do to fix it if you have a device that you just happen to see right on on your news feed in the morning is vulnerable there's no one for you to go to in Rural America to say hey I have a network I have 16 devices I'm a farmer like I you know I don't have time to go look into this how

do I do it who do I call that that just isn't a thing that exists today yeah I would add I think one of the differentiators in food and agricultural space at least if you look at the end users the farmers is the um awareness Gap it is uh dramatic and in fact I was talking to Kevin Kenny who's a kind of activist in the right to repair movement focused on AG agricultural right to repair and he talks about going out to Farmers and and who are running you know connected uh harvesting equipment uh heavy equipment kind of pointing to the cellular modem on the top of the thing and saying you know do you know what that is do you

know what it does I have no idea yeah it's like that is calling home to the manufacturer and sending all your kind of operational data Harvest data yield data they have no idea they did I mean they kind of know that it's a you know smart tractor that it gets software updates but they really just don't understand how any of this system works they don't really understand what's happening with this data but of course none of us all of us are probably driving smart cars and we don't really understand what's being done with our driving data right so you know should they so that becomes a secured by Design problem right well I mean I think it's in order to so

if this kind of gets to what what Emma was talking about earlier if if we're going to put the onus on securing their Farm on the farmer then their lack of understanding becomes a real big impediment right um and speaking as a right to repair Advocate right if they don't understand what's happening to their data or how their equipment works then it's hard to understand how they're being kind of manipulated and to get them to advocate for their rights but yes important okay but anyway I think that knowledge understanding Gap is a is a major impediment as well yeah so in terms of uh the age-old question of strategizing for cyberos is physical in the food

space uh the Cyber physical impact on what anyone could have to a remote system or a remote kind of uh any kind of technology that is present in terms of opportunity and cost what would what have you seen in terms of strategies do they call concentrate more on the physical side of things or more on the Cyber side of things that's a really good question um I'm going to I'm going to Turf that to AC to start and then I think I'm gonna I'm gonna steep focus more on the the Cyber physical or more on the sort of infrastructure piece yeah yeah that's an that's an also really interesting question um I I think in terms of like regulation and what

Farmers really do like boots on the ground physical right um I think a lot of government agencies globally are aware of of threats to things like livestock herds because of disease right and and things that affect crops and there are some I don't know a whole lot about it but I know it exists right and and when certain things are going through herds and countries like everyone's locked down right and there's procedures in place so I would say currently like by far physical in that aspect um I I think right on we're on this sort of Event Horizon of of like cyber security stuff versus phys SEC but both of those kind of have to blend

together right because you could just as easily like yes you could turn the thermostat up on on a herd of chickens right but what if you just delete the data feed about that lock of chickens oh is that better no I said I think it's a flock of chickens yeah yeah right a pluck yeah um but it's like what if you delete their health data right and then they go to market buy more chickens and spread some kind of disease and you need that Fusion really of of both and I don't see that happening so much yet but maybe just a quick thought um you're the traditional information assurance kind of approach is pretty well known

and while it's not known to Farmers that is an area of practice that uh that there's already a body of experience around and I do keep coming back to what is unique within the sector and and in particular what are the uh operational technology-enabled physical processes that you wouldn't know about unless you're an expert in that area to even know that it needs to be handled or protected uh in a certain way we have seen incidents where the the issue at hand had nothing to do with operational technology um you know Colonial pipeline was one and one of these green elevator ones was another one where they just didn't have access the information that they needed

to do billing or even to to know how much product is being shipped here or there and so they shut things down right not in the case of the one of the grand elevators it was a small operation they just did it manually right but you take you scale that up and it becomes a problem but um but there's a number of scenarios that can play out and and that's where I think getting that sector-specific um expertise especially on the the physical process issues uh is is quite important yeah notwithstanding the fact that these other issues can also have an impact yeah I think the takeaway from Colonial what was just that that you know to disable a industrial you know a

company that uses ICS industrial Control Systems a critical infrastructure company you need not compromise your industrial Control Systems you can just compromise their Billing System or their you know manager you can just shut the company down via their traditional IG Network and that's probably going to spill over into their their infrastructure the other point I'd like to make is we talk about Colonial we talk about JBS we talk about um you know the the co-ops one of the patterns is we these cyber criminal gangs are kind of tripping over these critical infrastructure providers in in the course of basically extorting them for money um but their their interest is not in sewing as much destruction and

disruption as they can they're really just out to get paid and so they're a good reminder to us of these vulnerabilities but they have not shown us what a motivated capable State actor could do and what the blast radius of that attack would be and so saying oh well Colonial pipeline we've learned our lessons about pipeline security no we haven't lost we just they just tripped over uh particularly literally I don't even think they understood what they were doing but anyway yes just just to jump on that real quick um going back to what we were talking about before I think a lot of farmer getting was actually that yes so yeah that kind of

activity that was being successful in terms of making money in the healthcare sector um kind of prompted and pushed Along by covert all of a sudden you know cool where's the GC Target where my stuff will work exactly oh whoops it's a pipeline yeah right or the grain elevators at Harvest Time that's right that's when those were coming yeah yeah yes um you have kind of a a kind of an interesting opportunity though because on Electric Co-op boards they're often made up of the farmers that you're talking about that are making comments about security it's both a problem and an opportunity though I have seen cases where the farmer on the board suddenly learned that there was a problem with John Deere

and suddenly applied that to the co-op that they were working with for electric supply and the cycle was they started to improve together because there's opportunities for that and things like water as well and food to all work together but you need to know who all the people are first so right right interesting

yeah just pulling the thread a little bit on dependencies and kind of Workforce Development there was a question earlier about kind of those dependencies do we understand what normal looks like like take away the Telemetry the data aggregation like in the movies before there's a famine or drought there's the farmer here's the crows and the dogs barking and then there's like a Thunderclap but as we become as you know eggs a complex system of system and so as you know we gain that Telemetry to make better decisions for yield for let's say soil Health do we really understand what normal looks like and are we still learning those skill sets and the example I'll give is is soil health

soil Health you can you can adapt and adjust with phosphate if you look at who owns most of the world's phosphates you know you have some friends in there and you have some other folks with you know different ideologies and so how do we know what normal looks like and how do we trust the sensors and are we still learning what normal looks like in a way that's at like that doesn't require digital and network than automation I think normal is the same except that thunder cloud drops four months of rain and two hours and golf ball you know and grapefruit styes to hail basically seems to be the new normal that we're dealing with

um thoughts on on that question yeah what what is there a way to um to Baseline this I think really right um you know what what should we be um yeah I guess it really depends on what where you're looking and what you're trying to Baseline but sure that I mean all of this technology and these these approaches give us more information to work on to to bring efficiencies I do worry that we're we're unlearning how to work without that yeah uh just in the same way that the grocery store can't sell the jar of peanut butter um perhaps we can't get to the point where water plants can't run a manual we do need to learn we need to retain the

knowledge I've heard Tom Fanning from The Southern Company talk about running exercises where they're going to run the Grid in Georgia manually you know we need to continue to learn how to do that because one day right it's going to be required right and farming without access that information as well and sectors you see like in the medical like in the healthcare sector right most I think most hospitals certainly and probably even small doctors offices do rehearse if our computers go down and we need to take care of patients just using pens and papers they're able to do that although I feel like with the migration to electronic health records like that's that's actually coming under serious

concerns yeah but I don't know that under other Industries go through I think some do and some don't I'm not sure about agriculture whether that's something that they're actually engaged in um yeah most of the farmers I know running Windows XP and like all the machines I just thought I'd have a look at the nfu guidance so they're interviews in National Farmers Union just kind of guess related to what you're saying um and they've done some work with the ncsc in the UK on cyber security advice and it's not very good I just read it right have you looked at what type of uh um it just chucks in the word Farm in a couple of places

but have you guys had a look at any advice that's been put out by governments and and what do you think about it yeah so so I know that the new food and agriculture ISAC has released a um kind of um guide cyber security guide for small um small uh entities and or small and mid-sized entities and it is exactly what you say it is basically sort of basic blocking and tackling uh you know patch vulnerabilities use two-factor authentication use endpoint detection and response technology you know all good stuff but nothing sector specific um there have been specific products like in response to a particular threat activity where sister FBI or NSA or some combination of those sometimes with with

other countries will put out an alert that is uh specific to that issue but I can't say that I've I can point to an example of of uh of a stellar work product along the lines you're describing right and that's the other problem and so if we're going back to it being distributed and a lot of what's going into the supply chains provided by lots of smaller players you're going to run into that issue like even with the most basic you know hygiene guidance um I think the amount of players in this industry that are functionally below the pop of your lines actually Implement any of that you know financially or even in terms of what they understand is pretty

huge so you know I do think the the flip side of that um that guidance being simple is that you know in those cases you have to actually start somewhere um is that a perfect solution no one more question and we'll we'll kind of wrap up go ahead all right so I'm going to follow up on the question about resilience um how do you square the sort of desire to be able to operate stuff manually and continue to be resilient against some of these sort of modern agricultural practices with the other side of that where a lot of the reason we're doing this is that it's really really efficient it saves a lot of money it

saves a lot of time you know like drone mapping and spraying is something that saves 30 40 percent when you're spraying a large farm and you know you can Target more specifically and stuff like people do it for a good reason and that was going to be my closing questions I promise it's like Kismet it was gonna be rhetorical right um yes can we I'd like to Turf that to AC I think that's a great question for you you've got background um how do you balance um the productivity advantages of precision agriculture with cyber resilience so um you know not all your eggs in one appsec basket yeah so I guess I'll I'll start out by

saying right the um I think there's a J in general as people hear about this and keep hearing about it over like the kind of the last three years there's not a great understanding about what a machine will and will not do without all the technology turned off right so probably like this is kind of a swag but for sure there are more running machines in the US and globally today that are not like high technology machines right they're existing machines doing their thing um for the stuff that's really high technology assuming like your engine ECU does not die right in some kind of attack it's still going to do the job most things you're losing like up to low

double digit percentage of efficiency right really like single to like low double digit um and so from from that perspective right there's still hope at the end of that tunnel that like you can you can Harvest and produce food which is which is great um for sure right is is like land use changes among the global population the population keeps Rising food production in general is a big thing and it's like yeah the current bet is um Precision AG is what's going to lead us through that right to significantly increase the yield um how do you balance it I like I don't know if any one person could answer that other than saying that's kind of the

typical security balance in any industry right is is how how are you secure enough but you let the industry function without being kind of too heavy or overbearing um and honestly I you know the path there to me is like application of what we already know about good security practices to the new things emerging on the market right but like the the very basic level there's nothing unique or magic in like AG Tech that doesn't already exist everywhere else and follow the same rules I'll ask I'll ask you guys to weigh in on that question um thoughts on balancing um you know cyber you know transformational capabilities with resilience typically for planning purposes your you're looking to survive a short period

of time without the without the technology at hand and so uh it's especially including in normal you know business continuity recovery it is not a five-year problem it's how do I get through the next month uh and I think uh you know if if if the uh you know the more technologically enabled um uh equipment is built to run without the stuff working that's fantastic that's great news because you can always do the next step um we're not you know the the efficiencies of of of all the things that have been described in terms of knowing where to place the chemicals or knowing how to rotate the crops or or the irrigation levels you know you're

not you're not going to screw it up completely uh in a month um but uh you know if to the extent that that equipment can get uh damaged and is not replaceable uh through through a Cyber attack you know that that would be more concerning because that does implicate future Cycles given given the the cost and the replacement Cycles but I would say I mean I've seen this time and time again um you know going back to what you're saying in terms of some of the birds eye view across different verticals that my job affords me um yeah the pursuit of efficiency is almost the natural enemy of security um and when you think about it as a as a

capitalist running a business I want to make things as effective hypothetical solution yeah sure why not

um no the idea is that companies want to make more money like that's that's literally how the free market works so you know therefore you're going to end up with this natural incentive that's kind of almost pulling by its nature against what we're talking about right now it's it's Fragile by Design because that's actually the thing that is more profitable because you're getting more automation from it and it becomes easier to use and all those different things so you know to me really what that comes back to is that it has to be regulated like there needs to be things that get pushed from a regulatory standpoint to at least you know Josh I think mentioned the patch

acts before um stuff like that that's kind of a solid foundational Baseline that actually becomes a product security design issue that's forced in from the start you know acknowledging the fact that vendors probably won't do this on their own in pursuit of resilience because yeah it's hard yeah I'm going to Echo that and I'm going to bring this back around to the 10th anniversary of I'm the Cavalry and I think one of the most important significant things that I'm the Cavalry did I mean obviously the patch act and your contributions to that but was to insert this notion of public good when bits and bites become Flesh and Blood right into the discussion around cyber

security and cyber risk which honestly and I I don't know I've been in this industry for 20 years like was not really a part of the conversation and putting on my kind of right to repair hat here what I would say is Precision agriculture is amazing digital transformation and agriculture hugely important we need it but as with other things the price of cool features and connected features cannot be monopolies that in the end are damaging to the public welfare and the public good either because they're exploitive and and and damaging to you know communities and and farmers and so on or because they create fragile brittle ecosystems that make us all vulnerable to malicious actors right so that that

basic balance needs to be there not just protecting consumers but also protecting our Public Health but I do want to say thank you to Josh and Bo I think for really foregrounding that conversation like you know Public Health Public Welfare public good like these should be our motivating uh uh our motivations as we talk about cyber security yes okay well please join me and thank you this fabulous panelist just wonderful and thank you AC appreciate it thank you very much we are we're back in here in 60 minutes and we're going to learn about water water water water everywhere and none to drink possibly and we're going to learn about the pathways to Public Service

which are not always straight someti