IATC - CVE CVSS NVD OMGWTFBBQ - Tom Millar, Katie Trimble, Art Manion, Seth Carmody & Josh Corman

hi so I'm Tom Millar thanks everybody for coming to this panel we put this together with the goal of sort of putting some names and faces on people who are responsible for coordinating software vulnerabilities and device vulnerabilities between you know where there's a multi-party coordination issue or where there is like safety life and limb I am the cavalry style bugs and things everybody here has a role to play we also wanted to we want to put like you know the friendly face on sort of especially like government sponsored or government-run vulnerability coordination activities and how we do those things and I'm really looking forward to this being actually sort of after we do our round robin and

everybody gets to talk about their you know browsing and complaining and griping about the current state of our ecosystem and how we're trying to make it better I want it to be a nice freely freewheeling conversation with all of you so this isn't just basically one of those panels where you get five different feels about a topic and I'm also going to strive to make sure that we don't just all agree with each other so so look for that and if you do see us all agreeing with each other and you want to ask a controversial question that is incredibly strongly encouraged all right with that I'll get to so I'm Tom Millar I'm from DHS I work at

Homeland Security I was previously and currently unlike supporting our undersecretary as a technical adviser which makes me special projects dude but prior to that I was the chief of the communications team at us-cert so if you ever read a u.s. cert current activity or a technical alert or I know everybody loved that gravely step grisly stuff publication that was the team that I helped build and run for a number of years at US cert we also did we also built a partnership Engagement Team there to work with all of our different stakeholders sets so that you can have not just so you can have like sort of like local expertise or the communities that you're working with like our

international partnerships or our partnerships with our core constituency the federal executive branch so that's what I've done and continue to advise on doing and with that I'll now let the rest of the panelists introduce themselves

Seth Carmody US Food & Drug Administration and we're just doing intro were you launching into our skills you can go on your spiel - all right yeah okay a little bit of context so cybersecurity program manager which is a title I totally just made up at the US FDA Center for Devices and Radiological health we had we got started a few years ago in terms of policy work about 2013 which culminated basically in a couple of policy documents that you may be familiar with it's we refer to them as the pre-market and post-market guidance but it basically says what are the requirements for a developmental device before was on the market with respect to cybersecurity and what are their what

are the our thoughts and requirements in terms of what people should do after our devices on the market to manage risk one of the things and key elements in that document the post market guidance was leveraging the concept with CBS CBS s with a common vulnerability scoring system and the reason why we leverage that particular scoring system which is perfect in every way of course and everybody knows that is that it had moved people that are really familiar with risk management in the concept of likelihood and severity and trying to get the likelihood component a little bit more teased out because it doesn't necessarily apply within cybersecurity you can't predict your adversaries you can't predict or model human behavior

you can try but how do you get people who don't necessarily think like that using terminology and using a valuable tool to start looking at risk through a different lens and that's why we've pointed people to CBS s and we know that it's imperfect which is why we actually have current efforts underway to have a rubric which prompts people to think about CBS s or leverage CBS s and the tools they're in and the scoring they're in and then think about the clinical context so what is what is the application of clinical use with respect to say an IT Enterprise severe scoring severity tool so that's where we are that's context for why and then

honestly very humbled and and honored at the same time to be here and thank you thank you for the invite all right I'm Katie Trimble these things freak me out so sorry about that if it's a little weird um it's weird for me too it's weird for me too so I'm Katie Trimble I am the section chief at the Aiden kick for vulnerability management coordination so in kick national cybersecurity communications integration Center yes I said it right the first time right so I am also from the government and here to help I do have my fans on today and I'm sorry I have a black jacket and I blend and I am from the government I'm here to help it's

easy to spot me just cool for me so I'm prior military I was in the Air Force for 12 years I refer to myself as a secret old I look much younger than I am so I've been with DHS for about six years I'm new to the in kick I was over in another part of it of DHS for a long time doing telecom and how telecom interacts with the intelligence community and how that works between government agencies and non-government agencies and private sector partners so within my portfolio I actually took a couple different spots from or post portfolios rather from Tom so October of last year the in kick reorganized we no longer have ics-cert or us-cert those

brands I know it's sad I'm sorry those brands are gone so it's now just the end kick so with that we have several operational portfolios so my section VMC vulnerability management coordination we are the ones that sponsor the mitre CVE program we sponsor nvd cert CC and I own Elle's ics-cert portfolio so all of that falls under my office so I'm here and very very privileged to be here because I would love to hear I'd love to hear feedback and I'd love to hear how we can make the system better right now we're in like a golden age of reorg and that pain period where we're trying to find ourselves which is a perfect time to

institute changes that need to happen across the community so I'll turn it over to art thanks hi everyone art Manion at the cert Coordination Center I am the vulnerability analysis technical manager / principal engineer much more at the manager at the moment than the principal engineer anymore but I still occasionally try to understand how how a vulnerability is working and what to do about it let me just art off by saying it's great to be here with my DHS sponsors and they're awesome and and are we gonna get a check soon and a couple of things I'll we're gonna be talking about today but um myself and the others up here I know think a lot about the sort of this

vulnerability ecosystem right somebody finds something and it gets reported and eventually publicized and it maybe gets a CVE on it hopefully and makes its way into the NVD and gets the CBS s score and now it's prepared for fall management to go off and do its job and tell you you have a lot of red and you should patch it and a lot of US government a lot of the world for that matter depends on this system sort of working and if they only knew some of the problems with the system I don't know what happened i yeah i think actually people are kind of overwhelmed in the first place so if the fire hose

got i've had more volume it may not help much nonetheless these systems are not they're functional mostly not in great shape they need help but i think that's what we're here to that's what i'm here to talk about at least uh probably today CVE briefly in my mind I'm on the CVE board so I have some hopefully knowledge about this sort of thing I yell a lot about the scope of CVE being to identify a vulnerability we've an abstract concept right it's not a it's not a pen or a notebook for a cup of coffee you have to describe it in some way and you have to identify it to be able to do anything

else with it do a CVS f-score detected manage it talked about it with more than two people distinguish it from other vulnerabilities that are very similar in the same product and the same kind of problem so really that's what CVS job is I think CD has gotten a lot better in the last two years ever since I started running it what - and then and then even more so after Katie took it over yes even better after that right seriously there have been significant changes in how CEO operate it's more federated and distributed now we're pushing the responsibility for issuing the IDS and filling them out on to the people who are finding the stuff

and the CNAs and the vendors and the researchers so a CNA is a CVE numbering authority naming Authority yes I think there's this I think there should be a special place in hell for putting an acronym in an acronym yeah I see what you're saying I thought I didn't do it but it used to be candidate numbering Authority way back I think that right I think you're right but yeah anyhow point is CV identification importance doing better positive good thing please continue CBS s mm oh no I'm actually I'm very excited and anxious to hear the results of some of the work that the FDA because I'd love to see if someone has figured out a way to sort of improve CBS s or

make it make it useful I know some of the folks involved I trust that there is some solid work going on there so I'm really excited to when that stuff is shareable in some way my initial comment on CBS s is that it may do a decent job of describing direct technical severity of a vulnerability right the quintessential example is heartbleed and it's it's a partial compromise of confidentiality because it leaks bits of information it's like it's like a medium five or something like that what I think a lot of people want what they think CBS s based scores given to them by NIST NVD mean is a risk score they can just take and run with that is so completely wrong

that it's almost I almost would prefer that the scores were not provided just provide the vectors and let people make up their own way to put the scores together so that's my main complaint with CBS s Seth mentioned likelihood yes right am I going to get populist vulnerability that's what you want to know CBS s does not measure that well the fact that one measurement has for it is inverted and not provided in the base score system and also what's my loss gonna be CBS s doesn't know what your loss is gonna be you do and you have to bring something to the table to get a good risk of score so CDSs may solve a technical severity

problem that i think most people don't really need and luck threat like good exploitation and your loss are the driving factors in your risk equation so CB SS have some issues and I'll stop for a moment there and turn it over to Josh oh I think we're gonna have speaking of issues I think we're gonna have a TV issues oh okay so freak out for a second so I want to apologize in advance um if you've been in this track for a while you know me I try to be that helping hand it's in the pointing finger I try to employ what's right with something I'm not gonna be egregiously or gratuitously mean but part of part of

love is tough love and sad but love I basically asserted a while back that CBE is essentially amnesia word cyber but essentially it's cyber critical infrastructure so many things we do depend on TV and a score for it and just like it we bad if our bridges and tunnels we're crumbling I think CBE has been kind of crumbling and you don't fix something until you state the problem right instead step one of the scientific myths say the problem so sad with love is I think it's a vitally important resource it's just one that we're not doing a great job on I wanted to see if I could articulate what some of the problems are visually so this is BA

Brutus is a data visualization chief data scientist at rep seven now he did this as volunteer just want to put some volume in color to how bad things have gotten again not meant to be negative just more meant to say if you really care about this thing we should really invest the love it needs and the resources it needs don't have the solutions but I want to give some shape and scale to the problem so here's a bunch of CVS there's 480 dots here specifically the colors are directly sourced from CBS s totals so these are the actual distributions of the CBS s scores and most people trigger this for I'm gonna say the top three use

cases for CVE are number one some sort of vulnerability scanner triggers off one of these things to say let me go write a check to see if something's vulnerable on this right use case number two is maybe some sort of blocking technology says let me write some sort of shielding virtual patching something or another number three if your practitioner is probably the most important to you which is you're gonna say am i affected and where am i affected and should i prioritize some sort of remedial action and we trigger off this so since most of us don't have infinite resources to the point that was just made previously that cbss is kind of a mess and we don't really treat it

like you know a number dot a number most of you just patched the nines and tens so see all those 418 they're gonna go away ready Boop whoo so at best we just patched the nines and tens which is about 80 of those in this particular sample so out of this many bugs because we have finite resources we're going to prioritize just the ones that are really red or maybe have a logo right and the problem with that is if you know if you guys know what OS VDB the artist formerly known is OSP DP that was a completely volunteer organization and and they were out pacing and out stripping the CVE execution for quite

some time and again not mad at the people doing the job it's just a hard job that's growing bigger than it was maybe originally intended in the face of a flat budget and the dynamic growth problem you got you know you have strain issues so if you look at prioritizing part of prioritization implies that you know all the truth but the thing is if you look at like some of these paid for feeds like voluntee B or security focus or Sakuni or whatever they'll have 800 known vulnerabilities for the 480 better in CVE at any given time and they'll tell you if you oppress them that they don't know them all either so they'll actually have

a blind spot - there's like a thousand known vulnerabilities of which they have eight hundred of which there's 480 so when we go to prioritize those limited things we can actually action in patch are we prioritizing the most important ones are the most important ones within our filtered purview and then also there's the unknown you know we just don't know how many there are and things like disclosure programs and well Disclosure bug bounties are helping to drain some of that swamp but even if you just go back to see ve the thing that bugs me more is not the relative percentage right if it's you know sixty percent of 80 percent of the truth it's

really the coverage that scares me and if you look at the coverage distribution most of the bugs and CBS s the same colors are really skewed towards enterprise market share so if you're like an enterprise technology in a corporate perimeter environment with a c iso and a whole bunch of stuff you probably have CNAs and you're probably getting bugs but if you're an industrial control system there's very little in the way of CVS for those if you're a medical device very very very few and most these are running on a Windows XP box so theoretically they should have about the same CBS s distribution to be representative probably worse because they're on older stuff and then open

source libraries like things I basically said a couple years ago it's open season on open source right most of these attacks aren't after your bespoke web code it's after patchy Commons collections Apache struts to bouncy castle it's all these open source libraries right are being increasingly targeted they're very very poorly represented in the this distribution and foreign language and then Jericho from attrition not org basically pointed out and voting machines like there's no voting machines or at least at the time we made this so depending on the topic you care about not only is it a subset of the subset of the known universe but it's also really heavily skewed towards enterprise stuff and obviously in the I

am the cavalry track we care about stuff that can kill you so I want to see something that can sustain the awesome responsibility of telling the public what are the vulnerabilities where are they how important are they and how do we prioritize fixing and marshalling the correct resources to their correct issues so this is not meant to criticize any of the parties involved it's meant to say if you're using CDE and CVS s in its current form as the basis of what to fix and when and where I think we have a lot of work to do is that the the houses that we're building upon this foundation or you know are gonna stand up so again

said with love thank you for Bob Brutus for making this data visualization but we want to make sure it's invested in sustained and supported as well as possible and hopefully we can get some creative solutions from the rest of the group on how to do so that good what's that phrase for fear of trip to phobia trip to phobia what's it the bureau yes so so there you go I'm gonna stop using data visualization so we can continue to go and do our respective spiels before I start acting and asking directed questions to the to the folks on their panel though I wanted to know if there are any questions that have already arisen from the audience because

like I said I really want this to be a discussion with you all so you can get to know us better and we can share ideas yes sir yeah just curious if you can flesh out how the hey form the ratios between like Von D B and C V like I know it was a representative case but like how was the percentage selected to give you the visualization I think the question was what was the ratio the 80 the 60 and 80 I saw an analysis to spend on year-over-year probably the PDF but basically I showed the exact the absolute counts in a in a given calendar year doesn't mean they're better necessarily but just in broad

brushstrokes it's training better the gaps closing a little but it's still a pretty friggin big gap you know it's about path the truth right and I think one of these I forgot to say but it's really important I hope this room knows it once you have vulnerability that becomes known it's infinitely more likely to be exploited in fact I coined this thing HD Moore's law everyone knows Moore's law has too few power doubles every 18 to 24 months I said HD Moore's law is that the strength of an unskilled adversary grows at the rate of the Metasploit project and then Bank workman a data scientist over it kind of proved it he basically showed that if you patch

a random CVE had a certain percentage likelihood of being exploited in the wild if it was in medicine it was like 30 times more likely if it was a Metis play in exploit DB it was like way off the charts so we tended to a bad job at prioritization of likelihood but I think once it's a known thing to somebody or anybody it's way more likely to get attacked and that's why I want to make sure we don't any blind spots on the stuff we can stop did I answer your question so art since you're on the board and Katie since you're you get all the fun emails from the contracting officers representative do you want to

take a pass it you know take a shot at defending sort of the state of CVE and how it's changing to address Josh's concerns sure I mean I already you know talked about this a little bit in the introductory comments but there is awareness among the people operating CBE that the scope and coverage has traditionally been I mean CV goes back to like late 90s ninety nine so back then guess what the internet was you know think there were not these things that are connected today that were kicked we care about now right so that culture is part of the history there's a focus on traditional compute IT that's a thing there is recognition that the

world is bigger the sort of federated distributed model of having the CNAs and others who are assigning and creating vulnerabilities and reporting them writing up the CVS is helping with that I suspect there still needs to be some sort of concerted effort to get into those newer domains and and get the people doing research and the vendors in those areas aware of CBE as a standard practiced am I've got four abilities reported on fixing them I'm issuing updated software CDE as part of that just trying to make it a simple practice we're working on making it easier not that github is easier for everyone but there's an experiment where I write up a JSON file and we're working on tooling

to do this and I issue a pull request against the master mitre CVT list in public github and that's where it all goes so you can automate that if you're doing about any kind of volume so tool support it should be easy you can have a webform and not have to edit JSON by hand which is what I've been doing a little bit so increasing scope increasing coverage Federation better tooling faster faster creation of CVEs are all going in the right direction more needs to be done but those are very positive signs and these are all you know 18 to 24 month old activities I mean I think art pretty well nailed it down so we I think several years ago we

recognized the state of CBE and the state of the of the ecosystem if you will so it's really more than just cv cv and MBD are kind of inter interconnected in a way that's very difficult to separate so one of the questions that we get regularly when Tom talks about the angry email I get a lot of angry email I'm right with it they hired me for my diplomacy skills I don't know so I'm willing to take angry email because hidden in that angry email Thurston tends to be a kernel of frustration and it's about a real thing so let me get to that real thing and maybe we can make it better for everybody but one of the

angry emails that we get on a very regular basis is the disparity between your paid for subscriptions so your your Valdivia versus your CV and why is there that in Josh's graphic there 400 entry difference and there's a lot of reasons why sometimes there is the CV is sponsored by the government and at the speed of government pace it's funded by the government we all know it didn't scale well so that is why we federated it the last two years and tried to make that a little bit more on to the community so that we can get a little bit of help because we're fairly limited by a resource constraint so I mean that's one issue the other issue is ACB

is run by a board of directors so we're getting and it's run by a community so all the community needs to agree in order to get things moved whereas there isn't that one you know for there wasn't that one director that says move yeah and I paid for subscription which you may have there's also the the essence of how things are counted so within a CV you may have one CV or one vulnerability that we would consider in CVE to be one but it may contain ten vulnerabilities oh well that would only get one entry within CV whereas we're in a paid for service it's all based on numbers so if you do a pure numbers poll

you may end up with a disparity simply because one CVE may contain ten vulnerabilities and in a paid for subscription that would be ten entries so it's all about numbers there so how you count really does matter especially when you look at it like that so we've moved this the set of a federated environment which has caused our tok point to actually move a little bit more so now rather than the choke point being in CBE the choke point is in in VD because that is still manually done one at a time so there there's improvements we're working on it I think we all recognize that there are more improvements to go so we're looking for

we're looking for input ask someone to sorry there's nothing super important but oh yeah more so I mentioned vulnerabilities where abstract things there's a great presentation from a while back about why you can't really compare vulnerability counts because people count differently there's a lot of detail as to how people count differently but it's very very true I have a very simple slide I'll show in a minute that's my attempt that just just the raw numbers count KD of what different databases report that they're counting per year it's basically just a conversation starter to show that you're gonna see the graph it just goes all over the place numbers are all over the place and there's a lot of reasons for

that and it's not that someone's right or wrong they're abstract things and they're hard to count some people count one thing is one or two their relationships between the vulnerabilities so right these are you know Secunia and and what is currently with space security are the high high watermarks there you can ignore Surtsey see for awhile we tried to play with everyone and we decided in late 2008 I think to stop trying to count because didn't wasn't useful to us you can see the supposed you know rod numbers gap here between between NVD and Secunia and and risk-based security and semantics still is is trying to count some things there's a couple of new ones

I don't have on here I haven't don't have the history for it but the issue is people account differently and you can read almost nothing into that really because we don't count the same way at least something if there were such a standard as CBE and see if he does have rules as to how you count I may not agree with them but if we count the same way we can at least compare you know counts do you want to share an example of how counting rules can differ like basically like where a phone ability may be counted only once in the NVD and actually it's counted as multiple phones in a different one yeah I don't have

anything specific but that's entirely true I can speak for my organization at least at that cert we have a our identifier is V you pound and some numbers after it and very often we will publish a bu pound which is a basically an advisory for one or more vulnerabilities with multiple C V's in it so in fact some of our thinking on how to deal with this part of the counting problem instead of trying to solve it in and impose everybody gets one way to count and you know you must try and get people to do that we looked at a structured set of relationships between vulnerabilities so I could say cert view pound one is a parent of see

ve a and C V EB and if flex era accounts differently they can still still a relationships between a flex era entity and I risk-based security and D could be made in a systematic standardized way then you can sort of make a map of what's going on but we decided it was probably too hard to force a standard way to count abstract things on the world and expect the world to follow that but there's lots of examples of count counting differences you know one vulnerability is here is 5 over here some of them are complete misses some are one-to-one mapping all possible combinations of sort of set theory of life I don't want to lose the

absolute count is one thing I was much more concerned when you look at the coverage distribution yep and that's gonna be more factor of things like CNAs whatnot or you know there was that are you always remember that it's better than I do what was the router vulnerability that was actually exploited but wasn't even a CV yeah so you Ubiquiti Networks that one I think the story is something like if I remember a researcher reported to ubiquity through a bug bounty program followed all the nice CVD rules coordinated vulnerability disclosure rules got paid to bounty ubiquity fixed it put a fix on their website somewhere I think I don't know if they could push stuff at the time or they push fixes but

they may not basically twelve months later all the ubiquity forums are all on fire with malware going around targeting ubiquity routers for that vulnerability so all the proper disclosures stuff happened the vendor was responsive the researchers the right things between Chicago I think paid or a t-shirt or something and there was no CVE and there was no sort of broad notice of this thing and that guys had a whole year to go figure it out and attack it before we've got enough volume to be noticed and I don't even know if it has a CD today I don't think it does but I sort of meant to go do that but I don't I mean that's that's there's a downside to

cv but I'm assuming this room is a bunch of people that care about ICS about high-speed rail like is there something this room can do to try to close the gap on the coverage problem make your employer into a CNA become a CNA please become a jumping on answers to questions and doing a terrible job but I do it like Seth since we haven't beat up on you enough I think this is actually a good time for you to stay you can spike basically like you know like in your in your particular sort of well it's not really an issue but in your particular sort of sector do you see people like becoming interested in becoming CNAs as

they're establishing sort of medical device B certs and stuff there has been discussion I would say was limited and I in the reason why the the coverage discussions a bit new on in medical devices because as the regulator yes for us FDA has sort of purview over the medical device manufacturer who's producing this technology right so we can say hey by the way you're you're doing the internal testing to drive down your vulnerability numbers as opposed to dependent on a coverage problem from some other upstream supplier I mean they have their supply chain issues but we're not necessarily having that particular issue right now and we're very early in the maturity curve in terms of you know you

need to be testing your devices for vulnerabilities infected prioritizing them through some known risk calculus or made-up risk like whatever you want there's a couple of them out there cbss there's one of them people take the that rubric and then make their own scoring system for a device to provide additional context there's another one that I just saw or become aware of risk scoring system for medical devices RSS MD and then fixing and prior prior to prioritizing fixing those vulnerabilities within their device before it's actually sold into an enterprise like a health care delivery organization so we don't necessarily have that particular high level problem but that's that's the kind of thing that we're trying to work on now I want to

give the audience another opportunity to ask questions before I dive into my opinions okay top Tom Kay can I take okay so it this brings up a thought and I thought that maybe I should share it so I thought it might be beneficial if I explain so what Josh was saying we need to get more people into the community to close that gap I thought it might be a good idea to explain how we actually do vulnerability disclosure within DHS because DHS is within ics-cert so ics-cert is industrial control systems but they're actually the armed with in my office that handles medical devices so specifically looking at medical devices we are the we are CNA and we can

publish CDs for medical devices or for anything industrial control system related that doesn't have a CNA already so I thought it might be beneficial to explain sort of how that process works because I think there's a key opportunity in there so within the vulnerability disclosure process responsible disclosure a researcher from the research community and I don't it doesn't matter to us if that researcher is somebody in a basement somewhere or some very official security firm doesn't matter to me they bring a vulnerability that they find in whatever system I industrial control system to us at DHS we have a little button on our website you can go and check it out it says report up owner

ability here we then take that vulnerability from the researcher and we require a proof of concept so you have to prove to me that it really works because we get a lot of noise so zero-day vulnerability I should specifically say cross-site fir'd forgeries vulnerability but not one that we do because we're not the internet police zero day vulnerabilities come to us we require a proof of concept we get that proof of concept from the researcher we then contact the vendor then we didn't work with the vendor to create a patch or a fix to the system the researcher and the vendor both disclosed at the same time so the whole thing is basically a researcher brings

it in usually there's a little bit of a depending on the relationship with the vendor sometimes we have to really prove that it really is a vulnerability in that system because a lot of people don't like it when you bring a vulnerability to them they say oh no my software's perfect this could not possibly be and that's where you get that like I'm gonna sue you if you that's a thing too so DHS kind of acts as a referee sometimes between the research community and the vendors to come to an equitable applicable agreement so that that vulnerability would be coordinated between the researcher and the vendor and and DHS and we all stand and have a

standoff and point a gun at each other and say you don't publish til everybody publishes so nobody publishes till everyone publishes and then we all publish the researcher will release their for their report the vendor will release their patch and their security bulletins and DHS will release a technical or all within a minute of each other on the same time same place so within our office I talked about researchers restraints and I know that sometimes it's kind of like that that Charlie Brown want want want want want wanna research restraints won't want want want the big government doesn't have the money we don't have the money I have seven vulnerability handlers that's it it's all I got

and they're the ones that are making those calls so at any given time the maximal amount of tickets that they should have is fifteen and I didn't even a given time they have thirty to thirty five tickets that they're working to try to get those patches out so we rely very heavily on the research community to bring that to us because we don't have the time and we don't have the skill set or the the resources to go and find those vulnerabilities ourselves so that is a key opportunity for the research community to understand that like we are here to help there are some parts of the government that may not necessarily love to work with the cyber community it's

not DHS we're here to help so I think that that might be we're getting better we're trying to get out there to some specifically targeting the medical sector get out there and try to build bridges but we do need help turn back over top okay okay so I'm gonna say my opinions about stuff now so I think so I have figured out late in life that everything is always actually a people problem or or a cable it's either a cable or and I think what the challenge we have here and is talking with uh with one of you earlier today about this in public grounds so thanks for hanging out what we mean we really see these

coverage problem boils down to like there's not enough good peace certain people out there like when there's companies are building P shirts and a lot of these sectors that are under covered as per Josh's research and visualization and well does everybody know what a piece or is when I use that word I apologize it's a product security incident response team so it's like a C cert but it's focused on the products that your company makes specifically vulnerabilities in those products and how you get the word out to your customers so at at a lot art can go into more detail in me because he's actually helping write curriculum guidance for people who were building or standing up

a products incident response team at their firm because we want we need to build capacity in that area like massively if we want people to be good you know good stewards of a pile of CVEs that they get to assign their own products and we want them to be responsive to the researcher community so that we can start to their to share the love and how these things get coordinated and discovered and reported you know we just we need to have more people come and learn about what a piece it is go make one out there you know company go get involved in the CVE numbering authority program or you know if you invited you're consulting

with clients and you're doing pen testing for them on a gig you know tell your client like here's some things you know here's some steps that you can take to you know be a little bit more proactive and maybe not rely on me to do everything for you all the time and we need all of that right when I say that everything is a people problem and it is basically we still have a dearth of the right kind of qualified candidates to do this work you know we all know that that's tough it's tough out there and when we say that the government cannot afford like necessarily leave the right kind of pentesting stuff it's not that we

couldn't go and ask you know proceeding like the classic line you're supposed to use with Congress well if if given more I can do more right so you know break out the appropriations checkbook and but in our cases specifically like you know like as far as you know like being able to spend the money on a federal employee who is going to be able to like do like series out pentesting it's just not like after a certain point that person's gonna realize they can make more money just playing a bug and bounty programs and you know can't keep them down on the farm so there there's a there's also that but that's also an effect of sort

of like beacons that incredibly I realize I and I know the room I'm talking to when I say this but they're very shallow and not especially wide talent pool that is available to do this umbrella incredibly important work so that's the trendline I'm most concerned about because I do think coverage is moving in the right direction it albeit slower than we'd like I do think CBE is catching up and its ability to sort of scale and issue things faster and the growth of the the gross growth of the CNA program over the past two-and-a-half years has been completely astounding we went from 10 - like over 80 of just you know pulling in companies 87 87 that you know

that these are these are firms that signed up say like I want to issue CVS for my own products because I'm and when we got Oracle and s AP into the program I was like I have got my white whales of enterprise software you know like now it's time to move on to the unusual suspects and I think maybe there's another person in the room who is remembering these conversations sin and we yeah we talked a lot about you know like we need to go to the other information sharing and analysis centers like for financial services or for automotive or for chemical and find out do you guys know what a piece or it is

you should probably have some here's some curriculum guide here's here's all the stuff you need to know and same problem with critical infrastructure everywhere and a lot of you probably do know this is basically they're just like we don't we don't have the budget for that you know and it's sort of like you you have to convince me well you don't need to buy more tools you need to hire more people and here's the extremely difficult and challenging interview process by which you will select only the best possible candidates and certainly know and you know amateurs who just sort of like know how to use tools um so that's sort of like I'm really concerned just basically that we're not

growing like sort of our human capital talent pool quickly enough to answer that but the rest the rest of everything else is far easily just because I've been part of part of the improvement so I had to have personal and personal bias that I invested a lot of time and getting these programs to be better than they were when I found them um but yeah I think what keeps me up at night about all of this is just like do we have enough of the right qualified people doing this work and do they know about the tools and resources and jargon words that are available to them okay okay so that was my little like not being a

moderator I'm gonna pretend I'm a panelist bit do you feel better yes clearly so back to questions I want to keep the discussion going with you guys input we said all right I knew it man

so I have a little bit of a background in vulnerability management's one thing that just happened in a discussion with someone from a major bank just yesterday that came up touches on the point that you just brought up vulnerability management vulnerability handling vulnerability analysis as a discipline or as a specialty or whatever has no certification programs has no real training programs people who learn learn on the job learn trial by fire so that's something that I just like realized yesterday whereas there's you know entire certification programs for intrusion detection and malware analysis and all that so it would be interesting to hear perspectives of the people on the panel you know starting with you perhaps Tom

as to what perhaps could be if first of all do you agree with that and then secondly what perhaps could be done about it I'll start and we'll go down or we can go we can we can start with Josh and comes back and I know there was another question we will definitely get to you so you should come to narwall normal dot B tomorrow night and I'm giving a 30-minute talk on professionalization specifically an incident response and sort of defender stuff because I think the state of professionalization in the defenders face is woeful and I covers more than just the vulnerability analysis counting identifying understanding how to coordinate stuff much more than just that but that is part of the part of the

problem I'm trying to identify and sort of scope correctly because I know we can't professionalize necessarily like application pen testing because that's a thing that seventeen year olds are really good at so I'm just I'm not gonna I'm not going to take that away from seventeen year olds and say actually you have to have a master's degree and follow this code of ethics before you can do you can call yourself a real life bug hunter like that's ridiculous but the defenders face the people who deal with the vulnerability reports from the 17 year olds who are cranking them out at the bug bounty program and not having to go to college or paying their way through whatever

like that that's where again in like where I'm worried about that capacity building that's not happening fast enough for my taste right so like that defender space of the person who gets the vulnerability report assesses the severity of it or it says is the likelihood of exploitation realizes what the impact is to their ecosystem of their users and customers or realizes it for their own enterprise like that person we don't have enough sort of like a professional you know a very particular set of skills that all of those people should have and sort of an our code of conduct for them to adhere to so actually what I what I didn't tell you about my title was where I actually

reside within Center for Devices so I'm in the emergency preparedness and operations and medical countermeasures which is a we're working on the title right now but the point is is that to Steve's point is that our foray into cyber security was very much an emergency operation of one that we we didn't have any tools or folks pre-positioned to respond to I won't go into the history we couldn't talk offline about that but what I will say is that what you see is that people that are that are baked into the emergency response and they they know it in terms of other hazards like natural disasters like hurricanes earthquakes they have a lot of the tool sets that are generated

are directly applicable yes cybersecurity is a very interesting beast and and for all the response items that I've participated in there's a lot of uniqueness to every case but I think that those base skill sets can be already be found in a preparedness and in response regime and there's training out there actually that can get people prepared and I've taken it myself that are and you see all the parallels it's really quite interesting but I know you have a soapbox I'll turn it over to you okay Bernie I feel like Tom therapy session time like I'm so excited about this okay so my soapbox so I am a hiring manager at DHS um and we have open billets yes

we do it takes about two weeks for someone to quit in about a year for me to hire somebody and that's a problem and like we recognize that that's a problem but it's it's a it's the big government beast and it's it's going slowly so we have a couple programs that are trying to incentivize people staying because what happens is like Thomas said previously like you you get somebody in there who's a real good real good handler real good analyst real good whatever they're doing and they go do a couple big briefings at you know a bank and they get offered and then they're gone so we're trying to fix that by offering some pay incentives a cyber pay

is one of them you can get up to a certain amount money 25% of your pay in a bonus which is cozy and very nice by the way so that's that's helpful that's retention incentives so we're making progress we're trying to change the way that we hire so currently we're under the GS schedule the GS schedule doesn't let us a lot of flexibility and to your point you can be asserted out the wazoo but I don't need somebody who is a certified ethical hacker for some of the tasks that we need to do I don't need somebody who's the CISSP or somebody who spent a lot of time and gone to college and has a master's degree in cyber

security sometimes I need somebody who can pick up the phone and call another human being and not take no for an answer and sit I know sit and that dial do you know how many times like it's like social engineering we have to convince vendors that we're not trying to social engineer live imagine hi my name is Kate I work at Department of Homeland Security in the industrial control systems and I'd like to talk to you about how your software is broken what happens with that they go thank you for your time click I don't know where to transfer you to click you should call it this other line click like my guys sit there they go through this telephone

game like it's imagine it's like trying to call your credit card company or the cable company imagine trying to cancel your cable that is their job all day long every day that sales pitch for a job ever yeah if you if you like a challenge or know someone who does but the point is that like there's a certain skill set that is overlooked and the problem is that I can hire in two ways I have to series as I can hire for I can hire for twenty to ten which is an IT specialist or I can hire for a zero one thirty two which is an intelligence analyst and if I advertise for either one of those I

might get the wrong person I'm if I advertise for 20 to ten I'm gonna get somebody who's doing tech support and can do that Tier one analysis and transfer you to you know escalate your situation if I advertise for an intelligence analyst who can write a paragraph about what this all means I'm gonna get somebody from counterterrorism and like I don't know I don't need that either so there is this this half way land that we're trying to get to so that we can better scope and better hire employees to be able to do those jobs so I would say that we're getting there but I'm getting the hook over here so I'm gonna have to hand off to so I have a

soapbox if anybody wants to talk to me about like my hiring problems I had a guy who has a master's degree in cyber security but dude's got a got a bachelor's degree in theater and he didn't make it through the cert process it's like I don't need somebody with a master's in cyber security I need someone who would talk to somebody that masters in theater that's helpful so arts gonna pull up some some CBE's we created today so we each made a CBE but I want to pivot slightly the the guy that asked that last question when I said there's a lot of love whether you know this or not Steve's basically the father of CBE so

the fact that we're having any sort of argument let's clap I don't think anybody knew at the time how important this would be and how we could bring some order from chaos and not have redundant names for the same kind of thing and it's got a lot of growing pains as anything valuable does but I am so grateful to him and his colleagues that kind of put this into the debate and the fact that we want to talk about it want to improve it hopefully is maybe the way to show some love to him besides a hug or thank you is start us to start contributing thanks by the way for the record I'm an organic

chemist so you don't necessarily have a official credential to be in cybersecurity for medical devices and my boss is a surgeon so I get back on my soapbox if folks are interested in the work we're doing for our CBS s for medical devices in the in I think that actually I'm gonna point you to overrate over there to Steve Christie Kohli he's heading up that effort in our work with mitre colleagues who've been tremendous so if you have any questions ask him and no Adam Brandt you had a question did you forget it

[Music]

[Music] so the question was for CNAs is there any vetting of contributions before they're added to the authoritative CBE list okay okay there's an entire like process that goes into vetting CNAs you have to be approved and you have to go through and submit that you really are going to participate and not just be a slacker sitting there like on getting the inside information you got to actually participate so when a CNA is published there is an opportunity to dispute that CNA and so we or that's CBE rather so we have CNAs that publish of CVS and they may downplay or we've gotten away from that there's a pretty good code of contact with the CAS for

providing accurate information and being honest with themselves about it so that's that's also very helpful but there is a status for disputed so another entity can be like nah I don't think so I'm pretty sure that's a way more critical than you're making it out to be so that's it that's a part of the of the system or did you want to just real quickly yes there's there's QA but with the growing pains and Federation you know the central authority can't check you know every single character very carefully so there's probably some loss of specific quality overall for the volume you're getting but um again a federated system has to be able to sort of police itself more organically and

have the central authority just check everything you know exactly down which leads us into our plug for NIST's new publication vaunt ology so this will be putting out a new standard within the next couple of months to the next year talking about standardized terms for vulnerability for vulnerability discovery so hopefully what that'll do is standardize some of the talking past each other and once that is adopted community-wide which over the next couple of years hopefully it will allow us to do some of these things a lot faster and do some use some machine learning in there to get the the Seavey's more automated all right and now we have bad jokes to share with everybody so this was a Josh's idea to

give us all a homework assignment before the panel and come up with a fake CVE and make a funny cvd description of it some of these are supposed to be some of these are meta like as in actually CDs about problems with CVE itself some of them are not I'll unless anybody wants to read their own I'm just gonna go ahead and do this and try not to corpse up here so Seth came up with CBE 2020 because he can see the future 5:07 and Smothers Brothers model 187 smart IOT ICO WinXP Burt chained is seen on TV pillow Rev doll have a vulnerability that allows an unauthenticated attacker to flip the pillow over and apply substantial

pressure for five minutes and yeah the CBS s on that one is 86 which everybody will automatically understand means death Josh can see even further into the future so he did 2025 5150 cyber critical infrastructure resource under flow chained with apathy overflow leading to binary implementation in the form of piss-poor patching and CVS did you have the logo that was just a triangle or did you have the the one that was a turtle oh did I just make you nostalgic okay that's what I thought would he'd bypass all logo I was just alright Katie is it actually timely 2018 867-5309 swipe right tender iOS and tender Android on mobile devices Sims version 1.02 current allow attackers

remote access to users with profiles while unbeknownst to user automatically reverse all swipe inputs yep I'm not sure which is worse the consequences of that or the pillow flipping one and arts CVE 2099 binary stuff surprise another vulnerability do we still think these are special and chop them by hand resource exhaustion among vulnerability response and management leads to denial of service and this one goes to 11 and I have my contribution is CDE 2018 11 D hashtag meatballs and complete knowledge of vulnerability ID and coordination is shared between nerds allowing for ignorance leaks so effective my everything is a people problem thanks everybody we'd be obviously like available for conversation and chat and the limited

time we have remaining together but I really appreciate everybody who came in it's great to see how full the room got [Applause]