BSides Munich Keynote - InfoSec's Dirty Little Secrets: Jennifer Janesko

hey my name is jen genesco and i'm here today to talk to you about infosec's dirty little secrets but before i get started i'd like to take this moment to say thank you to besides mesh for having me today for giving me the opportunity to speak with you all about a topic that i hold near and dear to my heart i'm jen janesco once again i am originally from the us but i'm living here in germany i've been working in it since 1997 and in the infosec space since 2013. i have had different roles within the space of information security so i've worked as a penetration tester i've done secure architecture and design i've also done

security reviews for critical infrastructures and currently i work as a lead in a software security capability at a large retailer and so without further ado let's dive into infosec's dirty little secrets

so on this slide you can actually see a number of different headlines now these are all breaches so health information breaches personal information breaches supply chain breaches and also system breaches and all of these breaches boil down to one root cause credentials that have been publicly disclosed in the internet and so usually the disclosure is through some sort of git repository and when i get repository it's usually through github or git lab or bitbucket and so the focus of the presentation today is how do these secrets get into the wild and what can we do about these secrets now before we even get started though really how big is the problem um here we have a

few examples of of big breaches that occurred as a result of credentials being exposed but really is it a problem like how quickly could an attacker really leverage a credential that's out there and is it really a one-off kind of scenario and so uh andrei jacques actually had exactly this question and in november of last year he actually posted a really interesting twitter thread where he dove into okay i am going to release some aws credentials to both github and git lab how quickly if at all will those credentials be used to compre compromise aws resources and so he committed uh the credentials to both github and git lab and for github it was 11 minutes until

compromise in gitlab it took slightly over an hour for compromise that being said slightly over an hour for a personal repository where credentials have been pushed to the repo is super fast 11 minutes is amazingly fast and so you have to think in order for that 11 minutes to occur there has to be scanning at scale being done for git repos across the board then in addition to this in order for that 11 minutes to work there has to be some sort of automation in place in order to check to see is this a valid credential and so those two things combined indicate that the attackers have taken the devops credo of automation to its fullest kudos to them

but it's super bad for us the defenders because 11 minutes to compromise doesn't leave us a lot of room for error but really okay so the attackers have automation in place they're looking for for for credentials out in the wild they can automate testing to see if the credentials are valid how many how widespread is this problem so for this um i've picked out two studies for you so in 2019 some researchers from the north carolina state university put together a study and they looked at github only and in github they found four easily fingerprintable secrets that slightly over 500 000 had been leaked into the wild of those slightly over 500 000 there are slightly over 200 000 that

were unique so that's 200 000 unique credentials out in the wild in 2019 in github so fast forward to 2021 um there's a company called get guardian and they did a study from 2017 to 2021. um and keep in mind get guardians sell services so secrets management and secrets detection detection services but they did a study and there in the past year they've seen an increase of 20 percent of git of secrets being exposed in github 20 increase so we have automated detection and exploit we have increase of secrets being exposed as an information security professional i just want to shake my fist at the universe and scream why can't we get this right why can't application engineers keep

their secrets out of the get repos that are public so we're going to take a look at that a little more closely now because if we can answer that maybe we can figure out a way to help them so that it doesn't happen as often and we can reduce the likelihood that a breach can occur that will result in pii data being compromised supply chains being compromised or other systems being compromised so meet ng ng is our application engineer and ng works for a pretty big company and they have a really cool app and the customers uh who use the app really like the services that are in the app and this app generates quite a bit of

revenue for the company ng is super proud of working on this app uh works very closely with the product owner to develop the features that have been prioritized for the app and to gain feedback from the customers so that they can improve the features it is and she really likes actually building apps because it's building something from nothing it's building something from code that people can use and actually benefit from and ng is super proud of the work that they've been doing on this app but a security review was performed on the app and the security engineer working on this this review noticed that ng had accidentally pushed code to their public repository their personal public repository

and in it was a database password for the application now ng remembered actually pushing that code out to their public repository um they wanted to actually take some of the work that they had been doing internally push it out there so that they could actually open source some of it and so once ng realized that they had accidentally pushed that database password out there they went into the code they deleted it they committed the changes and pushed it to the repository and ng felt like they were done

but the security engineer came back and and said um no no no you can't do it that way what about the get history when i run a scan i i can see that you've you've removed it from the latest commit but in the git history that password is there multiple times so what about that now ng a little bit enervated but not completely beaten down decides okay uh i need to figure out how to clean out my git history so that this password isn't there so spend some time googling uh reading through some medium articles and then did some trial and error and then actually was able to delete those passwords from the public repo

now the entire history was no longer intact from that particular repository but the passwords were gone and so from ng's perspective the history is clean and so there's nothing more to worry about or is there and you started to think okay i accidentally pushed this code to my personal repository which was public and that database password was there there's nothing stopping me from accidentally doing that again so maybe it's the case that i shouldn't be storing my database password in my git repo okay so then angie spent some time looking online to see okay what are some options uh and one of the options that made ng most comfortable was the option of storing the database password in an

environment variable in the production server and manually setting that up on deploy that way the database password wouldn't be stored in in the git repo and so it would be safe from being accidentally um pushed out to the public and then it would be self-contained in the production server but the security engineer came back and said you should not and don't store your application secrets on your production server if an attacker were able to compromise that server and gain access to that server the attacker would be able to gain that clear text password straight out of the environment variable and then use it to access all of that private data stored in that database just as if you had released that

password out in the public

so i'm going to take a second to stop here we need to take a breather and we need to think about the way that the security engineer has been interacting with ng think about the language that the security engineer is using you shall not don't you shouldn't these are all negative formulations saying what ng should not do but ng doesn't make money for the company by not doing things ng brings value to the company by creating things by doing things and so at a minimum as an information security engineer we need to start saying you should do this you should by telling an application engineer what they should do it gives them the opportunity to go and

exercise that possibility it gives them direction of where they should look to solve the problem and there are many information security engineers who already get this and there are some people who are sort of changing and crossing the line and realizing that this is the way that we need to start talking to our application engineers so that they have the possibility to secure all the things and the first step or the common step when you start talking you should and this is what you often see in security reports is referencing standards and so here in this case we have an olaf top 10 proactive control and the control is protect data everywhere and if you look

at this control there's actually an entire paragraph dedicated to application secrets and how to manage them an entire paragraph think about that application secrets management is actually quite a complex topic but there's there's a paragraph there at least and it says you should and in that paragraph there are a list of things that should be done with a couple of examples of what could be done although it lacks context but when we're thinking about how to manage an application secret we actually need to take a step back and think about okay i have this secret but really what kind of secret is it because an application secret depending on what kind it is will require different types of treatments

and so does it have crypto material so are there private keys involved if so then maybe we need to think about an hsm is it a build secret uh this is a type of secret that you use to actually deploy the application to allow you to assemble the application and push the code and the system out to whatever environment for deploy it is only used during the build process and so if it's that kind of secret then it probably belongs in the build environment and needs to be managed in a very specific way is it the type of of secret that's a run time secret the type of secret that allows a particular system to access another

system get information or do initiate some sort of processing so that the customer using the application can do dynamic things within the application if that's the case then it's a runtime secret and in this case for ng it is a runtime secret so the database password allows the application to go access a data store pull data from it so that it can build a dynamic experience for the customer runtime secrets are often managed within application secrets managers and so when looking at the slide on the top right hand corner there's a list of four different application secrets managers there so aws secrets manager gcp secrets manager hashi corvalt and azure vault these are four different

secrets managers made for four different contexts so these may not even be applicable for um ng's context so if ng is running in la cloud these may not actually apply there may be something specific to ali cloud that ng wants to look at so the this problem space becomes quite complex quickly depending on the type of secret that you're managing and then also what space you're working in and where your code is going to run and so to give you an example of how this complexity turns into time um i was recently speaking with a friend who's wise in the ways of application security and wise in the ways of application development and it took him one week to figure out

how to connect his application to the secrets manager inside of aws so that he could actually interface with that and actually get the secrets out so that it could be used during runtime one week now compare that to setting up the variable or the database password inside of an environment environment variable where it would take two to four hours to set that up and get that working one week versus a half day keep in mind that ng is going to be evaluated on the features that ng delivers to production and the the benefits that and the feedback that come from the customers one week versus a half a day if ng is under pressure and she's

probably going to go with that half day option not and not the secure option we need to figure out a way to reduce the security friction so that ng can deliver security value and feature value at the same time now keep in mind that one week didn't take into consideration other application secrets security issues like creating strong passwords generating random values securely sharing secrets securely between teammates managing access to secrets once they're in the various places deploying secrets securely rotating secrets retiring secrets and recovering secrets this is all part of the secrets management life cycle and if ng spends a week on each one of these things no features are going to get developed customers will stop paying attention to

the app and the company will start losing profit and then maybe ng's job goes away and this is not something that we want for ng we don't want it for our for the app that's out there because it's generating the revenue that's paying our paychecks as well so we need to figure something out we need to reduce that security friction so we need to give our application engineer some options when i say this we need to reduce the problem space for the engineers so that they have limited options for their context limiting good options for their context which reduces their lead time to secure changes for their application so they should be able to deliver the application in a secure way

within with the least amount of security friction from within the development process which leads us to information security's dirty little secret we can't know as information security professionals and engineers all of the options for all of the contexts we still need to help secure all of the things this is a conundrum in information security where as an information security professional people often look to you for all of the information security answers and the challenge with that is there are so many different languages there are so many different frameworks there are so many different build environments there are so many different target deployment environments there is no possible way that one person can know them all

and so we need to figure out how to work with our application engineers so that we can give them the options that they need in an efficient way and not put too much pressure on ourselves to know all of the things so how do we get there how do we how can we tell them here are some options in an effective and an efficient way first we're going to start small pick something pick something doable i don't care what it is but pick that one thing or that small subset of things pick something small that you want the application engineers to be able to do and then define it if you don't have any ideas

look to some standard that's going to give you sort of a maturity model approach to organizing the way that you roll things out so here for example we have the software assurance maturity model from once again owasp and you can see that this is actually a scaffolded approach to secrets management there's maturity level one two and three and so if you don't know where to start go to sam look at maturity level one pick a small subset of the things from that list and roll it out try it work with some teams but first select and verify whatever tooling you need before you actually start doing the rollout if you don't have the right tooling

in-house make sure that you get it if you don't have enough of the right tooling in-house look to see that you can get more licenses or figure out how teams can get more licenses make sure that you have the right things in place before you start saying here are your options because if you don't have the right tooling or a way to get the right tooling they don't have the options to use it so make sure that you have that in place it seems simple but it is absolutely imperative that that part is squared away remember we want to reduce the friction from securing all of the things have the right tools in place so for example if you wanted the the

teams to actually start scanning uh their code for secrets upon commit or push to repository then you need to look at okay what tools are out there what tools fit with the different contexts of the teams so the right operating systems or the right build environments or the right frameworks or the right languages what functions with what and then make sure that you've picked the tooling that will fit the context if you don't know what the context is work with a couple of teams to figure that out so that you can make sure that at the start you have the right tools to actually fit what the teams actually do then once you have picked something to

do and tooling to support it trial it with some teams don't do all of the teams at one time pick a few teams a few key teams to help you figure out does this approach work and when you're picking the teams you can think about trying to pick a variety of teams so that you can get different kinds of feedback from different teams try teams with different languages or try teams with different build tools or try teams that use different frameworks or try teams that use different methodologies take a look at your application security landscape landscape and your application development landscape within your organization and then try to pick key teams who are going to help you

build that profile and help you test the solution that you're suggesting

once you do that collect feedback from the different teams and if it works and you haven't rejected the actual approach that you initially picked out document the success so document things that work and how you got it to work document the things that didn't work and what what the team should avoid also curate any code or configuration artifacts that come out of the process so why do we want to to do this it all boils down to something called internal search so uh there is a study called the state of devops which comes out on pretty much a yearly basis and it's from an organization called accelerate and what the study does is it focuses on

high performing devops teams or high performing agile teams and it looks at their practices and what contributes to their success and one of the things that contributes or actually adds to the productivity of of teams is this thing called internal search and this is where teams internally share good practice and they share good code examples so that other teams within an organization can pick up those good examples and build them into their into their own code base and into their own products and by doing this what they do is ensure that it's easy to find they can actually employ it and it fits their context because it's usually the organization context they worked because they've

it's within that specific context all of the niggly problems of onboarding and getting things to work have been worked through by another team and so you have kind of a streamlined way of onboarding to a particular process or a piece of code and so this is called internal search it speeds up everything so if you can somehow build the security process that you have worked with other teams to refine and make better and to make code available for make it available in that internal search so that other teams can take it build upon it amplify it refine it and make it even better

once you've gone through that process and you have something working and in place and it's being amplified and it's being replicated and it's being improved repeat the process go back to the beginning pick out the next thing and start the process again don't worry about the fact that initially it's only going to be a little bit that you're implementing that little bit is more than where we were when you weren't doing the implementation and over time if you keep repeating and keep building you're going to build up a set of practices that are doable by teams that teams are actively using and actively improving and in this way we end up fighting against information security's dirty little

secret we can't know it all we have to work together we have to join forces with our application engineers and we each need to use each other's strengths to secure all of the things we need to be avoiding you shall not we need to avoid you should and we need to get to here are some options so that we reduce complexity and we improve lead time to secure changes when all is said and done we will work with our application engineers and we will have secured all of the things thank you for your time today i hope that you have a great rest of the day and yeah i hope to see you again next year