G1234! - Guardians of GitHub - Joshua Danielson & Dileep Gurazada

cool thank you so the talk for today's get her a Guardians at github and we'll be diving into a few of the security uh I say assurance control we saw they're missing some of the default github accounts but a brief introduction before get things kicked off here myself and dilip hi I said I'm an application security engineer network on many projects that revolve around different areas of security and I'm more passionate about vulnerability management and solving or designing security solutions yeah thanks for welcoming me to the V sites Las Vegas and this is my first time here yeah and so the leap and I are both part of Copart delete focus on the application security side I'm heading up the overall

info sex functions so definitely not as technical as dilip so he'll be answering some of the questions you guys get probably towards the tail end of things there to better describe the problem we're trying to solve is right now if you have github particularly at the non Enterprise version there's some core functionality security functionalities they're actually missing from that and you can get those from getting to enter github Enterprise version but MFA is one of the things that we absolutely had to have I mean there's some things you can probably give on when it comes to credential scanning and some other pieces but you have to have MFA from at least from our use cases that we had

internally also on Prem is a strong nice to have that you don't have that option unless you have enterprise and then also probably maybe the kicker for a number of users we had within our environment it was about a hundred thousand dollars a year in order to be able to maintain github enterprise so this is the problem that we were trying to solve and I think we have a few others that were probably trying to have the same struggles as well to be a little get a little bit more color to this one issue we came across was with credential management where developers and the like would be embedding plain text credentials and also access keys right into the code

also leakage of intellectual property where a lot of times that we'd have our developers not being malicious but sometimes just forgetful or just not even practicing good data hygiene you know it opened up the repo and to be open to the entire public so that was another risk we had I think there's a lot more breaches for this in the past a year so particular Amazon s3 but I think there's a few issues I think Apple had an issue with the github if I remember right maybe I'll six months ago and there's probably gonna be a few more in the next few months or so and then also how do you actually secure access to github itself

so this is where that multi-factor authentication piece comes into play talking about one of credential management as I mentioned before a lot of developers not being malicious they don't have a lot of solutions instead of embedding the credentials username and passwords and plaintext into uh repositories so that was one of the biggest problems I think we saw a lot of movement in this space cuz we developed it's probably about a year and a half ago and the past year and a half now you probably see a lot more options for this if you want to pay but we want to solve this problems as well how do you ensure that the controls are actually secured

so you can make sure that maybe they're encrypted but how do you know that not using a different solution or not using your authorized one so this was another problem we were looking to solve too and last but not least we have a team of several hundred developers so how do you ensure that that each developer is actually having this sometimes they have their own rules they may have this safer non pradhan another safer like prod or developers have their own solutions right if you're in a lot of you grown a lot before mergers and acquisitions they have your own products and solutions in place too so you have to be able come with some customized reporting that was

one of the other issues we had with a IT group it's about 500 people strong and intellectual leakage of intellectual property as I already kind of touched on before but right now there's tons of repos that are being exposed to the internet right now and they're leaking everything from credentials to access your environments but also any type of intellectual property as well depending on your organization this may be more pertinent than the credentials it just kind of depends um I think I already kind of touched on that point there really was no great solution to secure it by default in some cases we come to some features that allow you to click a button and you can

do it for like I'm sure no public repos at all but that's not really manageable for a lot of organizations you may have some repos that have to be public so you don't really had a manageable solution for most companies and in secure access to get hub accounts we were finding a lot of issues with password reuse or reuse and this is where we actually able to uh use github Guardian as well to help us out with that piece in particular when it comes to MFA that was one of the biggest pieces but also around API key management we have some service accounts that have that aren't able to use MFA how do you actually ensure that your developer is

actually properly managing their keys that you're actually rotating out maybe on an annual basis or whatever policies you guys stand stand up internally and managing of still accounts I think I did kind of touch on that piece too so there's a few solutions that are available the first one I talked about was hundred K a year for a github Enterprise I don't know about you guys but I think I'd rather spend my money somewhere else if you've got 100 K there's plenty of other products or even people you can probably use their third second third party solutions we saw like Presley in the last year there's a lot more players in this space that are able

to solve some of these problems but not all of them are doing all of them like we found a lot of players and you can just Google into that are doing like credential management for github and you're charging like $2 a user I mean I thought that was actually kind of expensive for just doing that one function and there's tons of more special players into space just in the past few months to a year or so and then third is to develop our own in-house solution so we went with number three so the solution that was developed GHD for legal reasons we had to change our name multiple times but we said oMG hd1 with

other things are all settled in what GHD we're able to ensure that no plaintext credentials are available in code and right now as soon as the developer actually does make a commit and actually has plaintext credentials into that code we actually can send an alert we're working some improvements right there but now this has been really helpful for us in fact when we first started this we found so many we found quite a number of plaintext rentals we have to turn this off before we can get everyone else out into place and then we're able to start actively monitoring for that piece private repositories I already beat this one to death about talking about ensure

repositories are publicly not publicly available right and then enforce multi-factor authentication and this kind of goes back to the other piece because if you actually are familiar right now you can enforce multi-factor authentication but you have to do it for all the accounts you know what I'd say it wasn't manageable multi-factor authentication so this is one of the other problems that we're looking to solve with GHz as well so to recap multi for the features we included with angie HD so you can see like everything that we're including is multi-factor authentication credential scanning for both username and passwords and also access keys like stuff you find in AWS third-party management of vulnerability vulnerabilities this is

actually an interesting that when dilip was actually creating this tool we didn't have a use case for this but what happened was probably bought a year ago when the Apache struts vulnerability ID came out one of our developers came back and said hey can just I know we're using Apache so it can't just tell me like and it's vulnerable he just tell me where it's at and we didn't really build GHD to be able to do that but within about a day or so the leap was able to make a few modifications to it and was able to tell the develop like here's a list of where we actually have all of our vulnerable versions of Apache struts so

it's kind of like the tool kind of evolved over time when we started having an introduction and working with the developers and they saw the value of this too and then forth a private repo enforcement so if someone actually does make your repo public we GHD can go back and automatically make it private and we have a few configurations that you can tailor to and we'll tell a few more details around the stories of how we actually got there made actually usable so with that I'll transfer things over to delete yes so what do DSD do about multi-factor authentication right like what as a tool as a solution what does it do so MFA enforcement so that itself

says that whenever a user like if you are not using it an enterprise account suppose you are using a regular organization account and then if you want to add an user to this organization you don't know whether you have MFA enabled on his account or not so unless you add him to the organization so that's an issue right now we are facing across so who is actually going to go there and see whether this user have MFA or not so ghe is an automated solution that this actually looks into the organization and once you add these users like it finds out if there is no MFA enabled on its account it's automatically going to delete that

account or remove him from the organization and this is what they access in lifecycle management is and then for the credential scanning so how do we identify plaintext retentions right like there are a lot of false lashes when you try to scan across the code and github or any of the repos so for this credential scanning right now we are focused on AWS keys because most of our applications uses AWS as a third party solution for at least email sending services or whatever it is so so this GG what it does is it actually looks for aw skis like access key IDs and all within the code and then alert us it can be there can be like lot of false positives

in it but then how can we fine-tune it so we were able to fine tune it by using like the configuration properties or where do we keep all these in the XML files so look at only the XML files and find out what are the access keys and what are all stored in that and then alert us so it is right now we are around 90% of like not having false positives and we are good with the solution and then the other thing is like finding out the third-party solutions like we mentioned like Josh mentioned before so what saves as developers use in the prod environment and non Prada environment so is there any authenticated solution that we are

going to propose that we use this as a common across the whole environment and then managing vulnerable third-party libraries so this is similar to what do we do for AWS keys so it looks for any of the Apache struts like Josh mentioned and then finds out the version numbers in there and then we can actually say that okay so this is a version number and then is it vulnerable right now within the like globally known CVE or any databases that you know and then you can get the version numbers and then compare it with what are all the projects you have in your github and then say these are all the repos are the applications that are vulnerable to this

specific issue and then we got the detailed reporting once it sees that disease the vulnerable version within this repo it can actually send out an email to the people so we have covered all these in the demos and in the further slides and then private repos enforcement so Gigi automatically looks into our organization once you like get it running so it actually looks into our organization and if you see some developer is making a repository public even though that code is proprietary to your organization so it actually scans for all those repos and see if anything is public it makes in private right away and then it alerts us as a security team in the end so that we kind of follows up

with them and then we have the exception management like it says so if someone wants to make a repository open source so you have to allow them but then before doing that we should have to do the credential scanning and all these security things that we have to do so this is really helping us on that side too so yeah I also switched over to the demos real quick so we'll start off with the AWS key management piece yes right now here you see this a demo organization called G open source and it got a lot of these repos demo 1 2 3 GHz and GZ demo like once you set up the GG a tool and you start make it

running it actually scans for all the repos and then sends out an email saying that okay so this repo and this file contains the access key IDs and secret keys so if you have a proper authentication to the repo as a owner or of the trip or as an owner of an organization you can actually go into that file and then you can actually see those access key IDs that are actually like written as plain text in the code and then submitted into your organization what it does it actually looks at it and sends you an email as a security team or the owner of your repository and then this is for the multi-factor authentication enforcement

so we are trying to see that this is a non enterprise account so how can you add an user or invite someone to your organization so this is something like you can just say like this is a user can you add me to my organization and you not know about his details like whether he of MFA enabled on his account or does he have like proper like access and also you sent him an invitation and then he accepts it till then you don't know his attributes like whether he has MF enabled or is he part of our organization or not so I use sitting there waiting for someone to accept invite and then look at him and say like

okay so this guy doesn't have I'm afraid he doesn't follow the security policy so we should remove him it is basically GG automatically looks at all these users like when once you it is a time-based or even based you can actually set it up like that suppose you say like run every 1 hour depending on whatever reason you have so it actually looks at it and then see like once you get added to this the list of users it actually finds out that there is no 2f enabled on his account and then in in some time like it actually scans for these users and removes him and then sends out an email to the user as well as you saying that

this is the user who got removed and this is the reason why he got removed into from your organization so you can see that it gets removed and you can actually have the MFA white listed for some accounts map if you want to use it for service accounts suppose you say and then they want to use a hardware token or something so they don't necessarily need it to FA so you we are also providing that kind of capability here and then you can see the notification there yes so this is one of the other features like we mentioned before the private repos so like this is basically automating what I use or the security guy should work on every day sitting at

their organization account and looking for all these things so we are actually creating a repo and subsidies a your organizational repo so it contains your intellectual property which you are riding to the repository and if you're making it public that's not good so if you see 2/3 is not a private repository so like I said like if you make the GHz tool based on time paste or an even paste it can actually go in and like look for all these repos and make it private automatically so you don't need to go through it and then make it private or right for someone to do so these are all the capabilities of gog and many more to come

but this is what we started with oh thanks too late so we realized we're not the only company that's solving trying to solve these types of problems and if anyone has any more use for this this is why we decided to open source it I think we when we first started it only solved a few use cases that were maybe niche plays and wasn't useful to maybe everyone but at this point we develop enough we start getting a little feedback from other other people we worked with in the community they thought it was worthwhile that they actually shared this so we decided we'd do that now we still have a lot of improvements be able to make and some

more keep they'll be rolling out in the next few weeks in months or so but I'm hoping that some people will get some use out of this at the current state it's at today so set up so the requirements are pretty simple if they'll it takes about five minutes actually set up all you need is to have owner permissions for the github account you're trying to manage and then we use Amazon's SCS service if you're not familiar with SCS it's at AWS is service for sending out emails to send out the notifications as well we got some feedback that some customers want but some people wanted to be able to set up like their own private

SMTP server I mean that's that's cool but we haven't I think everyone else when you said just use AWS most people have AWS even it's a minor presence they were cool with that too so those are the only two things you need there's some minor configurations that dilip had mentioned that you'll have to actually consider do you want to be time-based do you want to be in force or actually alert a time-based this is one that you can say put it to zero make sure as soon as anyone uh makes a ads in a repo that's a public you have to turn to private immediately any time a user adds into your organization that doesn't have

MFA in place make them actually kick them out if they don't have it so we crown some more feedback that some custom customers some other people were working with said like well we want to give them an hour or we want to give them a day or something else like that I mean being flexible I mean it's up to you to determine what's best for your organization in your use cases but that's why we put in a little bit of a timer in there instead of making a time-based instead of a vent base so a lot of customers we found that was actually helpful and if you're willing to live with that risk of having a user

account without having MFA in place through a few hours I mean that's up to you guys to be able to decide that for your own so with that the leap is gonna take us to the demo for the set up and show you how simple it actually is I mean we have a readme text in our github repository so which is going to go through it but it's just only this configuration file that you need to fill it I mean I know these are like like wow you're giving the plain text passwords here and then like plain access key IDs and secret access keys but just for sake of my use I did it like you guys are

feel free to like change the configuration file to like any safe or encrypted passwords in here so just know you still fill out this configuration file and then you should you can run the modules independently like you can only run MFA module or you can only run the AWS key modules we have that like flexibility of you in separate modules and thanks to get help they are providing us the API so you can use in leverage whatever you want and for everyone who's gonna try and I know look at this right now I'm trying to access this SCS account using this key they're already expired so I know you're still gonna go ahead and try

but they're doing there longer yeah it says example in there so yeah I just better use it take our word that's a cool I set up demo so what's coming up next one of the other features that we really like from a github enterprise was the ability to sync with Active Directory so when actually someone leaves your Active Directory which most organizations do a better job at manage that under third-party applications it'll automatically sync up and you'll be able to remove any of the accounts that are in github based on the Active Directory status that's one of the other capabilities wouldn't be looking to add in next few weeks or months or so the second round is a API security which I'd

mentioned before right now I mean key rotation is our focus but if there's any other feedback or any of the suggestions we're always open to listening because if you guys got the use cases it might solve some of our problems too that we didn't even think about before either so we're open to or opened for more suggestions on API security too and then lastly credential enforcement for encryption with our environment we haven't got to the point that we're able to reject commits base on putting text passwords being in place but we've been very fortunate we built enough awareness that is not that big of a problem we have a very few from time to time to actually have developers

actually do this piece so for us it's an alert capability and it's been good enough for us but we're looking to move that into an enforcement capability as well - thank you guys and I believe there might be time for questions I think we went really early for this one yeah oh

I can run my cool alright alright maybe you can elaborate on how you deal with false positives or you know perhaps whitelisting and an example I'll give is I have a cryptography library that I need to test key import from a bunch of different packages old versions of OpenSSL and whatnot and I actually recently was contacted by someone soliciting a bug bounty because they thought they found private keys in my repository and they were looking for a payout for them telling me that this was somehow a security vulnerability so I think this tool may suffer from a similar problem where there are actual cases where you do want to private key material inside a repository and a unit

test or something like that if I'm understanding correct how can we remove false positives suppose if it just says key in there and someone comes to you and say this is actually a key that's what your question is right so like we are doing right now in the end like what's next when Josh was mentioning about it he says credential encryption right like once you enter it or get into as like a pattern that says all of your keys when you encrypted they comes with a cipher or something in there so you actually look for non cipher words within the keys and say this is not an actual false positive or a false positive like you know what I'm trying

to say right like all these keys if you make them either cipher or encrypted and then get them to a same format like it says started with a cipher or like after you encrypted it so to get them into a single format that will help you to reduce the false positives as far as my concern like because when I started I have the same issues but understanding my are our application configuration and say like all of these XML files only have the passwords because applications take all these from the XML files so if if it is coming from XML file saying that this is an alert saying that ok dot XML file is having a

plain text password sent then so then I'm like at least 90% sure that this might contain a password and then on top of that like once you encrypt it and then since the application only takes the decrypted password so it should come up with a format and then when you decrypt it it should be able to be understandable by the application so try to get into like a specific format to get alerted so which we are doing right now we are not yet in that place

you mentioned stale account management what are you doing right now or what do you have plans to do in that in that regard well I mean around the it's really around Active Directory peace we were able to better manage some of the capabilities around the MFA token which would then hence get your ax kick you out of get up but if you don't it's not really a halfway solution so it'll return make the access but it's not really practicing the data hygiene we wanted to be able to do that's why the permanent solution or I would say the real solution is gonna be the sync up with Active Directory so it's it's it's a very very helpful right now but we

understand it's it's because it terminates the access but it's not where we want to be all right thanks everyone [Applause]