GitOops! All Paths Lead To Clouds

thank you so much all right everyone i'm alex and today we're going to smash our way through some github organizations so i want to start by qualifying what this talk is and isn't about this is not a talk about gaining access to github we're going to assume that maybe we found a github token laying around somewhere maybe we've compromised an engineer laptop that engineer has access to github maybe you've compromised an application that's got some permissions on github somehow you have some github access this is also not a talk about what to do once you've pivoted into a cloud environment or some other kind of managed service this is a talk about what happens in between how can you

translate github access to access to cloud environments or whether whatever other kind of service you'd be interested in that has some kind of integration with github all right i also just want to point out that this is like an intermediate level talk there's nothing technically difficult or complex about what we're going to talk about it's all straightforward and simple stuff but we are assuming that you're familiar with ci cd pipelines terraform things like that okay cool so quick rundown of the agenda we're going to start with a brief introduction to ci cd attacks we're only going to scratch the surface here very simple straightforward stuff but then we're going to kind of wonder like if we're targeting a very large

organization or maybe even a github enterprise with multiple organizations uh we're going to face a bit of a needle and a haystack problem all right so how in that situation do you find the repositories that you're interested in that you can exploit and to do that we're going to build a graph shove all that in a graph database and then we're going to query some attack paths cool all right so let's actually get started with the real stuff ci cd attacks a quick overview of this section we're going to start by discussing secret exfiltration from circle ci and github actions does anyone here actually use circle ci or has used in the past okay there's

some circle ci users quite a few okay i was hesitating i thought i would maybe cut that out of the talk but i won't since there are some circle ci users here so start with secret exfiltration from circleci and github then we'll see how we can attack some terraform plans and then we'll look at a couple of tips and tricks these attacks all have one thing in common the only thing you need to do to pull them off is be in a position to open a pull request so assuming you can open a pull request against the right repositories you can pull these attacks off with some caveats all right so circle ci context so we

have some circle ci users here you guys probably know what circle ci contexts are actually for both github actions and circle ci there's like two ways that you can manage secrets okay so circle ci one of those ways is to use context and contexts are essentially a way to bundle environment variables just a heads up i'm going to use the words environment variable and secrets interchangeably for the rest of the time okay so there's sorry contacts and circle ci's way to bundle secrets and reuse them across projects and let's take a quick look at what this looks like on the web panel all right so this is the organization settings and you can see here we have a

three context defined um among the people familiar with circle ci can you tell me what's wrong with this picture anything that strikes you as probably maybe bad all right so we have three contacts here and you can see that there's a security column and the security column uh tells you which members of the organization can access this context okay and when you see where it's all members as you probably guessed this means all members of the organization can access this context and if you see something else that means that it's restricted to one or multiple teams okay so in this case the front end context can only be accessed by the front-end team all right and we see down here we've got

an org context as the name indicates this is probably intentionally accessible to all members of the organization but here we have an admin context and this is probably not intentionally accessible to all members of the organization so why can't this happen it's an unfortunate thing with circle ci is that anyone in your organization can come and create a context and add environment variables to a context but only a github admin can come and restrict those contacts to a specific team which is quite quite unfortunate because either a couple of your github admins are going to spend their days juggling context permissions or you're maybe going to write some automation to do this which means that you're giving a

service github admin privs which is also not something you probably want to do so cool i don't know if i actually have secrets configured let's go and look at front end all right here we have a secret token um in the front end contact that is restricted to the front end github team so if you're a member of this team you can pull this token out all right that's circle ci context um an interesting thing with circleci as well is that there's no concept of branch protections for secrets you can pull those secrets wherever whichever context you're running your job in whether it's after push to main branch or a default branch or production branch or whether

it's just a pr feature branch or any anything really um so these may have team restrictions as we said and here you can't really see this very well unfortunately but this is just a basic config.yaml that would show you how you can exploit this so here we're defining a job that we've called x field creds we specify that that job is going to use the front end context and here all this job does basically is exploit the environment variables and post that using curl to an endpoint over the internet okay so depending on on where this is running like you may not be able to exfiltrate things like this maybe people are using self-hosted rona's they're running in an environment

with some tight egress controls so you may have to get a bit more creative to exfoliate these secrets but chances are like if you're if you're running arbitrary code uh in a build you can probably find a way to extract secrets even if it's just like encoding them and pasting the output and then just reading the log output and decoding the secrets cool so that was circle side context um circle ci projects pretty similar but instead of like reusing these across projects this is a way of attaching secrets um to a specific repository um and then having those accessible to the repository so let's go look at these all right so here we have our circleci

pipelines let's look at this project called aws infra so there's a one-to-one mapping between projects and repositories so there's an aws info repository on github if we go look at the project settings environment variables okay we have a secret token here another thing to note that's quite interesting with circle ci and this is relevant for something i'm going to demo a bit later is we can actually get the last four characters of our secret or environment variable out of circle ci this is just quite useful and we'll see in a bit where that may be useful all right so pretty straightforward but again there's no concept of branch protections here um and here to explain

these secrets all you need to be able to do is open a pr if you can open a pr against the repository you can get whatever secrets are configured there and very similar here we can just exfiltrate these by exporting the environment variables and posting them to an attacker control endpoint or maybe you have to get a bit more creative depending on your environment cool that was circle ci um github actions uh the line is a bit blurred between what's github and github actions so it's all kind of the same now but github has two ways of attaching secrets to a repository the first way is very straightforward it's the original way the one that was around since github

actions was launched and that's repository secrets so let's go check this out all right here i have a fake github organization we're using for demo purposes and i think it's this one console spa let's go look at the settings secrets all right so i'm going to ask you to ignore environment secrets for a second here and just look at repository secrets so repository secrets directly attached to repository and very similar to circle ci project environment variables there's no concept of branch protections here and if you can open a pr then you can pull those secrets out and here we have a sample of a github action config exactly pretty much the same idea except you need to change the way you're

getting your secret out it's no longer exposes environment variables it's going to be in kind of this like secret object cool the second way you can get secrets in github actions is to use environment secrets and this was launched relatively recently i think about a year ago december 2020 if i'm not mistaken and this actually introduced several ways that you can protect your your secrets and make sure that they're only accessible to some specific environments so here for this repository we have two environments to find we have a prod one and a dev one you can see that they each have one secret if we go into our secrets here we can see that there's a

dev secret token in the dev environment and a broad secret token and the broad environment um and like i said this does offer some ways that you can protect your secrets um the one that's particularly pertinent and interesting is this deployment branches one you can think of this as branch protections for your secrets so you can restrict secret access to say uh pushes to your deployment up so to your production branch or your main branch or your default branch or whatever um you can restrict product production secrets to those events cool so in this case pr may translate to secret access depending on how lucky you are i should i should point out that you

know as you can see here configuring this is totally optional so you may very well find that some places and some teams are just not using these protections because you actually have to go out of your way and do it cool um something a bit different now we're going to talk about terraform plans and how we can abuse these running plans on pull requests is a bad idea but it's something that a lot of people want to do and the reason people usually do this is they kind of want to see like how a a changes to terraform are going to affect an environment once they're merged in and for that reason you may think that

it's a good idea to run a terraform plan on unreviewed code and a pull request and then say hey this is what the plan is going to do to your environment all right but there's a problem with that and the problem with that is that if you're familiar a bit with how terraform works terraform is actually going to refresh its status or its knowledge of your environment when you run a plan and that means it has to have access to your environment so terraform has access to your environment when you're running a plan so how can we abuse this um let's go and look at some infrastructure as code here so we have this repository called github as code

and make this a bit bigger maybe cool um so if we just go check the pull requests here uh this thing make change and if we expand this we can see all right clearly we're running some terraform plan on a pull request okay now let's go and check what this terraform is doing all right here we have our main.hcl and we can see that we're using two providers here we have an aws provider and a github provider as you probably guessed by the name the idea here is that we're managing github as code with terraform which could be a good idea in some in some contexts um and here what are we doing with the aws

provider you can see that we're actually we're pulling a secret um called github token out of aws secrets manager and then we're passing that token to the github provider and that's how we're authenticating to github all right so where can this go wrong in a plan well you can leverage that access to github or you can exfiltrate the github token and how would you do that here's an example of exactly how you do that the um there's a thing in terraform called the external data source so you can use the external data source in terraform if you want to interface between terraform and an external application to get some data that you then want to use within your terraform

if that means nothing to you what this essentially means is that you can run arbitrary scripts in terraform okay and here we're running a script that we passed with this pr that we added with this pr up here um and we're calling the script and we can also pass some stuff on standard input to the script and in this case we're passing the github token to the script okay and here what are we doing well we're reading uh information from standard in uh we're also getting the environment variables because why not while we're at it and we're posting all this to an attacker controlled endpoint over the internet all right and just like that we

actually traded the github token also anything else that happened to be in the environment variables um and this is why you shouldn't do plans on terraform pull requests or where they can be dangerous um this is actually maybe not the best example i could have given so let's think about another scenario here let's say that you have a postgres database that is in a tightly controlled environment with tight egress controls and one of the only things that can hit that postgres database is your your cicd runner because maybe you want to manage that databases code which is you know maybe a good idea as well um but in that case your your terraform actually has access to the database

right it's managing that database so it has some kind of access to it um so by opening a pull request you can maybe just start like reading customer data out of the database and then excel trade that somehow okay um so that's why running terraform plans on pull requests is a bad idea um a little uh pro tip here um if you are doing this in an offensive engagement you may want to consider opening draft pull requests draft pull requests will typically trigger ci cd pipelines but they won't alert repository maintainers so a lot of teams these days they'll have like integrations on slack if someone opens a pr on the repository boom they'll get an

alert i'm not an alert but like a notification and they'll maybe go check out what the pr is about so you probably want to fly under the radar if you're doing some kind of red team or whatever so consider opening draft forward pull requests if you're doing these shenanigans cool like i said this is only scratching the surface of the kind of things you can do with this um github actions has received a lot of attention recently so here's a couple of blog posts you may want to look into if you're into github actions um but there's also a lot more than github actions like if i was bored and i was looking for a research project what

i'd probably do is i'd list all the common ci cd integrations that people are using these days maybe look at the integrations that your company is using and then go poke at those because i think this is like a very under researched area and there's we're plugging everything into github these days and things are going to go really bad at some point cool so a quick recap of this first section we see that we can often pull secrets out of repositories or ci cd systems just by opening pull requests in a lot of modern environments like it's very common to just give a bunch of people uh permissions to open pull requests you know because it's all about

collaborations and you have to help other teams and fix each other's bugs and everything like that um we've also seen that production terraform plans on pull requests is uh risky or maybe just a bad idea uh draft pull requests don't alert maintainers if you want to fly under the radar and there's a whole bunch more ci cd issues that are just not explored here that i recommend that you explore and if you're looking for some research ideas like this is this is right cool so now um this is all cool but let's say you actually want to exploit this if you're an attacker or you're a defender and you want to find where you have these problems in your organization

you're going to face a bit of a need on a haystack problem um so back when i was a pen tester i had a couple of occasions to go free roaming on some very large github enterprises and i'm talking about 10 000 plus users like they exist they're out there several organizations dozens of teams thousands of users thousands of repositories and it gets very large and very messy all right um so how do you how do you go about finding these problems in those situations well um i think we're all like familiar with uh graph dbs they've received a bit of attention in in the security field there's tools like bloodhound and cartography out there

so i figured like why not why not just take that idea and use it for github and ci cd so essentially we built a tool that queries github and circle ci apis we're focusing on github and circle ci because that's what i work with um we query a bunch of data out of the apis we massage that data into some schema that makes sense for what we're trying to do and that gave us our tool gurus i'll show you the repository uh at the end of the talk so cool i'm going to speed through this because it's relatively uninteresting but i just want to introduce you guys to the schema that we're using here um hopefully you can see this so we have

three users alice bob and cassie they're members of some teams and those teams have permissions on some repositories and one thing that i've not illustrated here is that users can also be direct contributors on repositories so have access to repositories without team membership without going through team membership okay cool so users teams repositories we want that information uh what information do we want out of repositories uh one of the first things we do is we pull configuration files for uh known popular ci cd systems it's like config.yaml or anything and github workflows etc we also do this for further cic distances not just circle ci and github actions we also want to get web hook information

so looking at the web hooks that are configured can give you a whole lot of information about what kind of integrations or what kind of magic happens on events to repositories and we also want to go look at status checks so status checks are these things that you can see in pull requests um they're actually present on every push not just a pull request so we pull status checks for pull requests and uh pushes to the default branch that may be extended to support other branches in the future cool um as far as github actions and secrets goes uh this is what we pull out so we pull uh secrets that are directly attached to repositories these

repository secrets uh we also pull the environments for the repository and the secrets attached to those environments as far as circle ci goes we've seen that there's context and project secrets so in this case we have alice she's a member of the admin team the admin team has access to the admin context in our case we've seen that pretty much the whole organization actually has access to admin context but also the admin team and the secrets attach the admin context but there's also a project a circleci project attached to the repository and secrets attached to the project okay so this is basically our schema the tl dr uh not very interesting from a technical point of view but query query

a bunch of apis massage that data into a graph tv boom john done all right now let's look at querying some attack paths so what i'm going to show you here we're using neo4j as our backend uh graph tv here um and what you're seeing here is just i think it's called bloom which is the front end for neo4j okay um first query we're going to look at we're going to say let's imagine that we've compromised alice or alice has been compromised um what secrets can you guys see that is that all right is that good all right so we're just going to imagine that alice has been compromised and what secrets would the attacker be

able to access after alice has been compromised let's run this cool so it looks like there's a lot to unpack we're going to go through these branches one at a time there are some labels up here that help you match colors to nodes first observation here as we can expect from our query the central node or the root node is our user alice and the leafs are the nodes on the periphery and these are secrets cool let's start with this one here so we see that alice has direct permissions on a repository called cquery and this repository has some aws credits attached to it and this is great so if you can open a pull request against

square you can pull these credentials out alice also has access to the swiper repository which has two environments dev and prod and they also have aws secrets presumably development aws secrets and production adobe secrets um as you can probably expect from the dev one like there's no branch protections on this one and this this may make sense um maybe you do want access to um your dev environment from a pull request to run some tests like this is a very sensible thing to do um so you can see that the protected branches are set to false down there we can actually see in this case that there are no uh protected branches on the broad

environment and that that may be a problem so you could also run a query like for example you want to find like all um all environments that don't have protective branches in your organization you could do this with this tool cool let's look at what's going on down here alice um she's a member of the organization so she has access to this artwork context which is available to everyone in the organization and there's a data dock token there no big deal she's also a member of the admin team through which she has access to the admin context and from where she can get an auto factory token she also has access to the web so

repository by being a member of the admin team that has a web source circle ci project configured for it and there's some github secrets here um pay attention here that like i said with circle ci we can extract the truncated value of the secret which may come in handy um there's also a little interesting story here this is completely tangential um if you if you go looking around and you start finding github tokens um often you'll find a github username next to the github token the historical reason for that is that at some point github allowed you to authenticate to the api with your username and password and the sdks were just like developed in

a way that you know made it very clear that you could pass a username and a password to the api and at some point obviously they figured that tokens were probably a good idea it's like 10 years ago but those sdks in some cases look exactly the same so you have some developers using some github sdks you think that they have to give a username along with the token when in reality that's not the case but the interesting side effect here is that when you start finding github tokens in the wild if you look around like you'll usually find a github username next to it so that's quite cool um and we'll see what we can do with

that in a bit all right so here we've asked ourselves like we've compromised alice what secrets do we gain access to another question we may ask ourselves maybe more if you're a defender or even in case an attacker who do i need to compromise to gain access to something all right this is the opposite question right here we're looking for paths between a user and environment variables uh where the environment variable contains artifactory so here we're trying to figure out like who do i need to compromise if i want to gain access to auto factory and maybe pull off some kind of supply chain attack cool so let's try to organize this um as a tree to make it a bit more easy to

interpret let's give bob up here chassis over here all right this looks a bit more like very little tree all right so we've already seen this path to auto factory from alice alice is a member of the admin team that has access to admin contacts that has our factory token attached to it and here we've found also that bob and cassie members of two different teams but what those teams have in common is that they have write permissions on the monarchs repository which has a github repository secret attached to it as well so that's another path to auto factory that we've identified you can also see that this user called fakebot as you probably guessed this is a fake

github user it also has access to the monax repository um there's another interesting observation here is that like as a consultant and as a security engineer so far in my career i've never encountered a github organization where there wasn't some fake github users used as bots that's because there's a lot of limitations and things that you can do with github apps so quite often developers just like create a fake github account and use that personal access token for that user as a bot but these are very useful if you're trying to like move around the organization as an attacker and that brings us to the next query now a few stars need to align to pull this

one off but i thought i would bring it up because it's quite a fun one to do if you can do it in practice so we're going to try to predict the access that we can get by pivoting through github bots that we have access to in the organization all right and we're going to use everything i just said about github bots the fact that you often find a username next to a token and the fact that circleci gives you the last four characters of your secrets so what we're doing here is we're looking for paths between alice and environment variables where the environment variable looks something like github user which means it's probably a github username we're going

to get the truncated value for that right and this is the last four characters all right then we're going to look for paths between a user and environment variables where the user's login ends with uh the truncated value oops all right does that make sense to everyone what we're trying to do here cool we're trying to pivot through github bots and predict the access we get and this is what happens in this case so we can actually see that starting from alice all right we're starting from alice but we see that alice actually has probably has access to um a github token and if she pulled that github token out then she'd get access to fakebot and we know that it's fakebot

because we have the last four characters of the github username and that fakebot has access to these two repositories which also gives alice access to data dog and artifactory indirectly like i said a few stars need to align to pull this off in a live environment um the reason i bring it up is because i've seen it with my own eyes um cool let's move on to hub repositories this one's uh probably interesting as a defender if you're trying to secure a github organization um what we're defining as hub repositories are hub repositories are repositories that have a large number of inbound relationships okay so the repositories that a lot of people in the organization

have access to usually intentionally and they're very collaborative repositories what a hub repository is may vary depending on your organization like maybe you're just going to say like you know if 20 of my users have access to repo i'm going to consider that a hub repository or maybe it's going to be 50 or maybe it's going to be 100 users you know whatever makes sense for organization but what we're looking for here is an understanding as a defender like um we all know the security is not about eliminating risk right it's about managing risk so you maybe want to manage risk around repositories that a lot of people have access to because there's a high percentage chance that if

you face a compromise one of the compromised users is going to be able to open a pr against those repositories so we're looking for uh hub repositories that have like here are just looking for token but it could be anything else that you know is particularly sensitive in your environment where the user count is above a certain threshold all right the results are not particularly interesting in this case but we can see that in our case uh monax is a high risk hub repository and this may be a repository that we maybe want to manage secrets differently for it and ensure that we're not giving everyone access to sensitive stuff cool last example i wanted to show was these

terraform plan attacks that we've mentioned before so here we're looking for relationships between a user and repository so users may have direct or indirect access to repositories so we're looking for up to two degrees of separation we want those repositories to have a status check on a pull request specifically because a pull request is a requirement for the attack as we outlined it where that status check looks something like this and this is a bit of a funky regex but essentially what this says is we're looking for things that have tf or terraform and fraud in them as long as the prod is not preceded by non so we're looking for things like tf plan prod terraform product plan

anything like that is going to get matched okay so let's run this boom so what we see here is that david and eliza are members of back-end team which has access to the flappy repository and the flappy repository runs a production plan on pull requests maybe something that you want to fix or maybe you actually want to go see what that terraform is doing and make sure that it's not too crazy awesome that was it for the examples do i have much else to discuss here yeah so um yeah this was only like a sample of queries um but i actually had a lot of fun with this and there's a whole lot more stuff that you could do

and may want to look into you could do pretty banal things like looking for repositories that don't have branch protections but have some kind of cic configured for it so you could look at like web hooks status checks ci configuration files to identify those and look for branch protections being disabled i don't think i showed it in the schema yeah we didn't show it in the schema we actually do pull branch protections configured on repositories um another thing you may want to do if you're a large organization is audit the indirect access that your external contributors can get maybe you work with some contractors who have access to the organization so you can look for users

who are not members of your organization but who do have access to some of your repos and make sure you understand the impact of that um you can also go hunting for like who's using legacy ci cd in your company like is there some jenkins boxes laying around in the corner um and you can also do some regex matches on ci files if you know exactly what you're looking for like we mentioned there's a couple of foot guns with like github actions you can maybe go and find those foot guns like this it's maybe not the most efficient way to find those uh but it could work cool i do want to give a shout out to my

employer um kalooza we built a smart energy platform we're doing smart things in the energy space they gave me a lot of time to work on this and also supported me and come and talk to you guys here we are hiring engineers not just security engineers all kind of engineers and finally i know exactly why anyone comes to cloud and devops talk so here you go the pooping unicorns for you all i'm alex k underscore alxk on twitter please hit me up i love to chat about these things i will show you our repository [Music] if this loads [Music] maybe i'll just type it manually all right that's right here actually cool so good oops uh overtex get oops um

the best name you've ever heard for a hacking devops ci cd tool i'm quite proud of myself you can see the readme is really quite slick it's a pleasure to the eyes our documentation is also quite complete um instructions on how to install build and run this examples we've got some more example queries over here um stuff not covered in the talk our schema is fully documented here if you understand exactly what we're ingesting and how it's connected to other components blog post that pretty much covers the topic of this talk and that is it from me i would be happy to take any questions from anyone

thank you if you uh like to put your hand up for questions i'll bring the microphone around

uh really great talk thanks um thank you i what i see a lot these there's people using github and circus ui to try to manage mono repos is it even theoretically possible to secure a monorepo using the built-in secret management of those platforms yeah that that's an excellent question and it's one i don't want to comment on because i have no experience with mono repos so yeah i mean yeah it sounds like it would be a great blog topic because i have no clue how you run a mono repo at scale sounds crazy to me anyone else getting the workout

[Music] so um i just learned that it's really bad running plan on prs what would you recommend doing instead yeah not running plans on pr's but there's actually there is um and actually at um in our company we've actually done some work um towards safely running telephone plans um i'll tell you this much there's gonna be blog posts actually if you if you go check out our blog if you type overtek on google you'll find our blog there's one article that explains how we do this for um infrastructure ci cd um and actually sorry this blog post is gonna be coming out soon explaining how we do this safely i'll tell you that it's contorted

it's not fun developers will probably complain but it is doable thanks

great talk thanks alex um have you tried doing any of this against um as your devops and it's github repos or things like that no i'm tall would you like to would i like to yeah sure is this a job offer um what i can um so i i have heard from people more in the know that me that basically github actions is like rebranded azure devops maybe a lot of the same foot guns are around there but i don't know i have no experience anymore

what's the worst setup you've actually seen in the wire or door on the company you've done a pen test for yeah um so my consultant days are a few years old like full disclosure i've actually had this um this tooling idea in mind for about like three or four years um dating from some of my last pen tests but i would say it's pretty much the things i've discussed here except at a massive scale like the way we've just like started adding integrations without any thought or any concern for security um is really it's it's everywhere um and obviously the bigger and the older the company is the worst it tends to be um there's actually an awesome code that

i i tried to fit it in this talk you guys know who robert heinlein is he's a guy who wrote starship troopers if you've not watched that movie you should go watch it and then read the book excellent stuff very relevant teams today's political climate but his favorite court is famous for is saying that progress is made by lazy people um looking for better ways to do things and i think that very much summarizes everything you see in devops so you can just imagine very smart very bright people but very lazy people who just got fed up with manual deployments and everything like that and i think that explains why there's almost no concern for security and cersei

thank you one more thing and then we'll shoot you on time [Music] hi thanks for the talk i just wanted to ask i don't know if you said it already how you populated the neo4j database like you navigated through the pipelines and github yeah yeah so sorry what what's the question specifically i maybe did a very poor job at explaining this but we basically wrote this tool good oops that queries github and circle ci's api and massages all that into a graph database so i didn't show you the popular the populating step because it's very boring it's just going to terminal do an api request and massaging stuff into a db but yeah that's the answer and what kind of

access do you need for that uh yeah that's a very good question actually um so if you're if you're a defender like try to get your org admin to give you temporary admin access to run this and then you'll get the omniscient bird's eye view where you can see like everyone's permissions and do these queries if you're an attacker you can still get most of that information but from your perspective so you're limited to what you can see so if you don't have like um access to some repositories and you're not going to see who has access to those repositories you need like maintainer or admin access to repo to pull that information now but then like in circle ci the

permission model is pretty weak so you can get a whole ton of information about the entire organization um so yeah the answer to that is um you can run the tool both as a member or as an admin you'll just see a lot more as an admin thank you i think that brings us just up to time awesome thanks everyone alex says excellent presentation