Chris Farris - The Cloud is Dark and Full of Terrors

all right so for the next uh you know 25 or so minutes i want to walk you through some of the ways that being in public cloud can really come back and burn you um you know in a lot of cases when your company has moved to aws azure google it happens organically it happens through developers saying hey um we just opened this aws account to go run our marketing website and corporate i t infosec aren't involved and next thing you know oh hey we've got public buckets on the internet um you know we've got a whole slew of terrors and risk in our environment so

bad computer um who am i uh i've been doing uh it for about 25 or getting close to 28 years now discovered the cloud around 2014 and had been moved into an infosec role about four years ago mostly i work at media companies so you know not finance not health care not government um so where chaos tends to be more um yeah chaos just tends to be more prevalent um so code spaces we're not talking about the github product but we're talking about code spaces from roughly what we say 2014. um you know they uh had some admin keys that somehow got leaked not sure how could have been fishing could have been you know

laptop left on a train uh they got a ransom they said we're not paying this ransom and that was a mistake because everything you know got deleted everything got deleted their customer data was deleted their backups were deleted everything was deleted and all they could basically say was sorry everybody i hope you had backups of your source code on your local systems we're out of business tons more examples s3 buckets have been a problem since 2016 2017 you know national reconnaissance office left a bunch of classified images open in s3 buckets dow jones had a bunch of credit cards stolen that were left in an s3 bucket not sure why verizon had a third party

uh third party leave a bunch of their customer records out there but it's more than that you've got open kubernetes you know tesla got hit by that their kubernetes was left exposed somebody got credentials they were able to push crypto mining containers into the tesla environment um and then exactis you know they had the first example of a public elastic search cluster you know aws will just let you spin one of these things up and aws up until recently at least would let you make those public so the cloud is different i think everybody kind of knows that but you know i try to describe it as defending your network and if we look at castles and moats and

firewalls and everything you're defending your network and you still need to defend your network at that two-dimensional space but you can have that third dimension that cloud dimension that cloud plane that you have to defend um and so you need to not just be looking out on the horizons for armies coming to attack you you need to be looking up uh you need to be looking at the clouds to figure out whether or not your network is going to be attacked case in point from the cloud i can go and remove the firewalls that you implement through this one command now there's gonna be a whole bunch of uh s3 aws commands all of those are posted on my website so

you don't need to screenshot i just take the last screenshot of the last slide and that'll have a link to all of this um and so but there's even more than just the the network plane and then the cloud plane there's many perimeters of cloud and part of that has all to do with the cloud providers giving you now managed services you know cloud providers are all about hey how do we move the undifferentiated heavy lifting off of customers so you're not running an elastic search cluster that's not what your business is about your business is about analyzing data and you know providing value to your customers alas to search is only there as a means to an end

and amazon and azure and google are really trying to help you get rid of that undifferentiated undifferentiated heavy lifting to um yeah get rid of the undifferentiated heavy lifting so you can move forward in doing what you want to do um so what you end up with are you have identities how you are authenticating to your cloud service provider you have resources those resources can have permissions directly attached to them um and then you have what we traditionally know of as our network boundary and so all three of those end up being the things that you're looking at and the things that you're having to defend

um can't do a cloud security talk without at least bringing up one example of shared responsibility model um everything that i'm going to talk about here is really on your side of shared responsibility lots of stuff out there especially within the last week or so around or last month or so i guess around cloud provider vulnerabilities azure google have had a couple of those there's even a remote code execution that came out and some amazon aws related software um that is the client for their desktop as a service thing but all of this is really attacks that can be run against your environment that you're responsible for making sure don't happen another concept lateral versus vertical movement

because as you look out across your castle you know you can certainly go over land to attack you know another castle you can move from host to host from network to network but you can also move from a compromised host to the cloud and you can exfiltrate cloud credentials you can use those cloud credentials to maybe move to another amazon account and once you have permissions in that amazon account you can move back down and take over another host and so there's this ability to laterally move both at the network plane and at the cloud plane and then going up and going back down so before we jump into specific attack commands um what do you need to get started in this

how hard is it really to find a foothold into to an environment um so for thinking you know a very high level miter framework um initial access pretty easy and in a lot of cases initial access and discovery are one in the same because if you have an s3 bucket on the internet that clear your objective is to find and exfiltrate data there may not be any need to exploit or anything else it might just be there for the taking so discovery can be part of that initial access once you find the resources that you're looking for as i said resources can have their own policies so that allows them to be open to anyone

any ip address it allows them to be open to any amazon customer and those are very cl distinct distinctions that you need to be aware of is that maybe i can go and pull your data out without ever authenticating but maybe i just need to put in my credit card become an amazon aws customer and now i can go and pull some data out um devops means that credentials to cloud environments are all over the place um and so you know go look for what's sitting in a github repo go look for what's sitting in the ci cd server and then you know privilege um you know we'll talk about uh that but you know

if you've got what you think are harmless permissions uh they can be very powerful for example the all-seeing aws read-only access this policy is provided by aws and it is intended for you know hey i need to go in and look at something um a lot of times folks are like okay well this is a reasonable thing to give all of our developers it's safe they can't change production which means that you know that that's cool right yeah except that this gives you the ability to list all of the im users in the environment which is great if i wanted to launch a phishing attack i can enumerate all of the privileges that all of those users have

including my own to see if there's any privilege escalation that i might be able to execute um it can enumerate all the resources it can exfiltrate and download data out of s3 um there's aws feed only access which is the most powerful of these but there's also ec2 read-only access which you may also think is oh well you know it can't access s3 but it can tell you or can tell an attacker who has access to these permissions what are all my on-prem vpn endpoints what ip addresses are my vpns um and you know what are some of the third credentials that are there that those vpns need so even ec2 and s3 read-only access

are are very privileged and should be you know should be used sparingly so depending on the attacker's objectives you know they might have been able to get what they wanted just with read-only access but if they really want to mess around with your environment they might need to privilege escalate and there's a whole slew of of research around aws i am privilege escalation that i don't have time to really walk through in 15 minutes or in 20 minutes so i'm not going to dive too deep into these i would recommend looking at some of spencer geetsen you know rest his soul his work from rhino security um and then the other thing is look at

cross account trusts where in your environment are you trying one aws account to another because that tells you how to go and you know execute lateral movement attacks in the cloud plane all right so low hanging fruit is aws s3 buckets right um what is amazon s3 it's their storage service it's been around since 2006. uh because it pre-dates iam it had its own original authentication platform objects are immutable it's a public service which means it is not really defended by your network by default there's there's no firewall you wrap around s3 to protect it um as i said there's two ways to misconfigure it because it predates i am and it's a global namespace so i own the

s3 bucket for b-sides augusta by the way if you need it you know let me know so all right easiest attack is if you find a public bucket you can run this uh command no sign request and that will list you all of the contents in the bucket you can also do the exact same thing with curl if it is not an anonymously accessible bucket but is accessible to any aws account then you can use your own aws account using you know dash dash profile um and again you can pull down any s3 object itself just with a simple curl command um the first one is anonymous the second one here is anonymous and works well with tor

um third one you know this one you want to be careful of because you know that that's tied to your credit card um if you start exfiltrating data um through uh your own aws account you know somebody's gonna issue a subpoena and amazon is eventually gonna tell you stuff um there can also be publicly writable buckets so this happened to the la times uh they were tricked into you know um or they left their bucket publicly writable somebody uploaded some mage cart uh and uh la times you know served that up for about three weeks uh they were the first but they weren't the last so what does that look like how simple is it to

execute a publicly writable bucket exploit yeah it's running the exact same command if the bucket is publicly writable you can drop anything in there just with a simple aws or a simple curl command and then to verify that your object is there again use that curl command if you're looking at a bucket and you see a bunch of poc.text files that's somebody saying hey i noticed your bucket's public and here's my proof of it you might want to go fix it and if you see those poc files that means they haven't probably fixed it because they haven't cleaned up the evidence that somebody had proof of concept of the exploit um aws account ids we're going to talk

about and a couple of slides ahead but one way to figure out who owns a particular bucket is you can just kind of iterate through looking at error messages all aws account ids are 12 digits long it's been a while since i've done high school math but i think that's 12 factorial which is not that big of a number um and it's reasonably quick to brute force um you know what account owns a bucket simply by creating a pattern or a policy and then just iterating down through the 12 digits until you find the correct error message and then the we are clouder folks uh you know have a whole tool around giving you give it a bucket and they will tell you

the account id another fun area of exploit with s3 is subdomain takeovers so typically what happens is hey we're going to decommission something well what's the expensive piece the expensive piece is s3 okay well let's go delete s3 okay we're we've shut down the system we're saving money except you didn't delete the pointer to that bucket remember how i said s3 is a global namespace well now i can go and register that as three bucket so this command here will list go through if you have aws read only access it'll go through all the hosted zones and it will look for all of the records and it will look for anything that's pointing to s3

at which point i can go and create the uh bucket um and now i'm serving content as you hopefully i'm serving non-malicious content so tons of things in the cloud can be public um more and more things are becoming public every day because amazon and azure and google the major cloud providers you know they're releasing new things all the time and so the horrifying thing for me trying to defend my environment is what the heck did amazon just released last week because i've spent the last week at a big hole in the ground in arizona what did amazon do in the last week and what do i need to be telling my development teams about so

you know container registries i can make my containers public i can leave boot volumes i can let anyone invoke my own lambdas um you know uh topics uh the pub sub infrastructure the queues heck i've even seen i am rolls that are publicly invokable allowing folks to basically make a public bucket without making a public bucket so what does it look like if we wanted to do a container registry exploit um you know all you pretty much need to do is as any account but you do have to do this you can't do this anonymously you know get a login password and then log into somebody else's um ecr if you get a login successful you can

list the images uh and you can download the uh the containers um so that's one um if you've got a little bit more permission you can um you know write this little json policy and upload it with this command and boom now you've just made somebody's ecr public so now you can go and pull things out really easy to do um if you just have read-only access and you're looking to see what policies might be there you know are only some accounts trusted you know you can you can run that that'll give you the pop that'll give you all of the um other accounts that this environment trusts and so now you have targets for

lateral uh movement in the cloud plane um public last search clusters you know an aws managed elastic search cluster now they're calling it open search i think um you know it looks like that customer gets to define something amazon will put something random on the end and then it'll just be es.amazon.com um if you just curl that endpoint it'll come back with hey you know for search what elasticsearch does um you find one of those in the amazon aws ip namespace you've hit potentially a jackpot go look at the you know go look at the aliases and that'll give you the indexes and from that then you can go and look at the uh the content of the index itself see

if there's anything interesting in there or if somebody else has beat you to it and all there is is a you know ancient ransomware note so as i said aws has made it impossible now to make these public i've tried to break things and do it the wrong way and haven't been able to so i think we're safe here um but we're not safe here because hey you know you can actually have public disk images so hey this has got all of our backup database on it um anything that's cached to a disk you know your developer could make it public so if you have a target and you're trying to see okay hey are there any

hard drives i can go after well you can describe snapshots and those are the real time things or you can describe images and these are typically built as [Music] for auto scaling with completely ready to go everything code deployed configured database connection strings could be in there or whatever so both of these would allow you to just enumerate all of the disk images available within within an environment and then of course there's secrets tons of places to find secrets a great one is user data so here's a command um you know it goes through all of the ec2 instances that are in a particular region um iterates through those and then this command here aws ec2

describe instance attribute says give me the boot configuration data that is passed in by the user and this is typically where you'll find a bunch of runtime stuff database connections strings are common ip addresses of other resources i've seen kubernetes uh you know uh secrets in there yep yeah anything that's kind of like a because you'll bake one of these and you'll run it in your dev environment and you'll pass in for that you know dev environment credentials and then you'll run the same image you'll promote it to production and then that'll have the production environment stuff in it so user data is a great thing it comes back originally base64 encoded so yeah just pipe that through base64 decode and

hey you've got a shell script that this machine is expected to execute on start um so if we go back to what read-only access can give you it can give you access to all of the secrets so here we're going to list all of the secrets that aws secrets manager stores iterate through those and get them um boom boom boom uh four commands and you've pulled all of the secrets out of an aws account simply by having read only access um and then lambda code you know another great way to you know maybe there's no secrets in the lambda code but you know you're going to find requirements.txt you're going to find a bunch of other things that you know may

hint to vulnerabilities whatever um again fully available to you with read-only access but you know you pull a list of the functions you run the get function and you ask for the code.location and the url that's returned by that is anonymously anonymously accessible so you can go and curl that down and you've got the functions zip file

lambda environment variables are how many times runtime config is injected into the environment before secrets manager and parameter store which are two aws services came along really the only way to say hey this thing is prod versus this thing is dev is through environment variables and the one thing i've realized about aws is developers and really security folks architect for the aws you have not the aws you want um and typically you don't end up re-architecting so if you get into an account and you have access at the read-only level you can list all of the functions and you can pull out all of the environment variables for those and so this will dump you a nice little json

file that'll say dbconnectionstring and you know whatever that is or the endpoint or the api token or whatever that lambda function is supposed to talk to all of the dynamic configuration runtime stuff tends to be in lambda environment variables which are really easy to pull out cloudformation is amazon's managed infrastructure as a code service uh so with this command you can describe the stacks and pull out all of the parameters so if there was a secret that was being passed into some infrastructure or machine that was being created here's how you pull the secrets out if the machine itself was creating secrets then it would output those and so this allows you to pull out all of the

outputs

[Music] there are many better ways to do this now but there weren't three years ago there weren't two years ago even so your teams aren't constantly refactoring their code to take upon the latest services so in a lot of cases if you were to look you would probably find things like access keys or database passwords coming in through parameters and outputs

github actions and really github in general i think github in general is like the next frontier of cloud security uh because so many folks are moving to github for get ops and github actions for deploying code that you know there's just a whole slew of places that that people haven't yet poked hard enough at on it but very simply you know any secrets that are in github actions on a repo or at the organizational level can be pulled out with this you know one little command and while github tries to hide secrets in the github console it's not going to hide it if you netcat it out to your own ip address um and then last but not least you know

your keys are stored in plain text in aws credentials um aws metadata abuse you know we can talk a little bit about that one that's how capital one got hit um and you know that one was overly permissive role they got in they pulled the metadata out um they didn't detect the data exfiltration and next thing you know the the senate is you know yelling at them a lot of these are available in a tool called paku which i think has been just kind of described as metasploit for for aws but the whole point of this really is to hit on what can happen in your environment to make sure that you know to go ahead and

protect it and if there's one thing that i would recommend is get yourself some form of cloud security posture monitoring whether it's scout suite or prismacloud divi cloud cloudsploit you know there's a ton of them out there they're going to tell you if your elasticsearch clusters are open to the world they're going to tell you if your eks is exposed and with that since i am pretty much out of time i will take any questions all right i sucked so fast nobody has a question

who is horrified i didn't horrify folks yet yeah i i they're already horrified of me yes

yeah um i think i think docker hub is really a a subset of supply chain um how do you know that the docker container that you're pulling down or for that matter the amazon ami that you're pulling down comes from a known trusted source um you know and again if you have permissions to write to somebody's docker hub then yes you can you can overwrite their containers i would say yes but i would put it more less in the cloud security field and more in the how do we fix the supply chain and the supply chain trust problems

does aws offer any good defaults um well as i said they no longer allow you to make elastic search clusters public so they've gotten good on that [Music] um if you create them through the aws console you have to explicitly say i know what i'm doing i want to shoot myself in the foot please let me um if you created through the command line or through infrastructure's code the the please don't you know please let me shoot myself in the foot isn't there yeah um so yes if you're doing things artisanally in the console they have gotten better about it but like you know if i launch a machine it's still going to put ssh or rdp open of

the world for me

a lot of the logging monitoring stuff still isn't turned on by default um and a lot of the logging monitoring stuff is kind of free so it kind of should be on by default uh well it's kind of free it's negligible in expense which is probably why they don't do it cool all right so i'm supposed to give away practical packet analysis so i will ask there is a way to do packet analysis in aws and it's an ec2 construct does anybody know the name of that i've got a lot of novice aws users in here um so how to use wireshark to solve how would you get wireshark data out of an ec2 instance

without being on the instance

okay um yeah that would work so there's a way to do it at the cloud plane which is called vpc flow law uh no not uh vpc traffic mirroring uh who said that you said ps remote i didn't hear a filter sorry um yeah vpc traffic mirroring is basically a command that can be run with aws credentials and it will basically take all of the output of one network interface and send it to another network interface cool all right well thank you everybody