BSidesAugusta 2017 - Track2: Security Automation in the cloud by Toni de la Fuente

thank you it's also like a smile it's my first time here besides her gusta thanks to the organization for having me here and let's see it for for coming so today I'm going to talk about many different things it's probably too much to cover in just 50 minutes but for our first goal is a software company open source and we do staff in the cloud me a long time ago so yes I have done and also I've done so I love security I was guilty and so this is why I'm here story during this last two years so I got this email from my boss that you here you go just pinkie and also I had another email

doing something like a for scanning something so this is what you have [Music]

all the time whatever you are working you know things about these are forensics in the cloud and then with where we are gonna see some attacks and or others incident response and also of course your environment your infrastructure etc first of all let's talk a little bit about one of the statistics of the cloud is so stations etc in the cloud you cannot do that so you have the same for this is a sure so I have used some example of AWS assured and Google cloud but prolly is the same so when it comes to deploying applications

[Music]

Braham may take months that means more environments like pci-dss

[Music] so we have to pay a lot of attention our systems so let's see some some generics for example the wick wa t as we have seen so the lega lega jurisdiction of course is different because we may have information in many different countries the elasticity is also interesting point to discuss because when you have

[Music] and many other planets I'm not going to go through the to the entire table but it's good thing good least consider it's like other differences from the identification preservation position etc so like the other day way too high a few details here like documentation of the sing of course in the cloud you don't have any exist in the cloud so you can save the time while you are documenting this part you don't have to do in this case some sometimes some tasks are better or faster in the cloud and others are more difficult also we need to know where is the formation in depending of course on the calendar you can find information in

many different places but pretty much all the clock vendors have same kind of services you need to know the key points on the object stories what is an object storage different tasks that you have to perform depending on where you are going to find also the broad storage EBS volumes orbital size etc so we can find information please just we can find information in many different places and also remember here ephemeral hard drives which is information that when you terminate an instance because way so this may find interesting information there as well or databases or queues cash engine services etc so this is not just like different services and once - aww specifics I have worked a little bit

we maximize your my job is being a double yes so I'm gonna go straight to specifics to tell you about some attacks so first good so in order to understand most of the attacks and how AWS work protecting services and protecting the different options that the offer is we need to understand the keys so first of all when you create you should have protected by a multi-factor authentication code hardware or software MFA the most important point here is the I am which is a username and password but also the access keys remember an access key is what it's a pair of keys you use the disk is for everything but you may have the STS which is a token that gives you

temporary access to something to our resource of course and is used to be attached to an instance even if an instance or if you use Samuel authentication like there issues Robbie you have also your user have also okay your access keys depends on photo kiosk as well and the key pair that is nothing to do with better press this is all related to a Darrell properly and the keepers are the SSH keepers okay sometimes people is confused with different things up SSH the keepers are only to access but it's not the same so when I'm talking about keep API keys okay but also from the metadata server we can get access keys so now that we know some basics about

let's see some attacks some common incidents that we can fine are the access keys compromise of course as we have seen there are many ways to do that and we are we are see some of the these ways phishing attacks you can send emails trying to to you know to force an user to change password or to give you keys etc through compromised resources using poison am i or ec2 instances many different ways as you can see here also the subdomain the covert applique applications running in a role I will show you some applications some examples and misconfigurations everything is at the end of the day or not everything but in most cases are errors related to the

configuration ok so of course I couldn't miss the s3 leaks you have heard this probably in during the last months many times like the Time Warner issues Verizon auto lender and also there is a very good list here about all the the latest s3 leaks that has happened and also if you want to see how far let's see from a bucket you can look at this tool it's a good proof of concept of how you can discover buckets and discover the permissions of the packets I will also show you a tool a problem that can help you to figure out the configuration if you have properly or not configure your packets but in terms of security

but this is kind of an attack tool or recon tool is very very interesting if you want to take a look to it and this is some aces of new tool from Amazon and it's the amp the AWS answer to all these issues around the PII information leaked from s3 buckets etc so it uses Amazon may see what it does is to look at you inside your pocket and also in the your packets configuration gives you very good information regarding as I said the configuration and if you have kind of credit cards in the data or another life using machine learning other places some places that you can find and look released is so places where you can find access keys

user data of course the user data is if you are familiar with Amazon when you launch an instance so this is the same thing with confirmation or the metadata server itself public EBS volumes public am eyes you can attach to your instance you may have information personal information or even keys etc and also in your workstation if you use a Linux or Mac you have prudential here or in a window windows here and many other places as you can see here even you can search in github look at this truffle hog in github you can see how a good example of how to find keys not only in github Amazon looks of for you to find try to find keys and you

know so what's up with that

[Music]

shortcut Hey and this I've been playing with this so don't tell my my mates but I'm going to use this so once we perform an attack we can do or we should do persistence try to move to do a lateral movement or whatever so we can look as I said to the meta-data service like this is just doing a core I don't know if I have yes cool call here to the metadata server service this is as I said AWS is specific Google OpenStack and other vendors in Microsoft Azure they have kind of metadata service that are five inside every instance it's not dynamic but it static but it is also in every instance okay so this is not only a kind

of an AWS issue it's not an issue is more related to configuration but it's just something that exists okay so if you have as I said a role like that stanza control station okay so if you get these keys and you use them in other attack automation tools you can access that you can perform in downtown or as we are going to see to perform malicious activities so the fake latter moving snapshots attach volumes to your own you can you can get if you have permissions to to do that you can share an EBS volume to attach to another easy to instance you don't even need the root password to do that because it's like you connect and if you just want to

server this is why is very important to have EBS volumes encrypted for example and you can also use Metasploit to make some of these things so server less who is a living server less here you know what is ever less you are familiar more less with server test server less is a service to run functions different functions in Python or C sharp or many other languages like no GS without having to take care of the servers in my case a few months ago our developers wanted to do so and they say okay Tony we are going to move this code to production so what can we do should I I didn't know what to do pad but I try

to make sure that these it was in Python this Python code was in backdoor or something I am doing something weird like charging too much or doing any configuration change and also in terms of the the permissions function comprehensive and we what I do is just to make sure that there are no many stars in the policy because in most cases at least it's more expedient so and they allow everything so this everything so you have to just take care of that so this is another very interesting attack is a little bit old but it's also using Saba and just having the kids the access keys you can do you can disable of course with the

proper permissions disable call trail encrypt all trails generate new developer access keys stop instances terminate instances and bar them all everything with just one button so this is why I said before so you can automate the creation but you also can automate the destruction so this is just a screenshot of of this and look tired some companies have to have had to close yes because they didn't have properly configured a infrastructure or not seen properly splitted in different accounts etc this is another another very good attack is kind of a proof of concept using lambda and if you are interesting in lambda security watch this video from the coast computer conference but decode is not a

value but the video is still interesting and there are many other attack tools or provoking sets very interesting that you can see here I'm not gonna go through through them because I want to tell you more about Harlan I'm incident response so once you find an issue or or you have the issue how can you identify the issue first with AWS or the vendor notifications they are they have very good security teams and they are looking after everything more than while you may think and very proactive all the time also looking at IEM activity and a new the new cloud ideas which is the billing activity if someone take your key your keys and start creating instances you

are gonna see that in your building of course API logs as I said everything goes through the API so you have to look at the logs and of course dedicated tools we are going to see some of them and the traditional tools like Norris ricotta or pursue which is a fort on asteroids we with office a query or a hunt reality and everything that you already know so yeah III know this is is too big but is

[Music]

[Music]

still to do and also depending on the case you may need to leave the instance life or to stop the instance open maybe you already have the instance topic and they call you to to look at it once in stop it so we have automated the memory capture and some other data like like metadata etc and also you can also use forensics distribution in the cloud same more or less the same way you have to attach volumes to do to create timelines etc so of course we need to do a lot of more things but this is not bad so I'm gonna show you a demo how to how we have automated all of these things and in the

cloud you have also to to acquire more information like not only the twelve Trail which is a API trail the VPC flow log if you are not familiar with PPC is a way to create networks private neck Orson in the cloud three logs RDS locked everything that you can get but also the instance profile information endpoints if you have n points configure the syslog screen and metadata from the hypervisor standpoint we don't have in the cloud of course information from the hypervisor but Amazon in AWS you can get some information from the visor like even an S screen screen shot of the the console in the in a particular moment windows which may be interesting when you are

doing an investigation and many other information you know that data like limits in this account you need to look for resources created from a given date from the data for example that you figure out that your keys were stolen for example and everything from inside every every instance so you have to to acquire our information and you can automate pretty much everything of this so the tool I'm going to show you a space was presented in black hat and it's based more or less in the idea that I wrote in an article back in 2016 and with some with a proof concept of how to automate what I have shown you and I had this kind of example shell script using

the AWS CLI and the guise of red responds presented this to lab and also a new version this year you have these all the tools here and documentation etc and what I'm going to show you is how to collect the how to respond to key compromise and how to respond to an instance compromise what we can do is using plugins you can gather the instant you can see here host but it's actually an instance metadata string consul etc also to tag the horse to make sure that your information the gains and is tagged and is under investigation to add an AC if it's inside our VPC dragon is easier to make sure that only the examiner

investigator can access to it the same way that isolated it's like to move do you need to keep the instance running but you have to make sure that nobody's been or nothing is going to get out of the instance but your investigator instance for example through SSH if you want and eventually you stop paying see if you need to so let's see first key compromise in this case is not only two we see here you are familiar with the with AWS console you can see here the one account the video is not running I have it here

by the way this is where you can find everything I'm talking here the video of the innocence compromise I'm going to go straight to the sound compromise so here you can see you can see an instance with it sounds a guilty group etc so what I do let me stop it a little bit what I do here is running it away higher which is the the command to perform an action in this case in that in some compromised case I give it a bucket name to to do to live there the the memory dump you have to give it room to the users SSH key and plugins plugins at the angle a is the order that different or the workflow of

the actions that you want to perform like colored horse etc and in it depends of course the size of the instance what you do is just to take the to perform all these different actions in minutes so with the target which is the instance is compromised it is try to find this instant in all the different regions AWS regions and perform the different actions for example using I don't know do you know a paramiku paramiku is an SSH Tool Library for Python so threat response the suite of tools it has one two just two for the memory damn remote memory gathering which is called Margaret - of time shotgun is based on parameter so here

you can see in this case was a one gig one kilogram instance but this is why it was very very fast the instance the memory dump and the different options the different acquisitions so the console metadata also the screenshot etc so here you can see automatically you have changed the security group only to the isolation security group which is your investigator and it's stopping they of course they the instance and you have also unsnap shot create automatically so what you can do with this snapshot is attach it to kind of the Sun shift for example or any other foreign six distribution in the cloud and to perform other tasks like to create a timeline and export this at I'm

going to play so for example etc and also creates the well in the s3 bucket you can find the memory dump I don't know if you are familiar with lying but in this case what it does also automatically is these guys have done a repository of kernel modules based on pretty much all the kernels available in the market and they have it available in an s3 bucket somewhere these two taste every looks at your linens and take to perform done you know to be able to do a memory dump in Linux you need to install a module so this is why everything works so fast and you don't need to do anything manually and now

with this damn using Rico or volatility you can perform other other tasks okay to find what the issue was exit okay let me go back here okay this more or less how to automate the data acquisition to change some things in the cloud in the in an automated fashion hope you can prevent persistence so having good configuration first of all but look in the tell to your user data there are ways to make sure that you don't have anything like keys etc in your user data so there are security best practices when you design the templates with confirmation also you can prevent an application like Apache or job application running with the user

[Music] if you have everything with a CTS token is you make everything more secure temporarily unless from inside from saying the same policy cold trail should be enabled always so I think from since a month or two months ago every account new account that you create in Amazon has a klutz I'll enable so make sure that it is enabled because it's the only way that you have to to see who has done what through the API separation of duties using multiple accounts of course to implement a good policy about security in the cloud you need to go from the ground up from the instance security to the network security tab to the provider as we we

have seen some here some tools that we we used and we decide also to use I'm gonna I have another list of tools but for example every instance every ami that we build should have the cia's benchmark for the operating system to make sure that it's properly hard and the hardening is applied no configuration if so if you don't need to configure it you don't need to access to it despite some users have SSH access but if you use a configuration management system you don't need to - SSH to any instance and also local tools like query was who are key hunter etc and of course sed knows always enabled and remember that you can collect

everything from your provider but every provider has also a API called limits a ee-aw has a columnist depending of the actual different actions and of course red teaming you know APA calls I really have talked about this who when what call what resources and from where happens the this is Claude trail and also every every cloud provider has a tool but also Microsoft has the operational insights which is more than just API calls but they they have tools for for everything and also there are proprietary and open source tools yes yes it's not bad but we have we have played with just one project with inspector and also we are using was who the latest version of was

who as I said was who is a fork of Cossack comes with with an integration with opens cap opens cap is also a hardening and compliance tool that look at your configuration etc and can gives you clues about if you have properly hard secure your operating system base of cis benchmarks or even needs or different standards so we we use open sky but we have also used your inspector it's fine inspector at the angle that you look at the also benchmark etc also as I said to automate hardening auditing etc you can use many different tools I'm going to highlight a few of them only and all that all the links are in the github page and also is are in the

presentation everything is in github already I like to highlight first of all security monkey I cannot live without security mount so give the monkeys at all by Netflix it works for aw sir recently for a Google cloud as well and what it does is to look for changes in the API and your configuration and alerts for for bad configurations or some some changes it has some patterns to to know if there are some bad configurations and gives you some advices and alerts about about those configurations Amazon has done good things about some changes like config rules and other other tools like I'm also amazed here as we have seen before but security monkeys is for me is

the best tool to look after the configuration itself know the API itself and you can use it for multiple accounts not just one also this is another good tool cloud custodian to control policies about your infrastructure they also plug in for AWS and also ask a - or the AWS see is benchmark which is a Python code does more or less the same as problem I'm going to show you a demo of problem which is as I said like space also in the benchmark and this has less checks than Startalk but it's a little bit easier to run because it's a shell script you don't have to install anything just having the AWS CLI work

time and just to highlight another thing get secrets is a hook for get to prevent your developers to send code when they do commit to github to send keys okay so it's a good recommendation for developers yeah finally what I'm going to show you a little bit about Pro layer as I said is a shell script tool that I have right here it's open source so I'm not gonna sell you anything this one so this demo is about automating the hardening itself so you can this is about hurting and probably I have bottle thing in the same video so you can take CloudFormation template which is this one from the URL they show you of a WCS benchmark and it's a good

example of how to automate the hardening of your AWS account to change account policies like password policies and also alerts in case someone changed V PC configuration or security group or open a security groove SSH for the entire internet etc so you can automate all this stuff you don't have to do your hardening manually okay and then we move forward because we are and this is probably so you ran Prowler and you get some checks like to make sure your password is it has the different values the length of the password and you have to change the password every a number of days etc and also last I don't know about probably two weeks ago three weeks

ago I added an extra checks which is beyond the CIS benchmark to check the security of your three packets to make sure that you have properly configure your s3 packages you don't have your litter box with a CA public or with policies remember these three buckets have a CL and policies both of them and you may see in the ACL side that everything is fine but a policy may have your packet open so you have to look at both of them okay and the same for EBS EBS volumes the beef developer we has been medication sometimes he wants to share what they have done with someone else they just make it any BS volume public they don't know or they

didn't figure out that they had keys or called or many of different things at this public so i am i added this check to prowler just to make sure like I can run pro rata gains one account all our

locations so you have to do to prevent things happening in different regions but also to prevent bad configurations policies etc so this is a way the way we did we do internally all this kind of assessment at least ok so you can see here if you have properly configured VP CSV pcs also with the flow logs enable etc just to make sure that you have the basics regarding security in your in your account ok this is what I have takeaways so everything that they have shown and templates and also the presentations director cuts so so we the presentation I had was too big and but it's available also here I think that's all yeah remember to automate everything you can

automate everything because at the end everything is an API call encrypt also encryption everywhere any liar even if it's inside a V PC and get everything on print on transit everything enable a consecration never use the same account for everything use different accounts not only for products developing etc but also for different groups or different work groups is much better to use multiple accounts and AWS in the case of AWS they have a new service called AWS organizations and helps you to manage multiple accounts least privilege if you ask me what is the best thing the first thing that you have to learn going to the cloud it's about the Identity and Access Management P okay go to imitability or a fumarole

resources and of course buy bitcoins just in case that's all I know if you have questions now you should have questions if you want

[Music]

[Music]

[Music]

[Music]

[Music]

[Music]

[Music]

[Music]

[Music]

[Applause]