Infrastructure as RCE: How to abuse Terraform to elevate access

well thank you all for coming out to this talk uh thank you to b-sides NYC uh for having me so today we're talking about infrastructure's remote code execution uh so quick intro I'm Mike McCabe I'm the president of cloud security Partners uh we're a small firm that helps companies with their Cloud migrations Cloud security challenges application security you know pretty much everything in the kind of modern web iot mobile space so we've done a lot of work with clients who are using things like terraform and doing Cloud migrations utilizing infrastructures code and as a security professional I'm super passionate about this because I feel like it's a huge benefit to people in a way to really

secure secure their infrastructure I know you guys just care about remote code execution so so what are we talking about do people here know what terraform is or have heard of it or no infrastructure as code some of you probably wrote cloud formation and then cried about that but terraform's a lot better so terraform is infrastructure's code so what does that mean you write code it is interpreted and then you create infrastructure out of that so it's it's codified it's consistent there's a standard API for how you write terraform um it's free and open source the hashicorp the people who created also have terraform Cloud terraform Enterprise so if you like paying for free things you

can do that as well so what are some of the benefits of this so I think my favorite term in this space is Click Ops I mean everyone familiar with that basically everyone went to devsecops and now we're going back to click Ops which is basically people clicking in the AWS console to create stuff so this is trying to fix that this is how you centralize your deployments of infrastructure it's really mostly focused on infrastructure not application deployment so it's not that part of the CI CD pipeline you can deploy consistent infrastructure so what we've always struggled with is people deploy something it changes or they deploy it once manually they don't you know commit their code for doing that

then they deploy it again and it's different somehow and we have breaking changes and you know every Dev team does it slightly different so you just have a mess in the cloud so things like terraform have the benefit of giving you a codified consistent way of doing it the other great thing that what I was talking about before is you can apply security controls for preventative measures at that level if anyone does appsec you know we've we've always tried to do the SAS model of running tools against our code to try to prevent things like SQL injection or process scripting all those types of things you can do the same thing with infrastructure as code

and I'll be talking about some of that as well what are the challenges with infrastructure as code and especially terraform one of the big ones that we see is uh terraform or pipelines are often given really high privileges so if you right now have your pipeline assigned to admin access in AWS you're doing something wrong so if you're if your pipeline can create new users can modify policies you can do everything under the sun you're doing something wrong but that is not very surprising that's that's very typical for what we see for people using you know centralized pipelines to deploy infrastructure or applications um and like the the title of the talk there's multiple ways to execute code

with terraform again it's a infrastructure provisioning it's not a you know endpoint provisioning it's not an ansible it's not a chef for puppet it's just meant to provision the infrastructure it's not meant to do and uh you know end system configuration that doesn't mean you can't do it and we'll get into that it's also a really great tool as an attacker or a malicious Insider to gather information about the environment and I'll talk about some of the ways you can bypass some of the controls around terraform as well so what is uh our experience with terraform infrastructure's code so we've worked with a lot of large financials who obviously have a very you know they

have a regulatory and compliance regime that means that they have to do things in a very secure very structured way um and they've moved to these types of pipelines where they're deploying things through terraform through uh cloud formation other tools like that so we've helped them build out basically programs to create patterns and security rules and security tools in their pipelines to deploy out infrastructure in a more secure way and the reason why I like infrastructure as code is someone who's you know struggled to help people with appsec for a long time to kind of reduce appsec findings and I've always found SAS tools and code review just to be not something that scales that well on the

cloud side you can actually scale this up really well because it is a standard API everyone writes terraform mostly the same way the providers have their own you know API documentation so it is a codified set of set of things you write against therefore it's very easy to check if something is true or false or something is this value or is not this value so actually in some of the environments we've done this we've looked at something you know like a csbm a cloud posture management system and seeing that if they deploy through this through this terraform pipeline if they have all the security controls applied they have zero Cloud findings and if anyone has worked in the

cloud environment or has looked at one of the cloud security tools it's all read all the time basically so it's a huge Improvement for some of these places that have a really big cloud footprint so it's a very powerful preventative control and you can map it to your internal and external controls your CIS baselines your whatever it is you want to map these things to it's very easy to do that so um so what does terraform usually look like in a pipeline so this is kind of a typical this is terraform Cloud so this is their SAS platform not just the typical you know terraform running in a Jenkins pipeline or some random devs laptop that you see here and there but

um it's very similar to any other pipeline you have a centralized system for source code GitHub unfortunately for some people gitlab I'm just kidding um bit bucket if you just are struggling um I'm just just joking but um so you have your infrastructures code there your terraform committed there and then you have a hook from terraform cloud or whatever it is sem hook to detect changes in your repo that runs the terraform plan and then then eventually apply and then that reaches out to your Cloud infrastructure so that could be AWS that could be Azure that could be on-prem there's a million different things you can use terraform for so that's the general that's a general kind of layout of how

people use it either it's seriform cloud or just running internally um so how does terraform actually work so there's kind of a few main pieces to it um when you first when you you write your terraform um you make sure the formatting is good make sure to have your secrets in there because that's how we get stuff done um you run something called terraform plan and that's kind of part of the magic of what terraform does for you is it looks at what you what you wrote and it looks at what you have in your Cloud environment and it basically tells you what are the changes that it's gonna you know what it's going to do is it going

to create something is it going to update something is it going to destroy something and that's the kind of plan piece that's the we haven't done anything yet but this is what's going to happen once we do the apply it gives you outputs for what things are going to look like after things are created it creates a whole dependency tree to kind of um show what has to be created before something else has to be created determines unknown values which we'll get into that because that's important from a security perspective um and then once you have that you do a terraform reply if everything looks good and that's where things actually get updated created destroyed whatever it

might be so that's when it reaches out to your your cloud apis makes your updates creates your ec2 whatever you're doing um and then what this actually looks like if anyone's never seen terraform before this is a super super simple example but this is an AWS ec2 instance you don't have to give it much in order to get stuff going it's not a very complex language hashicorp did create their own language because everyone has to have their own thing but it's very simple to get started you define a provider you define the resource you're trying to create in this case it's an ec2 you give it a couple different you know pieces of information the image it's going to use the size of

the instance and then you run terraform plan and this is what you get out of it which is basically a spec for what's going to be created so you can see the action is create it's an AWS instance which is an easy to all these other somewhat interesting Fields you can see things like security groups they're key names that's SSH key there's a fun one get password data so you can see all these individual fields for the resource that you're creating so that's the plan and then once you do the apply this is actually provisioned or whatever you're doing in your environment and then kind of the final piece of this is terraform state which is uh very

important because part of what terraform does for you is not just provision uh infrastructure but also track differences between um what you deployed and what's currently out there so a lot of people will deploy something and then go into the console update it which kind of breaks terraform because the whole idea is to manage it through terraform so you have everything centralized and codified but terraform state is basically that map of what did you deploy what is the current state and then you can update that over time obviously when you run changes that updates they're also doing drift detection soon with terraform and the thing that's most related to Security in this is secrets so any secret that you create in your

terraform obviously has to go somewhere it gets sent to the AWS provider and in this case AWS provider and it also ends up in your terraform state so if you're creating an RDS instance and you have a you know default master password if you're creating an SSH key if you're creating you know secret values in SSM or Secrets manager that's all going to end up in your terraform state which um you know if you're if you're running open source and you have this you know on Jenkins or if you have this back to S3 now all your secrets are sitting in whatever you're having your terraform State backed up to uh which is obviously a pretty big issue

um and terraform hashicorp I should say the people who create terraform are very well aware of this might be hard to read but this is a GitHub issue on um on the terraform repo it was opened in 2014 uh I took this screenshot a few weeks ago it says commented 20 hours ago and this is an update from a PM saying we're working on this this is coming real soon we we've had you know eight nine years to work on this but real soon we're gonna get this so um and someone said I shouldn't shame the PM so I'm sorry Omar but uh you put yourself out there so obviously it's an issue it's not one that's solved

from hashicorp and terraform themselves so it's something you have to solve on your own by storing that terraform state in a very secure place and there's actually systig the endpoint monitoring The Container company they did an investigation where an attacker actually found a terraform State used the secrets in the terraform state to access other systems and pivoted basically so it is a real a real world threat and you have to have one meme per talk and give everyone a second um so that's that's just the before we've created stuff just the kind of terraform infrastructure aside it gets much more complex so terraform is just a language specification it's just you know a tool that takes your inputs and creates some

kind of outputs a lot of the magic happens in what are called providers there's AWS providers Azure providers these this list right here is not comprehensive and these are just the ones that Hazard Corp creates there's obviously Community providers people create their own providers and you can do you know if you write a provider to interact with an API you can then use terraform to manage it what that means though is any of these providers are huge they have a huge internal API for how to write code against them I don't remember the size of the cloud the AWS one but it's you know hundreds of thousands of lines so all those apis have interesting edge

cases and edge cases and things you can do with them so it's again when we talk about third-party code um you know sourcing from open source we also have to think about on our infrastructure as code side we're taking all this community code and putting it into our environment as well and there could be malicious or they could just be you know dangerous code practices or functions inside these as well so um another fun thing terraform obviously has its own API so if you're using something like terraform cloud or Enterprise it has its own set of apis um the one that terrifies me the most is as I talked about with State it's your it is a representation of what you have

in your environment if that's cloud or otherwise uh the thing I always think about is you can do something called a terraform import where you import like current running State into your state and you can manage it through terraform even if you haven't provisioned it through your terraform I just always think if someone could import a production you know all production rds's into their state and their own run terraform destroy the Havoc that you could you could create with that so these apis are obviously pretty privileged you can do things like manage State you can do runs you can do all kinds of things so it's another attack surface within the terraform ecosystem you uh you have to think about

um terraform logging this is a fun one this is something we're dealing with a client right now which is terraform obviously has to have credentials into your environment and I'm not sure if people know what that is but basically right there is an STS assume roll session so when terraform assumes a role to work in your environment it has to get credentials back from Amazon to assume that role to that interact with the apis all that is dumped out to the logs if you have verbose vlogging turned on so if you allow developers if you're using something like terraform cloud or Enterprise to set a variable they can dump out the STS role and then have

direct access to whatever role you give terraform so that's another area that you have to kind of worry about and this is true for things like Secrets as well you can dump those out to logs um and it's uh another area that you have to lock down and think about when you're building kind of your threat model and your user privilege model for for terraform so with all that said what's that for common for use by competent people only um so that's kind of the lay of the land with terraform um you know I wanted to talk about some more malicious use cases and ways you can abuse um terraform in an environment either malicious Insider or an attacker who

gains access to a pipeline or just to a git repo that's hooked up to a pipeline there are dozens these are just kind of techniques that we've we've found and we've helped lockdown but there's dozens and dozens of different ways you can use terraform and either the built-in apis or the provider-based apis um to kind of do different malicious techniques so these are just some of the ones we're going to talk about but there's a lot more so I'm not this is not a complete list of things you can use to abuse um the first one up remote exec so like I said terraform is not meant to be um like a systems provisioning tool they

actually have a they have one called Packer which you creates you create images with it you can reference those images in your terraform to build you know ec2 like Linux images things like that or containers but terraform is not meant to be that endpoint management system you take an Ami you reference it in your Z2 config and that's how you move forward with terraform but they do give you a few options um hashicorp very strongly says don't use these but we give you these options which is really handy um so remote exec is basically a way it's like an ansible provisioner where you can run scripts on the endpoint so you boot up an ec2 you run remote exec

and you can run whatever you want to as that user on that easy too and this might not be relevant for maybe places that don't have a huge amount of uh kind of security controls when it comes to endpoints if every one of your developers has root on every one of your production instances you don't really care about this but we work with a lot of you know financials where they do care about privilege access management they spend Millions to make sure that developers can't run things randomly on uh on production instances and this kind of bypasses that um so really in this example creating an ec2 instance logging on and then running netcat to get back out so if you're

thinking of a malicious use case someone has access to terraform they can't get onto the actual endpoint but they could provision an ec2 run netcat through remote exec and then get remote access back into that production Server Like I said this is basically a way to run any kind of script you want on that endpoint that you set up so it's a great way if you can't get remote access into a production network if you know you'd actually do have privileged access management well well managed and people can't just get access into production this is a great way to bypass all that you could do pretty much whatever your user your user has rights to so if you want to uninstall you know

crowdstrike if you want to uninstall whatever kind of endpoint tools if you want to run scripts if you want to get a remote shell back out because no one does egress controls well um you know this is this is your tool to do that and this is built into terraform this isn't something you have to like get a custom provisioner for this is all part of the base API so if you're using terraform right now you can write terraform to do this that's one of the one of the techniques which we usually lock down and I'll show some examples of how we um how we lock this down in terraform pipelines and environments we work

working so similar local exec so whereas remote exec is working on the remote instance the ec2 or whatever whatever endpoint that you're creating with your terraform local exec runs where terraform runs so in some environments we've worked in again you give terraform a privileged role to do what it needs to do and then you either do appsec reviews on the terraform you run SAS tools you run different kind of tools that I'll kind of talk about to lock that down but then local exec basically give gives a developer or malicious Insider or whoever it is the ability to run whatever they want to run as that terraform role uh uh so remote exec works on the remote

endpoint vocal exec runs wherever you run terraform so again you can run basically whatever you want to run so shell scripts python whatever it is in this case we're curling the STS the metadata endpoint um and then curling that out to a remote endpoint so the idea being if you can run this you can pull out the STS session for the terraform Runner and then send that off to you know whatever host and then you can run that externally outside of the environment to do whatever you want to do um so the problem with you don't want to mix scripting and terraform together there are two different concerns you have appsec pipelines that deal with

some of these malicious use case and you have terraform pipelines that deal with you know terraform so when you start mixing these two things together it gets really messy but there are there are some tools that can help with this how many people are familiar or use some grep okay cool um so sem grab as in the name it is a pattern matching Tool uh obviously much more I don't know if I'd say more powerful than grip but meant for a certain use case very easy to integrate with pipelines it's pretty fast it's open source um and now they have a lot of Open Source rule sets as well so a really simple example hopefully you

all can read that but you can write a simple seven grip rule to check for all these types of things the great thing about terraform is there's not a lot of obfuscation you can't you can't obfuscate kind of what you're doing you have to write it and that means you can run tools like this against it to check for it so in this case I'm showing on the right the terraform example of vocal exec and on the left a super simple 7 grep rule to basically check for a provisioner and break this so if you have even if you're not using something like terraform Cloud if you're running terraform in a pipeline and you have you know your code

being committed you can run a GitHub action at Jenkins whatever bit bucket does these days I don't even know um you know you can run in action to basically run these these kind of checks against it and make sure that people aren't introducing malicious terraform into your pipeline I'm sorry here's a zoomed in version so again it's super super simple to write these types of rules and prevent these types of um prevent these types of use cases um so those are kind of some of the ways you can run code but there's other things there's other things you can do um again we see places that give a very wide role to a terraform instance and

what does that mean uh well there's data data methods within terraform there's kind of resources which are what you're creating then there's data which is what you reference and that can be things like SSM parameters that can be DNS lookups that can be the state of of different resources but the great thing about if you give terraform a really wide role is it can pull all that data out and give that to whoever's running this malicious terraform and there's a lot of built-in providers and just resources that terraform gives you that can be used as um as methods to get that data so here's one here's an actual data method it's HTTP so basically it makes a get request

it's not fully featured it's just a basic get request and it can't do a whole huge amount but again if people are familiar with the metadata service they're referred to before you can hit that metadata service and pull those STS credentials out for that terraform instance if anyone has heard about the cap one hack that was you know someone using ssrf to get the metadata metadata credentials and then using that to to kind of pivot elsewhere so this is a data lookup that you can then get credentials with but there's a lot of different ways you can use data hookups to um to get more interesting information and here's just an example of what what that would look like a fun thing

about this too and I'll talk about this but this all runs at plan time so not at apply time and that's important when we talk about some of the tooling that's out there but you can see we're able to pull out um pull out this data and output it to the plane so again the solutions for this are super simple if you've written some grep or if you've written grep you can write these these rules to prevent this type of behavior it doesn't have to be super complex it's very basic patterns in terraform so it's very basic seven grip rules but there are other ways you can abuse this too these are this is the fun of

terraform there's just so many apis you can use to kind of gather data um so this is an external data call which you would think the data means data and not data means running code but not according to hashicorp so this is external data where you actually can point to a script in this case a python script and you can see on the line five what we're doing is we're pulling in a data reference which is a Secrets manager secret and then the python script just basically X fill that out so use the terraform role to pull out data out of Secrets manager and then you know use your script and external data to um to X fill that so again really simple

examples of how you can abuse the built-in terraform apis to x-fill data or gather data so how do we go about preventing this in a more complex scenarios so Sam grep is great it's super simple to get started there's already rule sets out there for different resources so I definitely suggest checking that out but if you want to do a more in-depth kind of testing against your terraform you kind of hit some limits with some grip where you want full code logic there's a couple Solutions out there uh I'm sure people have heard of opa but open policy agent there's a language called Rego if you want to break your brain go read about disjunction in Rego

um if anyone doesn't know about that that's just crazy um but it's a it's an open source standard open source language you can write Rego to do a lot of different things besides just terraform but it does have terraform support also has their own tool they sell you the sickness sorry they give you the sickness and they sell you the medicine right so they have their own tool which is called Sentinel which you can get as part of terraform cloud or terraform Enterprise um which is somewhat similar to Opa and is meant for securing terraform so what are some of the benefits of policy as code is is what these tools are called you can write very

fine-grained checks against your environment they're embedded you can write multi-cloud rules you can get external external information um and both of them I think they both offer ways to do basically like hard mandatory soft mandatory rules and so what do these actually look like so this is an example of Sentinel it I would say it's like similar to python sorry if there's any pythonistas here who hate me for saying that but it's a very simple syntax very simple language you do some simple Imports you define kind of in this case the size of an ec2 instance you want it checks the plan what we talked about before finds those resources for an AWS instance and

then does a filter to basically say are there any instances that have uh the incorrect instance type so again this is 16 lines of code with um with you know white space and you're basically able to lock down kind of the sizes of your of your ec2s so they can be very simple and very powerful this is a somewhat more complicated one but so not that complicated where you're just validating the role so again these are simple rules that you can do pretty powerful things with and this is how you build a really hardened infrastructure as code pipeline to make sure that things don't go out the door in an insecure way um and we've also written rules in

Sentinel and an Opa to do things like prevention of remote code execution with remote exact or local exact or data lookups you can do all that there as well but one of the problems is some grep is a SAS tool it runs against the actual uh terraform runs against the HCL their language these tools run against the terraform plan so against the you know the generated um structure of what you're going to create and like I talked about before you have terraform plan you have terraform apply um then there are these things called after unknowns uh it makes me regret not putting a Donald Rumsfeld there's unknown unknowns but um but after unknown so terraform does a

best effort to basically say okay here's what the current environment looks like here's what we're going to create here's what we're going to update um but then there's some things that can't know at plan time so things like orange for newly created resources references to some resources can become after unknowns if you have these like very um referential relationships between modules that can become after unknown so what is the problem with things being after unknown so if you think about your Sentinel or your Opa that runs on your plan if these values aren't known how do you validate those those things so it's basically a way to bypass all of your security security controls at that point

so if you run a terraform plan but all these resources in this case there's an RN that's after unknown as true if those are after unknown your checks don't run against them or they just don't uh they don't come up as being a violation this is actually brought up to hashicorp with one of our clients too they didn't really have a solution so we had to come up with a solution which is you actually have to ban all after unknowns in in like a very um strict way in your Sutton or Opa to make sure that these types of parameters aren't after unknown because otherwise you can just bypass everything it wasn't really a known issue from

hashicorp when it was brought up so they haven't really thought through how their own tools work that well so it's another kind of area where you can build your security tools and then there's bypasses on the security tools so someone could write terraform to intentionally make something like an RN or some parameter after unknown and then bypass all of your Sentinel or Opa checks so you have to have kind of a balance of your SAS tools like sem grip and then your more in-depth infrastructures code tools like Sentinel or opa so what are some of the solutions to some of these issues that we've that we've seen and what we've done to kind of help clients with with these issues

um obviously defense and depth is what we always talk about and you do need to layer your your defenses with uh with these issues um I am is is kind of one of the main ways we build more secure and more siled infrastructure but there's a lot more thing a lot more things we can do than that but to kind of start I am obviously you need to build very strict uh kind of roles policies ways that you have to define the way that you want someone to interact with your infrastructure so if someone needs to deploy ec2 and RDS they need to deploy they need to have the rights to deploy ec2 and RDS if they need to

deploy like just an S3 they need just to deploy S3 in their policy we're not very good at this every Security review that we do in a cloud environment it's one of the first findings that we that we throw up is people have policies that are way over privileged over privileged they give cicd systems admin access and then there's no real limits at the IEM level to limit these kind of things uh but it is probably the most powerful way to restrict the damage um in these kind of malicious use cases is making sure that your IEM policies roles Etc are properly locked down and you also have to make sure you're monitoring these changes it's one thing

to create a perfectly structured IM policy enroll but if these things are changing all the time and you're not paying attention to that then there's no point so you have to make sure you're monitoring especially your roles around things like Ci CD or terraform making sure that um you know those aren't changing and IM gives you a lot of great ways to restrict how roles are used there's conditionals on IEM where you can restrict it to an ec2 you can restrict it to an AWS org to a VPC so there are ways you can lock this down to really limit the damage of even if someone does get a terraform role and starts using it elsewhere

um and a lot of when people are adopting terraform and deploying Services through terraform a big Focus that we do is on patterns um and this is going back to kind of having good architecture of how you want to how you want to adopt these services so how do you want to deploy something like RDS how do you want to use ec2 and having those patterns really well documented and then having those guard rails called out and all of those guard rails then become things like IM policies and guardrails and that they become Sentinel rules they become sem grep rules and that's how you go from having you know your your uh red check boxes your red

boxes on your cspn to having all green and no findings so you have to have really in-depth patterns to make sure that when you're deploying something you have those security you know parameters checked off which is not always easy and then hardening your services it kind of goes hand in hand so having that Baseline for how you want to adopt a service whether that's ec2 or S3 you know you have to have a hardened version of that and that the great thing about terraform is that you create that once in terraform you just redeploy it over and over and over again and developers or devops or whoever is using that terraform they get that for free and

they don't have to think about the security necessarily of these services um the next thing we've talked about this a good bit but code review you have to have Tooling in your pipeline both for malicious use cases and for your just general patterns of for infrastructure as code so having Automation in your pipeline is great if someone has the app SEC team that's big enough to review every terraform change you're very well funded I'm very jealous but most people don't have that so you have to automate you know automate code review thankfully there are tools like Sam grip like Opa um there's also this great tool called grip it's free it's built into Linux you

can do a lot of damage with just that um I'm sure Windows has something similar I refuse to learn so um but you can build these you can build these tools in to do automated code review you know we've been trying to do this on the appsec side I think there's a bigger opportunity on the infrastructure side than on the abstract side because again it's down to a very it's a specification that you can write really high fidelity rules against and we won't annoy developers about false positives so I think there's a huge um there's a huge benefit there there's also things like unit testing that you can do against terraform and against things like Sentinel for every test you

do in Sentinel or in sem grip you should have unit tests to make sure you have good and bad examples of it so that you know it's actually working that's something that we're not great at in security is actually doing you know developer like practices like doing unit testing or regression testing or kind of full feature testing so we have to get into the practice of doing that with our with our security tooling as well um and then monitoring uh no one no one has great monitoring um someone asked me after I did this talk before you know how do you monitor it for this kind of stuff and it's it's not it's easy if you have your logs

turned on super high and you have alerts set up to to check for remote executable exact things like that no one basically does that um so it's easy in practice or easy in theory it's harder in practice but we have to monitor for our infrastructure changing in unexpected ways like terraform getting new roles or someone else using terraform or a new trust policy being added we have to monitor our very trusted systems for changes around them as well so if people are changing the role for your terraform infrastructure then you need to kind of be alerted about that as well and also just monitor for the things that it's doing in your environment so if it starts provisioning new resources

you should be able to tell that too to make sure it's you know a new resource or a new region um that it's deploying to make sure it's an approved change so conclusions um I think despite everything I've said I think terraform is a great tool I think infrastructure as code is a great strategy for us to start provisioning cloud in a more secure way and move away from spreadsheets full of you know insecure S3 buckets and unpatched ec2s it is a great way for us to build a more secure Cloud environment especially at huge scale but it comes with a lot of challenges as I've talked about so we have to kind of go into with our eyes

open to know that even though it's it's relatively simple to get started there's a lot of work to do around securing how things like terraform are used um you know it's a really high signal and effective tool and security it's not just for devops you'd have to think it thinks it through very well have multiple layers of controls as anything and then monitoring so that is me thank you for listening

um if you want to hear more rants about infrastructures code I'm on a podcast called relating to devsecops if you're really into that and then I'm the president of cloud security partner so just reach out if you want to talk about infrastructure as code or Cloud security are there any questions okay you first um

development platforms um I feel like that can be very helpful in terms of separating right who has access to run these things uh for the developers aren't able to apply locally or plans with some of those other problems you were talking about yeah yeah I mean I think when it comes to I've seen it go both ways I've seen people go from a service catalog model to a terraform model and I've seen people go from a terraform model to a service catalog model so it's kind of like people you know they they do one thing they're like okay this is great in some ways but we're having challenges here because service catalog model is too strict it's

too rigid we can't you know build out a whole Suite of infrastructure with a few with a few deployments and then terraforms the other way you can deploy out everything at once but it becomes it can become more of a security headache um yeah I think there's a balance there I mean I think building really good patterns for how people adopt services and forcing them to use those patterns even even if that's more rigid than say just giving them terraform access to run I think that's the way to move forward so you have the flexibility of writing code to deploy infrastructure but you have those guard rails in place on top of having things like sem grab or

Sentinel or whatever thanks sure um

tax services and and like put in like a script out like that you know in amplify library or disabled almost here have you ever since like you ever feel like that type of attacks or destruction yeah I mean do people use things like Packer or other kind of golden image systems to produce yes I would say yes are people using like Alpine images to deploy everything in certain cases yes but I think it's always a trade-off between security and actually having applications or whatever else it is running um I think it's kind of like entropy where people Trend to okay here's a super lockdown image okay we have to add this okay we have to add this we have to

add this this breaks here um so yeah people definitely use it is it I wouldn't say it's quite perfect but it definitely helps reduce the attack surface there's a lot of people who just take Amazon Linux and run that straight and then might have some EDR or something like that on top of it so sir um TF SEC TS scans boss on them um so I'm like uh they're good good projects definitely they have a good rule set i'm I'm a firm believer and if you're going to do something you generally have to do it somewhat custom from the uh from the large kind of environments we've worked in they've had a dedicated Cloud engineering team to build this stuff so

if you're a startup if you're a smaller company you don't have the the budget to build a whole Cloud security engineering team definitely start with something like tfsec stem grep has built-in terraform rules these use cases are not actually in those rules though I have a rule set on GitHub that I publish that catches some of this stuff but the malicious use case is not really caught in most those tools I don't think tfsec has anything for remote exec or local exec but it's definitely a good way to to get started I think the problem is as you scale no one's you know no one's environment is the same so you can use tfsec to start but

like if you're going to want to enforce tagging standards or naming conventions or you know where things go in your network you're always going to end up customizing things and so you can't use those tools forever and so then it just becomes build custom tools McCabe 615 it's very original yeah uh ask me today's winning the prior enforcement that only developers won't use the high level Passions

uh open source no but in projects we've done it so in environments we've worked in it's like very customized so it's hard to describe from like a just raw terraform perspective but you can enforce things like where you're pulling modules from um again it gets back to we all trust npm we all trust GitHub we all trust all these packages then you have to start ratcheting down to what are our trusted packages and then how we enforce the usage of those so we've done it we're basically like here are the modules you can use and you can't just go to GitHub and like use some random module so if you're going to deploy RDS you have to use the RDS

module and we have enforcement there so you can do it with things like semrep or Sentinel to basically look where resource is coming from and make sure that it is coming from that module but it's somewhat complex I I haven't seen a open source tool that does that for you so I think

I have not personally used it I've read some Twitter threads and made me feel like I don't want to use it because it's a combination of like infrastructure's code in your code which yeah again I'm not going to say I have a well-informed opinion I'll just give you a hot take because that sounds horrible and I've just um read bad reviews of it I think they're two very different things I think you should think about infrastructure as infrastructure and and apps as apps obviously there's a lot of there's more overlap than there used to be um but yeah I don't have any experience with that sorry

[Music]

um the benefits of using something like terraform cloud is you just get your infrastructure it's like serverless you just get your infrastructure for you the nice thing about it is you can have things divided up into workspaces where you can say like this app is in this workspace this app is in network space which is really important if you do things like vertical segmentation which at startups you don't really care but at large Enterprises there's things like you know socks where people can't talk to each other's apps can't talk to each other's data so the cloud gives you that it gives you I'm not going to say it's a lockdown environment but somewhat more locked

down than a Jenkins server running terraform um but then again you're limited to what they give you in their apis um there are ways you can run things like stem grep in their pipelines as like a pre-plan a pre-plant basically action like a GitHub action but it's their Runner um so it's kind of a trade-off of do you want to host your own stuff and be responsible for walking it down or do you want to just hand that off to them and don't worry about it I think there was one more yeah

so it depends on where you back your state up to so there's you define a back end with uh with terraforms you can do something like S3 you could do local I mean yeah in the case of S3 treat it like it's a Secrets manager at that point like have whatever bucket you use be very locked down um and just treat it like it's your secrets manager so I mean Secrets manager itself is all I am based so you have to treat you know those policies and those rules with um a lot of scrutiny so it's the same thing if you use S3 you just have to make sure that that bucket is well locked down

monitored and has limited access to just like the terraform instance that's that's using it

[Applause]

great question I mean a lockdown it's a it's a object so a lockdown Object Store I don't think Esther's bad the only thing about the only thing that's bad about S3 is that you click a few buttons and it's now public so it's an internet Pace I'd say it's probably it depends on where a terraform runs but I'd say it's probably better to have it on a server that's locked down not on cloud infrastructure that can be made public no like a ec2 or something like that so but yeah sorry one of the advantages of infrastructure as code is that there's a basically a git log tracking all the changes in corporate settings what threat model

are you considering that like someone has access to write this code but they don't care if they're you know git users is identified the question is what what do you think of like a malicious Insider use case or what is the thread model that that you're usually considering in this regime I mean the two that we think of is a malicious Insider which is a developer who has access to terraform and preventing them from doing more than they should based on whatever that you know companies operating model is for developers and then the other is someone getting access to either terraform directly or um getting access to deploy things out you know again if they can push to a

GitHub repo if that has automated applies then they can they can run terraform so those are the two of the threat models and there's a lot of different layers to that obviously you have to think about like someone having server access to terraform someone having just API access to terraform someone having just access to code there's a lot of different layers to it but those are kind of the two main two main ones that we think about cool anything else thanks a lot thank you