Does Serverless Mean Harmless?

okay

[Music] okay um thanks everyone who joins uh joined uh my name is tal i'm a senior director for cloud native security research at contrast security and i'm going to talk to you about serverless security um just a few words about why me or who am i why am i doing so why me let's start with that i uh created or i started last year i started a company in israel called cloud essence we are doing cloud security testing um for serverless applications and we got acquired by contrast last year and previous to that i was heading the research at protego labs another serverless security um in protection that was acquired by checkpoint a year before in general you

can reach me through social media on github twitter or my personal links i'm also teaching cyber security in the quinnipiac university a connecticut-based university in the united states which has a master's online full online program so if you're interested shoot me an email okay so why is service that important uh we'll get to server specific uh soon but cloud native in general servos is part of cloud native cloud native in general is presumed to be the future of application development idc predicts that this year by end of the year uh 70 of all newly developed deployed enterprise application will be deployed as cloud native that is based on uh architecture which is hyper agile which uses containers

microservice and service functions um in general we already see a massive transformation into cloud native we can find at the top irobot skyscanner netflix our company that already utilize everything almost everything as a cloud native environment and we can see some big companies not only startups like capital one walt disney ebay coca-cola etc they are moving there and they are still working in some kind of a hybrid solution some of their application are cloud native and some are monolith but cloud native what does it mean it's not just a development pattern it's not just using containers and servers it's a different a completely different architecture so we don't have one big flow that what we are used to know from the

monoliths from the traditional application uh everything is deployed into one big logic instead uh what we do is we utilize microservices or nanoservices which basically tear everything apart to small pieces and then a lot of work in connecting them together into one flow so it's a combination of code and your infrastructure that together builds um the cloud native application the cycles are also different it's not a water flow life cycle which you build test and continue uh with the flow but instead there it's more of a devops or dev secops now it's also called uh operation where you work in a hyper-agile environment and every cycle is uh in in short cycles and you keep going

with them so uh the process is different is last manual you don't have uh hey today we're updating the environment so uh let's have a full server update or uh downtime instead uh the process is complete almost completely automated and the decision is also taken by the developer themself which take cares or owns the security and code into production it's not managed by high hierarchy as before serverless architecture in general it's like a big puzzle it's a lot of resources could be hundreds in some applications that we saw it's even thousands of resources that each is connected to another using some kind of a glue over infrastructure or a configuration and all of that together

becomes one big application but if we're talking about security here so we need to understand that these resources or services in many cases they are stateless or they don't really understand the entire flow but they own only the specific small piece of logic which means that in many cases we will have to apply security to each one of them in a separately so it has a lot of security challenges and we will discuss that okay when i talk about serverless what do i mean we'll ma discuss some of the biggest cloud providers but in general serverless means that you don't own the server where you run your application so the server is given the server we'll talk about it in a

second is given by the cloud provider aws gcp et cetera and uh when the code of a serverless function for example needs to run basically the cloud provider spins up on demand a container runs the code executes the codes and then shuts this down the triggers for these type of executions are based on cloud events cloud events is so it's an event-driven architecture cloud events means that something happened in the cloud it could be an api like we're most familiar with but it could also be someone uploaded a file to a cloud storage somewhere or uh someone sent an email to someone or logs that arrived and collected or analytics and all of that is configured

to allow the serverless function to run the environments are usually read only so you you cannot really change anything in environment in the container that runs the code it's a read-only environment in aws you have the slash temp directory so you can write the data there but all the environment itself is a read-only other cloud providers have a little bit different and we'll talk about that the code or the environment terminates when the code finishes the execution so once the code is completed the container itself shuts down so you don't really have access to it you cannot ssh to the to the environment to the runtime environment of the function it just executes the code so when it

spins up it has the code already inside the data is also temporary so when the container shuts down or the execution is completed the data that was inside is uh deleted or just eliminated and we have a while an asterisk there because uh in order to um to maintain performance the cloud providers really just put those containers aside and reuse them so in some cases even though it's not controlled by us one execution can have data from previous execution but again we cannot rely on that or know that it's random from our side the code as i mentioned reside already inside the environment when we create the the functions uh also the keys for the function in order to be able to

interact with other services is also inside the environment so when the environment spins up the keys are already in there so when you need to grab data from the database you have already access to that and we'll talk about that from a security perspective all right so i mentioned an event-driven architecture what does it actually mean it means that uh something happened in the cloud so api f3 bucket dynamodb iot rule etc that executes the code this is what we own this is our code it executes our code which usually processes some data and interacts with other services and there are many services that we can interact with and the security risk really mostly reside here

all right so let's understand what is this single purpose container better so we have aws azure and gcp and there are many others but of course we cannot cover everyone those usually have the most uh the the highest part of the market share uh ephemeral so as i mentioned aws really just eliminates the container after it runs so it doesn't leave anymore on azure and gcp it resides a little more uh depends on the configuration aws has the read-only environment other than slash temp on gcp you have the slash stamp and flash home in azure it really just continues to leave again you can change that configuration different triggers aws is uh of course behind um

in front of everyone with i think today's already almost 30 different triggers so email code sns sqs so queues authentication third parties etc uh azure and gcp are a little bit behind the source code is located under slash bar slash task you can also find it find it in this folder in aws as we know from previous monthly from application is www root and gcp had a user underscore code folder which holds holds the source code okay so what are the challenges there so when you would when you do some things in a small scale it doesn't really uh problem you can investigate you can put your efforts into it you can have security teams looking at it

but when you need to scale that's a little bit different because uh if developers shift new or create new functions on a daily basis or at least change them on a daily basis the security teams cannot really keep up so you have a lot of services on aws you have almost 200 different services frequent deployments i know organization with hundreds of deployments to productions in a week that because you don't really need to wait for the entire service server to to update the server but instead developers push new small functions or changes into functions which run separate uh into production so think about how hard is it for security teams to follow uh what who is talking to who or exactly

what is talking to what as you we've seen before with a complex uh puzzle there that the security teams really don't know what's going on in the environment yeah they know if there is an api that is exposed through the web the front end through the web application but in many cases it's just back back-end code that runs on based on other types of events which are not exposed to the world and security teams have no idea what's going on this is from experience many developers small security teams we know that in the best organization you'll have one security person for each 10 developers this is for mature organizations so it's hard for the security teams to

follow how do you know what's important that's tricky questions you don't always know what's important unless you really talk to the engineering in this case you'll also have to talk to the devops or to the infras teams to understand really what is going on and uh the maybe the biggest uh question that we have is is the security even the same if we're approaching uh now security assessment for service application does my previous expertise uh enough or are they enough for understanding the security challenges here and uh as i mentioned part of the application is the infrastructure is the cloud so if we have a cloud infrastructure team of a devops team an engineering team and i have a security

problems uh security issue in the cloud or in the application who takes care of it is it infra is it engineering who should i talk to okay let's try to understand what is the state of security in general so um recent checks that i did was showing that uh sorry serverless is keep keep rising of course this is quite old but it if you'll look it will be it will continue to grow as expected but the security really remains uh the same no one is actually interesting interest not no one but there are not enough um information or people that are checking specific security for serverless uh you can see it jumps between two in the

best cases to one and i'm pretty sure this one is me from that period on i'm sure there are it can reach two or three but not more than that so we talked about uh is security the same so can we use the same appsec application security to uh to serve our serverless application does it apply let's uh understand some of the risks that we have in serverless applications so this is an example of a function a lambda function and aws lambda function is is their server's compute so small lines of code usually really the function doesn't do much here but you can see here that it it communicates with an orders table and the put item here is basically the uh um

the api call that makes right into the database so what happens is the developer needs to give this function the permission the im policy for the function to interact with other services usually the developers as we know uh they will do something as quick as possible for it to work uh they will look into documentations and they will see what is what they need to do in order to make it work so if we look in the documentation or in stack overflow or things like this what we'll find is that we need to assign dynamodb uh access with a wildcard because the wildcard means that this function can do any operation so that would work

of course when i'm saying that this is what developers will do it's not every developers of course we know that there are developers or organizations that take care of security or i am in a different manner but we'll talk about the problem at scale uh so the developer will assign a dynamodb wildcard policy here to any resource on the account id any table basically that's what what happens with this is that if this function is then identified as vulnerable someone who has access to this function can can execute any activity any action on any table in the cloud account so it's not just the application it's not just the database but basically it's any operation

in the cloud account um it can be deleting tables modifying tables uh creating tables uh different even if they are not related to that specific application okay so as i said the wild card is uh bad because it's a bad practice because that gives the function a lot of permissions which it doesn't really need so what do we need to put here what we would need to do is to convert the put item here to the put item action in the aws uh im service and then put the table name here so in that case what will happen is that the function will only have access to write a single line each time it's it's a execution to this

specific table think about this as a granularity that you would not be able to achieve in a monolith application because this really takes just the right action to the database just the activity of writing into the database and limits the attack surface here or the uh permission the access level to just this active action so if you could do if you take them a monolith application like a traditional application and you have thousands or millions lines of code and you could say no but inside this code this function this uh part of the code can only have permission to write to the fund to the database and this part could only uh will only be able to

read this would be would have been great but it's not possible in monolith's application in serverless it does so it has a big opportunity here but the problem is when we try to scale things because if we have one function with 20 lines of code then it's easy to understand what's the security or what we need to put here but if we have a new code written every day and hundreds or thousands or tens of thousands and i know organization with millions then it's going to be hard to do it manually so you have to have some automated process there but how would you know it's if it's in the code another challenge that we have

is um the loss of perimeter uh if before we had one way in well usually http or some kind of a gateway or a load balancer uh but we always had one way in what we would do is put all our security tools uh in front of it so we'll have dlps and firewalls and gateways and configuration everything in there but when we think about serverless application really we lose this perimeter and what happens is that developers can have access or sorry users can have access or users or maybe the system could execute code from various different of um of entry points it could be analytics it could be direct use apis you could be

actually could be just calling the code so i can actually use an api to execute the code without going through any network component could be static assets uploads downloads or whatever mobile or whatever that is

okay so let's get understand how we gain access to resources in different environments and by that i mean i have a function that runs the uh how from the function itself i can again access to different resources or services so to the database for example so if we take uh gcp inside the code we have a metadata service internal so metadata.google.internal where we can we can connect using the uh made to the meta flavor header with the access token and get a better token this very token or jwt can then translate to access this token can then be used to access or retrieve data so you can see here i'm doing a call to this metadata

service i'm getting the token and then i'm doing a curl so in a simple request to a specific storage on a specific project and i'll get the data back so if i have access to the runtime and i can run this command i can get the token if i can get the token then from the internet no this is from the internal service but from the runtime of the function but this can be from my own computer and i can access the data on azure stuff something a little bit similar we'll do an and print and we'll see the endpoint and the secret and then we can use a curl command to an internal service to

get the token using the secret header with the secret that we got and then we will get an access token from that using that access token we can then con connect to the other system or apis from the internet and then get get data so again if i'm running inside the function and i have access to uh to the to x to the execution to the runtime i can run this command get this data run this command get the token and then from my own computer and we'll see that in the demos in a sec in a few minutes i can interact with other information sorry the information that uh i can reach depends on the

policy that is given to the function as we discussed before on aws it's a little bit similar running the end command will print the environment variables i will get the session token the access key and the key id these are permanent this is changing um and then using the aws cli or any uh api call that i want to make i can call aws dynamodb these tables get the list of tables using this token okay so we talked about the security challenges and we talked about how we get uh or how we connect to other services when we're in the runtime now let's try to understand how can i get into the runtime i told you there is no ssh to

the environment so we cannot uh hope or do some network uh probing and try to understand maybe default credentials and and log into the ssh terminal and then do that no we cannot but uh the runtime execution if there is a vulnerability inside the function or in the configuration we can then uh try to execute commands in the runtime so this is an example over a scenario that will show a show now so there is a user interacting with the slack chat bot the chat bot communicates with the slack api of course which is configured to go through an api gateway the api gateway will execute the lambda function which will grab data from the database

and we'll print it back to the user

so this is the slack chat there is a chat uh chat bot here that just replies back to different messages there is a vulnerability inside the function the way that i know it is because i explored a little bit before this is a cve it's not like a specific vulnerability cve known using a specific dependency and you can see here i'm trying to run a command injection attack required child process and then run some some commands so i i'm doing here a curl or an an execution to uh curl to a hacker.androg.io which is uh an http tunnel that i have here and what i want to print is a test now that i figured out that

it's vulnerable this request is coming from the function itself so now i can change the test to get some more interesting data and i can do unless for example get the file the function of the file inside the code inside the runtime now that i know the file i can try to read the file right so cat index.js and i probably want to base64 because we don't know what's inside maybe it's even binary so i got the base64 now i can go to the uh anger console get the data that i received uh it's basically a base64 a conversion of text i'll just copy that and decode it on my own computer

then i got the source code of the function itself and i can see the api calls to the slack channel so now i know that there are some keys here that allows me to communicate with the channel and i can see in a dynamodb so a database call here so right now using the code i crafted sorry using the code i crafted a specific payload

that i don't need uh to go through uh the the terminal anymore i can interact with slack because the code you can interact with slack so what i'm doing here is i'm creating a new dynamodb client this is just a syntax so require aws sdk allows me to create aws api calls i'm doing a scan scan brings reads the entire database and the table name as i saw here is inside the environment variable dynamodb table i don't even need to know the the name the bot token he's in the environment variable i can just use it because it's in the runtime and then i'm going to send it directly to the slack channel here and i got the entire

the entire database this is because there is a vulnerability in the function but also because the function has um permission to scan or to read the entire database if this database if this function only writes to the database i should have only uh allow it to write into the database in the policy not that it's going to prevent the attack but it would limit the attack surface here all right so we saw one uh the entry point was in a third a third-party application but basically there is some kind of an api behind it now let's see something is it a little bit different than api which is a cloud storage what happens here is

there i can upload files the files are going to sit in an s3 bucket once the file is uploaded to the bucket the bucket is configured to execute a lambda function so it's some code and the code uh sorry i'm uploading it through an api web api um the function gives me a token to upload directly to the bucket and then the the bucket interacts with the or triggers the code so let's see the demo sorry okay so what i'm using here is dbsa a damn vulnerable serverless application so it's a service a vulnerable by design serverless application with hundreds or dozens of services and resources behind it and there is a feedback channel here which like a page

here which i can send messages and also uh add files so if i want to attach a file i'll just select the file and i can see from the um the response the request that i'm sending uploading a file and the response i get a talk a url which is a pre-signed url to upload a file directly to the s3 bucket i'm not actually going through the application but i got from the application the uh the assigned url and i'm using the sign url to upload uh to a cloud storage but when i upload to the cloud storage uh it it uh executes a code that processes the file and this uh lambda function that executes that runs the

code is vulnerable to command injection so uh what i'm doing here is doing a command injection that calls again this http channel i'm doing the http channel so i can get back the information because it doesn't come back in the response uh many servers functions are not synchronous so you don't have a request with the response data but instead it's more as i said asynchronous event driven so now that i have the file name set like this what i'm doing is i'm printing the environment variables which we saw before what they hold so i'll run that [Music] and you see i got some requests here and now that i have the data the environment variable i will decode it

and i get the tokens as i said if we copy the session token the access key and the um secret key we can basically run commands from our own computer to interact with the cloud services that are allowed by the function so let's see how that happened aws cli interacting with s3 service i'm using the profile defcon here and now i got the all the buckets or all the cloud storage now because the function has permission to do more than that i can continue to uh interact with the uh with the services and get more and more data and you can see i got some receipts here and i can even modify a receipt and upload it back

to the cloud because i the function has access to do so so my tokens my access keys also have access to do that

so basically i uploaded a file and the case here would be that maybe i uploaded some file that i purchased something and then i can call uh support and tell them nothing happened nothing arrived and they'll see the receipt but anyway this is just an example okay let's continue so how do we understand or how do we find security vulnerabilities in a service environment can we use the same security tools uh that we used before to test for serverless application we can but it's not going to be as good as you think it would most of these tools are really ignorant to uh to the environment so they don't really understand anything else other than http

or some kind of a tcp based interaction with the code they have the uh the request coming in so the sync and the source they follow but they don't really understand things like interaction and configurations which is cloud-based so they don't have the context uh if a function has access to uh has a vulnerability let's say but it cannot do anything because if the permission doesn't allow it then even if there is a vulnerability a potential vulnerability it's not uh it's not exploitable because the function cannot do it and it is going to be blocked by the infrastructure itself so they can produce would produce a lot of false positives and a lot of false

negatives because they cannot see anything that it doesn't have uh a sync and a source or an http usually uh if something happens because there is an analytic service running in the background that uh and every uh and every part of of data processing runs uh a function then what happens is that when you run tests they cannot test it you can if you're using a tool like a das service for example you cannot even put point it to the endpoint because there are no endpoints in this case so what would you do slow executions it's not working as hey i have a new version of the server let's run a scan now to see that

everything is in place no new vulnerabilities no different lambdas are run are added to production on a daily basis so it's hard to scan or to run some security testing for them so basically that stops blocks the developers and disruptive to the ci cd so security teams uh so developers and security teams don't really like using traditional tools and if they do it's because we have they have the licenses but they don't really have the ability to get the the best information and it's hard to scale in such a big environment uh what you should do uh is do something else because you have context of the cloud you can actually identify automatically everything that happens so

if a developer pushes new or creates a new api connected to a code you can actually go through the entire process because these are all sitting in the same or sitting in the cloud and you have if you have the right tools you can actually monitor all of them and understand the flows and what's happening beside them what we do at contrast is that we send test those scenarios with malicious payloads and then we verify that each uh each of the resources to see if they are vulnerable or not and if they do we know also what is the impact what is the blast radius for the entire account there is a talk that i gave

in black hat about attacking serverless function using alexa device so with my voice what i did was a sql injection using just my voice of course this is not something that you can run dust on uh or even sas because they don't know the uh the the source or the sink in this case um so this is a different approach that you should try to understand uh okay so we saw some examples but there are many more risks for serverless i'm not saying that service is not secure i'm just saying there are challenges and security risks which you we should account for and we should understand first of all so first of all is the

event injection which is basically running injection attacks through events whether they are synchronous are synchronous direct or indirect authentication and we mentioned before we need to make sure our entire system is going through authentication zero trust maybe even if the services and resources are stateless sensitive data exposure relies everywhere over privileged function we discussed that vulnerable dependencies we all know i don't need to uh add more about that insufficient logging and monitoring so we don't have uh the ability well we have but we need designated tools to be able to monitor and log cloud-based application or serverless application because the data resides in locations that we do not own in most cases or as services so for example in aws you have

the cloud trail and cloud watch logs which you should then uh monitor and get the information from open resources so basically resources which are misconfigured to have access externally denial of service versus denial of wallet shared space i mentioned before lambda lambda functions or functions in general can have data from uh previous executions by mistake and of course secret management you can read all of them in the os servers top 10 project right now it's an interpretation of the original top 10 uh into a serverless environment we're working on creating a new serverless and tailored top 10. uh you can participate if you work with servers so we would love to have your information uh participate in this open

uh data data call also if you want to try it yourself you can use the os dvsa project it's an open source project you can install it and practice there are videos there are uh examples of how to use it and you can try it yourself make sure when you do if you do do not install it in production because it will deploy or production or any important aws account with sensitive data because it will deploy functions with a lot of permissions there which are vulnerable um with this i will uh conclude thank you very much to uh everyone who participated here uh if you have questions i'd love to answer in the remaining few minutes

thank you very much oh that was uh interesting i knew about the serverless the dumb vulnerable set in this application but i have to say i have never played with that uh yeah like most of the staff is like you know it seems like the wheel repeats itself with all the misconfigurations and uh that's usually kind of the the main issue uh with all of this so i was wondering like um if you could share like any type of story where there was an incident where the like the entry point was one of these serverless applications um so the example that i gave was from an s3 bucket and and basically an api gateway those mainly would be the most

common ones because uh they are they are many times they are configured or misconfigured to be open but there are other services i fi i have some uh or i saw some security vulnerabilities in services which would be a little bit harder to exploit because they would not have an end point so you cannot really reach them from the internet they do have an endpoint but it's uh an aws endpoint so you need to have the right access credentials to them the the maybe the most interesting one was um using uh aws pipelines and code commits so when developers push new code there is a process that spins uh a function behind that basically what it does it takes the code

push it into an um a container and runs some processes in the background to see tests and everything to see that it's uh it's uh not uh or it's running and going through the uh end-to-end test or integration test and in the function there was uh um first of all the misconfiguration like usual especially because it was like a testing or a pipeline function so it was not exposed to the internet and they had paid less attention to security so it had multiple permissions over permissions in uh in containers and easy to use to spin up spin down whatever containers and it had a vulnerability that you could basically determine the location where the code

uh will uh will um be downloaded to uh will be sent to the container afterwards because it would zip zip it in a way that you gave it and then push it into the uh container and what happened is that you could change the code structure so it would override code in the container when it put when it downloads it into the container so if the developer or someone who has access to it um uh even that they don't really have direct access to run code on the container they have to go through the pipeline they could structure the code in a way that they could run their own code inside the container and do whatever

they want that was a very complex but very interesting scenario cool thanks for sharing that um so we don't have any questions on the q a and also we are out of time now so thank you so much for for your presentation i appreciate the time that you took to be with us today thank you very much