Envelope Encryption, the proverbial silver bullet - Sundar Krishnamurthy

hello everyone i hope you folks can all hear me i'm sundar krishnamurti i basically manage application and cloud security with expedia group in seattle washington to tell you the truth i'm sleepless in seattle thanks to all the crazy things that developers technical uh people managers all those that do do with implementing systems software security and for me the talk that i have today is all about envelope encryption the proverbial silver bullet you can see all data is always encrypted right it's obviously true whenever you have a breach and the auditors or the law enforcement has was this data encrypted you say yes it was rot 26 encrypted and you give them a smile they won't understand what you're

talking about but i hope this talk would be more entertaining i would i'll give you some bad jokes not some not so bad jokes and please glare at me give me comments if you think my jokes stink most of them they do so i'll make certain i give you good ones so let's get started envelop encryption the proverbial silver bullet so let's see what are we going to talk about we're going to talk about the fact that most companies around the world today go to extreme lengths to make certain they guard pci data pci is basically payment card industry so what the payment card industry essentially do is they make certain credit card numbers are

tokenized they are encrypted you use double triple encryption you've got a systems that are not accessible from any networks you don't go to insane lengths to protect cardholder data or chd but what about pii which is personally identifiable information do you what do you do to protect that data some some companies say we protect that other company say we encrypt that but the gdpr clause is very clear they say encrypt everything so my talk is going to be mainly about encryption so personal data pii is covered by many clauses like you have gdpr and ccpa like the joke is that gdpr is a four letter swear word it indeed is so that is the global data

protection regulation from uk and europe and ccpa is basically california's privacy law both of them say pii must be secured it must be not accessible anywhere and if you get breached via for gdp or ccpa you are going to face hefty penalties so the problem's hard and we're trying to solve it the easiest way that we can do it so dev product teams were never taught the right methods to protect this data correctly and the best method to protect according to me is encryption encrypt everything encryption will get you to a very good place but how to do it will be the focus of this talk usually what teams do is insiders are given a lot of

power and privilege with the implicit assumption that they will work in the best interests of their product brand platform or whatever is their startup idea they're going to work in their best interest but you give too much power to these insiders to these people like i was working at a startup many years ago and the thing was like there were 40 of us on the team and fall 40 knew the usernames and passwords to production all 40 of us that wasn't a very nice place at the time i wasn't in security i was like a startup software engineer but now i think outside the box and i'm like that was crazy we should not be doing that so

the last point is logical and physical access controls differ for most teams they probably never engage security teams in their design are built out so security always gets called in the end like we folks security architects engineers managers professionals we get called in like hey we want to launch in two weeks can you please dot all the i's check all the eyes are dotted t's across push check boxes and just tell us we can launch and we say no you can't you called us so late we're going to do a performance threat model we're going to see what are all the gaps that exist in your system what are the ways via which your insiders outsiders anyone can

compromise your system we're going to go deep into it so we've got to come and make certain we grab our arms around this so the thing is we need to move to a better place and we say encrypt everything that is stored at rest unless public so that should be the uh tenet a mantra of holy grail for us so the first bullet point very simply is there is no law anywhere we know yet which states you cannot encrypt certain kinds of data and must be stored plain text why did i write china in a small the question mark there china is basically a country where any data that's encrypted inside china you must escrow the encryption keys with the

chinese administration the concept is that the chinese need to have the ability to decrypt any and every data inside their network inside their country inside their systems and do it plain text if they can do it you are not allowed to do encryption in china that's the only caveat but other than that if you encrypt data everywhere you're good like i knew of this weird use case there was this team what did that team do they were storing orders for countries around the world and apparently there is some law in malaysia indonesia and some asean countries that says pii must be encrypted at rest so what did this dev team do they were looking at the order

countries and then checking if the order is malaysia indonesia or singapore we will encrypt it if it is any other country like india or pakistan we are going to leave it plain text because these countries have no law and i'm like this is stupid this is ridiculous was my my feedback to the team i just asked the team what will happen tomorrow if india come out and say our data must be encrypted address too the team was like yeah we are going to write a program it will go to the database find every record which is plain text from india and we will encrypt it i'm like this all this is a waste of time so my point was

just encrypt everything at rest you need not check if it's india be encrypt if it's singapore we encrypt if it's not then we don't don't do that it's overkill just encrypt everything at rest so that would be the best way out and most importantly one buzzword you keep hearing across all security podcasts and conferences everything is zero trust the zero trust infrastructure so when you come to the zero test model it's all around encryption but my talk is going to be more around key management and that is going to be what is the crux of the problem here if you manage your keys nicely encryption is there but if you don't manage your keys you have a big

problem so the problem is that we have vague and snowflake encryption patterns that's what a lot of times what is happening is you would say it is very secure because teams are using aes 128 or aes256 so teams are using rsa 2048 it is secure then it has the team where is the key and they say the key is actually a file that is checked into github and i'm like all bets are off github is no place for you to store a key use your key should ideally be inside a key storage mechanism right so what do a lot of teams do is they say we are going to perform encryption we are going to use

good algorithms but we are going to take the key and we are going to put it in secrets manager aws secrets manager or the teams are going to place it in hashicorp vault which is another good place to store secrets but then hashicorp vault and aws secrets manager are great place to store credentials passwords api keys database connection strings but not encryption keys that's a completely wrong way to do it it's like putting a square peg in a round hole it won't go in so what the basic problem is that there is no key rotation mechanism in place there are no logs for encryption decryption tasks so you don't know that key has been used to encrypt how much

data or what were the mechanisms that you had if the key has ever left that secret storage we don't know so teams have been doing this wrongly in the past they need to change so a lot of times what teams did was they encrypted by hand in the sense what i'm trying to say is they're not going to come up with their own encryption algorithms essentially that's what that's not what i mean encrypting by hand so what i'm trying to say is teams are using encryption libraries like bouncy castle for java or the microsoft dp api and so many other mechanisms that exist but the problem is that nobody managed these keys so what would happen is keys would be checked

into a secrets manager or hashicorp vault and then what would happen when one of those people leave the team so if one of the admins who quits the team do you have any governance mechanisms to know that person went to secrets manager logged in as an admin copied over the encryption key that's used for all your data and he quit the company do you have any governance there you don't know right it's a hard problem so that's why we tell that never maintain keys by hand ideally there should be no humans who should ever have access to keys if you can take yourself on the holy grail path there you're in a much better place so

some developers checking github keys into github and then they say they're saving costs because instead of going to secrets manager and getting billed for every call they're making one call to github and getting this keys answer is no it's completely wrong so simple tenets we need to adopt all encryption keys must live inside a secure purpose-built device which is called a hardware security module or hsm it can generate runtime encryption keys for you to use so you have a device it's like a card or it's like a proper vault it's a it's a server that's going to be installed on your network what would that server do that server's ability is to generate encryption keys for you so

you can take those keys and encrypt data with it right so the two ways to which it would work is you can either import encryption keys one time from the outside you can push in a lot of encryption keys or if you don't want to go that route you would rely on it to generate encryption keys within itself and keep handing that out to you for you to use so as since we're going to look at aws here with my example i hope my the demo gods are good upon me i have 10 minutes that have kept in the end for going through that we'll go through that so the two examples in aws are you have

one service called aws kms which is key management service it's super chip cheap it's shared across many customers aws would have thousands of customers sharing keys on one device which is good for them and the second one is cloud hsm which is way more expensive than kms because it's a dedicated fips 140-2 level 3 compliant device it means one device can only contain keys of one company no other company organization should ever have keys inside of hsn that is dedicated so here is where envelope encryption 10 minutes into my call i come to it so what does arnold encryption do there are two separate things in envelope encryption one is called a master key the other is called a data key in the

past this was called master but i think we changed we never use the word master database slave database all those words are gone so instead of saying master key we say it's a managed key so the managed key lives in kms what can they do it can generate data keys for you and you encrypt data with your data key so there are two keys here a master key or a managed key and a data key the two different keys this is very different from asymmetric encryption where you have a public key and a private key pair as what you have in rsa 2048 elgamal elliptic curve cryptography or perhaps x59 certificates that's a completely different asymmetric here both these

keys are symmetric they're unrelated to each other so our two keys exist with three acronyms they're called cmkbek and edek there are two keys with three acronyms let's see what of them what they mean cmk stands for customer managed key it's hsm resident aes 256 symmetric key that can only encrypt 4k bytes of data that's it it cannot encrypt anything more than 4k so if you have a 10 tb database you'd be wondering how do i use a cmk for it the answer is you don't use a cmk you use a dek a dek is also called a kek in the microsoft azure world or in the google world they call it kk aws calls it the

ek which stands for data encryption key a random piece of encryption material aes 256 32 bytes that you employ but it must never be saved or logged anywhere so what we tell developers is whenever you're going to get this key take the key inside the function it's a auto variable or a stack variable use it to encrypt your data and then toss the key away don't move the key to a object reference or store it anywhere on the heap because if you have a core dump the keys are exposed to the world the third one is called edek edek is the encrypted data encryption key so it is the dek encrypted with cmk this must be

saved along with the encrypted data or cipher text all this might be sounding greek and latin to you i'll show you in my demo how simple it is this is how the whole thing works create a create a primary encryption key or employ an imported primary encryption key within the hsm k1 k1 lives inside the xsm k1 can never ever leave the hsn there exists no api no mechanism probably the mossad know about it no one else does you cannot get k1 outside of the hsn it lives inside the hsm and it will always live there period now what do you want the hsm to do you want the hsn or the shared hsm which is kms which is

shared across all customers to say get me a key so you have a generate data key api which gives you two keys or it gives you two entries one is k2 with what you will encrypt your data and the other one is k21 which is k2 encrypted with k1 so you take k2 encrypt your data with k2 toss it away and save your cipher text along with k21 k2 1 is k2 encrypted with k1 now whenever you want to decrypt the data you give k21 back to aws kms or the hsm and say please give me k2 back it will verify whether your iam user has permissions for kms colon decrypt for that cmk customer managed key inside hsm

do you have the right authority if it says you do then it would decrypt it and give you k2 back what do you do with k2 take k2 decrypt your data with it and you have it plain text this very simply is envelope encryption so how does this improve our security it's very simple there is no risk of cm cmk data leakage as nothing else should ever get saved outside the dek should be tossed right away after encryption or decryption never logged to memory never written to disk never logged anywhere if you log the dek anywhere that developer is going to get a python on his face from me let me know if you know of

someone who's saving the deks outside i'm going to call them up late in the night and i'll threaten them in an italian accent so they don't do it so you can enforce policies only certain uses users are allowed to encrypt others can only decrypt and all events get locked so if the encrypted data encryption case k21 gets stolen by a hacker you're going to save k21 right if k21 gets stolen by a hacker he would never be able to use that to decrypt your cipher text because he does not have the permission to go to kms and say give me k2 for this k21 he probably doesn't have that i am user permissions or policies or writes in order to do it

so key rotation is very accomplished very simply by re-encrypting k21 with the new k1 so the it is the difference between this you are boiling a spoonful of water to rotate the encryption key you are rotating the encrypted key so 32 basically bytes of data is going to be re-encrypted rather than encrypting a 30 tb file a 30 db db backup which is backed up by a key if you say you've got to decrypt it and re-encrypt it you're about looking at many thousand dollars worth of ec2 processing power from aws because to decrypt a 30 db file to plain text and then encrypt it with another key it is boiling the mediterranean sea but then if you're going to re-encrypt

k21 with a new k2 it is beautiful you don't have to worry at all because k21 encrypted with the new k2 is boiling a spoonful of water rather than boiling the mediterranean sea and you get components out of this so set up automatic yearly key rotation within your hsm it will maintain a train of expired keys for decryption for older data that was encrypted and data custodians of compliant personnel can be granted a re-encrypt permission they can only rotate the edek they cannot encrypt data they cannot decrypt data they can only rotate the key once every year so you have compliance and there is no risk of key leakage ever unless developers do stupid mistakes that can be mitigated

via secure code reviews and education and encryption is available as a service right now so aws offers this service called ssd kms which stands for server side encryption using key management service other cloud services have also embraced this design pattern and it's available across gcp azure and the best thing about ssc kms is that no core needs to be written you just have to create a cmk in kms and then go to the cloud formation templates or terraform scripts for each of your aws services like s3 buckets dynamodb tables sqs sns document db kinesis streams aurora postgres sql dbs even ebs which is the elastic block storage and just say i want to implement sse kms and aws does

all the magic for you so aws is offering envelope encryption as a service and calling it ssc kms it's part of their waf which stands for well architected framework not the web application firewall waf is well architected framework it's one of the architectural best practices from aws it's a tenet so i'll just go over these points because i have a small demo i need to run in the next 12 minutes never use secret storage systems like hashicorp or secrets manager establish a pattern where only service accounts have the ability to encrypt and decrypt no human should ever have the ability so this is known as hosm which is hands off secrets management you make certain that humans

have no ability to view any data outside of dashboards you can view data on dashboards obviously which is logged in everything but you can just go to a summary to a database and say select star from credit card table you can do that so audit all your secret stores audit github and find out dev teams who have done encryption wrong and make them learn envelope encryption it's very simple to do and pci dss auditors love envelope encryption it might form an important prerequisite in the forthcoming pse pci dss v 4.0 that's going to come out sometime soon so you can what you can do as a good engineer is you can take a set of keys push them into aws secrets sorry

push them into aws kms and cloud hsm you can push them into your own hsm into gcp and then you have the same set of encryption keys everywhere so if one of those services were to ever go down you could go to another service and decrypt your data you are not like a cockroach upside down with the six legs in the air unable to turn itself back up you don't enter that scenario so here comes my dirty demo thankfully i have 11 minutes to run through this demo so very simply let me know if you folks can see my terminal here so i do a clear i do an ls minus l i see there is nothing oh there is lots

i just tested it so let me do an rm star i just tested this out yes yes

one minute let me go back here demo gods be benevolent upon me yes [Music] i think i just kicked myself in the foot just give me a minute guys just give me a minute

uh

i can do it here as well clear aws config

can you folks see my screen here can you folks see my screen here i think i just shot myself in the foot i deleted the key if i mistake that i have here so basically i'll just go over this what i was planning to do let me just go back here you folks can see my screen again here aws yes the command is working so what i basically do is i do vi secret dot txt i say secret sauce 101 colon wq so i've saved this file what i'm going to do right now is i'm going to create a key in kms so i say aws kms create key region usbs2 description sample key for

b-sides yvr it created a key for me so the two very important things this is how you create a key so first thing you do is you grab the key id this is the most important thing copy and then you come back here and then you create an alias for the key you see sample key yvr i'm going to give the name copy paste sample key why we are an alias is created for this key the next step that i'm going to do is i'm going to generate an encryption data key using the kms key i created before so i say aws kms generate data key key id sample key yvr key spec is 256

encryption context this is a encryption context is a string you must always provide this is how you make certain the hacker who even steals your key will not be able to abuse it because the encryption context would not be on i'm saying projects b sides by er here is my key there are two separate parts in the json blob first is the cipher text block this is k21 it is k2 encrypted with k1 i have to copy paste this because i need this so i'm going to just paste it here and the other and the most important thing is plain text this is your encryption key 32 bytes this is the key with which the data is going to be

encrypted copy paste it here what i'm doing here is just for the demo otherwise you should never be pasting the key anywhere human shouldn't be able to do this so what i basically do is i take this particular text copy it and i give it to the echo command to base64 decode it into a file encrypted key plain text copy this paste files there now take the plain text key copy it plain text data encryption key base64 encoded copy this paste paste it now i have these files now here is the most important thing that i need to do i use the open ssl installed program to just simply say open ssl encrypt minus e

is encrypt using aes256 the input file is secret.txt which is my secret sauce 101 file out is an encrypted secret dot txt and the key file is file b colon triple slash i have to change this path because i was in the aws instance that i actually shot myself in the foot i have to change this to say users

source repose i just make certain this works okay it works so now if i do a cat encrypted secret.txt you will see this is the encrypted file so this is my secret source 101 which has been encrypted with my file the file that i use for encryption is basically the in the secret.txt is the input file and data key plain text is what was used to encrypt it the next thing most important thing i need to do delete the secret delete the data key plain text and then do an ls minus l so what files do you have you have a file called encryptedsecret.txt you have a file called encryptedkeyplaintext.txt right so encrypted keyplane text is dk1

k21 and encrypted secret is the my cipher text now how do you decrypt it i still have four minutes i can run my decryption part very simply paste aws kms decrypt the encryption context first project is b sides yvr this is the secret string i told you about encryption context should be your secret string the ciphertext blob is this file and i've again got to change the path because i shot myself in the foot with the ec2 instance i'm going to using my local one so i say users [Music] source repos decrypted so what did it decrypt it decrypted my k21 and gave me the plain text k2 so this is the key with which we

encrypted so i copy this and i say paste it here and i see paste that particular k2 here into this file paste and this is our open ssl command to decrypt the file copy o since o did not come out and i again have to change the path

you get one dot pxt now if i do cat secret one dot pxt secret sauce 101 and that is my demo i still have two minutes for questions if you folks can ask me any sort of questions

do you have any questions folks

i think almost out of time that was my demo guys demo boys were fine instead of running it on my ec2 instance i was able to run it on my macbook because i shot myself in the foot and i deleted the ssh key that i had i was thinking i was already on the ec2 instance so that rm command was me shooting me myself in the foot and this demo was running it on the macbook thankfully everything saved and most importantly the last thing that i want to show you before you guys drop off is these were all the commands these are all links that i have and please note down this one if you folks can take a

print sprint screen or something you would get my entire slide deck at this bitly link if you can note this bitly link please do thanks everyone

you