PAL is your pal: Bootstrapping secrets in Docker - Nick Sullivan

i am strongly encouraging women to participate at passwords.com to get into information security i have been co-supervising a master student that did a talk here last year and she also spoke at defcon with a room completely full did a great talk and i really really appreciate to see more women get into this stuff so if any women have anything to talk about related to passwords in any way let me know okay very easy okay so this is nick sullivan he's head of cryptography at cloudflare and last year at mccarron airport i was seriously free and tired of las vegas i was going home and i'm just walking there like a zombie and suddenly there's this bearded guy

appearing me coming up to me like hey you you you're petrossian right running past yourself yeah that's me i want to go home and it was nick and he said yeah you know dude i'm kind of working on something and i might have something for you at a later point in time and look here he is so nick sullivan and pal is your pal go ahead all right thank you thank you everybody um and yes raucous audience uh thanks for coming it's 5 p.m for this talk and also everyone online thanks for watching uh so i'm going to talk to you guys about a tool uh a tool that we built at cloudflare uh to help manage

secrets and specifically secrets relating to orchestrated containerized environments and that that sounds a little complex but um i'll get into it uh it's it's very simple so we we have a problem uh a very basic problem which is uh our can we have a container that's running in in a server that needs an api key and our docker container is built from source and we don't want the api to be in source of the api key to be in source control and the solution we came up with was uh excuse me basically encrypt the secret before putting in the container and run a trusted agent on the host that's uh underneath the layer where the docker

container runs that has access to the decryption key for the secret and when you run your docker container have a special entry point that talks to the host to get the secrets decrypted before it runs and that's the essence of what pal is so if you need something that does that this is your tool so where passwords con right so what what does this have to do with passwords um well i i i it doesn't directly have to do with passwords but it sort of has to do with the broader concept of of secrets and secret management and uh typically secrets used for authentication come in in in three major flavors there are sort of sub flavors

and subtypes but you have passwords you have bearer tokens which are strings or objects that that represent an authentication and you have private keys or secret keys now passwords as you probably heard hundred times over the last two days these are these are stored by human memory these are things that people are supposed to remember and manually enter into computers and if you're sending them from a client to a server it should be over an encrypted transport or someone can can copy it and replay it by contrast bearer tokens are high entropy strings they're things that a human being is not going to remember it's something that you store in your computer's memory and typically you can

ext get them in in exchange for a human-oriented password uh authentication and um there's examples of this are abound right uh cookies are an example api keys are an example and these again are needed in encrypted transport um there's also an expectation of these of that they have a certain lifetime which is may may not be the case with passwords uh the third type of authentication we can think of as is private keys so um when these are kept on your on your client and they're they're they're often oftentimes pass phrase protected so they are encrypted on disk and unlocked with a human memorable phrase that's that's a lot like like a pass password and uh

when unlocked you can keep them in this sort of persistent memory on the on the system and uh they allow you to do some really nice things such as digitally sign requests so that that your authentication doesn't have to necessarily go over an encrypted transport um but this complicated nature of it's it's a key sort of a random long string makes it very difficult to be portable um another example of this would be your is two factor authentication so the the codes that you generate from two-factor authentication these are derived from a key that belongs to a secret key that belongs to the client so um passwords as sort of human memorable things they have these requirements of being strong

sort of unpredictable and uh different per site this is oftentimes very hard for human beings to to handle because now this is uh this is from dashlane's blog people have on the order of a hundred or so different logins so remembering a different password for each one of these is as is a serious mental challenge so um for remembering passwords and for dealing with authentication uh we've derived some of these tools that sort of enhance our capabilities to make us kind of superhuman password remembering folks and and password managers are one example so um they support things like syncing across devices having a pass phrase that kind of turns your password into the same security

style as a private key would so you have a passphrase to unlock it and you can configure permissions as to which applications have access to these passwords and at one time and they have another feature called strong password generation uh which some people use some don't but uh it allows you to have turn a password into essentially what a bearer token is like a long entropy ridden string that makes it a lot harder to crack but then it also makes it so that it's not something that is human memorable you still have to remember the passphrase to unlock the passwords but this is this is one example of a tool people use and um when you're when you're at in an

enterprise or an organization there's often times shared passwords and uh these are sometimes passed around with sticky notes or you know email the administrator and these sort of things very very bad practices but um another tool was built for this and this is the set of enterprise password managers and this is kind of as like a centralized police officer with a with a case right there that that has a list of policies that says who is allowed to have access to these passwords and um in this example someone wants access to the visa account and they had they bootstrapped this based on regular authentication so the user says hey i'm this person here's my credentials and the server

will say okay well here's a temporary use for for this sort of shared password um so password managers and enterprise password managers are really human-centric they're they're built with people in mind they're they're built so that you have manual interaction they're they're built so that individuals can manage secrets and um and and these tools are are very useful and very helpful so but but this is not what what my talk is about my talk is about the other side of the story not clients as as sort of computers or phones or something like this or individuals but it's more as as as servers so servers need secrets too and uh the way that people manage

secrets uh passwords and and these sort of credentials for servers are are very different than than the human centric password based mechanisms because uh computers are not people right um so servers need secrets and just as an example if you're building a web service or you have some sort of server up in the cloud that does some things for for customers or for for people you may need api keys uh you might want to talk to other servers doing so you might need credentials to access dedicated databases or you might want to be the the terminating layer for https or tls and and keep the private keys so um this is these are things i don't recommend but what some

people do in this case is they have well the very the very first thing that you can do is if you manually manage your server you can just log in and write the secret into the server but if you're working at any sort of scale this this sort of manual work is is automated by a configuration management system and uh what people can do is put the secrets into the configuration management system these are these are things like ansible or salt or chef or these these tools that make it simple to to configure your servers and then they take the secrets and they put them on disk or in environment variables and then they just trust

whatever applications are on the machine to access the right variables using system level controls so when you're talking about environment variables you you can define an environment variable ahead of running a command in unix and other other services can can find out what that is by looking to see what the command was to operate these these um these functions so you're really trusting a lot of system processes to keep these secrets isolated from one service to another as i mentioned system acls so you can have services running under different accounts um and and this this is really you know hard to manage and easy to break a privilege escalation is one of these one of these bugs that these

bugs that we see all the time um and relying on this is is potentially not the safest thing to do if you're working at with a bunch of developers you may have a development environment and a production environment and if you just keep your secrets in configuration management then all your developers have access to it as well if they want to build a coherent development environment and be able to test these things and that means that your configuration management is in a source control that's accessible by non-operators and that extends your trust to everybody at your company who's potentially a developer and this this is not really the this is this is the way that

some startups do it but this is this is not really a safe long-term approach um and and furthermore applications are developed by different teams so if you have someone on some team runs an application on your server that is maliciously or or not trying to access secrets that are only used for other services you don't really have this strong boundary level here now uh this this is obviously sort of a straw man but uh but people do implement their servers and and protect them in these ways so what are some potential solutions for this um well more commonly people do uh this thing where they encrypt the secrets in source control and then they take their set of secrets they

make a parallel set for development to talk to development servers and a set for production that's encrypted that can only talk to other production servers and the secrets in the configuration management um they are encrypted with a key that's only held by one secure box and this is this is sort of one one node that is deploying the configuration outwards and this is this you reduce your trust space from uh source control management which is accessible to everybody to some one dedicated machine that you can lock down the access to and um and when you enter you add new machines into your system they have to authenticate with this configuration management master node and it will be able to distribute the

secrets so this is this is not something new um there are secret management tools and this is this is something that has abounded over the last couple years in terms of open source solutions so you may have heard of vault or knox from pinterest or key whiz from square there's a bunch of other ones lyft built their own but they all have sort of the similar architecture and that is uh essentially you trust your machines on first use so whenever you provision your machine you give it an identity and then this identity is what's used to uh you sort of manually accept these machines as part of your system and and trust it on the first time you see it so

you create a machine identity a provisioning time and then you can create this identity key pair and and register it and this is this is a very common pattern um and as i mentioned vault keywiz knox these sort of take this to the next level but um the basic idea is trust on first use and this identity that's provisioned to the machines is used to bootstrap a secret management tool and um so so i mentioned a few uh configuration management tools before they have these plug-ins that do this so salt has a gpg plug-in where you give a gpg master key to the salt master chef has something called encrypted d-bags which is just really funny to say

and vault keywords and knocks are a little more a little more complex but they have really sort of interesting password rotation uh and account rotation features built in so once you once you log into vault you kind of can it can deliver all the secrets to you uh keywords has this nice thing where it mounts a fuse file system uh like that looks like a file system that holds the secrets and whenever you access them it does a secure tunnel back too a central secret management server but as i mentioned the downsides here are that you have to have a secure secret server already running so this kind of makes makes a two two class system really where you

have some small trusted machines and then this lets you bootstrap and send secrets to all the rest of them um so for for people who don't do a lot of server development uh there you can you can think of this here's an analogy for this which is uh kind of a permission system like ios or android has and you can think of uh your machine as an application your permission settings as this configuration management system so if you approve and disapprove different accounts to uh sorry different applications to do different things this is this is like configuration management and your secrets would be system resources so um if you get a new application it can try to access these

secrets if it is in the proper configuration management system and and that that's that's sort of one way of thinking thinking about it and this is this is great um but we're in 2016. so a lot of these things don't don't often apply so it's great if you have a single team uh if you have a shared configuration management system for your entire company or your number of servers scales to the point where you can manually approve them all this is the third one's typically the case but um this is not how modern server deployments are going in the future we've got this there's this trend uh docker docker docker docker everywhere you've heard you've heard about this a lot

docker's not the only um technology in this space but generally containerized deployments are becoming very popular so in the last 10 years or so it was more more typically you have a virtual machine that has a full machine image and in docker is sort of a more compressed version of that and if if you look at this diagram it sort of explains uh how the the in the old case which is uh you have an entire guest os and this sits on a hypervisor and you can have entire systems going one two three and you can use trust on first use to bootstrap the secrets for those but in docker it's a little more compressed you you have

not an entire operating system but you just have a small amount of just the code you need and sort of a shared substrate of of the system and this changes things quite a bit if you if you consider your data center um because uh there have been a lot of tools built on top of it to make an entire data center and a lot of machines look more more like more like one computer than like individual servers so mesos and and kubernetes docker swarm these are these orchestration frameworks that that really make it so that when you deliver when you when you deploy services onto your servers you don't know how many servers there are and you don't really care you're

just treating the entire set of servers as an operating system and you use mesos or something like this to distribute resources for it just like an operating system would so if you can if you compare your applications inside a virtual machine to your applications inside of a docker container there are some very stark important differences and uh in in the docker world your applications are ephemeral you can they can go away they're designed to to be able to shut down and spin up als all the time so uh for example in marathon which is the uh in mesos it's it's sort of the equivalent of system d they have another thing called chronos which is this is

the equivalent of cron but for marathon there's a button in the ui that says scale and you say scale and then you say how many versions of these do you want and then it just goes out there and deploys them all this this is not something that's really going to work well with trust on first use you're not going to scale 90 or 2000 copies of an application and then sort of approve them all because they could be deployed anywhere in your in your infrastructure so in the docker world you have one application for a container and a vm you have this sort of world of applications inside each container and the other important differences is this

docker world lets you have different teams deploy things independently you don't have to go through a centralized configuration management system you can say you know this is a shared operating system space or a shared server you can deploy stuff here you can apply stuff there different teams can manage these independently so you may have read this this is uh just last week uh vine had this docker registry issue um i can sort of walk this if you guys haven't read the news but but basically docker.vine.com which was their docker registry that had all their their uh docker containers was publicly routable and somebody found it and was able to download their live production uh copies of their back end and uh they

were using i believe it was python so they could just read the entire code so they had the entire code base of all vines back end and more than that they had api keys they had secrets embedded inside the containers and these were just all fully revealed from just just the fact that their docker registry was open to the internet and not requiring authentication so this is the worst case for deploying secrets in docker and there are a couple lessons we can learn from this uh and these are just basic very basic things is use access control for your docker registry uh don't register containers that have secrets inside and you know sensitive code in containers is

dangerous this third one is is kind of hard to get around in a lot of cases but the first two are really kind of obvious lessons that you learn so if you think about it we can go into the sensitive code i have an example on that later but um when you have docker containers the the application model in for tofu doesn't really work you don't have you know i understand these are i've i've deployed these my 60 servers and i know where they are and i know how they're configured you have applications in the thousands so docker containers how to secret how do we how do we how do we do it so an interesting way to um to think about

this is to to take another permission system analogy and uh if you think of how ios or android constructs things um they have these sandbox applications and they all kind of live in harmony on on the machine and talk to the host to get to get access to system resources so uh this is this actually maps up pretty nicely with the data center operating system model where every container is like a signed application and uh if you have a container and you want to release it you code sign it and you do code signatures and you do that at the mastering release time and your applications are sandboxed your containers are basically sandboxed uh i had yeah it

it's basically sandboxed and if your application wants access to secrets it talks to the host and the host can sort of check these entitlements and deny by default uh just like your application can ask for access to your camera and it would be denied by default you'd have to sort of manually approve this has practical differences between sort of an ios kind of single application holding phone to an entire data center because it's fully multi-tenant you have multiple teams developing different things in parallel so sometimes your secrets are unique per app too it's not like every application wants access to the camera secret it's you can have uh application specific secrets um so this leaves us in kind of an

interesting question and and in place is is is what's missing what's the missing link to to make this model model work and that's really container identity and this is this is your whale passport this is what docker containers use to prove who they are uh but essentially you have containers and they have to have identities that you assert and luckily docker has built this tool for this called docker notary which is just like a code signing tool so doc docker containers built in a bunch of layers this is a digital signature over those layers and you can build a pki on that and you can sign it at mastering time you can do key delegation all these sort of nice things

around that so given that you have identity for your docker application let's let's put this into a visual i've been talking a lot in bullet points and and sort of generalities but um let's look at it so we have a host we have a container running on the host and say you have a secret manager this is just something running on the host that has access to a database of all these secrets the container will say hello the secret manager will ask docker notary is this a real legitimate signed container and uh does it have access basically is this a legitimate container and if so it will give the container its identity secret and an identity secret

is is like a key that says this sort of very similar to how we described earlier with the host when you securely provision a host you give it an identity secret which is a key that says yes this is this is really who i am this is this machine it's been provisioned and then that is used to bootstrap secret managements in this case each container gets an identity secret that's tied to what the application is now this is a little bit heavyweight because this secrets database deploying this to every single one of your hosts is is kind of dangerous i mean you want to limit access to this this is this is potentially you know how do you protect this this is

this is something that has all your secrets so another way to organize things is to rather than having your entire secret database live everywhere you would have a key that is a lot that can decrypt your secret uh living on the host so um the container would spin up send the encrypted secret to the secret manager it would check the policy to say hey is this container which we've validated the signature for and the identity of is this allowed to use this key if it is then it will decrypt the secret and sort of inject it back in the container and this is kind of flipping this model on its head in that it's it's becomes not not a question of

database management but a question of key management now you can split this up really nicely so each team has its own key and each team can manage its own policy and uh you can have multiple containers running on the same host and sort of everybody manages their policy separately um it's also potentially worthwhile to just have a separate server that has access to these keys and and rather than having the secret manager talk locally you can talk to something remotely and this is what pal is so uh we're introduced this is introducing the project which is uh pal permissive active link access link which is um this is a term of art i guess from

nuclear weapons it was it allows you to connect the arming data to the bomb and uh it's it's uh it goes along the theme of some of the other uh open source projects and i'll i'll i'll show you those further along the line but yeah it connects the bomb with the arming data and that's that's essentially what this secret manager tool does so the basic workflow is this you encrypt secrets before you master the the container pal is the entry point for your docker container so rather than running the application directly you run this wrapper around the application that does secret unwrapping first and so it takes your secret sends it down through unix socket down to pal

d which is which is the component that's sitting on the host and uh actually more let's let's look at this more visually so so pal is your entry point and what what happens is when your container spins up pal will talk down to pal d which would talk to dr docker notary and verify that the container is who it says it is and then paldi has a bunch of plugins or different ways that it can decrypt the secret and this is this is really up to what secret management or what key management tool you want to use so for example pgp is one option so pal d can talk to pgp to a keychain and say okay

well decrypt the secret for me and send it back up so we talked about vault and and knocks and all these things before pal d is not a secret management tool it's a container identity bootstrapping tool and this is this is very important to say because we have a million secret management tools and every single one of them relies on a secret to get bootstrapped to the point where it can deliver secrets so secret management it is really more of a more of a term of going from one secret to many and what pal gives you is that very first secret the identity secret of of the um of the container itself or the docker

container that wants to get access to your secret management tools now um applications i'm going to walk you through a few ways this is so pal is something that we've deployed at cloudflare and we're using for a lot of our internal systems we have uh a data processing pipeline that is all built on docker and mesos and the reason that this came about was we had secrets had it needed a way to manage them and we did have a secure way to identify hosts and enroll them but just not a secure way to inject the secrets into these containers so let's let's talk about a few applications service to service authentication now if you're running a distributed

system that's sort of popular nowadays to call it a micro service architecture where you'll have multiple little tiny projects little tiny applications that that handle pieces of the puzzle and whenever you take a large monolithic project and split it into tiny pieces you need to they need to be able to trust each other to do the right thing and just being in the same data center is not enough um especially if people can can sort of take any application they want and deploy it and and spoof other applications this is with well within the threat model but um if you're taking your your service and you're splitting it up into tiny services they need to be able to trust each other and

this is this is what uh pal can do with if you have a service to service authentication framework so uh the way it works is every service gets an account uh a service account with an identity key pair and uh the private key is encrypted with pal and on launch the container spins up decrypts this private key and talks to the service the service authentication serv and service and gets what's basically a bearer token a token that allows it to talk to other services and sort of assert claims and say for example if you have one container that is your api and one's that your one is your database uh the api thing can come up

and then it it will get credentials to talk to the database and uh there there's a tool that that we we've started using and other people experimenting with it's a standard from the w3c called jose and um jwt is a json web token it's it's just a simple way to encapsulate a bearer token it's it's it's you have uh claims and they're signed that's basically what it is so um when your service spins up it can talk to the identity server the service authentication service get a bearer token and then talk with any other service and those services will know that this is who they said they are that's one application um this this kind of has an enterprise

password password manager analogy um rather than a person trying to log into their get an access to the visa account it's you have this robot this sort of automated machine and and he says okay here's a jot that says uh can i get one that has permissions to the corporate visa account here's my self-signed identity jot and uh this guy says no way jose you're not on the list that's my it's my jose joke for the day yes perfect groaning uh i appreciate that anyways okay so this is this is this is one application uh another one that a lot of people have been using is pki authentication rather than using bearer tokens to talk

to each other i mentioned there's passwords bearer tokens and private keys pki based private key is a way that applications can can authenticate themselves to each other and if you have a service spin up it can you can encrypt this identity credential in pal and then it can talk to a ca and get a certificate and then then it's it's absolutely clear that the service is who it is and who it says it is and then you can do mutual auth tls between different services and be able to to spin things up a third application that i had hinted at before was this secret manager integration so if you're running vault or one of these other tools i mentioned

they need an identity secret to spin up so that's that's what pal can give you is it can you can embed this identity secret into your application and then it can talk to your secret manager tool and get all your api keys your your passwords and all these sort of things hooked up uh which is nice then you don't have to use tofu to trust every one of your containers you really you're moving the trust from the container down to the host this third application this fourth application is is something that uh is is interesting and this would maybe have helped the vine case which is you can encrypt your code with pal and when your container launches then

you can decrypt your code so if you have a docker registry and someone has access to it then they don't get anything all they get is this encrypted code which makes it a lot more resilient to if you screw up your authentication or someone gets access to it it kind of protects your intellectual property now with all these applications of pal there are a couple sticky parts uh one's revocation which is you know if your secret's in your container you have to roll a new container to to build your secret to build a new secret that's that's sort of a downside to this and building the policy for who can decrypt what in terms of applications is

difficult um but luckily we have another open source project and this has been open for i guess two and a half years or so that is is really cool that allows you to do this and pal is integrated with it it's called red october um so i don't know has has anybody heard of this tool it was on our blog a few people um but it's it's basically an encryption and decryption oracle this acts as your key decryption service and it does have cryptographically enforced policies and by that i mean you can encrypt a secret such that two out of n people need to approve it to uh allow it to decrypt and this is

enforced cryptographically so this is oftentimes called the two-man rule the two-person rule um and red october allows you to do that it takes you can take a secret and encrypt it such that you have basically two people that can can decrypt it and we've recently expanded it to include what are called arbitrary boolean predicates which you can just think of just any combination of people need to approve you can you can define so in this case you can have a could be like a super admin you can have you need one a and one of a group of bcd to approve it so this allows you to to have much tighter controls as to you know a person

manually approving how to decrypt something um this has it has it has a ui and the features are i guess the workflow is that when you encrypt something the owners of the you can encrypt it to a set of owners of the secret and also you can give it labels so when you want to decrypt it the owners need to delegate their passwords to this service or delegate their credentials to this service and they can do so for a limited number of uses a limited number of times and for a limited number of a limited set of users to decrypt it as well as a limited set of labels so um if you have a secret you

that you encrypt you can say i want uh two out of all of the managers in the company to have to approve this to decrypt it and my my secret is going to be called postgres database secret or postgres database password and um then two of those people can delegate to red october for one decryption within a five minute window and uh and then and to the pal d container uh not the container sorry the pal d host to to as as a user to decrypt it so they can say okay i want this secret to be deployed on this host i'm going to allow it to happen once and i'm going to allow it to happen

in the next five minutes and i'm only going to allow this person to do it and it allows you to have some pretty fine-grained controls over how these secrets can be decrypted but it also allows you to set things up so that they're completely automated or you can reduce the amount of user interaction so you can say if you want you can delegate for say a week you can say okay well 100 this can happen 100 times in the next week and and any one of the machines in my production infrastructure can decrypt it and so then if you have docker machines go up and down spinning up or spinning down you can have it happen a hundred times within

the next week and you don't need any manual interaction but you do have the full logging and auditing of every time someone goes to decrypt this so um i had sort of alluded to that in that in that tangent but back to secrets is uh the idea is you provision a hosts a purse a per machine user username and password with throughout october and then you encrypt your secret with the label blog blonding to your container and a set of owners and then before you deploy two out of the end owners or whatever the boolean predicate is uh decide to delegate their password to red october and then the user accounts on the machines who are expected to run this

container and every decryption is logged and if you look at this in in a visual way it's it looks a lot like one of the previous models except in this case you have owner one and over owner two who delegate password user label time and at any time this is delegated you can you can kind of restrict it to a subset of your infrastructure if you want you can restrict it to to one place or another and this is this is the main use case that we have for for pal now i mentioned pal is a tool we are planning to open source the tool very soon and in the meantime if anybody wants to

come up to me after or talk to me on twitter i can get you early access to this tool but before i do so i'm going to show you a quick demo this is it's pretty text heavy but um it will at least show you the uh how pal d actually works so all right so i have not this nice pal d demo so um first step is to start red october this is going to be fun for everybody to read but um basically you can you start up the service and it gives you that web ui that you want or you can sort of manually connect to it with uh with a command line tool or with curl um

but start red october uh let me show you part step one is encrypt the secret so um we have two secrets here one is uh defected dawn secret two is one ping only and uh essentially you can encrypt it by just curling uh or sending the secret to red october and giving it a list of owners uh jack ryan marco ramius these are these are the accounts of the of the owners that we had created and it spits out this yaml file down there that says demo it has an environment variable and a file and so the way that you write secrets down in red october is you just have a basically a configuration file that says secret and then you have

ro and then a bunch of cipher text and uh if you have a file you have a file name colon ro colon a bunch of cipher text so all right this is this is this is our secret as encrypted it does get inflated quite a bit because you're encrypting to multiple people um so pal d is um as i mentioned it's something that runs on the host and it gets a unix socket and a small configuration and basically just sits there waiting for docker containers to try to connect to it um let's see so palti's running and listening which is good uh delegate authorization this is where marco ramius and jack ryan give their passwords to red october for uh one hour

ten uses and you can sort of see right here and the the label of the the account of the demo machine and the label of demo so delegate authorization all right so it's talk to red october and that's good run pal so as i mentioned pal is your entry point to a docker container and if you don't know how docker containers work essentially they just run one one command and then that command is usually your application in this case uh pal is your command and it wraps another command so if you look right um on the line right there it says root bin pal dash socket it says the socket that it's trying to connect to and uh

root bin demo is our demo application and this demo application is really just either reads a secret from file or secret from disk and it contains this encrypted yaml uh of the secret and um and prints it out it's so let me uh let me run this so run pal all right so it says uh it has connected to pal d sent it the secrets and pal d has con has sent it to red october to get it decrypted and then embedded it into the docker container itself and then the last step is query demo where we just uh curl this docker container the secrets endpoint that is basically just going to print out your secrets um you don't want to

build a service that just prints out your secrets it's sort of the that's that's sort of the other the the opposite of what you want to do here but um for a demo it's quite nice so environment secret defective dawn file secret one ping only and if you see here this is the secret file and the secret environment and voila that's that's that's pal in action uh thank you

all right so uh we've got a little bit of time left to talk about answer questions but we're basically at the end of so in conclusion powell this is this is a tool for bootstrapping your container's identity it allows you to securely have secrets and if you're building a production infrastructure don't leave your secrets sitting around in source control don't leave your secrets your passwords anything uh in in a place that it's not unencrypted and and if it is encrypted don't just leave that key lying around anywhere either what red october gives you is this nice abstraction where you don't have any keys sitting around you have a decryption service and that decryption service can be time limited

which is really nice so pal is complementary to vault and all these other password these sort of secret management tools it it helps bootstrap the identity that makes that secret management uh world possible so where to get it um pal d just as a shout out joshua kroll is the main developer for this and a bunch of other people at cloudflare contributed as as well as myself and it will be soon on github with um a very permissive license right now it's linux only and it does have built-in mesos marathon integration because that's that's the our orchestration framework that we use and if you want early access you can email me nick cloudflare.com uh that's it thank you very much

questions

when you're speaking about the interaction between the container and palsy and the paldi is supposed to bootstrap the identity of the container yes but the connection that you have over the socket leaves us things the question of who in fact executes a call somewhat open right right so how do you identify the actual caller on the other side of the object call right so uh when the connection from the container to the host happens over the over the socket you can actually inspect uh the uid of the of the the program that's that's making that connection so having that uid as an inspection point allows you to measure it and then you can use uh docker notary to validate

that that container well then you match to see whether that is the right uid that's been sent there this is the mesos marathon orchestration uh part you just you compare it with this is all the innards of pal d but you compare the uid with what it's supposed to be and then you validate the signature on the container on top of that so um there's more details we can talk about it after uh sorry so everything i know about docker i just learned uh right now um so i'm curious um as running services on a host is it possible to still use this like red october uh to manage secrets yeah absolutely um red october itself came

well well well ahead of using pal and um the main use case for it was sort of long-lived sensitive secrets uh maybe not ones that are deployed on servers but ones that are used inside of build systems or for code signing or for things like this so if you wanted to use red october to manage your secrets in a in a sort of traditional host environment go nuts that's i mean it'll it's it's great for that that use case so i'm curious i guess why uh why why did you write a new tool versus using something like consoles um vault or i guess chef vault or whatnot yeah i i think the the main the main

point i was trying to emphasize in this talk is that these tools work but they need an addition an initial identity credential to to work and so how do you deploy that and you can you you could you can do that very well on a system where you have a bunch of hosts and you have configured them manually and you can deploy secrets to those hosts but when it comes to secrets coming directly from the docker container itself there's no way to bootstrap vault i mean that this is this is the way that we found to bootstrap vault there was nothing else out there that would that would do it for us so that's why we built this new tool

um if i'm the chief technology officer or on a security auditor how easy is it to look at a a system that's been set up this way and understand what the policies are and and exactly what's going on uh that's a good question um yeah as as as occurred to me when i was just explaining all the different points of configurability for red october it is quite complex um but if you do have an environment where you can use just a simple subset of these uh it's it's it's simple to write down the policy uh but um yeah i guess this along with a bunch of all these other new tools are new and explaining them to an auditor is

potentially difficult but i would say that explaining how you manage secrets without something like this is potentially more troublesome you mentioned earlier about the the trade-off between storing secrets locally that are encrypted and passing them into a central secret authority or key management solution versus just storing the secrets and having them manage all of those there and then i feel like that wasn't addressed in when we talked about specifically pal what are you guys recommending jurors do in that situation right um so at first when we when we first we built pal we we sort of thought we could just stuff a bunch of different secrets in here and and you you can just put all

your secrets that you want for your container inside of pal and when it starts up they'll all get decrypted um but as we as we went further along it made a lot more sense to just make sure make the amount of things that you do with pal as small as possible just limit it to the identity of the container and and that's it and and i think having a large secret management system is valuable and uh and and that's why you still want to use vault or nox or something like or keywords with with with something like this but but restrict pal to just the identity of the container itself okay we we need to to stop there to move

on to the next talk so once again thank you nick all right thanks