Immunizing vulnerable web applications with AppArmor

hello everybody my name is Jeremiah chick and today I'm here to tell you our little story with a partner so why is a primer at all interesting if you use a primer properly and you take care of it and and you really put in the effort then it can be used to immunize your applications against their own vulnerabilities known or unknown wonder abilities and this is why it was kind of interesting to us but let's let's first look into what it is actually so just a little introduction if you know what SELinux is then just take out a lot of features and a lot of complexities and then you will end up at something like a

primer so you have process running somewhere it is trying to make a system call and a primer basically hooks into the system calls we are the Linux security module framework simple any kind of file operation network operations obvious exactly and of course signals be trace actually quite a few and then what I ever gonna look up a centralized like a central policy a profile for that process identified by the path of the executable and it's gonna ask this profile ok is it ok for this process to do the system call so basically the profile is a wide list it says what your application is supposed to do what it can actually do and of course if based

on that profile that application just cannot make that call then it will deny it and and your app will receive a permission denied error or something like that so it's like the Cliff Notes version but there's the story I mean our story with this begins somewhere else so I haven't told you anything about Prezi yet Press is this I'm presenting right now with Prezi it's it's basically an online presentation software that gives you this zooming interface and it we believe at least it's a better way to to visually communicate because it gives you kind of a spatial impression of everything and one of the features in Prezi is that you can upload a video file and actually embed it into your

presentation so let me show you actually I prepared a little demo for this oh sorry just one more thing this whole thing this video upload is served by a micro service with written in Python and it will actually take your video that does some processing on it and it's gonna put it into the present and I made a little replica of this service so that I can demo your really nice thinking about it

so this is basically the user interface of that service whatever and I'm gonna say okay let's scale my whatever everyday why I'm uploading it doesn't really matter and I'm gonna upload this cute kittens thoughts avi of course and the conversion is happening in the background and this is really what used to happen when you when you embedded a video to Prezi so let's see oh [Music] this is really lootcrate I don't know if you if you had a chance to spot it but this is exactly the contents of the etc password file okay so so this this happened I mean one day we receive a bug bounty submission we have a bug bounty program and and it was telling us that

hey this this works I mean they could actually read the password file from the nodes that were running the conversions for for these video files so just just for I mean I'm pretty sure you see what was happening this is just a recap of this there was a CV about this so the issue was in in ffmpeg HTTP live stream processing feature it basically did not properly validate the URLs and it allowed to access local files as well instead of perforated to teach me requests so that was the CV and the funny thing is that CV II was released three days three days after we received the valve anticipation so we were a bit surprised and also impressed but of

course we had to do something right one of the things I want to add here is that of course we did a lot of love digging and looked into and looked into all the ffmpeg logs we had actually pretty good logging about what ffmpeg dogs does in the background and we found hard evidence that nobody could exploit this to extract any kind of user data so this did not escalate into into an actual incident of course we found the researchers attempt which is great so so this this happened but of course we were ok ok this this was not not escalating to an incident but we still had to mitigate the issue right so we asked the question what the hell

should we do and there were a bunch of options like patching ffmpeg or container rising ffmpeg with darker C's roots whatever and we found both of the options pretty slow and I think this requires a bit of explanation so we handle all of these valid bug bounty submissions as potential incidents so and for for Incident Response we have an SLA that is measured in the order of magnitude of hours and patching this thing rebuilding these services redeploying it to a bunch of ec2 nodes all the this stuff validation testing whatever all the processes it would have taken as days so I think this gives you the context why why we found these two options slow but there was a

farm where we already had okay so I'm cheating a bit here because we already had some basic infrastructure around it and it was not the first time we we used it but it turned out to be like a viable option I mean we could do mitigation with that part more like in an hour or so so let me show you what we did we created this profile for for ffmpeg it basically says ok it's ok to execute the binary it can read the uploaded video files and it can write to the directory where we store the converted video files and basically that's it some common stuff is there that is basically often aided by by all processes basically so

and that was it I mean we came up with this now you're wondering ok but you could still read other people's video files but luckily it's not the case because then you would have to guess the file name which you will not be able to so that was our our mitigation back then and of course I'm not trying to say that it's a bad idea to patch it's always a good idea to patch but as I said because of her SLA we we wanted to find a quicker solution and then later on we of course moved on with patching a fan back to the latest version but this was this was pretty new stuff there then so and then I thought

to myself why not why not take this a little bit further I mean yeah now ffmpeg was vulnerable but this was a Python web service that could have been any vulnerability inside Python or any Python library that we were using and of course we do not really know so so why not why not confine the entire web service for pepper mark we have a really good idea what this web service does on a system call level we can we can easily come up with a profile like it's it's it's really not that not difficult so I started to experiment and came up with with this thing so what you can see here is basically two profiles the first one

says that it's that the web service can read this configuration file and I created a profile that this kind of profiles come on gene Aquarian applications I don't know if you know about Gina crane it's basically a Python web server and and yeah the last line says that it is allowed for this service to switch from this profile to another profile and this is quite important because this is how a part more makes it possible to do privilege separation right now it kind of makes sense to have like two phases for application 1 initialization phase where it's okay to access the configuration file and then a phase when your service listens to client requests and then in

that profile it's not okay to read the configuration file because we already read the secrets from there you see what's happening so so it means that when the processing is in this place then no matter if an attacker can can find a vulnerability by which it can read local files he won't be able to access the configuration file which of course may contain some some sensitive secrets there and of course then when when it started to listen to these requests it must be allowed to execute ffmpeg but only can find with the profile with which I just showed you earlier so yeah there's the basic idea of course you will have to tell apparmor somehow from your application that you

want to make this profile transition so for that you will have to use LeapPad Parmar which is basically the user space library for doing this kind of stuff it's it has a Python binding compiled by swig and it's basically one line of code really I mean it's really not that's not difficult you read out the conflict files and then you switch to a profile where it's no longer allowed to read the config files now demo time again so I have my little service here and I'm actually on the node that is hosting this thing and I'm going to say okay at murmur please and force the profile for this web service I call this profile one

variable never mind so let's try our little exploit again I just enforced this profile for a primer and I'm gonna upload my specially crafted cute kittens avi I know doesn't really matter about let's change resolution as well and it is doing the conversion again now with the profile I I showed you in the previous slides and that part my working in the background so let's see the results oops no more password fie there so it works but of course I mean I showed you that profile there it's not trivial to come up with this it's not like I mean you will definitely have to have some tooling around especially if your service is a live thing so we started to

come up with some infrastructure that supports developing these profiles by basically letting you know what the hell is going on inside that Parmar luckily apparmor has really good integration with the audit subsystem so you can find all the audit logs about access denials in the audit logs and you can basically just pours it we we do lock stash so it was pretty easy I mean I know it's not readable but it's really up in this repository I just linked on the other slide and of course you can stream all these logs into elasticsearch and then you can do some visualization in kibana where you can see your the all the system calls all the operations that

were denied all the profiles all the objects that the applications are trying to access and of course the individual events and this gives you like live feedback on what is happening and how good your profile is doing and of course if you're already advanced and we do this at Prezi already you can build some alerting around this whenever a par more says no too to some system call you can for example to your slack alerts and tell the security team this is this is pretty important because this tells me that either so either there's a new vulnerability we did not know about and someone is trying to exploit it or our profile is not correct and it's not

allowing some action for that service to to carry out even though that action is legitimate so these are the two things and both cases and both pieces of information are really useful so yep a lot of things was said I want to give you a little recap like a little summary of what we learned during this whole process of playing with that partner so one thing is that you probably shouldn't confine all the processes I mean even if you have all this tooling around it it probably doesn't make sense to create a profile for every single thing that you have instead you would probably want to do a proper risk assessment approach and and see see what the risk of an

arbitrary command execution or an arbitrary fire rate can be on that mission because it might not be that's critical in all the cases yeah so probably you don't want to go crazy and and start kind of finding everything but it is as I showed you a good idea to integrate it with your existing monitoring system because this is really the thing that will make a poor more a a viable tool for your developers and it really has to be the developers because the profile describes so the profile depends on how your application works under normal circumstances how this supposed to work so it really lives together with the code so so actually it could be regarded as part of the code

and and yeah so because the code is understood by the developers you really really have to give them some tooling that makes makes it possible for them to use a primer as an additional line of defense on the other hand testing it is not it's far from trivial I think it starts from having good acceptance tests actually this is what we did when when the ffmpeg thingie happened we basically started to iterate on the profile always updating it adding some extra permission and then running the acceptance tests and see where it breaks and then we iterate again and again and again and in the meantime we could monitor this whole thing on cabaña as I showed to you and

finally a primer is just as good as your profiles I think this came up earlier in the apostille assist talk that basically profiles our whitelist if you say in a white is that it's okay to do anything then you are not better at all and you are just adding complexity but not security and that doesn't make too much sense so it really has to be has to be maintained and it's not snake oil I mean it's not like your security solution for everything yeah so that was kind of it questions no question what about selinux in comparison so we considered as Linux as well but we found that the cost of a Scylla looks for an

organization like us like Prezi which is like a midsize organization is this way too high because of the extra complexities because of the whole difficulties around the cynics I mean I know that there are definitely some advantages it has a lot more features it has a lot better documentation but if we really want to distribute this and ask developers to actually develop and maintain these profiles then I Parmar is a lot more viable option because of its simplicity then then I see Linux so what you just mentioned about armed developer square in the profiles do you think that's realistic um or do you think maybe another approach might be to kind of enforce a profile and say the

application needs to conform to this like which approach do you think would work better so when you mentioned about the profiles and having developers create them do you think that's realistic or do you think maybe it might be better to I mean that gives them say your application needs to work with with this

thanks very much for that question because I actually wanted to point out one thing is that the idea of using a perm were in this case with ffmpeg it came from a developer one of our developers so I was I was actually really proud and and I and I was like okay so people are already already aware of this and and the are okay with using it and trying it out with all this extra tooling so yeah based on that I think it's it's totally realistic of course your profile will not be like at any certain time fully up-to-date and and perfect but we are probably probably good with not perfect but almost perfect so I mean it's thank you very much it

might be the case that developers are very relaxed about permissions and also we have to take into account that with each update of application things might break might not work anymore so I will advise also to put one point about put it on monitoring mode for the first part of installation first few days I would say week and then on with every major update depending on how bold our developers with major and minor updates have a look out about things that get denied just for maintaining operations

yep thanks for thanks for pointing that out so apparmor has a like a complaint mode where it only logs what it would deny in enforcing mode and of course I did not say but quite obviously we did not run when we were fixing FM pi we were not running it's in enforce mode like in the beginning we were iterating it in complaint mode so because of exactly the reasons you were mentioning because of preparing for possible downtime and now voltages and and just breaking the whole thing we really have to take a careful approach approach there but it's but it's totally feasible to do that so thanks for pointing that out yeah thanks for the

talk I have a question so in the first demo you created the profile to harden the filesystem and specify specific locations that it has access to and you're allowed access to the uploads directory only now this was working for you guys because the filenames in that directory we're using UUID as a file name so the attacker a possible attacker could not identify not guess the file name to access it correct what would happen if you didn't use UUID and you had other types of file names like sequence I don't know would it work so yeah if you if you take this perspective then basically this only works because there was no way for the attacker to

list the directories right because if we can look up any file name then even the UUID thing will not really save us so actually that is also something that you can control with a primer so you can say that you cannot released this directory so that's that's what you can leave and of course there is as I mentioned the issue that these paths might change the application might change how it handles these file names and everything so this is why you actually you're kind of you kind of have to adapt your profile to the application all the time what if the file names are predictable that's my point okay you cannot list the contents of the directory what if the

filenames are predictably like 1 2 3 4 ok so using parameter tampering you know I can get any final 1 so the findings are predictable than a primer will not save you because it has no idea of the context in the sense that of course it doesn't know about any kind of authorization that you might implement on the application level so it speaks system calls it speaks 5 permissions it doesn't speak user or stuff like that because because that's like like a much higher level concept so you cannot cannot really do that anyone will you publish for example the profile generator or the L configuration on some github or yes I mean even even the

application I mean I mean this little Python thing is is already app the profiles are already up I think the locks - config is nothing that's wrap or yet but I can push it of course and I will yeah sorry so that was in the Oh No

so this is the link sorry I don't know what's the good formats of sharing these after not doing what if I'm running my services on docker for instance what would be the benefit or the added value of apparmor in this case so the thing that docker is that sorry so with docker you can do all this without a parmer the thing is that the all these security related features around docker tries to build on security in depth so they are providing a bunch of security related technologies and they might be redundant they might do the same thing so for example with some volumes and SATCOM and and capability restrictions and are limits for example you can you can do

the same as with an epimer profile but the idea behind this model is that you might want to use some technologies for a specific use case you might want to combine these technologies just in case any of these might fail or not up to date or not correctly configured so that's that's really like just taking a security in-depth approach and actually docker really supports a perm we're pretty well now of course the dr. Damon itself has an epimer profile and it has a pretty good default that former profile for for the containers which you can start with and then you can then you can iterate on that okay so Joey thank you very much for your torch

you