CG - Techniques for Escaping the AppSec Labyrinth - Andrew Hay

we good all right my name is adrew hay uh I'm the director of Applied security research at Cloud passage and this is the escaping the ABAC Labyrinth talk uh might not be what you're expecting uh as you can see from the turnout so they're losing out so I I used to be an analyst an industry Analyst at 451 uh worked at an information security office in a bank in western Canada oh sorry let do that at a university in western Canada and at a bank in Bermuda I was also a product program and engineering manager at q1 Labs so I've been doing a lot of Hands-On practical application of security for quite a number of years and

this talk was really born mostly out of the information security office side of things where I was sick of people handing me servers with all the apps pre-installed and said all right you know put it on the Internet it's ready um um so what this talk is going to do is we're going to go through some of the steps that you can take to actually profile the applications that are on systems and actually know where to start when someone drops something in your lab and these are the very brief overview of topics that we're going to be talking about and there will be a tool as well Chris so the Labyrinth the story of the

labyrinth and this is probably the best slide in the entire presentation um the the concept of the labyrinth was built by datalist to hold the uh the minur in creep and datalist made it so complicated that he almost didn't Escape he actually needed a string to get back in and so the uh the guy on the left is Alex Hutz icus also part of different story but this was the best picture where I could show Alex half naked so the way he got out was by following the string and that's really the clue to Escaping The Labyrinth and I couldn't talk about labyrinths without a David Bowie quote so there's the the requisite David Bowie reference from the

movie Labyrinth so there's a number of ways to solve a labyrinth and uh there's actually a whole bunch of game theory around how to quickly solve labyrinths and mazes just by looking at them and over laying all sorts of different data on top of them that's a little bit too complex so we're going to talk about you know you could blindly walk through the mace which is really what uh security people are forced to do now you could start from the opposite end that's only so good when we're talking about application security and great you can work backwards but it's still probably going to take you the same time to get out as if you went from

the other end uh there is a known way of walking with one hand on the wall event eventually you'll get out it's going to take a very long time to do so but eventually you should get out of that Maze and uh then there's having a map like a turn BYT map to know where you're supposed to be going that is definitely going to be the quickest way to get through any sort of maze or Labyrinth and well what's better you might ask so of these four what's the best method turn by turn turn by turn because it's easy and we're lazy so I I'll give you another example so what's better if you were given a

Rubik's Cube and it's put on a table and you were told to solve the Rubik's Cube you could be given a book on how to solve a Rubik Cube or you could learn how to solve by doing by actually picking it up and fumbling around until you figure out the methodology but what's probably better is having both so you have the map and the instructions on how to solve the Rubik Cube and the Practical Hands-On knowledge to actually go and fiddle around with it to follow those instructions so this I'm terming this tactile application security um I don't expect it to have much uptake but this you know nobody I think has ever said that application security is easy it's

definitely not like riding a bike it's more like riding a bike like this so why tactile I could have said tactical I could have said any number of things but I wanted to really emphasize that Hands-On methodology part because you need to install the software you need to understand what's going on you need to look into the packages and the binaries to see what these applications are going to do to the host operating system and so the definition of tactile it really does um you can reinform your understanding based on book learning reading uh any number of blog posts anything on the internet because everything on the internet is true has to be practical knowledge of going

through and doing the installation redoing the installation finding out what's wrong and then just common sense tactics to go through everything so this is typically how anyone in an Enterprise will deploy an application so you'll select the platform application framework you'll deploy it and then you'll think of maybe securing it at least this is how it was at some of the bank well definitely at the bank that I worked at not so much at the University but definitely the bank and what you get is something like this a house that's completely in disarray and usually what you know your SharePoint deployment will look like even after securing so how this is how I propose you deploy applications you know you

select the platform the application the framework you deploy it in a test environment you learn what it's doing from selection to the initial installation you can then secure it in that test environment and you learn more that way you can then deploy securely in production novel concept I know and then you monitor it after the fact so this this is one of those Common Sense things at least to me anyone disagree with this am I missing any steps no um ultimately we probably should have another arrow that goes from monitoring and uh or at least from the last learning back to the selection because this process should influence future selections and future deployments kind of a lessons learn type of

thing so it's definitely not not impossible and the way to do it is to really interrogate your applications when you're putting them on systems and this tacti methodology of continuous learning I propose that you learn by looking at the system and the application before the installation after the installation and also during the installation um you want to know what the application is not just doing when it's sitting there stagnant and running on the system but also what it's interacting with on the network side of things so this is the very easy 10,000 ft view pre-install post-install running and you get data actually do stuff with and uh because I'm as lazy as everyone else in this room I actually did write a

script to help uh not necessarily automate the collection of the data but definitely do some processing with it after the fact it's the go button so in terms of pre-install information who here uses a red hat based distribution and anywhere so these are a lot of things that I forgot even existed until I went looking for this talk so RPM qlp for the package you can dump that output to a text file and it's going to show you everything that is installed and its location where the default installation will be after installing that package I thought that's pretty cool you don't even need to have access to that system you just need to be able to interrogate

the RPM and you now know where to look when you want to go and start hardening um and don't feel that you have to furiously write all these down I've got links to all these at the end of the presentation RPM minus Q scripts so there's a lot of RPMs that will generate uh anit scripts and cleanup scripts after they're installed and this will allow you to dump just the script to a text file so you can see what's started um there's actually a really good example here so this is one of the this is the pre-install script so it's going to add the uh group 48 Apache we're going to add a user with no

login and we're going to have the directory as well so this is very important information so before it's even handed to you if Joe Blow admin or Jane blow admin goes and installs Apache you know what's what's going to be installed whether they change the permissions of the uh the binary or any of the directories that remains to be seen um but this gives you you this kind of points you in the direction of where to look all from inherent tools also check config who knows what check config does anyone configures your startup phones yeah yeah so instead of creating scripts yourself this is actually adding httpd to start it's pretty cool a lot easier yum most people use yum now

instead of RPM uh especially if they're using S just because it's a lot easier there is a special util that you can install called yum utils and then you can run the query or you can run the uh executable repo query and if so time is the package you can dump that to a text file and what it's going to give you is the exact same information as what uh back here when we did the RPM qlp for the package it's going to give you the same thing the benefit of doing this is you don't have to first download the package this is quering the repository Ser server and giving you the information so you don't actually have

to have that RPM local even easier you're freaking me out that you're writing all this day I'm going too fast let me know or just say go back D package if you're using a buntu or a Debian based distribution D package minus L and I put some grep some egrep in there people hate it when I use EGP but it's just easier to visualize um and what this is going to do is strip out a lot of the prevailing garbage that this command gives you and it's going to give you that sample output in the same format as the previous commands so it's going to give you everything that is going to be installed the directories

paths file names now unfortunately apt you can't um you can't just do like app get show me or you know app show me the interior there's an app file that you can download and what it will do is you can interrogate the the package in the repo just like we could with yum where it's app file list WordPress and again I put some cut in there to get rid of some of the garbage but you're going to get the same format as before all the directories and all the files if you're sensing a theme you're on the right track now Windows is a little bit of a pain in general but in terms of finding

where things are in a Windows registry um you're not you're not going to find Microsoft documentation that says all of these registry keys are entered and these are all the values for all the keys you'll find some and those are usually ones that you can secure through um not Mom whatever they turn mom into and uh that's it so you're not going to know everything so what you could do is just dump the registry from the command line so regedit capital E registry. pre and what that's going to give you is this messy messy output here where it's going to show you the key and all the values and their all the variables and their

values so this is only so good not easy to really purse through um visually it's a lot easier to inspect or you just do a diff could get a little messy I've got a better tool for you um Has anyone used

malware.trace in the back end so instead of running your own cuckoo sandbox server you can just upload your executables here and it's going to do everything that is done on Cuckoo sandbox uh it's not entirely reliable you get what you pay for it's completely free so it does kind of go up and down from time to time I think I've had to create usernames like three or four times since it launched because they keep blowing away the database uh but you can you'll you will get registry information file directory information what files are dropped and where and then addition addition analysis of those files this is really the lazy person's way to analyze what a Windows binary

does so it 64 Meg Max upload but I'll tell you what it does so once you upload your executable and here's the cautionary tale of when you upload it they store it and they get it so if you're installing custom applications that were built and are um the property of your organization or have keys that are already put in there you're throwing that out into the wild and then they own it so this will you'll upload the executable they will launch a typically I think it's a Windows XP sp3 VM they'll install it and it's all automated so it'll go through it'll click next next next take screenshots of the entire process and then it's going to start

analyzing what has actually transpired once this piece of uh software was executed so you'll get things like this under files you'll see all the files that were created dropped modified uh registry keys you're going to see everything that this executable has done um this is actually a piece of malware so uh you also get very detailed information under drop files of all of your dlls all of your executable batch files anything that was created um you get hashes you will see if there are any v r rules that will match that are checking against this hopefully your corporate applications that you're installing are not going to match against the signatures for malware if it contacts any sort of

domain or has any sort of network connectivity you will see it here as well uh for the domains hosts any HTTP requests interactions IRC s SMTP again hopefully doesn't interact with IRC or any of these really other than domains and hosts uh there's also behavioral analysis where it's going to monitor what's happened from the point if you were double clicking on the executable and running through the process it's going to show every step of the way what was edited so you'll it's almost like running an executable in a debugger where you're setting break points along the way this is just capturing almost every break point as we go another easy way to do it on your own

system or in a VM is to use red shot free to download what it's going to do is record what happens on the Windows system as soon as you double click the executable it's going to record all of your files directories registry key values uh any additions CH changes deletions if it cleans things up after the fact it's all going to be recorded available on Source Forge very cool users on a Unix system you know this is pretty pretty common information uh I've just added some Cuts in there to actually get us that list in a nice clean format so Shadow for users uh on Windows I did upload a Powershell script to GitHub that will allow you just to run get

local user and it's going to give you the exact same information and the same output where it lists top down administrator through all your local users again I'm lazy likeing this user grp group mappings again Etsy group with some cut you're getting all the information I probably should have cleaned up the uh the semicolon but man you get what you pay for uh I also uploaded a Powershell script for getting the group uh this is going to give you all of the user groups on Windows system startup scripts this is my very lazy way of running fine to show all the executables in uh in itd and uh dumps it all to the file same format you may or

may not care about this personally I'd like to see what's created and what's going to start up once I reboot the system Windows didn't have to write anything for this although I guess I could have made it easier but the gwi wmi object win32 service that has all of your startup script information information whether it's it's a manual automatic or disabled so if you do this before then install a Windows application run this again then you'll be able to see what has been changed if it turns on printer uh the so back 10 years ago a lot of applications would say okay well they're always going to have the printer Damon running so I need

that to be running otherwise I'm not going to be able to start that's what they' used to hook into and uh I actually worked at a company that did that and when you're talking about servers you probably don't need the printing Dam and running all the time so uh a lot of applications fail so if you see things being turned on by the application that look odd to you question it now more of the file system you could obviously exclude a lot more than I have here but for the sake of brevity I've just gotten rid of proc because proc is always going to change uh and this is really this is as ghetto of file

integrity monitoring is you're going to find so if you wanted to do this in a Cron job go for it uh probably not the best thing and find also works on windows so find Star your directory dump it out sy so this is a lot of information to do manually I'm sure you agree uh services on Linux host netstat with those flags dump it running services on Windows get service where the object is running the status is running dump that out as well any questions about phase one yes why not use nut stat on Windows as well because it I could there there's a lot of overlap in a lot of the windows uh Services I wanted to give people

things that could easily be scripted later without having to go outside of Powershell or the wmi like a lot of the wmi commands you could just call remotely uh which could be easier if you're doing just kind of centralized monitoring with distributed hosts uh there's there's really no right or wrong way to get service or any sort of information out of a Windows system or Linux system and the problem is that there's so many like I could have put VB scripts up to get information out of uh the different apis in Windows but I haven't done VB in 10 years and that's how I like it any other questions about phase one like I'm sure there's other things that

I'm missing by all means if if you don't want to say anything now come through find me after I'd love to add to this list because I think this is great for someone coming into an operations group like oh you know if you gave the security person all this information then they're not going to hate you after you give them the server just a comment about people that use OS 10 for whatever reason usually during an install you can hit command I or command L gives you a log of what's going to be installed or where it's going be insted oh very cool so very simple yeah so command I or command L on OS yeah very cool yeah this has been uh

primarily created for Linux and windows but I should probably extend it because there's a lot more people using Max for servers like you said for whatever reason so during the install process monit has anyone used process monitor on windows before it's a great and very very noisy tool um unless you set your filters you can really turn this into an installation Watcher so you can have it watch you can set it just to monitor for your file name uh you know installer.exe and it's going to show you everything that happens on the system in relation to installer.exe wire shark obviously if you want to start monitoring what's happening on the network be alarmed if you start seeing

outbound requests when you don't expect outbound requests same thing TP dump going to work on Windows and Linux uh same as wire shark I tend to stick to wire shark or t-shark on Windows I don't know why just personal preference and then TCP dump on any sort of Linux distribution just because nine times out of 10 it's there and again install Watcher so any questions anything missing from uh while things are being installed I seriously considered about using um s trce on doing installation but I think you'd have to like be able to read The Matrix just by looking at things if you're going to start doing that that's a lot of information to try and process and call

through it would be too much fun fun yeah I guess if you got the time by all means do it but that's that that's quite the exercise in uhar oh yeah you you get so much information from running as trace and then the executable so post install I don't know how many of you know this but in V lib D package info you're going to see all of these scripts and installation information uh for application so pre-install it's going to show you everything that's going to be run before the installation post install any cleanup it does uh post RM when it starts removing things what process it goes through it's it's going to give you

the actual nv5 sums of the files which is you know kind of kind of useful to have uh and a list of all the files and directories created by the installation so a lot of your work is done for you here but you first have to install it to get to this point so I could see this being very valuable if you were doing this in a test environment you're going to know exactly what's going to be created for that particular uh deban package version and you're going to get the the md5 sums if the md5 sums differ on the different systems well then that is a little suspect you might want to see if you're running a different

version than they are or install a different version or if uh different different files are created in different directories that's a little worrisome to me then this is again post Unix uh these are all the commands that we talked about in the pre Unix Linux information gathering and uh I've just put them in the order that actually seemingly random order because this isn't the order we covered them in but this is all information you could just dump to the file uh typically I put either a pre for the beginning and then post for the aftermath and that way I can diff them a lot easier and keep them kind of sorted in my head and again with the windows

all the same commands so back to Red shot so red shot actually once you run it there's a little button right over here called compare and it's the O character and what it's going to do after you do the second shot is it's going to show you everything that's different from or everything that's newly created from the first snapshot so this is doing the diff for you and it's only showing you the diff this is what's been created so it's going to tell you the files that were added the directories that were added the registry keys that were added pretty simple pretty lightweight now there are some this is a shell a pow shell script that I uploaded it's going

to give you network statistics as you're running your systems or sorry as you're running your applications and it gives very detailed output where you're getting the protocol the address the port the remote that it's interacting with uh again you can get this from netstat but this goes a little bit further and gives you the the pit and the process name and everything this is it's just a easier way to digest what you're seeing and again wire shark TCP dump uh once it's running I would definitely recommend if you're going to install an application in a lab I would leave this leave your dump running because maybe it's got a wait timer maybe it's not going to start communicating out for 15

20 minutes 2 days it all depends how it's created right so you probably want to see leave it running for a week in a V see what happens see if it starts interacting with the network now this this is a really this is probably the first Ruby script I ever wrote so if you look at it and are want to rip your eyes out I completely understand uh I am not a developer but what this does is it will start enumerating your root file system and tell you the counts of all the files in those directories so this is good if you want to know um um what's being created or the counts that are differing

between pre and po install uh then you can do a diff of this information as well it's only so valuable it becomes more valuable when you want to start doing like file Integrity monitoring because very few people will say okay I'm going to do fim on rout and everything and recursive through everything because you're going to get screwed on proc and and some of the other things you know so this is this is just kind of like a handy thing to to get a count of everything so now that you have all the information what do you do with it uh you could just tuck it away for reference later on you can manually aggregate all the data to figure things

out which is you know ideally what you should be doing now with this kind of stuff to understand what's going on or you could use a tool that'll take things a little bit further um and now I'm going to tell you about the T tactile file profiler uh which is much harder to say when you're drunk than I realized initially so we're GNA call it TF tailer pardon me Tac toiler Tac sounds kind fun something something shark native all right so this is really you know I would call this pre-alpha it works for some of the stuff uh what it is going to do is it's going to take the input of your pre and your post it's going to do a diff

for you and allow you to it's going to take all your file permit all your file Loc and get the permissions ownership uh sgid suid um for directories it's going to do the same for ownership and permissions so right now it's just doing the file system permissions on Windows or file system permissions and ownership on Windows and Linux which was a good place to start so the caveat it's very poorly written in Ruby a little bit better than the uh the other one that I wrote but uh it is mult platform and you can get it at uh my GitHub repo so yeah thanks to inputs uh recursive find of the directory I've already been over that so it's really

easy you know Ruby tfb the first the pre-file and the post file and uh from there you get something that looks like this and I know this probably isn't the easiest thing to see but this is actually a tool that I use internally at Cloud passage when I'm defining rules uh because it makes it a lot easier when I'm trying to figure out file Integrity monitoring or um configuration security monitoring rules so what you have is it's always going to be active uh the element what it's going to be so in this case Etsy WordPress WB config PHP whether it's a file or a directory it knows to calculate or to figure that out uh a

description that is null uh file presence should we do a file presence check on this yes it's a file you can decide later on if you don't want to do that or not uh it's going to dump out the ACL of the file the owner the group owner uh if it has set u ID or setg ID set if it's a directory you'll have the directory ACL as you can see down here um and you know is it going to have sticky bit set these these ones here at the end are things that I'm going to be adding through the interrogation of the files so you know what uh what interface is it listening on what what process should

own it so that's kind of coming later oh and I forgot I highlighted everything so here's the map so file ownership file group purple directory ACL 755 directory ownership Directory Group ownership 0 so rout so what you can do is you can take all this information and because it's in tab delimited format because I think people that do CSV should die in a fire for parsing that's horrible tab delimited is going to be a lot easier so you just import it and it's going to give you all this information in this nice format obviously on the bottom all the nulls aren't that great depends on the file as I add more features and functionality to

it that'll get a little bit more full so the next version of that particular Tool uh I I really want to create an exclusion list because do you really care what image files are created or other sort of temporary files that are going to be cleaned up after you don't really need to look at that um I'm going to have more command line switch support because right now you can only it'll take two inputs and that's it uh you could probably try and do three I don't know if it's going to take the first two or the last two or the first and the third it's Ruby it's magic I'm not sure uh I'm going to actually have it

kick off uh some sort of packet sniffing and then feed that information back in to the tool for better analysis and uh like I was saying for the the last section of the of the output I want to see you know profiling of the service whether it's running listening ports ownership of that process and uh and just you know profile service related information which I think would be valuable uh and I also want to generate all of those profiling files that we talked about initially with just a handy command line switch where you're not going and running everything top down by yourself you would do like you know tfp D- pre and it would generate all the

files and then tfp D- post it would generate all the files and then maybe tfp D examine and it would take the two and and merge them together and give you all the information because I I want to make this easier because not a lot of people do this uh so I had no idea how long this was going to take because I had so many slides but uh so you know what I'm what I'm doing now and are there any any question questions about the methodology I've employed can anyone see themselves does this look too complicated to do right now manually are you going to wait for me to do the tool for you yes so just a question clarification

on objectives here you're primarily focusing on system Integrity so deploying application there and you're looking for the inputs to deter pre and post that that's one use case so the question was uh was this primarily for file monitoring or Integrity monitoring uh yes and no I could see it being used as a way to really document what the application is so if you're doing an instant response or forensic analysis exercise you're nine times out of 10 you're given a either a hard drive or a computer is just put on your desk like go forensic this which is fun um at least if you know where to start so if they can say oh well you know from the

logs we think uh that the web app was popped so okay well now we know what application uh was potentially compromised so we can look through the the post or the gold status State we could now run the file against the current file system and see if anything pops up so this is something that could help Focus an investigator on an incident but also if you bringing in interns that you're telling okay go install these applications you know what the expected outcome is before they've even started and if they have questions you can say well you know based on our documentation we have you know this should be set to this and you could also

verify uh file ownership so just because it's given to you with 644 on all the directories and files doesn't mean that that's the most secure state or if it's the web server is owned by the default uh www data that's great but maybe you don't want to do that uh you'll probably still want to like put uh put your web server in a rooted jail and you know just do extra things but the the primary purpose of this was to give people something to reference down the road um I know when I was building servers I would forget like just like when I'm developing these scripts I had to go through the uh not the tfp but the other

one and read through it like so that's what that does again all right I'm trying to think of what it did and it's the same thing with deploying application servers um especially in Academia they don't here work at a university or has ever worked in a university very rigorous documentation procedures nothing changes all the time everything's a gold standard image it's all yeah pristine never the case never the case so this gives a little bit of extra ammunition to figuring out what your applications are doing on the system any other questions yes so you I'm trying to put this into a context for like a big Enterprise company or something and I'm also wondering about you talk a little

more the context where how you're using it at and what sort of capabili process okay um so yeah how how we using this a cloud passage and what are we using it for so how do you sell your man on that well this was the kind of thing where I did in my free time while my wife was watching Dancing with the Stars and I could tune that out and uh then I just kind of presented it to them as a way to make the policy definitions for our product which allows you to do fim rules uh configuration monitoring file location information I wanted a way that I could do this profiling hand hand that

spreadsheet off to the analyst and say okay go build the policy and feed it back into the product so it it was really taking a lot of manual effort out of it for me because I've got to do that and apparently you know what I do also is marketing and you know sales over here and startup you wear like every hat on the rack so it this was just a way to make my job easier and I thought it would be helpful and you know could be used to help other people like if I had this when I was at the University this would have made so much sense for me to have just you know file for every

application that was out there instead of like oh yeah we installed this six years ago we're not sure what it does uh something horrible happened we we think one of these 10 servers has child exploitation images on it go come on that's not easy this is good stuff thanks well then it was worth it and I didn't have to watch Dancing with the Stars so it's a it's a t any other questions so just just to sumarize you know you can navigate this application and system landscape if you know where to go and hopefully I've shown you how to generate that map yourself obviously I'd say 99% of you are going to wait until I make

the tool just like with a big red go button so that you don't have to do yeah so you don't have to do all this pre-work it is a lot of pre-work but you can script it all and then you're probably saying well you could script it all too and then we could just execute the script but you're not a developer but I'm not a developer and it's going to be in Ruby not in Python because I don't care yes so you are developing the uh tfp on on GitHub right yes technically we can also assist you right you can Fork it make it better awesome make it good and you just have you can't make

fun of my code because it's really bad it's not like all full of go-tos or anything but it's damn close so it it's easy to get the information about your applications right now you just have to take that manual effort and do it um or again wait for me to do it for you and we really need to stop like I I could see this tool you could give this to your Ops people or your Dev Ops people and say hey you know what just run this script and give me the output email it to me I don't care and then once everything's done run this script and email it to me that's all I'm asking

you for just an email know now you know what's going on when they drop it on your desk you will have a plan in place of like oh yeah well like all these permissions are wrong and the ownership's wrong so I know what to go in you're not going in blind and because we really need to stop putting out unconfigured unsecured servers at on the internet obviously it keeps the security vendors in business but it's just not a very good practice to follow uh so the all the tools are on these slides um especially the malware.trace

this from somewhere and put it available for someone else to download so I I can't remember I'm going to say it was a News Group uh process monitor you get it from CIS internals uh process monitor actually used to be split out into regon and something else and I can't remember what it was but then they kind of merg them yeah that's right but they merged them uh wi sh TP dumped you probably have those already so thank you if there's any questions please yes can you script the process of sending the email yeah I'll do a Cron job just for you buddy no problem yeah he asked if we can script the uh the

email probably yeah you know what if you if you made it part of your build process I think that'd be great document you know I will not accept this as being built until this email gets sent out with this file output and get someone with a a v in front of their title to to sign off on it I think that'd be great good luck yeah good luck if you work in Academia never happen they'll throw that whole academic freedom card at you you'll never see it again all right well there's no more questions uh I'm around all day today and tomorrow grab me say hello that you get uh 10 [Applause] minutes