Tuning The Warp Drive with Laforge: New Tool for Building Security Competitions

hey guys welcome so today we're going to introduce a new tool called the forge it's actually a tool we've been using for a while but we think it's ready for like public release so here we go quick agenda we're gonna go over kind of why we got here who we are and like why we're even in this problem space then we're going to attack the problem like why is this so hard to do and after that we're gonna introduce the solution which is LaForge this new tool for doing competitive security competition environments and then we're gonna talk about the future and like what's still to be done with this tool and how you guys can all help and join so to begin

we have alex alex is a senior engineer at uber he builds a lot of systems he does detection systems he does intelligence work it runs the gambit and it ruber was a big sponsor of all these competitions we're gonna talk about and even building these tools and then I'm gonna senior red teamer I do a lot of ops and then also tool dev and we both work heavily in these security competitions so security competitions are actually how we met we met through the US you see the US Cyber challenge which is a great program there's a bunch of alumni in the crowd and a bunch of people that speak there regularly and since then we've just done a ton of

security competitions so we've done CCDC where we are part of the red team Alex was actually a blue team there yep and then what we're gonna talk about today is we are really heavily involved in a competition called CPT C the collegiate penetration testing competition so what is CPT see CPT C is a competition that started about four years ago at RIT in New York and it was put together by originally Bill Stackpole and a bunch of other people that were really passionate about security competitions they didn't playing CCDC for a while so what we're trying to do if you're familiar with CCDC that's the collegiate cyber defense program where the students have to defend the network's we're turning it on

its head we're building an entire environment an entire corporation and we're allowing the students to attack this corporation we set up logging we set up infrastructure we run it as a legitimate company and throughout the entire thing basically the students run a legitimate Penn test so they'll come in they have to do an RFP they have to submit they have to do all the papers to get in to the pen test then they win the pen test they have an external phase which are our regionals and then they move to the internal where they do internal pen testing and it ends with a report and presentation so it's very real-world we're trying to emulate real-world pen

testing and we're trying to get students involved in it because as much as you do this stuff in college once you get into the real world you're hacking boxes and you have to stand up in front of you know potentially the c-level where the security team and present your findings and we think that that experience isn't really gained in college so we are a big proponents of these competitions that give people this experience so this is a model of the infrastructure you can see we put a lot of work into this there's multiple networks there's multiple hosts and what you don't see here is this is one team we actually scale this by as many teams that compete last year we had

16 teams in the regionals and then we had 10 teams in the national competition 23 regional ok 23 in the regionals and then 10 for the Nationals so it was this infrastructure times 10 so you can see like we have a really big problem with the setup all these hosts and then manage them you also don't see in this diagram the all the jump boxes and all the teams infrastructure so this is just the competition network there's actually about double this when you take into account the you know the jump box is like every member of the team has a Windows and a Kali Linux instance running in the cloud as well yeah the vidi eyes are how they hit their

environment so it's all segmented in compartmentalize too so like they can't leak out of their environment and you know hack the school or whatever last year's regional was 1471 hosts and another thing I don't know if you can see it in this diagram is but we actually do it across providers so we'll do it in like multiple clouds and across different providers for resiliency so this is a lot of infra and it sucks pretty much this is a big problem and we've had a lot of developers and all the developers approach it from their own perspective so we needed a tool to kind of unify everybody and bring everybody together so we could write this stuff and the tool needs to be

simple it needs to be an easy language to write like this is the first language people are going to be writing so they can't be learning something new it needs to scale but it also needs to work across multiple providers so we couldn't just use something like a tree DevOps tool like terraform we had to kind of come up with our own solution and like I was saying we want resiliency so we don't wanna we don't want it to stick to one cloud provider we want to have multiple cloud providers in case one goes down so the solution LaForge yeah so you guys like this slide and this is a yeah I I ran into a nice

little slew of Geordi and data online little anime characters and so I just had to throw that in there yeah dan did a great job kind of covering why we why we needed to go out and build our own tool right this isn't in a talk about us hating on traditional DevOps tools like terraform and ansible they're really good tools right like the people have made terraform are brilliant people but they they made it for a use case that just wasn't ours we took a lot of inspiration from them we took a lot of inspiration from a lot of these other DevOps and operations companies you'd be surprised us being security people how little we

actually know about that space it's a whole industry that really operates just you know almost like ours does right there's DevOps conferences there's people that go around and talk at conferences about this stuff but they it's a science to them just like it is to us and we really don't don't take the time necessarily to go out and learn it so part of this on our side was going out and actually trying to understand the current space of DevOps and realize where we needed to be and what effort we needed to be we needed to put in on it so you know we're gonna build all this competition infrastructure well you saw how big it was what am I gonna just have

a hard drive with OVH on that that's not gonna work in today's age we need to have our infrastructure as code we need to have a reproducible we need to be able to say I'm gonna go build these five hosts that I'm gonna hand them to Dan and Dan's gonna press a button and he's gonna get the same five hosts that I just built and then I can add one thing ship them back exactly we can you know merge and redeploy a test down exactly and and that was something we saw as security people as we're not developers were not DevOps experts so for us you know a DevOps person might find terraform really easy we wouldn't

right they're just not tools were used to so for us to be able to have a group of people building this infrastructure we really had to just go out of our way to try and make it easy for them yeah most of our team is volunteers there doing ice on a weekend we you know they have a month to kind of help us put together this infrastructure so right and so let's kind of talk about the history of LaForge here for a bit so the very first version of it was this really embarrassingly large Ruby script that I wrote that just I I was writing terraform configs for CPT C and I got really sick of writing the same 25 lines

over and over and over again I was like oh I can just template this out with Ruby and and make it happen and then as we scaled the competition out all of a sudden to generate my terraform templates it was taking literally ten minutes so we went out and said okay let's let's how can we solve this problem it was about the same time for those of you I see a lot of CCDC people in the room I love you guys this is not a talk where I'm going to talk about how I hack you but we were getting into go for those reasons and we decided let's let's see maybe we can rewrite the

forge and go so we took a stab at it it went really well I would say it's kind of supported us in that that v2 version for the last two years it was really fast yeah we could iterate on we could ship a single binary anybody could use this single binary so it was nice it was like you know redeploy able it really was the biggest problem with with LaForge v2 honestly ten minutes thank you the biggest problem with v2 LaForge in my mind was that I just didn't understand my users enough I needed time to put something in front of them that they said this sucks can you fix this so after after another year of us just

watching how people use the v2 this this past spring we decided you know what let's throw that one out let's do it all over again some of the big problems we had and we'll kind of talk about it you know they don't really talk about on that it's fine are some of our problems were really around how we decided to make the structure right so when you use LaForge last year you had to clone a repo you had set up some dot files in your home directory you had to set some environment variables and you had go and you had to know where everything was in this folder structure and if you didn't know where something was you

might have messed something up and it would just break something for a lot of people and then you'd be like oh what have I done right yeah pull yeah and that uh that really was was a difficult experience for us it showed me that a lot of the problems with the forge v2 that we needed to solve in v3 were really around usability you know it's really fast to generating these configs it has a lot of extensible uses but it's just not people aren't finding it fun and easy to use so what we did this year is I went I sat down and I said okay what are the tools people really do like using that

they use everyday to collaborate with code with other people on right and the two tools I landed on were get how many people in here do a git command at some point in their week yeah awesome does everybody find that just easy yeah one of the things I really loved about git is you could be anywhere in a directory and it knows that you're still in the git repository even if you're five folders deep in that right that's the kind of just implicit awareness that I wanted this software to have the other feature of the other tool that I took a lot of inspiration from on this was docker I'm not a docker expert by any

means but as I've started to play with it the last couple of years I've noticed that they do this this sort of layered approach to how they build these containers up and I think that's a really interesting thing you don't you don't have to rebuild everything every time right compilers have been doing this for years caching objects and whatnot so figuring out a way where we can take the best of both of those worlds where the nature of it is just implicit you don't have to go and explicitly say things and the actual tool itself doesn't just have to constantly just do it state over you so we got rid of yanil just Gamal if

anybody's ever used UML in a distributed environment where somebody has Windows another person has Matt yeah I see Doug was over there just shaking his head right you get one person with with you know windows line endings and you're just you're done the whole thing done doesn't parse so we got ready Gamal and we're like oh that will solve our problems right it didn't because we still hadn't solved the underlying issue which we needed to make it easy for people to use so yeah so this is kind of the structure of the old yeah Mille here where you have all these folders and handle these yeah Mille files and again is this easier than writing a terraform

config absolutely do you need to learn terraform no you just need to read the comments in our yeah Mille and you can probably just you know really just figure it out but networks um host run some scripts exactly so so what we've done in v3 with LaForge and we're running out of time so I'm gonna be real fast with this we basically made LaForge contextually aware just like git is aware when you're somewhere deeper in a folder structure it knows that you're there LaForge knows implicitly where you are and depending on where you are knows how to load the dependency graph for that environment now what do I mean by that right we talked about those docker

layers this is exactly that concept here where every time you go to a different location within the folder structure LaForge really just knows where you are and can overlay the configs on top of the the dependence of the environment that you're in right so this is a little screenshot of LaForge now where I can just run a command LaForge dips in any of the directories inside of our infrared minutes thank you and it'll show me okay so I'm my main is is the build right there the build context and that's just inheriting down from those right and what's nice about this here's another command LaForge status shows you at any point in your repo just like git

status what your current context is here's kind of an example of why this is really powerful in this situation so we have our own little config language called LaForge the files or doubt LaForge files it looks a lot like hash II Corp config language we use some of their some of their parsers and ASCS to implement this so you have a host config which is just this block with an ID and some variables there right and that's in a global context over here in your in context which is yours this is shared this is you this is yours you decide you wanted to just make a change to host ext one so you make a

change well in new LaForge this just inherits this way and you just patch over the top of it you didn't have to rewrite his config you didn't have to go in and figure out what needed to be changed or what didn't need to be changed you just said that for my environment host ext one needs to just be slightly different in this particular way and that's what makes new LaForge so powerful is no longer you just having to drop hundreds of animal scripts in a folder and hope everything works effectively right there it shows you now in my environment that's what ext one equals it's non-destructive so anybody if Dan has in his environment if he's

extending ext one he doesn't see any of those changes and we can even say how they conflict and like what happens when that happens so another fragment that's inside of the new LaForge configuration language is a non conflict block you can actually describe if you are going to collide with an object how you want that to be handled so in this in the previous example it was just going to do a soft merge between the two in this configuration it's just not going to accept any of his configs and say no the only thing is valid is mine in that state as bare as it may be so why how does this happen how does this know what

it needs to be loading that's a hard thing to cover in ninety seconds but we'll give it a shot graph theory in 90 seconds great yeah so I took a lot of this stuff in school and I'll be honest I'm pretty sure I got c's and d's in it but tried my damnedest and I think I think I learned something here so the whole concept of like Dijkstra's algorithm and switching and cost of paths and stuff like that it's fascinating to me especially when you start looking at dependency graphs or software so when you have files that include other files at what point do those things get loaded in and that's exactly where the magic behind new

LaForge right so we have a imagine that like our in context configuration that's going to load B and C and C's gonna load D and E and D it's gonna also load F right so if you think like six degrees - seven - six degrees to Kevin Bacon right like there's a there's a stage that you can count incrementally on how far away from the center that you're getting with these and that's exactly what the Forge does is when you go and just at any time you're in your repo when you do a git status look forward status LaForge Debs it walks your entire dependency chain contextualized to the directory that you're in and starts at the first and

just like docker just starts building up the layers of the config one on top of the other applying the differences using the on conflict pragmas to make sure that everything is keeping sane right like this is actually what you end up with right layer one gets layered on by layer two layer three and that kind of stuff so your environment no longer has to be a bunch of duplicate code of Dan's I can actually just extend Dan's environment in particular ways and have everything that Dan just a little bit and now and I can just work on it at my leisure so there's a new preview of it it's live be gentle its alpha preview

v00 pre-beta whatever you want to call it looks you can help us it does work it's there I'll uh we want to step outside afterwards I'll give people a little demo of it it's it's really you know what Omonia laptop you can give the demo it's fine we're done yeah so you can go check it out the codes there it's in go if anybody years ago for please hop into it it's kind of fun and wacky TLDR of it we can build competitions super fast I think C PTC last year probably had more infrastructure than almost any collegiate competition I've seen in a long time we did it with a very small staff of people and it's not because

we're really good we just automated ourselves into looking so good yeah I know it's skills per team so stand-up one environment right you know 10 copies it's great for competitions that is the original thing with the forages I didn't want to have to write for 10 teams the same config ten times I just wanted something to build me ten configs and now the forge does that with a bunch of bells and whistles that make me not look like a failure in math class I guess and coming in on time whoo thank you thank you a big round of applause if you have any further questions since there's no one here you're welcome to stare and ask

questions but the streaming will be ending their question minutes five minutes great okay well cool any questions questions yes so the question was what are some of the things you can do in the career world example of something that you can deploy through this and how the deploying works yes I'd like a single host okay question great question I can zoom in on this okay so this was our dev environment for Gotham elections there are hosts that have dependencies here so for example DC o2 needs to dcpromo before backup so one joins to that domain there are DNS records in the get lab box because the get lab box deploys code into the production environment right we

don't build these competitions to just stand up some Metasploit able box we build real real integrated edification like a bug is gonna be catching logs from all these other services exactly and so I asked myself like I don't know how to use salt or any of this stuff like how do I deploy this stuff well I know how to write shell scripts everybody knows how to write shell scripts so we probably could have done a better job covering that if LaForge to actually do the configuration like yeah you do your your LaForge you know little markup statements of whatnot you there's a section there where you just define scripts and those scripts are just shell scripts or

powershell and it whatever you want you can define Python scripts and if there's a Python interpreter it'll try and push it in there so in the new version we can actually put dependencies on the scripts so you could say stand this up as infrastructure run these scripts but then wait on these ones until like those are done what the day you guys but at the end today configuring gitlab doesn't require me learning salt I just have to write a shell script that knows how to take a bomb to that no foreign builds get lap out which is great for volunteers because again everybody right she'll scripts they give it to you it just configs the thing one of the

powerful things about it that I saw last year was for the first time we were able to solicit volunteers to help contribute to this so costs actually was a prime example this came in three days before the competition said I want to help how can I help in any way we're sweating we're just like don't talk to me right now but now I hit me it's like he doesn't need to learn this Kyle go write meet shell scripts that like do cool and interesting security things that's a shell yeah and just make them small and give me like seven different ones so that I can just kind of scatter security things around this infrastructure and

then when you go back to score I do scoring at cVTC I just go through the shell scripts because I write there I'd be like oh yeah he put a vulnerability there there there there I got at the config and boom I have my rubric for how we're gonna score that year's competition it was really powerful we had probably a half dozen people last year at C PTC contribute scripts without having any knowledge of the environment how LaForge worked or anything it also lets you design the environment and the networks and the configs and then kind of push that downstream and get other people just stand up and config the boxes exactly yes so the question was are there any

pre-built sample environments or environments you can just get up and running with real quick right now so there would have been had we have just said yeah we'll just release v2 v2 code is out there but the v2 configs or not as I'm going through like I was up last night writing the readme on github I'm gonna be putting up examples and stuff like that on there part of my new development methodology with this is to actually just start writing an entire specification to test against in that spec itself will be an example environment there's also a sub command if you have if you want to know like how do i configure this you just

run the forage example script or host right you saw a little host block for example host will literally print a host block with all the parameters than you can choose which ones to fill and won't just copy that into a text file it's there so the one issue is you do need a provider so you need like AWS or like keys like that or you don't need it to play with the model okay so you can build all of those relations and dependencies you can't spin them up without actually spinning spinning up something in AWS right you can play with the language you can play with the configs you can play with inheritance and layering stuff on top of it we are

developing we've developed now we just have to develop the plugins itself LaForge is now all of like it's gonna compile the terraformer it's going to compile the vagrant that's all now extended out into a plug-in system so I'm actually writing the terraform one right now we've got an OL one I'm gonna do a native one that just spits out shell script it's it's it's a lot easier LaForge you wouldn't have been able to say yeah let me just write one for vagrant like that's not gonna happen but the new one totally will and if you don't like terraform or vagrant you want to use something else let us know what that is and if it's easy to do I'd love

to make a little little plugin for it thank you Alex and Dan for any other questions you've got side and have a wonderful night