BSides Oslo 2019 - Marit & Bendik - Technical challenges of creating a CTF

I hope everybody food and drink during the break and lots of going on so that's great are we all good at the back for video great so we have a talk now about not only doing a CTF but how to actually make one I'm sure we're gonna hear some mistakes that have been made and lessons learned so as someone who doesn't have a ton of CTF experience but very interested looking forward to this talk thank you very much moderate it an organ thank you thank you all right so I will no I will stand here right and I have to find a nice spot right so we are here representing tjak tjak is Norway's largest japanese-style

CTF I will talk more about that later on we are it didn't work then yeah we are or I am moderate I'm the leader of T Jack I also lead the hacking group at the University of Oslo called the u io CTF and what I don't do this volunteer stuff I'm work as a developer at suppress area and in CTF I like mobile reversing and I hope to get better at web exploit a web security and pole ponies bar exploitation panic yeah my name is Ben de Corrigan I work as a security analyst at mnemonic and integrity G hack I'm doing some infrastructure engineering and also creating some reverse engineering and forensics tasks all right so we are part

of a bigger crew we host a large competition so we are many people doing it this is our crew this year we are going to get even more people next there there's a lot of stuff to do this is the agenda for today the main part is to talk about the technical challenges right but it's very important that everybody in here knows what the CTF is and also what type of CTF t.j.s if we have time in the end I will be doing a demo a very simple web security demo so how many here knows what the CTF is yeah that's everyone how many here has actually participated in the CDF that's a lot of hands nice really cool

well I'm going to get into details here either way in what CTF sorry I didn't actually think that so many people would have participated in the CTF I have this nifty or nifty I'm not sure why I said that sweet and short definition of what CTF is and it is as following CTF is a hacking competition with a wide spectrum of security related tasks where the goal is to find a flag and if you haven't participated in the city a few would wonder what a flag is and it looks like this now we see that this is a little different than the one Kenny showed us but this is a normal type of flag that

we have in CDF's we had this static part over here in our competition this year it said TT 19 for every task we had then we have this nice curly brackets and in between the curly brackets we have a text that is unique for each task now I like to say that these flags represent access you're not supposed to have or information you're not supposed to see or just something that recognized that you actually solved or you did the hack in a proper way so this is the most important parts of what a CDF is and now I'm going to go a little further into details yes it's short for capture the flag it is a

team-based competition usually all city else you have teams for the CDF's and it contains challenges from the security domain there are two modes attacking defense and then the jeopardy style since TG hack is a jeopardy style competition that's what I'm going to talk further about I'll just grab some water alright so the name comes from this American TV show game called Jeffrey and the reason is that we have several categories and we have tasks within each category of different difficulties the higher the difficulty the higher the score these are the most common categories in CDF's we have the cryptography over here some reverse engineering binary exploitation which is called poll forensics and so web security and then in the end for all

the tasks that doesn't really fit in any of these categories are put in the miscellaneous category so there are two ways of giving scores to the tasks we are using fixed scoring which is not really the recommended way since team since the scoring is supposed to represent the difficulty of the task it's easier to use the dynamic scoring since dynamic scoring works that way that the more people who solves the task the lesser points it gives you so it's a nice way to visualize how hard the task really is right so to play the game you choose a task from the Jeopardy board you solve the task if you manage to I guess a lot of

people here has tried to sort some tasks that I didn't manage to when you get the flag you submitted and when you submit the flag you get points it's real addictive and to win this this is a dump from this year's competition so to win you have to either be the first team to solve all the tasks or you have to be the first the team at the top of the high score list at when the competition has ended if two teams has the same score the one that got the first wins that's the rules of CDF's now that was a short introduction about CDF's and i want to talk a little bit about what's so special with tjak just sum it

up I just told you that TG hack is a jeopardy style competition right and I also said that we have fixed scoring but that's kind of ironic because I said dynamic scoring is the recommended way so we're moving on to record two dynamic scoring next year and there was one more thing that I forgot so we just move on this is a picture from The Gathering it looks very nice right the thing is that it's an on-site competition at the gathering the gathering is the huge LAN party in the Viking ship in Hama next to being an on-site competition we also host the CTF globally so that anyone in the world can participate after competitions and then

we have the side open so that people still can learn from our tasks we have these two special categories which are called afk a noob afk short for away from keyboard and those are all the tasks that are on site at the gathering so you have to be in the biking ship and maybe do some SQL injection with barcode scanners or get a keyboard with colemak and you have to you cannot yeah whatever there's lots of very fun tasks to do in the ship the new category it's something I'm very proud of because this is where we we've noticed that a lot of beginners want to participate but the tasks are too hard and we we want to get as many people as

possible to try hacking try their best to to learn about security and hacking and so we make these tutorials so that people who cannot really solve the task and we the tutorial and then they should be able to solve the new tasks yeah so the tutorials I just mentioned we also have workshops at the gathering for on-site participants and when the competition has ended we had a walkthrough of the 15 most voted for tasks after the competition has ended we publish all source code or write-ups so that people can read the solutions for tasks and we also if we have time publish templates so that other people can make see can make CDF's as well and

this last point here I'm very proud of you heard can you talk about this platform City a platform called City FD that's one of the most common can we used CTF platforms but we make our own platform and it looks really good and yeah this year we have a little more than 1300 competitors among 709 active teams that was really cool it was a very nice milestone for us and there are two more things that I find very special with tjak and is that we have this focus on beginner but so we make all these guiding tasks for beginners but we also make very interesting and learning tasks for the hackers with more experience in addition

we have set this goal of becoming the de facto hacking platform in Norway which in it might actually seem like we're almost getting there or might even be there now still more people than I knew that seems to know about tjak and we see a lot of people using it throughout there yep that's it oh it's the first time I've been to this part so quickly then I can take it easy so this is the main part of this presentation the technical challenges that we've approached we can separate these challenges into four parts first of all in order to make a CTF we have to make tasks we make approximately 40 to 50 tasks each each year and then we need to

deploy these tasks right so we need some nice in front infrastructure and then some monitoring which panic panic we'll be talking about today infrastructure monitoring so getting an idea might be the first part of writing a task right we get a BS from lots of places we have lots of friends who know a lot of security we read a lot of articles there's tons of places right the next part is to really make the task and we have these folder structures that has to be followed this is one example of a task that has to be hosted on a server so this is a poem task called flip each task are has its root folder inside the

category folder so this would be like hone and then flip and we have the root folder of the task what we see in the top here is that we have a flag file this file only contains the flag and then we have the readme file this file contains the task description at the competitors will see they mined the amount of points the author of the task and these D files are very important to keep in the root folder because as of now we have not automated this part and we have to we have to put all the information into the CTF platform manually that's tons of work 50 tasks like that oh so we're going to automate this for

next year to make it easier now we also have this tossed the tunnel file I will talk about that later on and in the end we have the file that I guess most of us use most time on and it's writing the solutions for the tasks because we hope that people get to learn as much as possible from doing our hacking tasks so from the top here we have the server folder in this folder all the files that are needed to host the tasks on the server output no other files which is a really really important and I think benek will give an example later on so in the source folder the author can make

his or hers mess it's just to be able to generate the tasks and uploads folder contains all the files that the competitor will need to solve the task and that we put on the CTF platform oh I forgot one thing of course we use git and when we make a task we branch out of the master branch the master branch contains all the file all the tasks that are ready to be deployed and when you feel like your task is finished you can make a pull request and this is where our quality assurance comes into the picture in this picture the screenshot from github we see that we have a reverse engineering task which is a kernel model module and what's

going on here is that we have two reviewers one of them is the category chief the one that really knows reverse engineering and then we have the volunteer the volunteer is just someone who says I have time I can read the write-up I can try to solve the task and go through check is it political correct is it does the task description match with the actual task and can it be solved with right up the category chief has to make sure that all the files needed to host it on the server are present and that the task may be solved

so when all the everything is in place you can merge the task oh there's no picture but since we're such a bunch of nerves we don't just merge the tasks we do this rebasing and squash and merge all this kid stuff all right deployment now this is where the task to Tamil file comes gets into the picture the toasted Tamil file is just a file with key value pairs containing information such as what should the subdomain be called what port should run on stuff like that and then we have this and now I'm gonna quote the author of the script we have this shitty Python script that runs through our repository finds all of these tasks to

Tamil files and auto generates or generates some terraform and ansible files now their form is a tool that takes all of these auto-generated configuration files and builds the infrastructure right expert and then ansible it's yeah it builds the infrastructure so it builds all these VMs and then ansible hosts all the tasks on these VMs yeah I got it right nice now pendick will be taking over and talked a little bit about our yeah more about the infrastructure the infrastructure itself because this is one of the main problems is actually defending in a tactical infrastructure where we have people actively trying to screw with our servers basically so for our infrastructure overview as Mara said we are trying to implement everything as

infrastructures code returning actively to not do this tactic of hoping over the hack see we try to do security in-depth so for this we're using Google compute engine as our cloud provider and terraform to deploy these cloud liam's we were second scible to deploy the task itself that are darker containerized for our website which is basically another infrastructure is based on communities with helm as a deployment manager we also using traffic as a reverse proxy and subjects for our health monitoring and last we are using the elastic stack for logging so we are we are using cloud pretty actively to deploy our infrastructure and this is because it really helps us with the availability and scalability so I don't

need to drag my servers up to the Viking ship and also I can request more resources for VMs or more VMs or more IP addresses whatever I need previously and the only negative part mostly is the price because there's no limit on the price if you need 100 servers it's going to cost another negative point about using clouds for CTF management is the complexity and also you need to put some trust in the provider but we didn't have any problems with Google this year so I think we'll continue with that um so it's it's pretty pretty important to fight while all the things because we don't want people to pivot in our network so we're segregating what we can so as I

said the task infrastructure and website infrastructure are completely separated so if someone managed to take control of one of the tasks VMs they can't poke the website to change their score for example we're also kind of separate segregating the health monitoring and security monitoring and also we were firewall in all things no traffic should be allowed unless there's a firewall rule that says it the traffic is allowed and most of this firewalling is actually happening in the terraform script that are deploying this Google Google firewall tags basically this year we hope that we could limit all the egress traffic from the tasks containers but the two of the tasks actually required some external traffic so we didn't have full egress

control this year but we're hoping to get there next year to tighten everything down as much as possible for management everything everyth all we do with management is out of bounds we a jump post and next year we're looking into implementing more tasks based rate-limiting as we observe to some of the web tasks which where some of the competitors started using some web brute force errs and the web brute forces aren't actually the problem in itself because we implement the tasks in the way that they shouldn't be solvable by using automated scripts but they're creating a lot of noise and stealing resources from other competitors so we're going to tighten that down next year and for our

host earning this is probably the most important part of our infrastructure because we host some of the tasks which actually allows users or requires users to take control of the control of the tasks and actually execute code this is called RC remote code execution and we separate all the tasks that are that allows the that allows the users to execute their own code from the tasks that are not required to do that so every RCE task is hosted on its separate VM and all of the other tasks are hosted on multiple machines that are hosting multiple tasks so if if a user is able to as a pivot from a or not private pivot but escape the container from one

of the RC tasks they should be able to get flags from the other hosted tasks the containers are hardened somewhat they should be running as an unprivileged unprivileged user and with read-only files and no temporary temporarily files left with the only the task for exposed to the world and here is one of our failures for this year because one of the tasks we deployed actually had one of the solution scripts in the temp folder which someone found out really quickly and deleted it so yeah this is why we need to tighten down everything because yeah we have a lot of users that want to win also one of the cloud host hardening we are doing is

limiting the the privilege the VMS can have towards the Google infrastructure itself so the service account so even if you're able to escape the container you shouldn't be able to poke at or at our Google infrastructure itself however we still have some work to do because if someone are able to execute code they could use up a lot of resources and stop other people from actually solving the task pipe for pumping or yeah just using up all of the resources so we were looking at looking into using cgroups to tighten this down for next year as all of the tasks are also containerized we might also look into hosting all of this with kubernetes but i think i think we

will continue with our current set up for the next year

this is a picture from our website this year it's really nice and this is in the middle of the e CTF actually the website itself is running on sa s container on kubernetes & asmar said this is homemade which is why it why it looks really good and we are proud of this website this year we peaked at around 100 requests a second which is why we needed to scale up our back-end to eight containers and we could probably keep on scaling if we needed to this is actually the first year we opened up globally so our CTF was actually on CTF time which is why we received a lot more traffic than we are

used to so having a scalable infrastructure it's really important when you are receiving more traffic than you planned on however we didn't have a highway leti high well ability database this year so we need to take this down for five minutes the scale list scattered up this year so this was only the only downtime we had this year was the five minutes that the database need to be scaled we are also looking into implement implementing more caching and making the backend more efficient so we are able to handle more traffic on fewer back-end containers and for the monitoring itself this is really important when you have a thousand plus users poking around and trying of the

things they're learning from the internet and course running random scripts that they don't know what you're doing so I used subjects with dock picks which is docker ice topics to monitor the containers themselves we're also using Google Health sets on monitoring the cloud beams themselves and graphing this I used a farmer on the longing part we use the Google staff driver which is actually really nice and to get this enter around we M which we can use to further look into logs we used a event filter to take logs from the google stock driver into a message queue system called pub/sub and then we used pups of beat to indexes into elastic which ended up indexing around

one gig an hour which is a lot but yeah we're looking into implementing even more logging and monitoring next year us look look I love a lot of the things we implemented this year was actually done manually because I didn't have time to set up a better system we also use some cron jobs to restart some of the containers when we saw that users kept on deleting files from one of the failed tasks and this is another nice reason for actually using containers because they're pretty easy to just restart and everything works so never on the rest of it never underestimate the chrome script and also we were looking into next year using some kind of auto task solver

which done reads all of the solve scripts in the git repository and then tries to solve all the tasks and if it doesn't if it isn't solvable restart the task itself this is some of the statistics from this year CTF however some of the metrics are not entirely correct because yeah there's there were some weird aggregation going on but at the top you can see we used the Google health monitoring itself to check our website how quickly our website responded so we actually had alerting on alerting if the website didn't respond quickly enough so we could scale up the website so our end goal is actually having one button push to deploy all of the infrastructure with the monitoring

and security monitoring so some future future plans and lesson learned better time management don't do everything the following weeks before the competition and more automation this also helps with better tire management so we can focus on creating tasks rather than actually running our infrastructure we also are looking into open sourcing of some of the CTF platform and some of the infrastructure code that we use so other people can deploy similar CTFs so as a conclusion on some of the technical tasks technical challenges infrastructure as code is really nice you can reuse it and it's makes infrastructure engineering really easy and helps with deployment we actually ended up deploying all of the infrastructure the same day as the

competition started container is containerization of tasks also helps this as we are able to restart containers if anything goes wrong and also helps with the making sure that we don't need to install all the lips and stuff structure and coffee storage is important to make sure that you don't leave files in temp and in general make the tasks solvable and also never trust users especially in the CTF when they're trying all the new scripts so try to resource all of the all of the resources like network and CPU memory and stuff with C groups that's it for my part our infrastructure is still running in a somewhat limited state because we scale down to solves to

save some money but it's going to be up until January next year so we do have time for the demo now since everyone here has really has actually participated in the CTF or almost everyone then the task we were going to use is probably very easy but we'll go through it either way maybe we can do it together yeah let's do it together we have so many people here it probably would be too easy do it let's see here so this is the webpage TG hack that I know oh yeah and I made this test user and this is how it looks our web page when you get all the categories so the task I was going to go

through is this one let's see here what is it it's magic so the team for this the theme for this year was the with three world school of wizardry right so all the tasks had something to do with wizardry and here we see the tossed description and it says like hey did you remember to check your grades that it's magic so we can open up this link and check out the page so this is how it looks like yeah it's a reference I'm not really sure if it's okay but whatever it's yeah so the first part I'll just skip this one it's not an SQL injection we just do whatever a parent labs and some stuff here and we

log in and this is how it looks so it's just to get like a feel of that you have a user you're going to log in now do any one of you have a feeling of what it is I know that someone already do huh yeah I do ya know I hope that went well um so what would you do then

alright what you wanted to change it to alright 30 37 all right I won't do that at first you're totally right so basically the thing here is it's a it's a vulnerability which is in the OS of ten which is called insecure direct object reference I will go back here just to show you that oh wait it's my it's the time no all right it's the it's the time here at eyes right um so it says like are you stuck well check out this so I say googly insecure direct object reference and also have you ever heard about this big so when you check out insecure direct object reference you see that you're supposed to have you're supposed to have

IDs that you cannot increment right yeah should have a random generated IDs that's what I'm looking for so what people usually do it's like the first user or one of the first user in the system probably is the admin user right so whenever I get the toss like this I would probably just go to number one but it then we just get the same page it's different grades but then we'll do as that guy over there with blur I have a son and we get to fly so this is one of the easier toast that we have I just find it easier to go through here nice and then let's take the fly cop in

put it into the CTF platform down here and you solve the task nice so that's how it goes you are more than welcome to go into our page register a user and try to check out our tasks we have lots of different to toss in different categories so and will I spend excited will keep the page open for a while I think that's in we did go through it a little bit we actually we couldn't do another toast does anyone have any questions so far

okay have yes sort of considered doing monitoring of the participants and how they execute the tasks so that you can sort of I guess see what what what they've done to mess up whatever you have set up like do you did I that make sense no yeah it would be interesting but at the same time we need to ask the users for mation course yeah but it would be nice to like for learning purposes I guess to to follow that so we might look in to it because I guess running or running a task like this is almost like a capture the flag exercise in and of itself keeping everything up oh yeah yeah I would say something about

that as well we have actually talked about trying to filter out some input and check how many curse words we get I guess there's a lot anybody in the back don't start at the front I just quick question is this interface currently freely available or is it not open sourced yet it's not open sourced yet but might be next year yeah thank you when you guys are open sourcing you're gonna start accepting donations because what you're doing is contributing to the community so I guess we could also give back somehow monetary I'm not sure about donations but if you have any cool tasks you want to help out with we might be able to open up for that and said maybe for now

at least TG is paying for the infrastructure did someone at the back have a question as well what were they what were the running expenses during TG and what are the running expenses say this month I think it's 2000 2500 ish so it's not that bad for the rest of the year I'm not sure but we're removing the infrastructure actually away from the cloud to some of the TG or can do servers to save cash yes we don't need to scale as much now have you guys heard of or have you participated in something like boss of the sock it weights plunk do you kind of relate to because that's something that's from like the opposite side so this kind of

also things back to the question about about logging what people are doing and kind of investigating that have you thought about kind of collaborating with those guys not yet but that would be interesting so maybe but yeah we're also looking into logging more for our own infrastructure to keep our users safe this question is more related to that I see this is in English and you had it at then the the gathering which I presume is going to be all in a region and also considering the ages of the people who are at the gathering how was this received amongst and specifically people in school kids students well most of the people that go to the gathering are

there for gaming and they know English so it's and everyone can also come to the our spot the tjak area inside the ship to get help but we haven't really gotten gotten any responses like that yeah alright and NORs multi-part question so the next part of the question is this I know for a fact that sir schools inside of Oslo communi actress she those kind of areas are working more and more towards introducing cybersecurity inside the school itself as one of the themes that's one of the subjects and something is taken forward and Scott and sponsor from the community - M engineers also sponsored could this be toned down a bit because especially if I look at this change you've just

done it's a nice simple straight for challenge kids Kendra and those kids can do it and it gets them started definitely we do have this tutorial page here which is like anyone could go here so you can make your kids for three kids to do this no one but if they would like to we have like three tutorials here so that even though if they don't know how to approach a task if they they don't feel comfortable with googling which we are trying to we try to make people aware that you should google this if you're unsure then you could probably for instance do scripting in Python I'm I really like this one you can make your

kids do some potent exploitation with stack overflows so we have this text based and tutorials so that you can see for instance how the yeah how does that looks and it's explained so definitely we should we could use this or anyone could use it for kids as well at schools great any other yeah so have you already had any cases when somebody breaks in and messes up the infrastructure or something or can i I will say something first so last year TJ 18 we did have an incident in the back end where someone did do the script tag alert thing and I was just poking around looking at the users and then something I got this

alert right but I was so used to getting those alert that I don't really I didn't really recognize it as a bad thing at first and then I was like oh [ __ ] no this is actually bad thing but we patched it really quickly and then no harm done yeah if our us the other part of the question I think we're just depending on where they are how how well they gotten fit all we'd probably just redeploy some part of the infrastructure oh yeah yeah yeah we'd have to look into our logs to check out what happened I guess and then patch it and redeploy it

anybody else at the back again okay

or do you feel the need for any considerations for the participants security because if you have kids if you have people who are more experienced is there any way that somebody could actually attack or disappoint computers is that a consideration there is a possibility yes but we try to mitigate that as much as possible we haven't have any in problems with that yet but it might be more cases because we are opening up to the world so before we could go poke the guy who did it at the Viking ship that it's worse if they're in the US or something now all right thank you very much guys let's have a hand for moderate [Applause]