Chris Myers and Barrett Adams Building a Cloud Pentest Lab (and how we blew some up)

for uh coming out and sticking around I was looking back and I we've done like maybe eight presentations over the last year or so and like three or four of them then last on the last day of the conference so we appreciate having an audience to talk to so our presentation today is on building a cloud test lab and how we messed some of ours up along the way before we get started into things just some introductions my name is Chris I'm Bharath what we do currently right now is we provide simulated corporate networks for our clients to have a realistic safe place to practice penetration testing and doing that we've built a lot of cloud

and test labs all of our stuffs posted in AWS so we've had a lot of headaches along the way and we kind of wanted to do this presentation to share our experience with you there provide some tips and tricks to help you guys set your own up more easily and you know share our funny pitfall stories so we're going to be talking about is kind of the steps to build one of these that's using a cloud provider what are all the networking things that you have to set up beforehand launching your different lab systems setting up multiple apps and then you know why you should do things in the cloud and some of the disadvantages that has there and we kind

of reworked with this whole presentation since yesterday we had the bright idea that instead of just telling you now this is how you do it and what it'll look like we're gonna live build a lab during the presentation today so hopefully things go smoothly definitely a lot of switching between PowerPoint and doing demo stuff so keep your fingers crossed for us so first step before you do anything is to choose a cloud provider there's kind of a lot of the major ones that you've heard of out there and there's others too is the windows digitalization and that sort of thing in the end it's pretty much going to come down to personal preference if you're going to pick one of the major

cloud providers they all are pretty competitive with each other but there's some things that you should consider before narrowing down in the final choice first of all you got to figure out which operating systems you want to have in your lab some people just want a pure Linux environment or just one pure Windows most of the time you're going to want both with that directory and that sort of thing so that goes out the pure Linux providers like my node you gotta be careful since this is a penetration testing lab that you're following all the terms and conditions most providers have some language around what you're not allowed to do and ways that you can

do penetration testing in their environments but almost all of them kind of exclusively disallow things like DDoS attacks and attacks on the underlying infrastructure so you should expect to be able to do any of that you've got to figure out the cost which we'll talk about the cost of our example lab that we set up a little bit later they're all gonna be pretty competitive there as well and then some of them are different from Windows licensing but for the most part it's how to bring your own now Amazon and maturer will handle that stuff for you so we went with Amazon for all labs that we build just because we were kind of familiar with the platform

already going into it it's got a lot of different when it's flavors you can throw in there and a lot of other services that we use as well that we like about it so before setting anything up you need an account but it's super easy to do just need your basic contact info there and a valid credit card and phone and when you're setting up your account they will actually call you so it has to be a real phone number for you to get that account setup they also have a nice breach here doing your first year and if your live environment is pretty small you're basically going to pay nothing for it a lot of your services

are going to fall under the free tier type stuff it's actually have an account you can start setting up your network so these are all the things that you're going to need kind of to get your basic networking setup down so with Amazon sorry I forgot to mention we picked Amazon this lot is going to be in Amazon so all the examples and everything our AWS specific just won't really help you if you want to go do it at Google but hopefully ok they just do this so you're going to need VPC or a virtual private cloud to segment your systems into a local network there that's going to create a route table and an Internet

gateway to help some of the communication between the lab systems themselves and the internet and then you'll have different subnets security groups that will define what traffic can get in to and out of your systems and then you need some way to connect to the you see so you're gonna have to set up connection so the first step is to create the VCC so if Eric's gonna handle all the demo stuff and dive into that alright so this is already WS console here

[Music]

[Music] hey there's a mask on phone okay you'll see a really good service isn't here for what we're gonna do today we're just going to use ec2 and and these are you can make the whole short links up here so you won't see those so first we're going to create the BBC I will take easy as possible right here which is that little wizard button that gives you did you see with a single subnet and all I need to give it here is a name and we're going to all the IP addresses so it's going to make the BBC and all the associated resources underneath it so it will give you everything you need for have your basic lab environment

you'll have a subnet and Internet gateway to provide Internet access to the subnet and the route table division communication something here and something that is a rough table which we'll come back to once we add more stuff in accuracy we've got our local route and then route to the Internet through the internet gateway it made for us alright so first step done BBC is created you have your own little private local network that you can start to build out whoa so this is kind of just brief overview we'll just did we created a V PC needed to find the name cider block for the Bowlby PC and then the cider vlog for the subnet that's going

to get created you can create more subnets so it all the labs of we build we have you know five or six different subnets it helps kind of keep your lab more interesting and more realistic and you can do things like create ACLs between sub s which is not worth it for now certainly not in this demo because that gets pretty tough they don't let you play with security groups where it's going to make one of these when he does assist some launch in a second but essentially security groups are almost like viral rules and that they're gonna define your inbound and outbound traffic to your individual systems however they're not like ACLs and that if you define inbound traffic

allowed on port 80 to a system once that connection is made it's definitely going to allow the traffic outbound from there so you don't have to define it going both ways if you allow something and down and then make the connection inbound that trap is going to get back to you so now we have our BBC set up the first thing we need to do is to connect on so there's two two options to connect you can you go with the software VPN or Amazon has their own VPN connection that they'll provide you we usually give up the software VPN you can just use something like Open VPN there free which is why we like it it's relatively easy

to set up and that's going to be an actual lab system in the BTC that's handling you connecting and getting access to the other PPC resources if you go with an AWS VPN connection you have to pay for that it comes out of your civil Griffin every month and generally you need a static IP and to do a bunch of other configuration so we get a software VPN and that's what we're going to set up now yep so we're just going to launch a system in our V PC here and install and setup going to be depend on that so we're gonna hop back to ec2 in the console I can launch an instance so

if you've never been down this menu of watching distance before there's a lot of steps all these to be thought about answered so these are all the a eyes which is like system templates that Amazon uses so these are a bunch of quick ones that Amazon provides for you you can make custom ones there's a marketplace for vendors and upload once with free configured software and whatnot on them and charge money for those to watch out click and randomly a lot of times they'll be it'll tell you at first like zero dollars for the software just ec2 charges that all want to be a free trial so later it'll get you with the subscription and then

community is more or less anyone who can upload these images so you can find useful stuff in here but you also can't trust them to the same level as the other other sections in here anyways we're just gonna go with a brand new vanilla coupon to server micros definitely fine for the PM there's a million options we kind of stick in the first three or four there cover your general system for allowed these cases she was our VDC that we made only wanted some of that in there we do want to assign a public I'd be especially important here it's the DPI we need to be able to get that from here and the way we're doing internet access you need

to assign a public IP to a box for it to be able to reach out through the internet it doesn't mean that anything on the internet and touch it inherently unless we specify that but instead of creating an active way or something else for systems to talk out we're just doing them like these one other option we're going to make use of on this screen is user data here you can specify a script that will get rotten when this machine gets launched so instead of waiting for this to boot up login going through it would be ten configurations we're going to do it through this so this is a script I made it'll do with the OpenVPN install

and setup will link the gist Slayer in this presentation so you can use all this stuff let's hope you can can get real confusing the setup but we can take just a shortest path here so we download that script make it executable and run it and it takes two options which are the domain name and the DNS server so these are DHCP options that are going to get pushed down to our VPN clients when we connect to this so later when we connect our Khaled box to this lab it'll know from the network connection okay we're on the b-sides demo domain and the DNS server is this demo server is not that yet because we haven't watched that

system yet but so begun before that system so we make up an IBM ELISA later that's about it but we really need to do what's storage defaults time restoring opportunity and no tags necessary these are great for organizing things at you AWS resources it's not going to matter for this here we're going to make a new security so we're gonna allow SSH because we're having to download a VPN certificate this box is going to be the BPM self the default probe and be done is UDP traffic on 1194 and understo connector right here if you have a subject you know you connect brahmer just want to have it opened everywhere and then this one we forgotten a couple

times over the course the last two days you need to allow the lab traffic to hit your VPN box yeah we tried to reconfigure this VPN server and get the script working correctly because we couldn't get any traffic back from the live system but we just told Amazon to not let that happen so yeah we've automated all this and been several months since doing it manually so kind of reinventing the wheel on few steps anyway that's all we need there and we have to create a key pair I have to lead it them so that this would be like the first run-through that's going to give us an ssh traffic you so the key pairs of how you either SSH

into your Linux systems for all the dams on provided one anyways or how you get decrypt your windows local admin passwords to the RDP yeah this is how so I did that ProSource shows yesterday this is the keys that you guys were talking about when you were saying you had access to our boxes from SSH immediately he will bypass the logins because you already have the key pairs our store I presume it's the same thing but yeah for now that I'm okay anyway we're gonna copy that over to our attacker box here for many years

the right permissions on that for a cessation to let you do anything with it and then we can specify it in our connection actually we're not in SSH we're gonna SCP copy down yeah so that's good we read created and open VPN configuration files so that we can just use that to connect it so if we go look at our instance here you have a public IP address which we allows access to from just here oops

[Music]

I so that is nobody came to the duration about that'll connect to our content server

all right

mouth rolls downhill on this thing and then I'm at the end so the BGM creates a new network adapter Network device on here and you can see we have a 10.8 that's here at 6 address that's the default range that opens up things in you don't otherwise mess with the end thing it's important that those IP addresses apart within the submenu choice for your EPC because something like Amazon will get real confused about if it's talking to its own system error and needs to send that traffic to the video so we just went through all the different steps to launch a system it happened to be our VPN system and now we should have access to the environment

and be good we should be able to start you know getting C to channel and stuff like that that's what we thought for like sound like a couple days while trying to figure this out the first go-around so we had an issue where after we did all this we we had configured a Jenkins system that had no authentication we went on there we dropped a payload to get a C Tube channel and nothing happened we were like what's going on we don't know so we sent back a request to our IP now let me explain sorry this little diagram first the red boxes are big answer every which has a $10 dress and then the attacker

boxes the client connect there which has a 10.8 address so all your payloads and everything are going to have I be listening on the 10.8 address so that's what you're compromised Otis is going to reach out warm so how that happens is it tries to reach out to the route table and say where should you out this 10.8 traffic turns out we've never said anything there so it didn't know to fix it you got to go set a route for that traffic to your VPN so that it knows ok if there's something and that 10.8 range that's not in this BBC route it to this specific box and let it handle all that traffic so once you do

that you can get your normal c2 to that can and communicate just fine yeah let's go do that

so this is give the same name as I gave us we need to under networking change source destination check this is Amazon's checkup like is this network document for this box and since we're going to send packets to this box that Amazon doesn't think belong there we need to turn that off and then under our V PC specifically under the subnet and that subnets rough table we add a route to our media so we want all the 10.8 traffic to go once you remove that source destination check your system will show up in this drop-down so you don't see it there go

all right so now all of our systems that we launched in the lab should be able to actually talk back to clients that connect to the beginning because the route knows where to send that traffic to one more caution on the networking side of stuff and one of the disadvantages of doing things in the cloud is around responder and broadcast multicast traffic so if you recognize the screen shot this is from a tool called responder and what it does is it will listen on a local network for multicast and broadcast traffic and essentially allow an attacker to capture Network credentials and then you can go crack them and use them to gain access to the

domain so we try to do this in our labs and we were just hearing nothing it was crickets on our network and we didn't know why it turns out in ec2 it kind of takes over all of that latitude networking for you so there actually isn't any broadcast or multicast traffic happening so if you want to run a tool like responder or a similar man metal tool that's going to look for traffic like that on those lower levels of networking you're probably not going to see it because Amazon and I'm sure the other cloud providers are doing something kind of magical in there and to handle all of that so now we can move on to launching other

lab systems so for the Linux systems we just launched one they lost Ubuntu Amazon has a really good selection of other just available Linux flavors out there so they even have their own called Amazon Linux which if you're just trying to do something real simple like an Apache web server it's going to be super quick and easy to set that up you can do a bun to Rosendo us SUSE probably a hundred others you know there's a lot of Linux distributions and that kind of the point of a pen test lab is to have vulnerable systems there so there's a lot of horrible Linux distributions like Metasploit Abul or a damn vulnerable Linux and those you can get into ec2 as

well but they're not going to be available through the marketplace if Amazon doesn't want normal stuff out there someone would definitely complain about that all right put this system on my network and then it got owned and I don't know why so to do that you have to first download it yourself in your own virtual environment and then you can import that to Amazon and it'll be available as part of the my ami section so you can still do that for Windows systems the first thing we always make sure our labs have is Active Directory so anywhere you go everyone's going to use ad for all sorts of things in their environments authentication and access

control all that there's a couple different ways you can do it Amazon has a built in Active Directory service more or less that you can manage users in and manage authentication to your Windows boxes but that's not really what we're going for here so instead we're going to launch a system and create a new forest and make that system a domain controller and it will serve the service in our lab environment and also give us a nice realistic box to go back that's going to be similar to what you find on all of your actual corporate networks the licensing is taken care of and it WS they have to worry about that and you can take servers back to like

2003 so they have a pretty good selection there alright we're gonna go and launch that first domain controller and turn it into anime trailer yep so same launching process here we're going to choose a 2012 r2 server this one we're gonna go with medium you can definitely use small but for the purposes of waiting for things to happen doing it demo go a little faster but it is that same subnet we are going to need to give it internet access because it installs some windows features on the way to becoming NPCs don't you just be able to talk to home for that and again we're going to use this user data to specify a script to run on startup there

it'll look a little different for Windows we're doing PowerShell so it's got these powershell tags around it know how small that is

so we're we're installing this windows feature domain services importing the deployment module and the powershell and then making an Active Directory forest here which will make this machine primary domain controller role that it's also going to be the primary DNS server for our environment so in this network interface ii right over here we're going to specify that same IP address that we told our VPN server that the DNS server was going to be at

another useful tip bit about the user data is that those will run a system or as roof so if you need to do powerful stuff like turning this box into a domain controller you can do that from there that's a good way to make something happening without having to RDP into your box or anything since now we have a VPN and access into the network we're gonna make ourselves a new security group here just going to call it our local Platt Network and allow all traffic by just go inside

be good we use the key again here you'll see in a few minutes we can ask Amazon for the local admin password that it generates for that box and it uses the key to encrypt that so we have to provide the community birthday go soft we're gonna let that do all of its becoming domain dance to Windows systems in particular take a bit longer just spin up turns into a controller and we think that'll take a couple minutes and it's got to do a reboot so like five minutes or so here and we actually the OM Azam will not even let you get the password until like it says you have to wait at least four minutes or something

for this system to get it to life together nothing for you to have a password and be able to access it so in the meantime we have some stories of things that blew up and also a word on workstations so most environments are going to have actual user workstations in them so Windows 7 Windows 10 Amazon is not going to let you do that very easily so you can launch all the servers in the world and they do have a mechanism to do like a PDI set up it's called Amazon workspaces but you can't just launch those into your BTC and have them act as part of your environment so what we do usually is to turn a Windows

Server 2016 into a Windows 10 bucks or a 2008 server into a Windows 7 box to be clear you can run Windows 7 and 10 in Amazon but you're on the hook for dealing with the licensing which if anybody's ever done Microsoft licensing stuff it just gets terrible real fast yeah and even if you do like the developer licenses or anything like that that are free they'll expire after 90 days then you'll have to go back every 90 days and make this happen again so you can do a couple of things to Windows Server this is a screenshot of Server 2016 looks almost identical to Server 10 and in most respects it is kind of

identical to the Windows Store where does 10 sorry and you can also do stuff like limit the concurrent remote desktop connections and kill off all of the server specific executables like the server manager you'll notice isn't in that bottom bar there so they really do work and feel just like workstations so unless you're trying to do something really specific that's probably going to be enough for you some of our pain points on blab systems are around those key pairs so don't lose them if you lose them if not all is lost but it's super annoying to fix so what you have to do is shut off the instance that you lost the key pair for unmount the drive from it launch a

new instance create a new key period that you do have access that and the old draw out that you unmount it mount to the new system go in there place your SSH key or grab the admin hatch or whatever otherwise get access to that system I'm not that remounted back to the old system and turn it back on it's like a super long process and really annoying so just hang on to those key pairs and don't lose them yeah you can't read download those things from Amazon like when I generate it and downloaded it that's the only time you'll get it yeah yeah so a my passwords were keys they're generated kind of three ways so

Amazon will custom generate your Windows passwords for you so that just if you launch a new Windows Server from Amazon it's going to generate the password for you some other ami is out there on the marketplace will do some other custom generation or if you take if you take a server that's been launched it makes a random password and you create an ami out of that and then you relaunch that ami later it's going to have the same password that was a created that auto generation won't happen again so just be aware of kind of all the different ways that those passwords and access gets handled and make sure that you know before you launch something where am I getting the

password or the key to connected this with all right what so far domain controller is closed up so this is what getting though we just passed it looks like it's just a right click off of there you give Amazon back password

stick within calculator just have one connection that's where you always need quotes around the Amazon passwords because we just crazy characters in them and then the IP

yes

we can take a look at the nice stuff here so after this DC is fully up

so we're gonna go ahead and create a actual domain user here pop the user I logged on that was the local administrator for this which is now the administrator for the domain it's actually going on here so here's our domain a new guy

we like to check the password never expires once is that way to feed your lab for a month then come back you don't end up getting locked out with mix Bioware domain accounts we're not exactly worried about our purposely vulnerable and that's why being secure so we'll just attack back in make him into a nap and freezes a counter domain join the next system that we watch so he does need to have ways to do that so before we domain during that next system we'd have one DHCP configuration to make here so in our V PC you can specify DCPD options so currently it's setting this PC to internal domain name I'm using Amazon provided DNS servers so

this next system we want a domain joint which means it needs to know where that domain is that is joining which means we need to tell them that that DC is its DNS server so we're going to create a new one of these give it our domain name and our server funny story that days we tend to try and pick domain names for our lives that aren't real because it just gets confusing like you could make one called google.com and instead of domain but then you try and browse to it and if your DNS isn't fully happy was going to be the end and you end up on actual master just confuses all your stands and

opportunities for skimming things that are errors and things there's an RFC that says the dot test TLD should never be used so if you always like yeah yeah top levels pretty safe my my domain top test yeah so we make them up we were making a bank environment so every wanted to call it Eagle Bank and evil that back didn't resolve so we decided that was solid made a whole three domains to forests eagle that Bank and Golden Eagle that bank and then we go and set up a bunch of vulnerable applications and everything connect in there and errors and crumb can't load at any one of our websites turns out dot bang is on the HS TS

preload list in chrome so every dot Bank domain has HSTs enforced which mean do you know me visit an over SSL with the balance and none of our boxes have so we had to regain freedom and yeah no idea the preload lists or thing especially for an entire TLD anyway we just got to attach this do you see D Hawking said do you see so now we're going to launch another server for environment - there's two systems there we can show you how to domain join something as you're launching it so you don't have to bother with the look left and pass right at first and going in the same place another little PowerShell thing here this is usually we just

created example and this will domain join our server and also rename it to your server one and that on our same local security group

security groups get associated with your VPC so once you create one if you're going to launch something into that VPC again you can go back and pick the same security group but if you're in a different VPC you'll have to make it again so if you've lost one or something just swishing PVCs that's what happened to it this will take another couple of minutes to come up but it should be a little faster than creating a whole domain controller so now we're going to have two systems are in our environment we can get to like the purpose of pen test labs which is to have vulnerable applications and things actually go and attack and practice with right so it's

kind of a lot of resources out there for this already a few of our listed here my favorites buggy web app and what good web application is that are purposely vulnerable from Hamas you've got a dimple or web application as well and then you can also do things like Tomcat Jenkins J balls ColdFusion other applications that aren't necessarily vulnerable but maybe you can miss configure them to have default credentials so if you get an old version of Tomcat on there or a new version and maybe credentials default that's a common misconception so there's a great types of things to do as well something that people don't consider when building their live environment is to put in downloadable

applications I think it makes a lot really a lot more realistic and it's going to feel more like what we feel when we do a pen test on an internal network so having those types of not intentionally vulnerable applications can be pretty nice too and I wanted to say that just because they're not intentionally vulnerable it doesn't mean that the applications aren't vulnerable if you go to source borage or web gallery there's a lot of like really bad and open store ABB's that can be fun to learn because you know I look at it and you say this thing's definitely vulnerable to something but there's not any documentation out there on how to you know do it through steps one to ten on

buggy web app or anything like that so they can be really fun to help and then kind of an extension of that is to put in realistic data in your environment so after you have something you have to go find some target data in your corporate network whether that's credit cards or other PCI data or employee data so if you build your lab with that in mind and create those users and files and target data to go after it also makes it a more realistic experience and helps you kind of learn closer to what it's going to be like all right let's go check back in on our server here I think we're going to actually have

this lab built in the time doesn't have so that's good our example user

cool so that's a doing engine system we're gonna skip actually installing vulnerable software I just started being very time-consuming well we can take a look at kind of what what's he doing Active Directory

now we have these two systems that's like the most basic pen test lab you can go do we have our VPN we have a domain controller and one system joins the domain so if this is all you did you still have the minimum to get C 2 channels practice doing different Active Directory enumeration and attacks there and you know practice different lot of movement techniques and all that stuff so we'll try to get a CT channel here real quick just to show you that it does actually work and then G can do some things with it so this is PowerShell Empire if you're not familiar with it it's just a post exploitation command and control

framework it's free and open source so you can download it and install it without any licensing issues or anything like that and it's pretty good so we tend to do all of our demos and stuff with that so people can not rely on a commercial alright presentation so when you establish to a CT channel with Empire it's going to give me back what's called an agent and you can interact with that agent to do different things like gather information on the system

derivative show options we saw a feature there you have it you can also do stuff with ad and everything through your agent here I don't know if we quite have enough time to go into that too much yeah play around there you have some of the advantages the advantages of working with cloud labs that we found and kind of advocate them for it is they're really easy to back up your whole environment am sahn's going to manage all your snapshots and everything so if you create a lab snapshot it in a good State at the beginning when your pen testing stuff blowing it up you're going to be able to revert to that no problem

and everything should be kosher having it hosted in a platform like Amazon also means it's super available if they're like you know nine nine availability or whatever it is if people like Netflix and all those providers can rely on Amazon for their infrastructure your pen test lab would definitely be there too and it's almost always going to be better than your home network then the surprising one year is about cost so a lot of people have this misconception and there's fear that why would I put all my stuff up in the cloud I already have you know a server at home that I can use or whatever it's going to be so expensive well the lab we just built

which is the VPN and then the two systems if you were running it 24/7 will cost you fifty five bucks roughly a month but no one can happen for 24 hours a day seven days a week so if you're only using it a couple hours at a time and turning it off and on it costs you seven and a half cents per hour so you could run this thing for two thousand hours a year and it's still how to be relatable it's not all bruises though there's some disadvantages with the cloud so the backups are super easy but you have to be careful so this Brittany a forest with time travel if you snapshot a domain

controller and there's no other domain controllers your other systems go on living for a while you revert to an alteration of your DC for the opposite you get a newer version of a DC compare to the rest of the environment and throw it back in there all the trust between your domain controller and your other systems will be lost it's a time with just DC's and systems it's one of multiple DC's yes lost between them yeah so doing that can mess up your forest and you'll have to rebuild the whole thing we did that a few times it's not fun it's very available being on Amazon but it also makes you need an internet connection

anytime you want to use it so you can't take you know a couple VMs on your laptop and go back from somewhere without internet connection which is a slight disadvantage and then the only real one here that is pretty bad in my opinion is that it's hard to do any sort of hardware stuff so if you're trying to mess with ICS or raspberry pi is a lot of people like raspberry PI's you can't really easily integrate them into your lab environment without doing like a hybrid cloud home setup in which case you're going to go through all that trouble like just also put the rest of your lab at home also comment in that category routers switches your level

networking stuff all right so that's it for us we have a lot of environment there now all the gist all the scripts we use to kind of make that configuration for you easier up on github right now is just so they're in there and that's our contact info if you have any questions or questions no it's been alright and what usually with actual pen testing we'll have windows VM or just pen test from a host so forth we can connect using Windows there it's kind of a smoother user experience to just connect with Windows but it things like you have to remember to include the clipboard tag in your connection stringer you will have shared clipboard mini RDP session in

your host so there's some weirdness about it but it's fine a lot of people recorded roll with I never got to work yeah that's definitely the easy road all right well thanks a lot everyone

you