Shining Light into Container Cluster Visibility

everybody it's time to uh go for the next uh uh talk about before that I have more uh or another prize to give away so um yeah Mo prizes and this time I'm gonna enlist will to take out a ticket I think we need to get a bigger so okay [Music] wait what are we giving away this time oh yes good good question oh this question uh this time we are going to be giving away a one-year hack the box subscription because thank you for reminding me to actually say what we what we were giving away all right who has lucky number two three three eight two four six this guy this guy [Applause] get your email

um so real quick before our next presentation I have been asked to um make sure that if any of you ask a question please speak up a little bit we do have some people who are um checking in virtually and I want to hear your wonderful questions about horses and ducks and so on and so forth all right anyways uh our next presenter is nathanielquist he's a principal researcher at unit 42 in prismacloud his talk is titled shining a light into container cluster visibility [Applause] awesome hello it is so good to see people's faces it's awesome this is the my first conference in three years since before everything happened so um I'm glad you're all here

um some familiar faces very happy to see some of you others not so much and I'm kidding all right so um a little bit about container security and visibility on it so I just kind of want to just kind of pull the audience a little bit um who's using containers either Docker kubernetes right now awesome who thinks that we have awesome visibility into containers right now okay awesome next question who's used service meshes few hands all right cool um so uh I I left this relatively kind of high level there's a few things we can drill down into I'm always open for questions um and with that we'll just get started um my Beard's gotten smaller I'm trying

to be a little bit more professional these days I guess I don't know why I should just let it grow back and say who cares um but my name is Nathaniel Quist um sometimes I go by the letter Q I've actually just recently been promoted I'm now a manager of the cloud threat intelligence team at under Prisma Cloud um we are super dotted line to cross with unit 42 so like well I'm I work very heavily with unit 42 and I work with them I'm actually prismacloud so it's a bigger umbrella that does all that stuff um but unit 42 is a lot of fun really enjoy working there a lot of very very smart

people um so yeah a lot of fun um some things about me I like computers I like Cloud I like RC helicopters I really love RC helicopters they're a lot of fun um first time you go inverted and then you actually go up you're going down right to the ground and you break everything it's a lot of fun so so all right all right so uh all right a little bit of the agenda today we're going to be talking a little bit about the Dark Side of cloud we're going to talk a little about the challenges that we have with Cloud just kind of in general in the higher higher view we're going to specifically zero in on uh

service meshes uh and then we're going to talk about a couple other tools that we can actually use to get a little bit more insight into how containers are operating so the fun side let's talk about the dark side of the cloud I thought about doing kind of like a Pink Floyd sort of thing with this just because it's similar and all that but I was just like ah we'll just stay with our actual dark clouds so all right so uh first off where are we just in terms of cloud security all right um from an Intel perspective uh what we're seeing is we're seeing a lot of identity misuse so um IAM identity access management within

all of the clouds within hybrid clouds there's a lot of complexity that goes into these environments um so with that we have a lot of leaks as Greg alluded to uh in his very first uh keynote here um is it your very first keynote I think it's your very first one too right second one awesome third it's even better okay um fourth it's the seventh um all right so um identity is huge and it sold um readily it's easily compromisable um and so that's some of the reasons why we're going to be looking at some of this um some of these big attacks that we see um IBM um huge breach on clouds um you know hundreds uh it's like 120

million uh different credentials are stolen and readily available for cloud environments it's it's insane mother of all leaks within Forbes um the one that I really kind of want to zero in on the most right here is in um solar winds the bottom right hand corner right here solarwinds goes under the radar a lot actually it durability doesn't because it's very huge and very large but when it comes to Cloud attack you know specifically it was a solar a supply chain attack but it was really a cloud issued aspect of solarwinds so solarwinds Orion which is their Cloud offering for um you know hybrid Cloud environments hybrid on-prem Cloud environments um and it's how they operate that over a

larger structure and they were able to uh put a container an infected poison container into that supply chain make it part of the hot fix and that's how they gained access to not just um tens of thousands of organizations but 40 plus U.S government organizations as well including his U.S treasury um a number of different uh places on that so it's a very serious Tech and it's also apt-29 or nobelium um as as Microsoft is calling that um so very uh profound attack when it comes to Cloud attacks in general so some things that I want to look at these are I just pulled off of Docker Hub I actually pulled 50 of the very top

most downloaded containers of all time right and then I compared these things across Showdown and I crossed it Acro compared that across uh um exploit DB so here's The Showdown attacks as you can see here um uh Alpine it's like in a crazy amount of downloads so just count some decimals here there's one two that's uh 7.8 billion downloads of Alpine it is the most used OS within Cloud operations okay then Ubuntu and then obviously uh you know nginx python BusyBox postgres redis shows up and it's uh 3.8 billion so it's pretty busy pretty common um sort of thing um within Showdown these are how many exposed um you know instances or products across all these environments uh obviously

nginx tops the list by a lot um I used a a logarithmic scale on this one so um this is definitely in the 300 millions are that are are exposed um you know but you can see there's a lot of these uh particular uh instances that are exposed I was kind of interesting that memcache doesn't actually have any exposures maybe my Showdown search was off who knows but it's a it's pretty interesting um on this one so if we go over look at the exploits so these are how many are actually available to the cloud or to everybody if we look at exploits obviously WordPress is the biggest one has the most available exploits because

that's awesome um as was it how many exploits are popped like 10 10 every five seconds or so um you know but there's some interesting ones in there httpd obviously used uh pretty heavily uh my sequel which is pretty easy so if we go back and we look at MySQL MySQL sitting there over uh just over a million exposed MySQL databases on any regular time in a regular basis and yet it's like the third most exploitable Cloud container that we have out there right now so kind of gives you a little bit of idea on that I love this graph because it looked to me it looks very pretty so what I did is I took all the exposures and I

referenced and I cross referenced all of the different Cloud providers that are a part of this one and I'm just going to point uh to uh one of these particular things and I just kind of want you to see like which one that is right this one this one this one this one no which cloud provider is that digital oceans do we ever talk about digital ocean when it comes to x-fosed and exploitable databases on a regular basis attackers use it all the time sure I'll just set this up and we'll line we'll have payloads come from there we'll just use it as C2 infrastructure but how many people are actually using uh digitalocean as their their actual

cloud provider and the biggest one on there believe it or not is Maria DB how much data is held in mariahdb on all of the cloud infrastructures so there's some interesting layouts and Footprints these are just 20 of the top 50 um Cloud containers that most downloaded Cloud containers that are out there so I'm going to Pivot from that just that's the escape the scope that we're living in right now and I want to talk specific about Cloud threat actors Cloud threat actors is something that is very near and dear to my heart it's something that I want to Define specifically so we just created a cloud threat report from unit 42 myself and j-10 who is the other

uh researcher for this we looked specifically at who's attacking Cloud environments and if we can quantify them or index them in some way in order to do this we had to kind of Define what a cloud thread actor is it's not a normal threat actor normal threat actor from nist um you know 800 150 says that it's a an individual or group that is compromising uh you know infrastructure that's not their own so well that works for cloud sure to be a cloud threat actor we do came up with this definition an individual or group posing a threat to organizations through directed and sustained access to their Cloud platform resources services or embedded metadata you can be a threat actor and

tangentially compromised Cloud environments and cloud services doesn't mean you're necessarily targeting Cloud environments you just happen to spill over into Cloud environments so we wanted to look at what are the actors that are specifically zero Landing zeroing zero Wing in man that's a hard one zeroing in on cloud environments so this is what we came up with and we created the very first Cloud threader Cloud threat actor index now some of these there's only five on there right now and a lot of these you might realize that these people organizations are actual crypto miners at least that's where they started however there's something more interesting about how they operate if we want to look specifically at Team TNT which is the

very top one we're calling them the Adept and this particular asset X you know standpoint they're considered to be one of the most sophisticated Cloud threat actors specifically for their Cloud enumeration techniques and their credential scraping capabilities they yeah it's all bash scripts and it's all very boring and it's very open source and it's like oh come on and they're supposedly retired we'll see how that how long that goes on um but uh their ability to compromise multiple Cloud environments not just Amazon uh not also Google um very very impressive in how they're doing that Watchdog is considered the thief they're very technically Adept coders they do a lot of go language however they like to

steal things too so they use a lot of Team TNT access they'll use kinsing scripts as well just to kind of hide and mask their operations inside of those um can sing it's all about money they're also really good at Cloud uh credential scraping rock is an old time I've been around a long time like what they started out in like 2014 um as a as a ransomware called iron ransomware um and they subsequently just kind of moved into Cloud they're still not very good when it comes to Containers they're sort of okay um but uh um they're really they're pretty good at just mass quantities of compromises and then 8220 which is spin off of the rock

um they've returned they took a break for a while and they've come back and they've hit um pretty strong when it comes to um credential access and containers specifically so we kind of want to look at is we want to look at like the common tactics that these particular actors use across their environments um and we put these across miter attacks framework and we found that all of these particular techniques are common across every single one of these particular actor groups and lo and behold they're all identity focused every single one of them they're all looking for cloud accounts they're all looking uh for you know access tokens for sessionized application tokens if you look under the

credential access they're looking for specifically private Keys Cloud metadata apis they'll just go into that C2 node and just do a metadata poll and well they have Cloud credits it's pretty fun and then they'll look uh pretty heavily on Discovery as well but they all have access to escape a host and they all have access to a container and uh resource Discovery so this kind of led me to this kind of idea it's like we haven't spent a lot of time talking about the actual techniques that cloud actors actually use against Cloud against compromise containers so that's kind of the essence for this particular talk um all of the unit 42 atoms all these particular access groups are part of a

unit 42 atom um it's just kind of a common framework everybody's kind of familiar uh with six taxi formats and it's all Json encoded so it's easily decodable um build it right into your uh your particular security tools um and it's all free which is pretty awesome so please go download that um if you want links please let me know and we'll give you those those links so here's the challenge uh that we that I currently see within Cloud so here's the logic that I kind of want to go down there are a lot of containers used much more often in the industry uh you know we saw the top 20 list um there's only one of those containers

that being redis is the only container targeted uh application that we know of right now currently uh for cloud environments redis is the only one that all five of those particular actors use on a regular basis the other ones they used and even make the list was Oracle weblogic I think think PHP um you know struts patchy struts those are all very common those are all used pretty much by all five of those particular actors however there's you know 19 others that are more downloaded and more exploitable than those five that I just mentioned so there's a whole bunch more that's out there there's a lot more exposed containers in our industry than I think we realized no

one's really looking at digitalocean one because it's free and cheap and who cares um and because no corporations using it's all individual people right soda you know I know of a few organizations that actually do use digitalocean as part of their their actual production environment which I guess because it's cheap so um all right so um but my biggest thing is we are not seeing these Cloud threat actors specifically Target why aren't we seeing them Target WordPress you know are we just not seeing them we just don't have the visibility into their particular malware sets Etc so here's the the things that I think we're missing uh one is everything's fine don't worry about it we're all good

uh don't don't don't care just don't even want to look at it another one is that the actors are just better at hiding their tracks they're just really good at skipping across everything we're not seeing things organizations also have very large gaping holes in their security perimeter in their Cloud environments I'm not going to go into a lot of this why I can say that that identity and access management the identity the specific roles behind identity and access permissions that organizations across the board we looked at over 200 different organizations 99 of them 99 of IAM policies are overly permissive it's insane the number three most used IAM policy in all of the organizations right now is administrator access

within AWS in Azure the number two most used identity role right you know IAM role is owner that's access to everything in the cloud what happens if that key is stolen or compromised oh it's secure you know it's in our GitHub and GitHub is Totally Secure right sure interesting yeah all right so my argument is that we don't have enough visibility in two containers specifically and if we do have the visibility into those containers we are not doing a very good job at offloading that data and analyzing it in any way and why is that all right so let's go to the Rosetta Stone for cloud tools just real quick let me go over a couple

Cloud workload protection all right so it is the ability to see actual containers in their operatings in runtime right so you can actually see when a container stood up you can see the process is running you can see it's Network traffic you can see all that good endpoint detection stuff okay so let's say an attacker compromises the front end system let's say this is nginx because it's the most uh used one that's out there they're able to move laterally from any number of reasons either they go to um you know they they run do run C and they escape that and go up into the uh the host and they move over to whatever container they just move laterally

follow the traffic into that back-end database whatever it may be inside of that backend database not necessarily exposed to the public but most likely is as we saw with like mariahdb and stuff like that there's probably creds in there in some way or you got to the host and that host has creds available so we take that Dev key because most likely it's a developer engineer that actually just you know stood this up and they just have keys that are exposed in there just locked inside of that really nice you know dot AWS backslash you know configurations or credentials and they just take that key and then they can then if that IAM policy is compromised they can have

access to everything that that particular IAM role does if administrator access is that I am keep which it most likely is or owner is that I am role like it most likely is um they have access to all this stuff but that's kind of where it stops that's all a cloud workload protection can give you it's only an endpoint monitor it can only see the containers right you cannot follow that IIM credential into that larger Cloud platform okay that's the end of cloud workload protection so let's look at what can look at that bigger picture the cspn the cloud security posture management tool this is GRC or governance regulence governance regulation and compliance uh for cloud

environments it can see API connectivity you can see user logs in it can see hourly scannings of how uh your particular workloads are configured all of that good stuff you can also detect drift detection in a lot of aspects if something were to change in configuration from start to later on it'll detect those things which is great it also has the cool ability kind of mixes a little bit with uh Cloud identity Services where you can do IAM monitoring you can actually see users that are changing permissions you can see all of that sort of stock all in that one single dashboard so if you look at that look at that we can see uh AWS

IAM sensitive activities by user that's great where did it come from we might be able to see a container but that's great we don't know what started it we don't know if there was malware on it we don't know if the lateral moved to get that we don't know any of that stuff behind it that's the end of cspm It's Too Tall right it doesn't see the granular aspects so it can't see workload behaviors it can see IAM configurations and behaviors so we're going to look at a couple other tools these are not necessarily ready for my import uh this particular talk right now but um they're important to know CCS which is cloud code security infrastructure is

code scanning vulnerability fixes vulnerability scanning misconfiguration scanning real-time fixes obviously is a huge bonus you can scan infrastructure as code your terraform or cloud formation or whatever you have and it can it could fix it right away but it's like yeah go ahead and fix that awesome you know get rid of administrator access or this particular IAM role has access to every single thing in the cloud just that one specific thing least privilege um the cloud network security also very important being able to see that micro segmentation between cloud and your particular thing is very poignant to uh The Talk today we'll talk more about that uh and then the cloud identity security I am monitoring enforcement and

cloud and all its applications so these are all parts of this larger tool called cnap which I heard a funny thing uh so it actually stands for cloud native application protection platform but I had this a funny little thing where it was like as taking a nap in a comfy cloud and I was like that's funny because it's seen him anyway all right so um it combines all of those C tools together so your network your identity um your code um you know workload protections and uh cspm all together into one unified dashboard or UI which is very helpful and it can help us do things that are really cool like we saw this example

before with the cwp uh you have your front end you have that lateral movement over your database exploit the database get that credential oh now we log into that particular Cloud environment guess what our cspm now can see that so that's the power and the beauty of a of a c now being able to take multiple tools and see that chain across events I would say that yeah it's a buzz tricky term and I'm trying to leave all the vendory things out of it because I can certainly talk about it but this is important for all people especially if you're working in cloud in any way get a cnap make sure you can see visibility from beginning to

end as far left as you possibly can see in the dev process and in as far right as you can possibly see in runtime so now we're going to talk about service meshes um actually I want to stop just real quick any questions about any of that it's a lot of information we all make sense we're all happy yeah go ahead in the back go back a whole bunch in the beginning with some graphs why did you put star count as a drink on there star count The graft I was showing the most downloaded containers oh okay with the star count is oh okay so sure so all these uh were pulled specifically from GitHub okay uh star counts

um specific environments actors will will uh you know malicious actors will actually try to you know impersonate a particular environment so using a star count as a basis for pulling down your your containers making sure that their industry reputable I guess you could say uh you know having a really high star count means it's most likely to be actually legitimate just a metric that you can use um I like to see which ones are the most popular that way which one people actually use while Alpine is like the number one most downloaded one um Ubuntu is the most favored by people right so just just as an example any other questions on scene apps we're all good with cnaps

okay cool

lots of animations why I love my animations all right so um service meshes what is a service mesh a service mesh provides reliability security and observability through and organization through the organization monitoring and securing of each micro Services basically how do you organize things how can you segment that thing and then how can you Monitor and traffic how much is going to that one particular container they do this all of them do this through some sort of sidecar what a sidecar is is a proxy that sits alongside that particular container and is able to The Container runs essentially on uh localhost every communication into and out of that particular container runs on localhost the sidecar is its Network

Gateway across the environment that proxy connects together how about this this is better you can see Ingress traffic coming in here from the left it hits the pro it hits that proxy a right it goes into service a performs its function leaves service a goes over to proxy B for you know service B everything is monitored inside of that proxy that proxy allows us to do a lot of things that allows us to load balance traffic it allows us to do network monitoring on that particular set of traffic um I use the SD on this one on the bottom section so the control plane every service mesh has a control plane of some Port some sort uh this is just a

cods uh they have a pile of Citadel in the galley we don't need to go through those for this but that's just how they control that particular proxy and how traffic moves through that system there are other security mesh players there's console istio and Linker D these are the top three most used console is the most widely used across organizations and I say widely used the organization HG insights found that 5749 organizations actually use console out of all the organizations in the world we only have 5 000 that are using one particular service mesh istio has about 3591 91 organizations that are at least publicly you know stating they're using that particular mesh organization uh console was made by hashicorp

hashicorp also did like terraform we're probably a fair mirror with that with our uh our systems they also do vault which is a really good secret container for your containers as you're moving through um I personally like istio it was uh kind of co-founded by IBM Google and red hat they all kind of work together to kind of make this particular system it's super Tinker friendly you can just Tinker around with it and do a lot of really cool things uh may not be super robust when it comes to development administrative accesses um but it's it's it's like like a console and it's a little bit more power featured than Linker D Linker D is super

lightweight basically it's a proxy um that's what all it really allows you to do um but cool Cool Tools there's a few other ones that are out there as well um built into istio there's a number of monitoring programs um just a quick example uh kubernetes dashboard doesn't come with this Geo it actually comes with just kubernetes itself gives you the ability to see what your resources is CPU Ram utilization um all that fun stuff going down Prometheus is kind of a third party out of the box open source program it is basically the out of the box favorite for uh kubernetes monitoring we'll give some examples on that you can monitor lots of things as I'll show you rafana

is like a UI interface for uh for Prometheus um gives you pretty pretty Graphics as we'll see in a second uh Cali Kai I think it's Cali would you agree that's Kelly I say it's Kelly okay what chiali I like key alley okay we'll do that um and likes I like Jagger Just because Mick Jagger but I think it's Jaeger but that's okay Jaeger Jagermeister all right let's do that um all right so uh Cali's really good for container management it's actually I think an undersold sort of system as I'll show why in a second um it does a lot of good things for security people um and then uh Jaeger is distributed tracing basically being able to see what

one container how it's talking to another container for how long and how right so um we'll get into some of these more succinctly I'm not going to talk about uh kubernetes dashboard at all um so we'll focus mostly on Prometheus and the rest of them this is the basic structure for Prometheus um it's basically uh kind of like an Etsy Etsy SCD database stores everything from a container and then makes it uh queryable in some way some fashion you can pump everything over to grafana which is a palm ql visual dashboard query language for it you can do alerting through Prometheus if uh um you know CPUs got too high on a particular container or granulization

was over over utilized Etc so it does a lot of really good things um its dashboard is super basic really basic um basically uh you have this little thing you can say which specific individual metric you actually want to uh you know look into and then you execute that and it'll give you those results of that particular metric um there are hundreds of these I thought about listing every single one of these things that I can track and I was like that's too much I don't want to do that um all of these things are all searchable within uh grafana grafana can look at the exact same thing so I was zeroed in on uh API server

um uh watch events to Total on on the last screen these are the instances that it can see you can see that uh you know there's a specific internal IP Communications between which apps you're communicating with which apps this is all available within istio or whatever proxy you're using and it gives you this kind of insight into how communication or how containers are communicating with other containers in the environment cool little rafana we've all seek grafana so I don't need to go into that it just looks pretty it's what you put on when the sea level comes in and you're kind of like yeah I'm doing work see and uh and that but something's missing right I mean

this is really good insight and really good uh information for you know managing and administrating these particular systems um but that's really all we kind of have we don't have any endpoint connectivity we want to see what processes are happening we don't see who's connecting with any of these particular systems from outside or or whatever they're doing so why not um here's Jaeger um very similar I could zoom in on this essentially it gives us uh kind of the same information this is what one container how one container is communicating with another container uh this particular container this is how uh Kai alley is that how you said it I like Italy is speaking with Prometheus um

they had a one 1.2 millisecond communication span from which IP address they have their whole Header information we're getting close we're getting Header information we're getting some cookie information which was great but again it's just how containers are communicating with other containers if an attacker were to compromise a container and move into another container you could probably see that traffic in in Jager Maybe skin something's missing on these this is where uh key alley comes into it's a great tool for seeing how istio is working from a visual level this graph right here you can see each individual little graph this is just the the book info which is just a standard application that comes with istio just

gives you a web page and you can interact with the web page and see how data moves through all of these little cool things are they're all animations so you can actually see like little packets moving around and it's really kind of interesting but the cool thing that that is or like Kylie Cali lets you see is the um that little there's a little lock you can see it like right there and right there every single one of these communication Pathways between all of these containers is TLS encrypted there's micro segmentation between every single little container going through there which is great right from a security perspective you can't just leak data really you know it's not going in

in clear text it's all encrypted from container to container that's the beauty of a proxy that's the beauty of that sidecar technology be that istio or console or whatever you can you know encrypt all that and it's all customizable you can use default encryptions you can make your own decryptions you can use your own CA search you know you can do all the stuff that you want to use which is fantastic the cool thing that the the mesh allows you to do is able to see that traffic unencrypted because it's the it's the proxy that makes that happen it all that information comes out somehow and that's what we're trying to get to um so uh Kylie lets you do that and also

we're getting really really close to actually being able to see communication so what I did is I um who's from my own little uh Firefox browser went and queried my you know book info system and I'm now actually able to see my user agent string I can actually now see who's connecting with this particular system I can see the cookie information I can see uh Trace IDs I can see anything that that particular app allows me to communicate including IP addresses including domains if that particular system is configured to do that with a particular application I can now see all that information and it's all stored within um Kylie which is it's a monitoring tool

for administrators but we can use it as a security team to actually export that stuff which is pretty cool this is where this kind of comes in I I was like how do we actually make a a service mesh actual part of our security architecture and Industry this is a simple diagram you have uh what's that simple um but um you have Republican public internet here um it goes through your cloud service provider I don't care if that's AWS or G you know Google or whatever it goes through their internal gateways their internal load balancers before it gets to your system then it goes through your kubernetes system and then it goes into your Super Service

mesh platform and that will include its own internal egress Gateway its own you know Ingress Gateway it'll have destination rules so you can actually dictate how you want destination to travel inside of your service mesh if you only want if you're let's say load balance this is the most common uh reference for service meshes and how they're used in the industry for devops people if you want to load balance a particular new application a new version of the front end system or something like that you will say I want 70 of the traffic to go go to my old system and I want 30 to go to my new system just to see if I can crash my new code right and

just kind of load balance it that way you're not sending all traffic you're just kind of mirroring traffic off to see how much you can hit it with and break it how about we use that idea that Concepts to load balance traffic from our applications into security tools like cerakata or onion or Wireshark or elastic we save all information so that's what a if you can have an endpoint system have a proxy you can transmit all information via istio or console into whatever application you want all Network traffic so kind of a cool tap and it's built in and it's free and you don't do anything else pretty cool so here's istio load balancing first we have to make a Ser uh a quick

little virtual service you can see here in the red box on top essentially what we're doing is we're designating uh all HTTP traffic that is destined for whatever prod application host whatever version we're using or in this case we're using version one and all traffic so 100 of all of that traffic is going to be moved uh can't you know we're specifying where it's going to be moved to to the particular prod application and to a security tool so what essentially we're doing is we're saying okay istio take all traffic that's coming in destined for this particular endpoint and split it off into two things into that endpoint itself and then off into whatever tool our security tool

then we actually have to make the mirroring configuration again this is a service a virtual service and we're saying again all traffic from prod in V1 we're going to send that to security tool and we're mirroring 100 of the traffic that's as easy as it gets you try to do that in AWS you have to use port mirroring you can do one port and it's insanely expensive I don't know whoever has done AWS Port mirroring it is like ridiculously expensive to do that so you don't have to istio do it for you then you just feed that off into Splunk look at that we've got all of our Header information we've got IP addresses we've

got all that information now we have where it's coming from ports is coming over all that pretty amazing stuff um just pipe it into whatever centrally centralized information system you want I think that's pretty cool it's a way to really gain visibility into your Cloud uh into your containers but that's not the only two we have so I'm going to leave service smash here a little bit I'm going to look at other tools that we can use for something called ebpf anybody heard of uh ebpf cool few of you what it what it does is uh basically makes a Sandbox for Linux binaries and it launches them in its own OS kernel with that you can

see everything that is happening inside of that particular container inside of its own little uh kernel if you have his own little tiny little sandbox you get all the network information you get all the runtime information out a particular system you can see the kernels you can see all the library calls you can see I mean everything is happening on it and how trace traceability and serviceability here's a few different tools Falco probably the most common everybody knows about it um Aqua my competitor Greg's competitor they have really cool tools called Tracy there's another one um trafalgo I think is what it's called and it's it's made by psyllium pretty impressive um so yeah there's a couple different

ones a couple different applications again it can do tracing profiling monitoring it can run OS it can map fires basically it's a really cool Trace program for for containers or Linux environments you just happen to run inside of uh containers um this doesn't really work unfortunately for Windows systems but as our recent reports have found is that we're looking at like you know 86 percent of all of the cloud is running under Linux so Windows sorry Zach apologies but uh when you know go away no Linux is where it's at um all right so ebpf is really cool so the information that you can get out of this I know um I should have like

highlighted specific things on here but it's cool you get this nice little Json format of all this really cool information we're going to zero in on a couple of those things you can see here that uh look at this the origin is running from a container you actually put this on the host it can see everything on the host and it can see everything happening in every container underneath that host here's another example of that particular thing cool things you can see uh which node it's running on you can see that's origin on this one is running host you can see that that host is running PS the PS command on that particular host inside of the container you can see this

running busy box so that's pretty cool now we can actually see which containers are doing what uh in that environment you also get the nice cool hash values all that cool thing is here again run that into Splunk again you can use whatever tool you want to use um and uh you can see that everything the container is actually using and you can see everything the host is using underneath that which commands are wrenching those particular systems so uh the hard part is just installing the cbbf tool on your host and then making it report back to your particular interior I could talk more about that we're having a really cool Honeypot project that we're working on where we

go really deep into this and how to map all this out but some cool stuff so you have questions about how to make that work hit me up after this I'm gonna talk a little bit about commercial tools I'm sorry I'm gonna have to because there's some really cool things that are out there and I really want to highlight some of these things this is Palo um it's my company so it's just you know shout out to my uh provider of my paycheck um but uh um wildfire threat intelligence uh um feed is very very impressive unit 42 curates it um it's a very impressive uh tool really good threat feed um if you're using kind of a micro

segmentation technique like uh like a CN series firewall or something I'll get to in just a quick second um if the Wildfire your thread Intel feed feeds right into that that's the point that I really want to make whatever threat Intel feed you're using whether you're using proof coins or using whatever whatever containers or whatever domains are are being monitored by that you can pipe that directly into your prod and into that prod host okay so all your containers can start blocking those domains those malicious things right out of the gate pretty amazing big bonus if you're looking for just micro segmentation if you just want to go in line if you want to work with

istio or console or something of that nature you can have your uh your specific pod running whatever containers you want to be running on that running specific Services as it's going through the proxy have your envelope Envoy proxy send that to your particular um you know Cloud monitoring agent whatever that is in this case just for Simplicity using uh compute just because you can have compute start monitoring those domains those hashes all that stuff that's happening and then block those things allow them to go through redirect them do whatever you want as they're moving on to their other services across your environment pretty cool stuff so to recap containers are targets as we can see I

mean there there are very big targets there are big big actors that are specifically targeting these environments and I'm not even talking about the big actors right I'm not talking about um you know apt-29 or 28 Cloud Duke malware which is very impressive targeting a lot of uh Azure space using a lot of azure storage blobs and things of nature in order to host malware and go through that or specifically targeting Google creds um there's a there's a a new Chinese thing is no ap41 targeting specifically um Google passwords and credentials um so there's specific nation state actors that are doing very specifically targeted information not just the actors groups that we're looking at with Team

TNT and and and Watchdog which is kind of I mean let's be honest they really are just kind of like I'll just take as many credits as I want and you know I don't care who I'm targeting I just want information um and then they sell it or whatever they're doing with it but they're gaining access to that cloud that top level environment there are full spectrum monitoring capabilities inside of containers right now be that kubernetes dashboards Prometheus grafana Jager you can see performance capabilities and metrics you can be able to see Communications from container to container which could be helpful if they're lateral movement are compromised um Kylie is is really impressive and it's starting to give you the at the the

ability to look at those access logs within the kubernetes dashboard being able to export those out to your environment um but the key thing that I want to take away is that you need to send those logs from your containers to some sort of centralized environment that we can actually start monitoring on because we're not doing a super good job on that service meshes is a very solid option um you know the cool thing is it adds encryption it has a lot of security things as an organization it's not that heavy it sounds complicated the lift is actually pretty pretty simple if you know how to read yaml you can you can do it it's really it's really that hard

um but then please don't forget uh your other cool tools that are out there your ebpf tools your micro segmentation things don't forget your vendors having that cnap capability the ability to see things from that container endpoint all the way to that cloud platform from the very very tippity top will allow you to be able to see that migration of actors moving from container to cloud back down to whatever resource they're looking at changing Keys you know like you know there's thousands of different apis so that they could be accessing and if we're only looking at containers you know we're only going to see one aspect of it if we're only looking at the top level

aspect we're not going to see any of the initial compromise so there we go so that's it that's uh that's my uh my whole talk I think I'm doing well on time actually um so I think we have any questions uh available cloud cloud containers service meshes anything like that go ahead so uh in terms of like AWS API keys do you guys have any visibility into like where you think maybe those are being stolen from or or would you venture a gas unlike any Avenues where attackers are getting access to those um so uh just from the actor groups that we've we've been able to see and this is includes uh uh nobelium with solarwinds

is they compromised they compromised uh either first start with that uh GitHub sort of aspect you know take those angles take social media um your social Outlets um there's a lot of there's always Keys even though GitHub is really good about like scraping credentials really fast it usually takes them about an hour or so uh to start you know modifying those things um so they're always there initial comp like they're there somewhere I mean it's really easy shush uh shush kit is a really easy tool that you just basically just do an API token into GitHub and you can find interesting things um lots of Cool Tools in that aspect um compromised uh Cloud instances is

always a really big easy Avenue to get to um Team TNT is an example Within I mean it was just within a few days they had over 9 000 you know compromised Cloud endpoints and they do that specifically by scanning compromising gaining Cloud you know tools and they sent their tools off they sent their creds off to their C2 node you could log into the C2 node using their own tool and you can see how many and they had thousands thousands of cloud identities and accounts um so we're really at a point right now what's really basic that we're just storing our credentials in dot AWS backslash credentials doubt gcloud backslash credentials most of the clouds

in Cloud exposed clouds instances are just stored in those particular places if they're not there just reach out to metadata API pull down your credentials um it's it's really easy go if you want to try it go to your own ec2 instance go to um 169.254.169.254 backslash metadata backslash credentials look at that you got your credential um so it's pretty it's pretty it's pretty straightforward um so unfortunately um there's a lot of opportunistic actors right now and I kind of go through like uh the the business email compromise that Greg was talking about um really easy to compromise a lot of stuff um very simply it's not really that hard you can use you can just go ahead and go

to virus so we'll download somebody else's script and use it it works um it's pretty well just make your own wallet and then off you go but uh um crypto mining is it's on the decline ransomware is picking back up so um I'm just waiting for the actors to actually start using something else other than bash scripts actually start using more golang use more Health binaries to actually do their work which it's it's it's starting it's gonna happen any other questions what what would you like to see in terms of functionality so that you have more clear oversight of the communications that that's occurring like what what things do you think could make that easier or what

open source communities might be okay so um so repeating the question the the the question is how can we make it easier to communicate between cloud and security tools I think kind of potentially it's kind of your question um how do you make things easier um I'm not gonna like how it is complex and the complexity um makes people some some people very nervous about about doing things but cloud is 64 of our internet right now it's it's an amazing amount of of the internet that we actually use on a regular basis is cloud um we're already there and so uh being able to gain education visibility into this that's why I hope this talk is a

little bit helpful because things like service meshes are very helpful um in organizing compromising these you can only see that if you take all of those tools together all these service meshes together from One Source there's basically 10 000 organizations in the world that are using these tools and they're very helpful they really help security they help get the visibility into tools so um the education being aware of these things Cloud actors are compromising environments and they are making money off of it and they are selling your credentials for cloud environments and most credentials are administrator access which means your entire Cloud infrastructure is probably compromised um so it's that should be scary and I

want to be very clear if that should be very secure because it scares me um and I think that the cloud service providers are doing a really good job of I really do think that they are doing a very good job of trying to educate people and what that shared responsibility Matrix is what are you responsible for what are you what is the cloud environment responsible for essentially the cloud provider AWS gcp Azure is responsible for giving you the infrastructure full stop you are responsible for IAM you are responsible for how your applications are configured Zach up the compromised things you see like can you gauge what percentage of them are used directly by the actor who

compromised versus like the credentials are sold or maybe both you know where they're kind of double dipping um so uh it kind of goes into the uh see now visibility is a problem within Cloud environments on being able to know that um I will uh being able to see in the cloud environment and know if a credential was used by a specific person that was boughten by somebody else or not button that's terrible English bought by somebody else um I don't know if we can see that you as an organization would have to have a good record of where your identity access management policies or your keys or your access is coming from and then

notifying or flagging when something is amiss using some sort of anomaly detection that a cloud service provider uh or that a cloud service posture management cspm tool would help could could give you insight into um of like say team TNT Watchdog uh Rock we are able to see um their tools when they do compromise the cloud instance if they are then subsequently accessing um the larger Cloud environment team TNT most certainly is um rock we're not terribly sure yet Watchdog following suit with Team TNT probably is as well so um again visibility is a problem again we need to make this information more readily available

um but since it um if you're doing a service mesh and you're able to actually see all the data because of the TLs what are the ramifications for uh PCI or HIPAA compliance on the look access on the log for audits good question so the question being if you're using uh TLS for communication between containers uh how do you handle PCI for HIPAA or you know compliance for HIPAA or PCI um inner communication I guess it depends on how you uh you know you between you and your uh your insurance provider for your if it was compromised um you are providing doing doing diligence by providing encryption in into end um it is a matter of you recording that

communication or not um doing best practices to say if something was compromised how do you pull it down and actually know that it was compromised um again you have to be using a service mesh to begin with um that would probably be due diligence um on your part um but it would be it would it would satisfy PCI and HIPAA by having Inc uh communication and the audit requirements are access to the data which now require access to the logs right so the logs are stored kubernetes does store the logs in their analysis logs format which is a which is an SCD format um so it is there it is still it is stored and you'll be able to prove that it

happened and then it was encrypted or not encrypted kubernetes will hold back or you can pull it off to a third party we can maybe do one last question

cool cool awesome well thank you so much for your time appreciate it [Applause] so we are about to break for lunch um real quick um before we forget again um we we are donating a lot of proceeds from today's conference to Silicon flat irons if you're not familiar with them they're a really cool organization kind of based here actually if you're wondering why we're in a law school of all places um it's because uh we got got some help from them getting a great venue so um if you get any of the swag upstairs or if you see any of the various bits and pieces of silicon flat irons stuff that we spread about please consider

um donating through the uh PayPal QR codes that we've spread out all over the place um but yeah anyways there's lunch upstairs um but it's Nerfs that's it yeah it's Nerfs snarf sandwiches um anyways go nuts and we'll be back in an hour thanks