Popping the Penguin: An Introduction to the Principles of Linux Persistence

All right, whenever you're ready. All right, guys. How's it going? Uh, my name is Mark Kum. I'm a former Linux engineer with a managed application hosting company in Southville, Michigan. Currently working as a security consultant with Bio Point. So, is everyone having a decent time already? Yeah. All right. So, today's talk, we're uh called Popping the Penguin. Um, and we're going to do a basic introduction to um Linux persistence. Now why why this talk in particular? Um basically it's to provide direction you know and when I gave this talk in Chicago afterwards I had a question uh posed to me why didn't you talk about rootkits because we're not going to talk about rootkits at all in this um we're

going to talk simply about bash commands and commands available on the Linux system already and that's because if we truly want to be effective as information security personnel we really need to understand the concepts that we're talking about you know we need to be able to understand these tools that we are using how they work and why they work before we go ahead and actually use them in the first place. So by the end of this talk we should have a fully functioning u basic remote shell that we can use in order to maintain our persistence onto the system. So in this talk we're talk we're going to have two actors. Um we're going to

talk about the dichotomy between the CIS admin and the hacker. Now the hacker they're generally cool people. Their motivations depend mostly on their affiliation. So whether or not you know we have our you know famous AP1 or you know our script kitty at home their motivations are going to differ and of course they always wear a scheme mask. So our system admins are a little easier to understand. They have three basic motivations. The first is patching the second is heartening and the last is uptime. These three concepts together are the concepts that every CIS admin focuses on. They want to make sure the system is secure. They want to make sure the system stays up

and they want to make sure that nobody can get into it. Okay, so we're a hacker right now. We've gotten into the system. We've used our favorite kernel exploit. We're sitting at a root shell. So why don't we just use the same exploit to maintain our access? Well, like we said before, a CIS admin's main job is patching, hardening, and uptime. And in the course of performing those three duties, you will lose that access eventually. May take a week, may take a month, may take a year, but you will lose that access. So, how do we keep that access with minimal risk of being um detected? We use a back door. They're easier to hide and it's easier to evade some of

the logging um which we'll talk about right now. So in Linux all the logs for the most part are stored under a directory called /bar/log and in there you're going to see a lot of logs access logs etc. Two of the main ones that we want to look at though are bar log messages and bar log secure varlog messages is the uh log that had handles all the system level messages. So anything from the kernel all the way up to bash logging if it's enabled. And I can guarantee you on a properly configured system, bash logging will be enabled. So you really want to watch out for that. And then we have varlog secure. Varlog secure is where it's

going to handle all the user information. So any authentication. So let's say you SSH in, you get and you steal someone's credentials through social engineering that will be logged into varlog secure, which then they can then ask that person, hey, do you log in at this time? Or say no. And then they'll know that they've been compromised. So we also want to watch out for that. anytime you switch a user that will be in varog secure. So let's say you do su dash um and then run that and then somehow find that password that will be logged. So we want to watch out for that when we delete these logs and when we clean them out we want to make sure that

we're not deleting you know four hours worth of data. during an incident response at my uh when I was a Linux engineer, we noticed that a box had been compromised because it was missing four hours worth of logs and we noticed that uh one application checked in every 10 minutes and wrote to this log. So using that we were able to provide compelling evidence to get an a formal incident response that someone had actually compromised the system. So when we're going through and we open up this log in our text editor, we want to make sure that we're only deleting certain lines that are associated with that um with our events and with us. So we want to

look at the trends going on so that way we don't leave any traces that we were there. Now in order to combat that as CIS admins what we can do is we can use something called log shipping. What this is there's two main programs in Linux. One called S tunnel and one called our SIS log. They both provide somewhat of the same function. But what it does is it sends these logs to a centralized server in order so that if in that way if we do delete them, if there is a compromise, we can go through and we can say, hey, you know what happened in this time? We know that there are logs missing. What

is missing? And then that will show compelling evidence again that there was um an intrusion and then again your persistence will be lost. Now one way to leverage that to leverage that centralized logging is to use something called event correlation um and oh event correlation um using something called the SIM uh SIM which stands for security incident and event model. I don't know if any of you guys caught egg drop X speaking uh yesterday but he was talking about somewhat of the same thing except his had a blender which was way more cool. But what you can do with an SIM um and this is something that if you're intruding you really want to watch out

for or if you're being if you're a CIS admin you really want to um look into getting is that you can do something called event correlation. So let's say that you have um because v in varlog security it'll record every failed login attempt as well. So let's say we want to correlate 500 failed login attempts and with one successful login for that same user. That will generate an alert and say hey you might want to take a look at this. Somebody just tried failed to log in 500 times in five minutes and then suddenly got in. Somebody might have brute forced that password. So what can we do? What we as hackers what we can do is we can shut off that

log shipping. We can shut off our SIS log. We can shut off SIS log NG stop that service and prevent them from knowing that we're there. However, what we also want to look at is spoofing those logs because what you can do with an SIM as an admin is you can actually set an event to be generated. Say I haven't received a log in 1015 minutes. Well, that's pretty suspicious because I receive one every 30 seconds from this box. So, I want to generate an event and I want to generate an alert saying, "Hey, somebody might be on this box having stopped it." Or if not, I want to go in and actually restart that

service because I mean logging is pretty damn important. So, let's assume that we don't have any um let's say assume our admin doesn't notice. Let's assume that we've successfully mitigated the risk of being detected through logs. So now we want to um issue our first step in uh maintaining access which is going to be the rogue user account. Now in Linux if you don't for those of you who don't know um all the information for an account uh all the account info, passwords uh usernames etc are stored in two places etsy shadow and etsy password. Etsy shadow is going to contain all the password, the hash password with a link back to the Etsy password file, which contains the

username, UID, G ID, and a bunch of other stuff. So, how do we create it? Well, we're going to insert a line into Etsy password that looks something like this. This is an Etsy password entry. Now, to be honest, this meant absolutely nothing to me, but it's actually quite organized. So you look right here, M kicked up, that's going to be your username, followed by the UID and the G ID, both being zero. Now a UID and G ID of zero is going to be the root account, which is the administrator, or it's more analogous to the system level user in Windows. This part right here is going to be a comment. So what is that? What does this

user do? Is this the systems administrator? Is this the MySQL database user? Is this the uh Postfix user? That's going to be the home directory. So you see, we got to make that route. And then we have the user shell mash which is born again shell. So we insert this into the middle of Etsy password with an appropriate comment. You know so that way if someone's doing a cursory investigation of an Etsy password file for any particular reason they can look and say well hey this looks legitimate. This looks like it belongs here. We insert this into Etsy password and then we're going to run pw-rash and what's that what that is going to do

is that's going to generate us a Etsy shadow entry. So, it's going to assign us a random password into the Etsy shadow file. And then what you can do is you can also open up Etsy Shadow and you can take that line at the bottom that it's going to generate and move it up if you want to be a little more secure and have a little more guarantee that you're going to uh remain persistent on the system. And then all you have left to do is run PASSWD followed by your username and change the password. And now you have a root account. Great. So what does that do? Well, you see this zero again going back to the zero and the root

account. that zero links to the user root. So when we su dash into mka that's going to automatically toss us into the root user because the way Linux handles usernames and the way Linux handles user accounts is it handles it by UID. So that UID of zero is going to link to root and so we don't even have to have the username to root once we get in. We won't need to have that um barlog secure issue. We can just get right in. So, how do we mitigate? Well, there's a few there's about three different ways to do it. Um, three main ways. Um, how many of you in here are familiar with Tripwire? Well,

cool. A good number of you. So, um, there's two different versions of Tripwire. There's the, um, freeware community edition that was originally created and then there's the enterprise version. If you're trying to save money, as most CIS admins don't, you know, usually have a large budget, um, you're probably not going to be able to get the enterprise version unless you're, pardon me, uh, well, you know, multi-million, multi-billion dollar company. The problem with the freeware version is that it only checks in once a day. So, it'll generate a hash value for each file that you want to look at, but it only check in once a day. So, if we do have a SIM, if we do want to correlate

events with the time, it's not really good for our purposes. So, then we can move on to something um configuration management like Puppet. Any of you guys use Puppet in your job at all? Puppet Enterprise. Okay. So, Puppet is a configuration management tool. You have a Puppet server and then that will pull down a copy of the file that it should be. So, let's say that you that a hacker introduces a line that says um like the one before, it will completely erase that and replace it with the last known conversion of that file. Problem is is that that doesn't really generate an event. It just replaces it silently and moves on. So, our last option is something um like

an intr uh ids system for host based uh hostbased ids system and what we can uh what we can do with that is that we can detect whenever some of these files have changed so for example a um I know wolf light here he's he's actually the one who turned me on to in the first place very cool program and what we can do is we can send those logs to an SIM and we can go through and we can look at it and we can say hey this is correlated with you know known hacker activity somebody really needs to take a look at this right So a note on that the comment that you

made a couple points you talked about the fact that you could erase the logs and now changing the files the advantage with uh OC was the fact that the logs also get sent to a centralized server so the fact that you try and erase the logs or or change the logs on the local machine doesn't matter there's a master copy on the server exactly and again you can watch the file system for any changes the files that you want to monitor so creating a new user account while it might happen gets notified pretty darn quickly Exactly. By the way, um I know in 24 hours it gets noticed within like 30 seconds. Right. Right. Exactly. And that's the

advantage with asset by I know uh Wolf Life from Detroit and I forgot to mention at the beginning of the talk. If you guys have any comments or questions, feel free to interrupt me um at any time. Um you know, I'd like to think of this more as a discussion than lecture because it's lectures aren't any fun. I hated them in college. But he's absolutely right. You know, if you can cut that time of detection down from 24 hours to 30 seconds as a admin, you'll have a much better time. But as a hacker, you also want to look out for that. So let's assume that that mitigation never happened. Let's assume that like myself, I didn't know about OC when I

was working as a CIS admin. Let's assume that you just have a regular system. So now we have our user account and our logs taken care of. How do we make sure that we can get in to use that user? Let's say that a session is available. Let's say that there's a whole host of other reasons. It's behind a firewall. it's behind a um it's behind a load balancer. We can use something called netcat. Now netcat is a fantastic tool. It's describes itself as the networking Swiss army thing. And it can do everything from opening listening ports, connecting to open ports, scanning, banner grabbing, tunneling, and a whole other host of just awesome features. We're going to pay attention to the

first two though. Connecting to open ports and opening listening ports. So there are two main ways to open up a uh persistent connection a forward and a reverse. The first way for the first way with forward we would do um two things. One we would run this on the uh pop box. Now there are two flags in there l and dashb. You notice the dashb is in parentheses because it's optional. The dashl says I'm going to open a listing port. So fairly simple. And the dashp just adds verbosity. So if I want to actually see what's going on, the 4242 is going to be the port we open. On some versions of netcat, you're going to need

to do a -p, but you can just run mannc or just nc-h. And that'll tell you uh the syntax for your version of netcat. Then to connect back, we just do nc followed by the popped IP and the port. And we now have our tunnel open. That's great. So what happens if it's behind a firewall? Let's say port 4242 or any other port is blocked. We can't just stop a service and use the port that's open because that's just going to alert the SIS admin right away. So what we want to do is we want to do something called a reverse connection. So the host the pop box is going to connect back to us.

So we see we're going to run this on our machine on the hacker's machine. We're going to run netcat and then the box is going to run um netcat back to you and open that tunnel from its end. It's going to be the initializer. So you can see the flow is almost exactly the same except for which one uses which. So forward connection, you're going to use a high level port, you know, 50,000, 10,000, something like that to connect to port 4242 on this side. But if there's a firewall, it most places don't do egress filtering. So what we can use, we can use a high level port on this side and then connect to port 4242 on

this side where we control what firewall and what comes through. So what's the point of this? Well, this will bypass reverse connections, bypass, as we said, inbound firewall or I'm sorry, egress filtering. Um, does not bypass egress filtering, but it does bypass inbound firewall rules. Um, it'll help with load balanced environments. So, let's say you have 10 hosts behind um one single FI and you don't know which host you're in. Well, if that host is connecting back out, then you have persistent access to that host because if you hit that one IP and it goes to 10 private ones, there's no guarantee that it's going to be load balanced to that same host again. So what's the point? Oh, right. So how

do we get this host to connect back to us? Because I mean, if we're on the host running NC to connect back to us, there's really no point in actually doing um this in the first place. So the first method that we have are dams. What you would run would be nc-l4242 um then an amperand for example, which would allow you to have run as a damon in the background. not doesn't work too well. Dammons die, processes die, boxes get rebooted. Not really great for our purposes in terms of persistence. So, we have a better method, Cronab. Um, how many of you are familiar with Kronab and Linux? Awesome. I know you are. So, there's two Chronabs in the system.

Um there's Etsy cron tab which is the systemwide cron tab and then there's varspool chron cron tabs and then followed by a username and that's going to be the user specific ones. Now I'm assuming most of you who use uh chron tab on Linux use cron tab-e to edit and chronab to list. Well that's listing this one right here. Etsy cron tab you actually have to go looking through. you have to go in and specifically look at that cron tab in order to see if anything's been added because chronabl will just show you your user chron tab. So we want to use this first one up here in order to add this um entry. So for those of you who aren't familiar

with chron tab, we have a handy little chart here for the syntax prompt. Pretty simple. Minute, hour, day of the month, month and then day of the week followed by the command. So what this will do is this will run at 0200 on the 12th of every month provided it is a Saturday or a Sunday and it'll reboot. Absolutely useless command, absolutely useless entry, but it just demonstrates the um point in the syntax of how this uh actually works. So what's the point? We now have our connection. We can now send text back and forth. What do we do? Well, for starters, we can demoralize admins. Admins are very susceptible to trolling and we can make them cry.

Hopefully if they cry enough they won't actually be able to do their job well or and cry sorry crying admins are actually you know not necessarily what we want to do. So there's three things that we can do. We can do egress filtering total egress lockdown and configuration control on the cron uh tab to prevent this. or now let's assume that there is no egress filtering. Let's assume that there is no egress lockdown and let's assume there's no uh configuration control because let's be honest on most boxes they want to do it for a budget. You want your web server to run on a budget because it's not going to be the bulk of your business. It's depending on

what you do. So what we can do is we can use something called IO redirection to make this text flow actually useful for our purposes. Now, we're going to take a little break here from the code and we're going to go over how this actually works in Linux. Because with Bash, the born again shell, which is what we use to interact with the system in Linux, uses flow programming primarily versus object-oriented programming. How many object-oriented programmers do we have in here? Quite a bit. Now, how many flow programmers do we have? Three, four, five. Significantly less. So, I mean, flow programming isn't that widely used anymore, but it's very cool when we combine the two together. So, in

the end of this, what we're going to do is we're going to combine flow and object-oriented programming together to make a really cool shell. So, this is the first of our object-oriented or I'm sorry, flow-based um symbols that we use in order to change how the programming works. What this is going to do is it's going to take standard out on the left over here and then it's going to redirect it into a file over here. Pretty simple. This is going to take standard error on the left and redirect it to a file on the right. So two is going to be your designation for standard error. This is going to take both standard error and standard out and send it to a

file on the right. Again, pretty simple stuff. This is actually used quite a bit um with programming because um if you're doing some things in uh for administration test, you really don't care about the output. So you just send it into dev null and just absolutely ignore it. Now, this is probably one of the most used um flow flow programming based symbols that Linux has to offer. Um I know those of you who do flow programming will probably recognize this as the pipe. And what this will do is it'll take standard out on the left and pump it into standard in on the right. So, what that allows you to do is take the output of one command and use it as

the input for another one. So, and I apologize to anyone who actually uses Linux. This is going to be a bad example, but it'll demonstrate the point. Let's say you want to list every file in a directory under a specific directory. So you can do ls and then pipe that into ls. Really bad example because you could use find a whole lot of things you can do. You can use a flag in ls but it demonstrates the point of um using this pipe this redirect standard uh input. So it'll use any file over here and use it as a standard input for a file on the left.

This is what we use to identify our files. So, exec opens a file, the five right here, that's going to assign a file identifier five so that we can use a little couple more advanced IO redirection techniques on this file somewhat like this. So, this will redirect standard um output into the file with an identifier five. Then the last thing that we um use quite a bit in Linux are FIFO pipes. Now this what this will do is this will make a FIFO pipe called awesome. FIFO stands for first in first out. Really cool feature of Linux a bash. And it basically what it is is it's a queue. Whatever goes in first is the first

thing to come out. Just like if you're at Cedar Point, if you're in first in line, you get to be the first on the roller coaster. Yay you. So, here's our first command. You can see right up here we're doing make FIFO. And HDA is going to be a um just, you know, a generic uh device that we can make in dev HDA. It's not going to stand any scrutinization. There'll be a little P next to it. If you do a long list in Linux, but it'll, you know, kind of fly under the radar a little bit. And then you can see what we have here. We have our actual connection command. So what we're doing is we're connecting

using uh to our listening port that's going you know to the cloud take a drink later and then that's going to use um hit this netcat. So this command right here that's going to get processed by bin bash right here and that's going to get sent to the FIFO pipe and that FIFO pipe gets redirected back through netcap. So you see we have our this being replaced since this is a file. FIFA pipe as a file. We can use our IO redirection to send that back through the pipe. So any output that we get from running the command that's sent through here gets sent back to you. So that's our basic our very basic bash

shell. Almost done. There's a second way to do this. There should be multiple ways to do everything. This is going to be using a command compiled into bash called devtcp. Now what devtcp does is it calls the c connect command to connect over any port on any given IP address if it's allowed. The best way to think about devcp is that it's a tho pipe but across a network. So it's a networkbased fo pipe. So our make tho and our netcat we're combining it together in something called devcp. I prefer this way because it's a little more robust and it's just cooler to be honest. Also, it's very useful if let's say the admin has

removed netcat or she's not available on your system. So, here's the command looks it's a little more complicated than the other one, but we'll go through it. The um this can basically be broken up into two parts, initialization and execution. So for our initiation part we have exec. Exec opens a file. Our next part exec IP port. This opens a connection as a file as a FIFO pipe essentially across the network on an IP and port. And then all this is doing is assigning that file that FIFO pipe that network FIFO pipe an identifier of five so that we can use redirection on it execution part cat will take any file and it will print

that to standard output. So what we're doing here is we're directing the file with an ID of five. So our FIFO pipe anything coming through that pipe we're going to send it to standard output. This is your basic while while read line. So while anything is being read, we're going to assign it to the variable line. We're going to execute line and we're going to keep doing that until we reach an end of file. We're going to redirect standard error back through this network FIFO pipe. We're going to redirect standard out back through this network FIFO pipe. And so what we have is we have a line that will cat out a file with an ID of

five, redirect standard out to standard in of the while loop, reading the commands for line by line until attempting to execute them and redirecting the output and error messages back through the pipe to your machine. Pretty cool stuff. So this is what it looks like. So if you remember that here's our netcat command, our netcat flow just goes through and redirects back around. We're just adding one little part into it is that we're adding this little while loop that allows this connection to remain open for as long as we want. So it sends it back through the pipe, but then you can see the process actually goes back to the while loop waiting for more input.

All right, we are almost done actually little schedule too. So now let's say that somebody has actually put out or deployed an IDS and IPS. I mean I'm sure most you know most of you can see the problem with actual clear text commands going across the network. If there's anything that's suspicious, that's going to set off red flags among anything else because there's there's no time when you should be interacting legitimately with a system without encryption going on, whether it be SSH, RDP, and Windows. Anything. I mean, you shouldn't even be tnetting to be honest, but I mean, I know people do it. So, how do we solve this issue? How do we get around having our commands go

through and clear text? What's up, question? Okay. So, we're gonna use something called B 64. So, B 64, um, if you don't know, it's just like base 8, base 10, B 16. It just has 64 characters, um, in its, um, alphabet. [ __ ] what's up? Crycat. Crycat. Um, I actually because, well, first of all, since a basic um talk, I was just going with B 64, but I was building up. I'll just spoil the ending right now. It's a AES pipe. Um, honestly, I had never heard of uh CryCat before. I mean, is that a Linux based command? No, it's it's just encrypted version of net. Oh really? Yeah. Just just s Okay. Well, the um actually one other

thing that I forgot to cover at the beginning, one of the things that I wanted to do was to do this without um using any local certificate. So, without having to upload anything to the server, it allows us so that to have a minimal impact on it. I mean crypcat is absolutely and using SSL is absolutely key to having a good rootkit to having a good intermediate rootkit you want to have some sort of actual encryption. The problem with AES pipe is that it only works on objects. So like zip files, tar files, etc. So if you were actually making a root, crycat would be a perfect example. But for our purposes of understanding the um the actual basis of

how this works, we're uh we don't want to have to actually upload anything because we want we're looking at minimal system impact. It's a great point though and I'm actually I'll probably incorporate that the next version of this talk. Thank you. Actually come on up afterwards and talk to what's that? OpenSSL. OpenSSL. It's always Yeah, I mean usually it is correct. So to create our basic encoded um shell, we're going to just add a couple extra lines um couple extra commands B 64-d and B 64. So what's going to happen is then it's going to echo the command the basic uh the encoded command that you send through it. It's B 64. It's going to echo that into um

into B 64-d run the command and then re-encode that and send it back to you where you can then decode it. So this is going to be our basic encoded um shell. So, run a quick video for you. Um, I was going to do a demo. I was told not to temp the demo gods, and they're right. I am nowhere near this good of a typist. Being able to cut video is awesome. Just saying. So, you can see here, um, you can see, can I change the resolution on this or is that going to mess up your uh, you could try it. It might you might kill the slide deck though on here. Oh, all right. You know what? I mean,

you can try and dump it down suggestions for the next time you do the talk. Let's change this so that your shell background is white and your text is black and it will pop on the screen. I had orange originally. That's what I usually use and I realize that nobody can see that. I thought this would be a little better. Fortunately, not. So, I'll just talk it out loud. So, you can see here we have our netcat um listening port. If you can't, apologies. Um and using our verbosity over here. I know no one can see it, but this is our pop box and this is my freance box at home that I was using as a test subject. So, we're

not just limited to Linux. This also can work on Unix systems, anything running bash shell. Free does not run bash by default, but it is installed by default. That allows us to have this um process actually work. You can see we're um opening our devc file. Actually, most can't see. You see, we opened our connection successfully. And so now we're actually going to run the encoded version of our uh remote shell directly in the line into B 64. We're decoding, running the command, re-encoding, and then sending that back through.

As I said, a very good typist. Um, unfortunately, netcat-l does not like having its input and output redirected at the same time. Try it with a bunch of different FIFO types. So, as it stands right now, barring um I actually have a Python script running um to actually run the client at home. You probably have to use a Python script, but for our purposes, doing it manually demonstrates exactly what we're talking about. So you can see the response that we got that was B 64 encoded back from the pop box from our free nest server in Unix box. Then we echo that into a uh a decode command. We can redirect that into a file and

then we can list the contents of that file. As you see we have our free followed by list of our directory. So this can be pretty powerful if used in the correct ways. Um obviously you don't want to use just this. You want to combine this with other object-oriented um options like Python, like Crypcat, like OpenSSL. And you can combine all these different ways into making a very potent, very transferable um shell. Oops.

Okay, so these are the two operating systems we used. I connected from Fedora to a freest box and was able to successfully have a persistent connection through that using the tools that we described in here. So to wrap it all up, if you're a CIS admin, there's a couple things that you want to do, and it basically boils down to this. know what's going on in your environment. Monitor your flows, monitor your logs, monitor your events. Depending on how paranoid you are, and if you're a good CS admin, you should be incredibly paranoid. Should do egress filtering or if you're like um my friend back in Detroit, Wolf Gang Garlic, you can just go ahead and do total egress

lockdown. And I was actually having a pretty good talk with him about this before um besides in that what he it's a completely session based firewall. Unless you have a open legitimate session to that um port 80, nothing goes in, nothing goes out. I like I said, he's incredibly paranoid. So this is me consultant, former Linux engineer, played way too many games. Member of my you can give me on Twitter. Um here are the references that I used. Uh does anyone have any questions? Not we can go ahead and end early today but yeah go for it.

Absolutely.

You can just fine.

DNS,

right? One of the things that um that I like to do uh with this and the uh command in general was that I don't really like netcat, you know, I mean, I showed it as an example, but devtcp you toss it into a script, call some, you know, call it MySQL, call it whatever you like. Um you can kind of obiscate what it looks like if you really want to. You can uh sign zero in bash will actually allow you to change what the running process is. So if you do have a pearl script, you can go ahead and you can um change whatever you'd like. You know, you can change or in terms of what

it shows in ps aux-f uh whatever your chosen poison is. That's an excellent point too that he made was that Pearl is a great language. Python also is another great language in order to do this kind of stuff. Bash just shows you the basic structure of how it works. It's not going to be great if you want to do persistence um in the field, but you can apply the concepts from bash and use that in Pearl and in Python. Thanks. Thank you. So, now that you've added now that I went back and thought about your talk a couple times, I wanted to bring up another thing because I really want when you did your talk in Chicago, I'm like

besides Detroit, I'm going to hit him with an IDS solution he hasn't seen before that can stop some of his attacks because you weren't familiar with hostbids at the time. That was awesome. Um so, uh with regard to hostbased ID, mentioned earlier and you mentioned that it can watch the file system. It can also watch activity and things like open ports. So you want to open a port, a non- center port, fine. We're going to be uh alerted on that right away. Um again, not 24 hours later, but you can set your alert tuning to whatever you want. You 30 seconds, 5 minutes, whatever, whatever you need to be to make um make yourself more aware

as a systems administrator, but also um I'm a firm believer as a system systems administrator in whitelisting. So I don't care that you try to do an IP or a random port. If your IP is not allowed to touch talk to my devices, it doesn't happen. Absolutely. So, whitelisting is another great mitigation defense for systems administrator. Sure. But I mean, you know, in terms of like let's say you have a web server, you know, something going across 48, you know, um you know, you've got that it's kind of hard to white list that but but that's where you watch for um through your logs. You watch for you use OC to watch for activity that looks

suspicious, right? And you combine that with You have to be a smart systems administrator know what's what shouldn't be there. So absolutely you know and you combine that with a sim and you've got a pretty comprehensive solution. If you've got B 64 traffic going out of report 80 you probably have a problem right and to be fair um that B 64 um will probably only stop snort or some really basic IDs. If you've got a Cisco ASA if you've got anything fairly advanced you know you want to use something like CryCat like OpenSSL in order to prevent that from being on the fly. Another one called DNS for DNS. Ah, very good call. I I did not include

on the talk, but that is an excellent absolutely excellent idea. Yep. Absolutely. Anyone else? All right. Great. Thanks, guys.

Hey, not a problem, man. Do