Confessions of a Sh*tty SysAdmin

hello b-sides dfw my name is andy rainmaker thompson and we're going to get started with the track one today um i got to give you a fair warning this has got some explicit language so if that is a trigger to you check out the other tracks they've got some fantastic content throughout the day here so let's get started the title of my talk is tales from the trenches learning from the misery of others but in reality we're going to call it what it is confessions of shitty cis admins now let me tell you a little bit about myself before we get started again my name's andy i have a very interesting job i am a

researcher an advisor an evangelist with cyberark software so we do privileged identity management i have a bachelor of science from uh the university of texas at arlington and management information systems i have a host of different uh certifications which we'll talk about shortly and i again i'm a very active member in the dallas hacking community between dc214 dha ntx ntx issa ise squared you name it and uh i'm probably there uh but what i'm known for mostly is that i am what's called a travel hacker my wife and i we travel all over the world on a shoestring budget so if you want to learn more about that come find me on discord and we can chat but beyond all that i was

once a shitty sis admin but before i was a shitty sis admin i was a shitty help desk person i worked as a help desk agent funny enough about a mile down the road here in plano and that's really where i got my uh my foot i got started in my career the foot in the door and so that's where our story begins and it really starts with passwords as part of my job we were expected to reset the user's passwords in order to validate their identity we had to ask them a secret question now what was interesting about this circumstance was that the users got to pick their own questions so we had some

funny ones which i'll share with you today the first one was a guy who decided to be cheeky and pretend that he couldn't hear so he kept having the person ask the question over and over and over again you know what that question was who's your daddy yes so that poor poor person on the help desk side often it was me would have to shout who's your daddy who's your daddy who's your daddy and the answer me yes the next guy this one was a bit of a pervy guy but uh he had probably one of the most creative secret questions uh the question was what are you wearing right now and the answer quite correctly was that's totally

inappropriate so with that said we're going to get started with what i call the tla section of today's presentation tla stands for three letter acronyms so again take notes because there will be a quiz all right so the first of our tlas is sla does anybody know what sla stands for well you and the back are correct it is service level agreement this is the uh ratio or whatever numbers main you have to maintain to basically meet your contractual obligations this is a very common uh statistic in help desks msps you name it uh the second of our s tlas was our fcr this was the sla which we were graded by i know lots of tlas folks

but fcr stands for first call resolution now what do you think is an appropriate fcr uh percentage for an entry-level help desk agent like having to resolve the call on that very fir resolve the issue on the very first call would it be 50 that's that's a lot right 40 no it was actually 90 we had to resolve 9 out of 10 every calls otherwise we weren't really doing our job supposedly which again breeds discontent so as i worked up the ladder i became a junior a senior ultimately i progressed to the world of being a systems administrator this is where i met sean garnish sean was my uh my tutor and my my my senior superior

employee that worked with me uh he happened to have stage four multiple myeloma cancer and i was hired to replace a living dead man it's very awkward situation but sean was an awesome awesome teacher and he taught me a lot of things the one thing that i want to share with you today is the three envelope rule this one lesson has saved my career and by the time i'm done with today's presentation i'm going to share that lesson with you so as i progressed in my career i joined a large enterprise organization and a large it organization and as you can see regardless of what organization or what role you had within an it organization

systems administrators are often met with a certain amount of ill will if you may i don't know they seem to think that we happen to say no and block access a lot right well if you're familiar with this xkcd comic you'll see that sysadmins aren't mean they're just concerned about one thing and one thing alone what is that thing well it's uptime right uh many people think that we're doing a glorious glamorous job but in reality we're just trying to keep this ship afloat because are you familiar with the cia triad this is the this is like a cissp question folks this is the fundamental tenets of i.t and security for security to exist we have to have

three things what are they cia confidentiality integrity and availability now that's the cia triad specific to security as a systems administrator i had a different triad altogether and this was the aaa triad now my question to you is what do you think aaa stands for well you'd be correct it stands for availability availability availability yes what we're trying to do this whole time regardless of what configuration we're just trying to keep the boat afloat folks that's all we're trying to do trying to keep up time at those five nines that you often hear about which leads me to this how did i get the name rainmaker um it first off i don't believe that hacker

handles should be picked you shouldn't pick out your cool handle i think that these names should be bequeathed upon you and so i'm not doing the whole cash cash money money stuff that you often think of when you hear rain making uh i look at it like um being in there to you know do a miracle if you may uh in the circumstance where i got the name rainmaker we were doing a data center migration and our connection to our data centers uh ceased to to exist and some way lo and behold i was able to get that uh connection back up if you ask me why or how i couldn't tell you but from

then and that day forward my boss started calling me rainmaker so again the name stuck so let's get started with our first of our confessions at one time i worked for a large restaurant organization they had a bunch of different chains and different concepts and as part of a retail restaurant company we had to have multiple networks so you have to have your back of the house network you have to have where the managers do their functions you have the front of the house point-of-sale network where again all your point-of-sale machines are communicating with each other and then lastly you have your public-facing free wi-fi if you choose to provide that in this circumstance where we were doing

a grand opening uh all the hardware comes in about a week before so we have to do this huge massive uh push to get the it operational right before the grand opening well unfortunately i wasn't given all the necessary network equipment uh to get the entire system up to spec right so unfortunately i was mandated against my better judgment to make everything on a flat network so backup house point of sale even the front of the house was all on an open network anybody could dial in anybody could get access to that network and it was scary right and subsequently i ended up leaving the organization due to situations like this and i won't say

where or who this organization was but if you still go to those restaurants or that restaurant you'll probably find that that network is still flat so which again kind of leads me to this point here and i think this picture here really personifies this and this is something that i tell all my clients quite often any security control that inhibits the ability to do one's job is a security control that's going to be circumvented so that leads me to the next set of stories now if you haven't seen alt beers uh dc i'm sorry b-sides dfw badge it is awesome it's the cyber dolphin from johnny mnemonic and so that's really what the rest of this presentation is is

going to be mnemonic devices that will help me tell stories of my misfortune and others so the first one is world of warcraft what the heck well this was in august of 2004 when i was working the night shift uh as that help desk support person and this is back when molten core 40 man raids were rocking uh i was duel boxing my resto shaman and my prot warrior with 38 other people and i was doing that late at night at the same time as i was working so i was bringing my laptop in and raid leading that was bad you know you shouldn't bring foreign assets into a corporate network but what we're seeing now is the

exact opposite in order machines are so locked down now that many people have struggled to do their day-to-day functions and what you're seeing is data is being exfiltrated by the employees to personal assets so that they can actually do their jobs now this is a big no-no folks we definitely don't want our data to leave the controls of our organization but that's what's happening nowadays now the next story i want to tell you is about content filtering all right so imagine this you go to your uh your office you happen to go into some inappropriate websites and it gives you the website block right everybody has those well in certain circumstances you need to go

to websites that are unfortunately blocked maybe they're cryptocurrency sites or something like that um but just due to an aggressive policy the this web content is blocked so most organizations will allow outbound uh communication via port 22. i mean we got to access our aws instances right so what i ended up doing was instantiating a pro a sox proxy so i just sshed into my raspberry pi back at home and made a few configuration settings on my web browser and by doing that you can completely bypass your organization's content monitoring filtering so that's another thing do as i say not as i do right now the next one is about multi-factor authentication i have a terrible memory

i it's just it's bad i have a hard time remembering names and faces and losing my keys in my wallet and quite often i would forget my multi-factor token when i was going to work unfortunately you'd have to access the pci network through multi-factor authentication well what i was also at the time was the virtualization person so i ran in the ex esxi hosts and our vsphere implementation so by jumping into the uh machines via console access on vmware i was able to completely bypass any sort of multi-factor authentication so think about it this way folks if you're protecting these assets aggressively make sure you're protecting the virtualization just as aggressively because that's an easy way to pivot from

within the virtualization infrastructure into those vms so be wary of virtualization now the next one is does anybody see what's wrong with this particular picture here somebody just happened to walk away from the computer leaving it completely unattended and this is something that we saw all the time back when we worked in an office facility this is where i wanted to introduce the donut concept this is a great great opportunity to gamify security within an organization here's what you do you set up a distribution list called donut at your organization now if you happen to walk away from your machine for example somebody might stumble upon your computer opening up an email to donut at

your organization and unfortunately you're now obligated to bring donuts in the next morning so like i said this is a great way to gamify your security and have a little fun and at the end of the day be more secure now the next thing is a forest and it's and tripping over a wire what what is this well at the time i was responsible for monitoring the integrity of all of our point of sale systems there's a great great tool called tripwire that monitors every single change every um every tweak on a on a machine and reports back to the to the administrators as the change of integrity of this system um when you're doing this on a global level

you've got you know thousands of thousands of endpoints that change configurations ever so slightly on the daily and so we were having hundreds if not thousands of alerts coming in on the daily and for a single person to manage that who was enact improperly trained no less i figured out a way to automate this and really make it easy for me so as all these alerts were coming into my email i created this rule that deleted them all yeah so it really didn't accomplish a whole lot and so often what you see here is is that you can't see the forest through the trees you have to tune and tailor your alerts so that you're not inundated with enough

alerts that you don't no longer see the actual alerts coming in that are of the real value so again filter your alerts because it will save you a lot of hassle in the end now i'm going to share with you a video here that is a mistake that as a systems administrator i made all the time and open the file this is a 6.0 version you didn't upgrade yet did you know who this is just use your translations live and jimmy where's that move [Music] go to your chooser go to the printer pick your zone and pick your printer and then we'll i'm on the chooser it makes everybody get up is this the zone here

move there you go green saver code all right let's run attach just type in x y dot again violator slash [Music] [Applause] [Music]

with his credentials he was essentially putting his ntlm hash in memory of those machines so the better practice here is to reboot the machine after nick logs on that way it clears his cat it clears the memory and if the machine was compromised well the only hashes in that memory would be the person currently logged in and those service accounts so again nick should have said move totally fine logged off rebooted the machine and handed this system back all right so what do we have here in this next slide we have this mountain of trash and passwords so what we see here is quite often people are recycling their passwords in the wrong way we see

password reuse over and over and over again in fact did you know that when disney plus was released that very day there were accounts that were available for purchase on the dark web why was disney plus hacked absolutely not people were using the same passwords over and over and over again even the same passwords that were leaked in a data breach so again this demonstrates why password reuse is so bad but we also see password reuse in other methods the first one here is tux here holding the windows logo this tells me a story i want to share with you about password reuse one of my clients happened to have a service account password that was on

their windows environments they were using it also to interactively log on as systems administrators they were also using it again like as a service account as a database account the same credential was also used in their network infrastructure and their linux environment they weren't even 80 bridged and they were still using these same credentials so what would happen if they were required to rotate the passwords well they honestly didn't know where the passwords were being used they didn't know what applications would break in the circumstance of the credential change so what did they do nothing at all nothing at all they didn't rotate the password so in the event that it was compromised they single bad actor could basically pivot

across the entire organization and multiple platforms now our next story i think this one is hilarious this is kangaroo jack this is the avatar of the gentleman that was working as a linux systems administrator next to me super smart guy and this story is hilarious so audit required us to rotate the linux passwords every 90 days now if you're familiar with a large enterprise organization there's a significant amount of linux infrastructure um to rotate those passwords every 90 days manually was a huge challenge so we found out this uh we use this application called dish that would allow us to dish out multiple commands to multiple endpoints at the same time so what we would do is

go into the dish out the password change to the entire linux infrastructure changing the password thus breaking all the applications the service accounts you name it but what he did was he immediately dished out that same command to re rotate the passwords back to the original so essentially if you look at the last date of the password change it's it's compliant audits happy but in reality the password never changed so yes we don't want to uh we want to use our passwords correctly don't reuse them over and over and over again passwords should really be one of our not one of three things three of three things complex unique and the most often neglected frequently rotating because think about

it like this it takes 57 days to basically brute force an 8 character complex password if you have a policy that rotates the password every seven days or so that password's been changed multiple times over by the time that that initial password was cracked which leads me to this point i when it comes to compliance compliance is a good thing it allows an organization to have that carrot on a stick that pushes them to be more proactive with their security measures but what you see here is is that often compliance is the low water mark it turns into the minimum viable product so in my opinion a lot of organizations view compliance not seriously so if you want to find an organization

that does take security seriously and you want to learn more uh i would highly recommend you follow my friend hermit hacker brian mork is amazing resource and he's the type of person that personifies uh real security so if you want to learn more check out hermit so you've seen me on some of my confessions i absolutely have shared with you some of my my my dark secret confessions uh you're i've seen terrible things again throughout my career and viewing the mistakes of my clients and and co-workers um here's one that i i it's not my coworker but i want to show you uh a a video and i want you to tell me where the mistakes

all right

i brought a gift with me right here um anybody see what happened there this right here hold on is we'll let the room [Music]

which leads me to this don't have weak passwords don't use the built-in administrator passwords these sorts of things literally blow my mind every time i see them which leads me to gilfoyle here if you haven't seen silicon valley i highly highly recommend you watch this show it is amazing i love this quote if you're dumb enough to leave your login on a post-it on your desk it's not a hack it's barely social engineering it's more like natural selection which allows me to show you this over the course of my adventures visiting all sorts of clients i've seen a lot of exposed passwords just out in the open so again don't do this people and pen

testers and bad actors will absolutely go to the office facilities and harvest these sorts of passwords don't get smart and leave them under your keyboard because that's probably one of the first places people will look which leads me to this point if you're ever going to be interviewed if you're ever going to be on tv if you're ever going to be having some media attention don't have that in your office because quite often passwords are again exposed in broad daylight and these are all media interviews in which passwords were clearly exposed leading to actual data breaches so which leads me to this in the future what you're going to find is less vulnerabilities less exploits and

more misconfigurations i like this quote from gartner by 2023 99 of cloud security failures will not be due to vulnerabilities but due to misconfigurations on the client part we just saw that recently with a huge financial institution had a major data breach simply due to a misconfiguration of their waff firewall rules basically the rule allowed for uh reading of aws buckets when they had no business doing so and again this is just misconfigurations over provisioning and that sort of stuff another thing that we see is hard-coded credentials in source code within scripts and so i wanted to show you a tool called truffle hog which searches for strings of entropy in github repos now it not only does

that but it also looks at the commit history so even if you remove the source code or remove the credential from the source code if you didn't remove it from the commit history as well well bad actors are going to find it so let's take a look at my recorded demo here where we're looking at this idiot's github repo and oh my god look at all the ssh keys the private keys that are available here this happens to be my own github repo so no harm no foul but again it finds api keys passwords ssh keys you name it so any string of entropy truffle hog can sniff out and so that's just one of many other different

tools that are available to you free and open source which leads me to my next story here mr t i love mr t i had a little mr t doll when i was about six years old and uh this is a story of intrigue and being laid off yes as part of my career i almost was laid off and unfortunately i had survivors guilt because the large number of our e-commerce team our it it team were outsourced to an msp and uh our e-commerce team one of the gentlemen did not take very kindly to being laid off so what he did was plant a logic bomb tick tick boom at some point in time this code would execute and launch its

payload so what happened was on april 1st uh i can't remember which year but basically this was long time past this guy leaving the organization the logic bomb went off it basically manipulated a javascript code on the front facing corporate uh the retail site basically it flipped the entire website upside down now everybody in the inside was panicking going what the hell happened well in reality it was just a very simple javascript security found it but until then they were running around with the chicken like their heads cut off pr played it perfectly they put because it was april 1st they played it off as an april fools joke no one was the wiser until now now you know

now the next story i'm going to tell you is about a gentleman that uh you're you're gonna love this one so uh as part of my job previously i would go and do assessments for organizations and uh as part of the assessment we would find highly privileged accounts service accounts um sysadmin accounts that are running services which is like a big big no-no because simply put if that service account is running then that's the account that's running that service that hash is in memory so if you use like i don't know a domain admin um that norm that would log on and do that well that domain admin is in the memory uh as long

as that service is running so i found this and i reported this back and i didn't realize that that same gentleman happened to be in the room with me and uh i'm gonna quote this and again plug your ears it's not that i'm calling you a liar but i don't believe a word you just [ __ ] said now that will put you uh on your heels very quickly right i didn't know what to do so the first thing i did was grab the hdmi cable plug it in we already peed into the box and lo and behold guess who's running procmond i have never seen a man turn that red that fast that quickly

it really vindicated me but again it just goes to demonstrate that you know we need to make sure that we're not interactively logging in and using service accounts simultaneously it allows for your uh your incident response team your sock to determine how an account is properly used and when it's not being properly used now this is a good story about porno surfing or surfing for inappropriate content there was a gentleman where i worked that worked the night shift and had a propensity for not appropriate content what this person would do was save some movies and images to their local machine their my documents and didn't realize that the documents were also being synced to our san

back at hq now we stopped replicating that a few years back but it was my job to clean up the sand and kind of remove and archive some of the data upon that i stumbled upon his treasure trove of naughty bits and we had to report it to hr now this gentleman did not take too kindly to that accused me of planting it on him so that he would get fired so i got very very clever i said hey you've been here longer than i have right all right well why don't we do this let's go pick a backup you just pick any date any day before i started okay and we'll see what happens and so we restored the backup

and lo and behold um yeah his backup retrieved a bunch of naughty stuff now i'm not trying to kink shame anybody what you do on your own time is your own business it's it's not ours but don't do that on a corporate machine folks that's definitely a big no-no which leads me to my next point there are certain words and phrases that we really need to refrain from using with dealing with people within our own teams people outside of our teams and i'm just going to go through a handful of them the first one is ping now this one's a little weak but a ping is a icmp request beacon waiting for a beacon response and

that's the technical term it originated from sonar um it's not to reach out to somebody and you know hey i'm pinging you for this uh hit me back later it's a little confusing to people outside of the military and outside of i.t the next phrase i think applies to everyone to be honest right that implies at times that you're not honest so just do yourself a favor and just drop this from your vernacular okay the next one i want to share with you is burning the ships this is to denote you know marking a line in the sand that you just can't go past you know or you can't go back from back in the day the conquistadors of

yesteryear would finally make it to the new world and wouldn't have enough resources to ship back to where they came from so their stories that the conquistadors would burn their ships so their crew would know that they couldn't go back well one they didn't do that two if they did that would be really stupid right what they did instead of burning the ships was actually scuttling the ships think about how much natural reason not natural but how many resources would be available as a ship the lumber the canvas there's so much that would be available to these conquistadors and these sailors so again it's not about burning the ships and moving forward it's about doing that in

a responsible way so again no burning ships now the last one really chaps me i hate this term this is something opening the kimono was a term that really started in the late 80s and early 90s it's to denote a pure transparency opening one's kimono to expose themselves and and in my opinion this is not only racist but also sexist so uh just drop this entirely just don't use this word and if you hear people that are using this take them aside and let them know that this is just not acceptable so thank you do yourself a favor no opening the kimono last but not least i hate the term hacking when we're talking about the

context of cyber crime hacking is not a crime folks it's not bruce schneier defines hacking as using something outside of its intended use it's how and what you hack that determines whether it's good or bad also the the stereotype of the hacker with the hoodie and the guy fox mask living in their mom's basement or whatever these are just stereotypes you want to see what a hacker looks like well it's it's you it's me look in the mirror folks that's a hacker right it's not the guy in the hoodie okay so i've shared with you some of the terrible things that i've seen i've often seen some really good things over the course of my career the first is a

really quick and easy way to protect yourself from ransomware let's face it joan in accounting my grandmother there's no reason why they need to be opening up powershell scripts there's no reason why they should be running batch files or vbs scripts so by associating them with wordpad instead of the the actual engine that will stop these malicious payloads from executing it's a very quick and easy way to protect your organization also if you notice right there there is an easter egg so i highly encourage you to check it out all right the next thing is uh just a very simple change when you're installing software have that secondary data drive that you can change the

install path because quite often script kitty tools are looking for a default installation of c program files x86 blah blah blah blah so by changing the destination of where the software is installed it makes that job for an attacker that much more difficult and it also has an operational benefit as well what it will do is shrink the size of your incremental backups considerably so not only is this a more secure course of action but it's operationally sound as well so as you can see i've had a lot of fun over the course of my oh gosh 20 something years in the industry we've got jason street and his awkward hug i've got some time the spaghetti time

when i was working with the government and i've made some amazing friends this is another fun thing for april no it was june 17th if anybody's watched american psycho i changed all the time clocks globally to say feed me a stray cat so you can have fun on the job too so one of the things i want to share with you is that shitty cis admins make the best security professionals because we know where the bodies are buried we've made these mistakes and we know how to correct them which leads me to this there's a common question that i get from a lot of people starting out in the industry whether or not they should what what

they can do to get their foot in the door and it leads me to this the concept of experience versus certification if you don't have experience certification will help you a long way does anybody know who this person is this is dave ramsey dave ramsey is a financial guru he teaches a class called financial peace university and i owe a lot to dave as he helped me get out of debt i had like 35 000 in debt at one point due to credit cards and uh following his what we call debt snowball it really allowed me to basically become debt-free very very quickly so his concept of starting off with the smallest building upon that building upon that until you

have a giant snowball of of whatever we're going to take that and apply that to certifications so from a blue team perspective from a defensive we start at the basics you know just learning the fundamentals of it that's where the comp tia a plus leads to the security plus which then leads to the sscp or giac has a really good gset course that i find falls on this line with this with this stage but once you've accomplished the g-sec or the sscp you basically got five of the eight modules that are really necessary for the cissp that's the anyway from that's like the i don't know master's degree of cyber security but even then you can go forward and

specialize between uh health care cloud you name it all of these basically allow you to become the cyber warrior that you you're trying to achieve um the same thing can be done on the red side as well so starting out with this certified ethical hacker leading to the uh the was it sex 561 i think the g pen also that leads to the oscp which then again you can specialize by going into the osce whatever and at that point you're a dark wizard now this leads me to this certifications are only as good as the paper that they're printed on you really have to be able to walk the walk and talk the talk you otherwise you're just

wasting your time with these brain dumps and certification boot camps so really understand what you're learning because otherwise you're just cheating yourself now this typically is meant to be a live presentation where i would then walk around the crowd and say hey oh it's your opportunity to share your stories and confessions but since we didn't have that here today i'm going to share with you a couple of confessions that were shared with me from the b-sides dfw community the first one is this one does anybody know what you name s does on a linux system well it's fairly simple it checks the machine name properties of a remote server but what does that same command do on a

solaris system anybody know well it happens to rename the host yeah imagine doing that on a dns server you're going to mess things up real quick and it will be a bad time so know what commands are what on each operating system now how many people here have ever done this yes yeah there we go so i know some people have done this i have done this it's a mistake and we've all made it so a quick thing that i would recommend you to do is edit your dot profile to rm-i basically what this does is prompts you to confirm your deletions that way you're not making huge errors like this right now this one we were just talking about

this downstairs um a gentleman i'm going to not use his name was working for an msp that maintained several different clients this was back in like the late 90s early 2000s maybe and basically uh he got smart and said oh i'm gonna make this a a file that i can easily access from anywhere so what he did was he put the passwords on a geocities website all the usernames phone numbers passwords root access into these systems now he was smart he didn't link it to anything so you know hopefully spiders and search engines wouldn't find it but you know who did find it wayback machine yes it archived that page with all the credentials and usernames and passwords

you name it so yeah that's uh not cool right so the next one is uh from uh ben goers from uh fuzzy snuggly duck i think is the name the their awesome team this is a story that he shared with me of a situation he didn't make this but this is a story of a hacker or not a hacker a bad actor a criminal so what happened is this database this database was exfiltrated by this bad actor but not only was the database stolen he actually compromised they compromised the database by injecting their own email address into that database so what happened a few months later is infosec finally found evidence of a data breach and alerted

all the users in that database say hey your information's been compromised we've already taken care of it blah blah blah blah well that also notified our bad actor here not cool well you think the story ends there right no this is where the bad actor decided to take a step forward and use that to fish the employees again it's basically saying hey we know your data was compromised we've created the secondary portal if you want to get paid go log on here and basically that's how this malicious actor was not only able to compromise the database but also fish everybody to help so yeah so we're getting close to the end of my presentation folks i told you there was

going to be a quiz later so let's see does anybody remember what sla stands for yes service level agreement f fcr what was fcr first call resolution all right now the big one here the cia triad does anybody remember what that stood for confidentiality integrity availability that's right but what is the aaa triad of systems administration that's right it's availability availability availability and last but not least i just needed that in here so anyway again the takeaways from today's presentations are pretty simple uh specifically prioritize security uh any control that's inhibiting the ability to do your job is going to be circumvented so don't cut corners make sure that the people that you're entrusting to enforce

these rules are actually doing the same themselves also just remember that in the future um the futures now folks that you're going to find the case more often than not that there will be breaches due to misconfigurations versus actual traditional exploits and vulnerabilities they're still there but you're going to find more misconfigurations as the entry point versus anything else and just understand systems administrators we're people too we make mistakes and it's just part of life so a takeaway from today is understand that there is risk everywhere especially even as a systems administrator learn from the mistakes of myself learn from the mistakes of my friends and co-workers and and don't let this happen to you

oh i almost forgot the three envelope rule again i told you that sean basically saved my career after he passed away he said whenever there is the the [ __ ] hits the fan you don't know what to do everything is turned upside down and you don't know what to do go to this file cabinet and grab an envelope and trust me it will save your job so after he passed our database went down that was holding our timekeeping software so uh i didn't know what to do i i was out of i was out of luck i was at my wit's end so i went to the file cabinet i opened up the envelope and what it

said was blame the previous sysadmin so that's what i did i blamed it on sean said hey this was his fault i'm so sorry i'll get this operational and i finally got it operational but that saved my job now about six months later another system went down i think this was our sap system or digital asset management system for the sake of the story it doesn't really matter but the point is is it went down hard and i'm scrambling trying to keep my head above water i couldn't do it so i went back to that file cabinet you know what it said when i opened up that envelope blame the vendor so again that's exactly what i did i

said hey look it's the vendor's fault it's a problem i'll work with them just deal with it we'll get this up eventually and you know what it worked it totally worked they bought it hook line and sinker now again several months went by and another system went down again for the sake of our story i can't remember what it was but again the [ __ ] had hit had hit the fan i didn't know what to do so i went back to that file cabinet and i was at my last resort i opened up that envelope and you know what it said prepare three envelopes that concludes my talk today folks i really appreciate your time and enjoy

the rest of b-sides dfw thank you