← All talks

Confessions of a Recovering Linux Admin

BSides SATX · 201940:55395 viewsPublished 2019-09Watch on YouTube ↗
Speakers
Tags
CategoryCareer
DifficultyIntro
StyleTalk
About this talk
Title: Confessions of a Recovering Linux Admin Presenter: Ell Marquez & Allie Barnes Track: In The Beginning 08 Time: 1600 BSides San Antonio 2019 June 08 at St. Mary's University, San Antonio, Texas Abstract: Many of us in the technical community have heard the saying RTFM (Read The [Friendly] Manual) as advice given to someone learning Linux. In Confessions of a Sysadmin, Ell Marquez will share a practical approach to learning to navigate the command line along with some of the secrets she has learned along the way. Speaker Bio: Ell Marquez has been part of the open-source family for a few years now. In this time, she has found the support needed from her mentorship relationships to grow from a Linux Administrator to an OpenStack technical trainer at Rackspace. Recently, she took the leap to join Jupiter Broadcasting & Linux Academy as a Community Architect. Allie Barnes has been in the Linux community since 2011, jump-starting her IT career as a Linux Administrator at Rackspace and utilizing mentorships and community knowledge to eventually navigate into the OpenStack world, working on Red Hat’s OpenStack Product. Allie is learning to take the leap into the community by participating in more talks and community events in hopes of spreading knowledge and love of Linux and Open Source in general!
Show transcript [en]

Dukas admin you know we went back and forth on whether explaining why we decided to do this at a fiber security conference but really I think we'd like to tell you a little bit of a backstory about how this talk came to be this talking to be when I hit a pretty big wall of burnout and I was done and I didn't want to do Linux anymore because I was so tired the community I kept feeling like I kept having to prove that I was worthy enough to be part of the community that was technical enough that I was good enough and it got tiring it just got draining and I kind of started playing around with InfoSec and

figured out like this community is so different it was okay you're new so why are you telling me that grab your laptop sit down and we're gonna do this like and I talked to Ellie about it and started finding that even in my gusto to go and mentor other people and talk to them I found a lot of that negativity inside of me and I'd start making comments like making a lot of fun of alia for running arch it's my favorite thing to make fun of her about um and as like this isn't healthy so this talk came about for me as just a way to get this off all of my chest and be like this is where we are this is the

state of the Linux Union for me right now and kind of challenge all of you to help me change the world and Ellie why did you jump on cuz he told me to no no yeah so like oh I was just really burnt out at work and kind of like I have a career but I still don't know what I want to be when I grow up what do I do with my life I was like come to talks with me so and then we kind of sat down in the coffee shop and ranted for a really long time and this talk is born this is what we came up with so I guess we should to

introduce ourselves though so my name for those of you that aren't my friends who are coming to support right now is el marques and I'm the community architect or Linux Academy now really I could sit here and tell you all the great things I've done and the cool stuff I've done but if you know me you know that I just tell people I'm a professional new look I love learning new things and with that comes breaking them it comes trying to figure out how to do it imposter syndrome finding mentors trying to build a community around teaching it and really it's just the exploration of being new and having fun doing it hey everyone I'm Ali

I'm the cloud operations engineer over at info site where a small cybersecurity start-up I basically manage our infrastructure that's so yeah before we get started I'm gonna give you a resource I promise this is not self-promotion our slides are a bit lacking when it comes to words because I speak enough for all three of us so every resource that we give you and it comes to capture the flag when it comes to resources when it comes to Linux distributions can all be found on a github repo so you don't have to worry about taking pictures or trying to figure that all out the website is ello punk calm so we can kick off now I think

everybody should be in if you are looking to sneak in except for this one person who I'm gonna call out and that is we can go to confession number one and that you know when I started out in the Linux community we might imply X I'm reading it haha when I started out I really wanted really wanted to feel like I belong part of the community I didn't want to just sit behind a terminal and do my job by myself and however the more that I tried more than I tried to join meetups and come to conferences the more that I felt like I couldn't get past the front gate and really that kind of

brings us to our first lesson for you guys and that's the fact that gatekeeping happens like I wish I could tell you that we've solved this problem that cake keeping was a thing of the past but if you're gonna tackle something new if you're gonna show that vulnerability and show up and say hey I don't know how to do this and I am here because I want to learn someone's gonna look at you say well how you not know this how long have you been here and they caught that attitude it's gotten to the point that a question is simple as what is Linux has turned into a gate question so if I say Ally what is Linux

what would you say I tell you Linux is an operating system and then this is when you start getting people who you know they can't keep are their face changes they get a bit of a look you look they stand a little bit straighter and they're like oh no Linux is not an operating system Linux isn't huh really we've not matured past the point where we can have an honest conversation yes once upon a time Linux was nothing more than a kernel and then we open source and then we grew and then we learned and now Linux is both Linux can be referred to as an operating system there are Linux distributions there are Linux

kernel or below Linux kernel that is in development all of these are right answers and by shutting someone down with the question as simple as that we do absolutely no favors to our community anyway no one is immune from this I was lucky enough to go to a open source summit and I was looking at to go to open source summit and I'm sitting there second row as Linus cobalt is being interviewed and the interviewer looks at him and says - like how do you understand every single line of code that's in the kernel he looks at her and admittedly says I don't I don't have to and at this point the room erupted people pull out their phones they're

tweeting they're taking pictures and this is what I see on the news feeds the next day and this is a bit ridiculous because if they had actually put their phone aways and listened they would have heard that what he said is I don't because I don't have to what I need to understand is the big picture by understanding where I want the project to go where the project has been I can look at a line of code and see if it fits in within the vision so this is a lesson that we're here to share with all of you is yes gatekeeping happens you know what I have never not once furthered my career by trying to appease

those gatekeepers but I further my career by is going and finding the other people who say hey I'm you too but I'm willing to go yeah and through that all you need is a starting point you don't have to be able to do you know now a reverse malware engineering and get down to the bits from day one it's a ridiculous ask but if you have the big picture if you've learned a little bit of coding and you've learned a little bit of the OS and you've learned a little bit of no pentesting whatever your interest is it's easier to pick these things up and for those of you who are new and I just

view this out for you on ello punk comm I actually have a study group that I held to tackle that gatekeeping question of what is Linux we go down from the basics of the Linux kernel down to why is the mascot a freaking penguin anyways so my confession a lot of times I have psyched myself into thinking that Linux is a lot harder to use than it actually is I did this a few ways trying to use harder distros than my skill set allowed at the time trying to use the wrong tools for the job basically setting myself up for failure and for days where I was like I hate Linux I hate it I

don't understand it it doesn't make sense but that's okay um so I just have this I like to make fun of me about using Arnage you know what when I started out on Linux and I'm sitting there with my little blue 2 laptop they tell me you know a real Linux admin uses arch so it's always been a kind of yeah it was like a long-running joke but you don't have to use arch you don't have to start off on arch right away don't do that you're gonna make yourself hate Linux cuz you're gonna be like what is this this makes no sense um so picking a Linux distribution there are couple things to keep in mind

the first one is your workload so what are you going to be using this computer for is it are you gonna be managing Enterprise production environments are you gonna be browsing the internet and YouTube are you gonna you know I don't know travel whatever do you see tf's run VMs think about things like that when you're trying to pick a Linux distribution also different features you want it can be a whole nother talk in itself to talk about desktop environments so I'm not even going to go into it but things like that that overtime of using Linux you'll start getting your own preferences and things that you're comfortable with on your own and the Lessing is keep in mind your

skillset um some people do learn by installing arts first thing on the first day of using Linux and can just pick things up I don't know it's magic but in general well take baby steps learning Linux so keep that in mind you don't have to retake your laptop every time we probably already know this in here run beams if there's a new Linux distro that's out throwing a VM and try it out plug for LS challenge right now yes if any of you are thinking about choosing Linux I challenge you to join me every single month I kick to a brand new Linux distribution I started out with elementary OS when to open SUSE a

apparently running pop OS and the next one we're running is clear Linux do it as a group because we can support each other when we break things and we will break things yeah and also if you're already comfortable Linux keep learning it um don't lock yourself into one distro have an open mind about it because if you're like me you'll go into your career being like oh I learned on Fedora I support Red Hat that's all I know and then you get hit with a layoff and you go to in a bun to shop you can pick it up but it's kind of like setting yourself up for a learning curve yeah um so lessons we just want you to take away

from this is use the right tools for what you need work smarter not harder right um and don't lock yourself into one distro make sure you keep an open mind and keep learning of course there may be just rows that you're like I hate art I hate pac-man Oh gross but at least know I know oh you know if you feel like that layer about a certain distro we all have our preferences but just keep an open mind to so what you want all right it's okay when I started out everyone said use the tools in your toolbox right make use of everything that you had and what I found is the cuvees freaking easy

you touch a button and it works and this became one of the biggest gatekeeping things for me is they're like oh well real learning Linux admin it uses the command-line why and so like everything there is definitely an xkcd comic that goes with it give you guys a moment to look at this and many of you especially linux users are going through the man page in your head right now going back why if you are I have the option of using a desktop then you would go to the file and you simply click the button and it's out where you need it to be yes if you are on a server you might have to learn how

to use all this but one of my favorite things to do at a conference especially a Linux conference is to sit in the background and watch people try to get on the Wi-Fi because you see them committed to nmcli tap tap tap tap and you'll say like hey if you just go to the glute no no I got this look tap tap it gives you the options by that point I click the gear you clicked accepted and I'm on the Wi-Fi reading definitely goes back to the work smarter not harder and yes use all of the tools that you have available to you in your toolbox you know if I asked you to come over and

help me attain some frames yes I mean don't recycle all of the tools don't become I only use the GUI because then you limit yourself to not have the functionality that Linux gives you get there hold on stop with the spoiler alerts it's another slide but yeah if I ask you to come to my house and help me hang some you know pictures and you showed up with a screwdriver like all right I mean we can do this you can get the screwdriver and sit there and do it or the same way if I asked you to help me with my pipes then you showed up with a hammer I mean alright we can get it done but

this is just silly you're just trying to prove we don't have to prove you're impressing no one by yourself at that point which gets us to confession number four everybody in here already knows this confession technology sucks sometimes um there are lots of days where I really want to shut my laptop quit my job move to the mountains raise goats um you know have the local children believe that I'm a witch and never interact with anybody or technology ever again and that's fine we all have days like that to make my point I'm gonna tell you a story about when I was a baby Linux admin and I had two monitors that worked fine with fedora one day my boss

decides like okay you're not a baby Linux admin you're worthy of that third monitor so I'm like okay when I guess that's how you know that's a promotion yeah I got an owner I'm like cool great um well I thought I could just plug it in and it would work I spent six to eight hours probably fighting with xorg I cried my mentor was like how do you have a job here it was a bad day right so um I solved my problem honestly by switching distros to arch yeah I switch the arch that day that is the day that I switched Arch Linux honestly but it worked it just worked in that case which

kind of ties into this because when she told me the story I related to it you know most of the time when I'm trying to do something with Linux and it doesn't work I blame myself I'm like this is my fault I don't know I'm not technical enough you know I really should just buy a Mac and use what everybody else uses and when she told me the story I looked at her I was like you know did this happen kind of like at the end of 2015 kind of era because I remember this if Fedora pushed an update that broke the xorg files and it meant everyone on my road lost their third fourth and fifth

monitors but we weren't on the same team she wasn't able to get that communal experience and I got when I was doing the Denix distros and the challenges and I kept breaking things and at the time I found a guy by the name of Jason who does program called choose Linux and we started doing distro hopping together where I started documenting every change I made on a new distro so that when someone says oh well what you do you know how to break it it say oh look at my files my error logs you there every single command I've run on the command line since I got this boxes on there and it got to the point

where we ended up in a pretty big flame war if you follow me on Twitter with a major Linux distribution where they were pushing back that it was my fault I pointed them to the logs and we got some good changes going on in that actual distro but this happened because we did it as a community we did it as a group it when I said this is broken someone said oh it's broken online - and push their logs up as well so we challenged you that if you're going to do it don't always believe that it's yourself take a chance try new distros try new applications learn something new but do it in the OS challenge mindset do it

with a friend find a mentor there are many times where I'm like hey I want to learn how to do CTF Ally like can I have you on standby and I'll go do something and it won't work and she can jump on and help me troubleshoot what was going on - see is it me or is it that environment that isn't working you'd be amazed how many times it's the environment that's not working okay so next confession a lot of times when I have a question that I feel like the answer is something that I should already know I hesitate to ask that question and I'm sure that everyone in here has probably done this about

something but I just want to tell y'all it's okay not to know it's okay to ask questions I'm sorry but it's so good the confession so you know you're kind of cheating yourself you know a lot of people say there's no such thing as a stupid question but like L said earlier sometimes you'll say I don't know what SSH is can someone explain that to me and if you're in a room full of Linux people how do you not you might get a response that's like you know now what SSH is yeah like what are you doing and it's just like a crappy feeling right so it's okay not to know and it's okay dumb questions it's okay to ask the

simple questions because even giving this talk I realized if you're thinking a question that you probably don't know the answer to that you perceive as basic there's probably someone else in the room that's thinking the exact same like I don't know what this is so when not knowing the answers we all know it's a matter of Google foo like al likes to say as you progress throughout your career you will get better at googling things and pointing Google will probably point you to some documentation of whatever product you're struggling with the documentation is your friend and I would encourage you if you get you know an urge to give back to the community start with documentation

that's a great way to start contributing if you see something that's outdated on a community doc update it submit a pull request which leads us to 5a and it's the one that I hate giving cuz god I hate getting it as a response and that you know what guys sometimes we really do have to read that friendly manual yeah when I showed you guys the picture of the xkcd comic and I've actually seen this I swear to you I'm giving the talk somebody with your phone and through the table googling what the options are and like you know if you just go to the man pages and especially if you're on a terminal do backslash example you'll get

exactly what you need there are people who dedicated their careers to documentation who have spent so much time trying to make these tools easy for you to learn yeah you can go to stack overflow in four and a half hours later you'll have sixteen different ways to do something or you can go to or you can just go to the documentation for the product that you're using and here's the thing many of you're like oh I try to go to the documentation and the answers not there cool did you update it when you found the answer we have to write the docs we have to be willing to give back in that way and so our lessons for you

here is yes Google is your friend but so is documentation and I encourage you if you've had to do something twice and you didn't find the answer go write it for someone else you're not the only one that's wrong with us and it's the only way that we can truly grow an open-source community okay combustion number six this one makes me nervous so and a lot of times when job hunting and things like that I feel like it gets my profiles looked up but I don't feel like it displays Who I am as an engineer fully you know people walk up with these certs all the time and it's a cool label to have right like oh

wow he's an RHC a great um but do you know what you're talking about beyond the cert when you come on with Linux on the floor we can smell our own right so if you walk around with an RHC si for example that's great so we may know you know how to do with some basic Linux stuff but do you actually know what it's doing or did you study for an exam so that leads us to our next one which is the fact that you know going to bootcamp which is what a lot of people do it's great it can give you a start but it's not gonna make you a command line ninja and many times when you have that person

that has that certification and we think it on the floor eyes a li said you can smell our own they've been becoming a story that we all have and so I'm gonna share with you my story about the day that I met the world's very first OpenStack Java developer now if you don't know OpenStack it is 100% written in Python that's all that it's written in so I'm go in I introduce myself and this guy is standing so tall and he's so proud and he tells me about how he just graduated with a degree in OpenStack Java development now that's nice I'm an openstack trainer and this guy is just so confident that I'm under the

phone going texting the foundation going guys what did I miss like I miss an email like this is huge it's is going to change my job and then the texts are coming in with l-o-l you're being pumped seriously all you can't tell sky is so confident so I'm talking him I'm trying to pull it out and then I find out that he went through a bootcamp and got a certificate of completion in full stack Chavez you only go as far as you actually dig into the technology what will set you apart is that hands-on time because you know what yeah I came from a certification from a boot camp certification program and I can tell you

that my day started at 6:00 a.m. and went well past 1:00 a.m. I got very little sleep and I spent as much time behind the command line and behind the GUI that I could which phases to our confession 16 so neither of us have a computer science degree um we both agree that your knowledge is what defines you didn't mean this quick set yet to make this point we're going to tell you three stories really quick so I'm start with mine I'm a high school dropout I'm a double college dropout never completed any like formal education everything I've learned has been on the job through mentors through the computer community from the internet and that's about it

I'm a product of my mentors as far as my linux knowledge goes so yeah this is the perfect place for me to make this confession Mary's University we're from still high I have a college degree in Christian theology I went to school with great hopes to go and be a missionary and I graduated right about the time the Catholic scandals occurred with the sexual abuse and everything was not the right time to become a missionary and you know what I needed a job and I needed something to do and that's how I found myself in the Linux world but now I absolutely no formal training sorry camera guy I walk when I think we becomes to Linux however one of my

mentors Erik Joslin is amazing and he does have a computer science degree I believe he actually has a master's and what makes him amazing is when I ask him a question and I want to know the deep dive I want to know how those system calls are actually occurring for me to be able to understand he will show up with one of his college textbooks ten years after the fact he can turn to the right page show me the highlighted section and start talking like he was just studying this yesterday he became a product of an education that allowed him to actually dig beyond the command line he kind of started memorizing his books ebooks became a part of him and it was

just amazing to me as they like okay you don't you didn't just read it you actually understood it so whether you decide to go through a certification program you decide to teach yourself our lessons for you are pretty simple and that is nothing beats hands-on experience like you can cram for a test all you want and you can go and copy and pasta and basically SERP dump and it'll get you the job but like Alex said we can smell our own and you don't want to become somebody's story and also what defines you is not just what you know but how much can you impart I always tell people it's cool that you know that

but I don't believe you until you can teach me or you can teach others give back to the community nobody at me for this confession okay I used to do I used to use a Mac to do my Linux assignment work I didn't hate it as much as you might think so I was at a point where I don't know my workstation is kept breaking for me when I was running Linux no matter what distro right I just got sick of this struggle where I'm like man I have to come in and troubleshoot my workstation in order to do work that is so annoying and I just got enough seniority with Rackspace where I got an Apple computer

one day basically it was really random actually I can't blame art for that one actually 100% artists well no but I didn't hate it and I wanted in here cuz I think it's important you don't have to run Linux necessarily to be a Linux admin that's a great way to learn Linux sure but not all Linux admins and engineers obviously run Linux and that's okay that's totally ok I think which kind of brings us to the point made earlier that stole the thunder of my favorite Linux distribution is currently Windows that's because there are some amazing things being done with the Linux subsystem when it comes for Windows it's gone so far beyond PowerShell that I can tell you

that I am a complete noob to it but I can tell you that if you're interested in making money in this industry knowing Linux but being able to use it in a Windows environment down to a docker eyes din vironment when it comes to it windows containers for Linux Linux containers for Windows you're writing your own check especially if you can do that on the cloud with a sure we need to stop getting towards the point where we say we say we need to get to the point where we say we are system admins whether that be for a cloud environment whether it be for adjure whether that be for Windows whether it be Linux and I

can do it all on my Mac or my Chromebook and it doesn't matter it's your skills have I said that enough guys but we are going to go on a side note and a side rant here and that's something that really kind of becomes an obstacle for a lot of people because they say well you know I can't go run a Mac I can't afford one well let me show you this example of this guy a great guy by the name of Richard and he teaches a school in Ghana and he wanted his kids to be able to be prepared for being able to use things like Microsoft Word but they didn't have electricity they didn't

have computers so every morning he would come in and he would draw what a word environment looked like for the PowerPoint environment would look like and he would walk his kids through using it on blackboard this is one of those problems in it I think I was talking to somebody earlier about where we try to solve all human problems with tech and so Microsoft found out about Richard and they're like man he's doing some great things so they packed up a bunch of computers and they shipped it to him so that the kids could now learn how to use Microsoft in real life did I mention the part where I said the school doesn't have any

yeah so once again yes I do say that hands-on knowledge is what's gonna get you there sometimes we have to get creative on how we get that hands-on knowledge one thing that we talked about is when I started out in this industry I told you guys that I went through a boot camp what I didn't tell you is that I don't come from a technical background my very first computer that I got was what they handed me at Rackspace when I was hired I learned Linux through talking the local library to letting me use their VMs through borrowing friends computers through sitting there with my clunky a book and drawing things out trying to understand it really the only

two minute limitation you're gonna have is your own creativity so we have a little bit of advice for you guys so Ella what do you say for those I'm hearing when I get started with Linux for most choose Linux now she did say that you don't have to retake the computer to it but if you can it really is the best way to get started the way that I mentor people is if we have two computers one can be your daily driver the other one you've chosen one extend every single day pick up the Linux box and use it to the point where you've broken it do as much as you can then you can finish your day off you can go back

to your regular computer finish your day at work and then come back and learn how to troubleshoot no matter what your job is in the tech industry your troubleshooting skills are gonna be what gets you through so why not get them actually getting that hands-on experience by choosing whatever Linux distro that you want to so after you've chosen Linux next piece of advice we have for you would be to I put ansible here but it doesn't have to be ansible just learn some automation I say this because this great DevOps word right that's where the industry is going you're gonna get in the Linux pot and you're probably gonna have to write some automation whether the ants will puppet

chef whatever it's probably gonna happen so once you have that down this is the worst piece of advice for me to give them I give it to you because I need to take my own advice and that's learn a scripting language that will get you through and I always get made fun of because I can write on batch script it'll take me a thousand lines to write what you can in two lines of Python but you know what my script worked and that's what matters but realistically if you're gonna be involved in the tech industry regardless of what you're doing scripting can help you especially when it comes down to actually troubleshooting the little bit of

scripting that I know has really helped me be able to parse logs better and to really be able to understand what it is that I'm looking at I'd love to tell you that your environments are never gonna break but I don't care what distribution what OS you're using at some point if you're having fun and you're learning new things you're gonna break it just like you should people of you Greybeards in the room yeah all the time I'm a Linux master um so fine you can use arch oh these are not dating oops yeah all right all right I'll just read to you yeah so a lot of connection that's great okay so just keep learning Linux even if

you're running Arch even if you're a Linux genius I would suggest keep learning you know some things you can do is things like the OS challenge with L that bounces to different distros keep your mind open try new things um but also I would suggest trying to build your own Linux that's a challenge for sure and once you feel comfortable with Linux especially on like a senior level I challenge you to give back to the community and spread that knowledge that really is what's gonna define you as a Linux engineer and for those InfoSec people in the room and only how would you suggest yeah so I have this really cool slide that you can see with all of these CTF

platforms that used to be trying to fix it okay maybe no we lost connection to the server that's fine so okay 50 top InfoSec Newey's I got into things like this by playing capture the flags um Brendan I'm still ops engineer at Cypress place so I kind of took the approach of going into the industry by doing a job that I know how to do already and that way I can like learn from my co-workers who are doing threatened and stuff like that so playing capture the flags does everybody in here know what a capture the flag is does anybody brave enough to admit they don't know what it is okay we're at the reckless

yeah just kidding so we've got like eco CTF we've got half the box over the wire things like that you can Google like CTF platforms and get that information Ellen might have played a few of these if you're the kind of person who likes to learn in a group I would challenge you to try and join a CTF team for the women and those who identify as women a non-binary um girls taking over is an awesome group there's also cult of the party here and open to all CTF those are both CTF teams that are awesome and you can just like follow them on Twitter they'll probably add you do a slack channel that's how

girls taking over does it and you know you can just kind of collaborate and if you want to do a capture the flag you can I will note though that if some of you are like well I'm really new I don't know what I'm doing I've never done a CTF hi Ellie you know hey I started studying for my security plus I want to do a CTF and every girl in that group is like you should join hey how are you doing how can we help you so most of these groups are extremely inclusive regardless of where your tech skills are if you're brand new if you don't know anything joined anyway like it's bought

it you know people who already know everything in there are excited to teach you about whatever we're doing so I encourage you to join in it even if you have no expertise or knowledge here definitely Oh plug the CTF time if you have like a free weekend and you're like I want to do some CTF practice this weekend you can go here and it'll show you what remote enabled CTS are going on at a certain date yeah and we're kind of coming to the end of our talk so we want to give you the final lesson and that's something that you've heard me say quite a few times today and that's you know what you're gonna break some member

keeping this family friendly because my kid is in the background so we're gonna break some shirt and you should like I don't know where that came to be this communal knowledge this communal agreement that you should always do everything perfectly and you should be good from day one because you know what when you're going through it's hard and you're not learning if you're not pushing your boundaries so both of us are gonna stand up here and we're gonna have a moment of vulnerability and we're gonna tell you about a time that we broke a shirt yeah so mm like what probably a month ago right now Elle and I are sitting on a coffee shop talking about this talk that

we're giving right now I go home after I've got to do a maintenance so I wait and I go in log in to do my maintenance what do I do I accidentally delete a database note okay we figured it out I figured it out I can GUI I felt that so hard so that's her boss yeah so I I blew out a database note and so I that's the face I made when this happened and this database node was gone from the console all of a sudden that's that's my boss's reaction literally I told oh I just blew up this database know what i'm candi-cane he's like oh man you know if that's my fault

I should have had termination production on rebuild it we got backups let's just get this back online no one got mad at me no one blamed me they made fun of me that was about it at this other startup I worked at you know the first thing they told me you're gonna break production we're telling you right now it's okay that's when I learned about the concept of fail forward learn from your mistakes own your mistake it's okay but I'm not gonna break production and she broke production the next day yep all right so we go to my story and I can't let her outdo me so I'm gonna tell you guys about how I broke what I did not playing

with it how I broke production the very first hour on my first day on my first job at Rackspace and so they sit me down I like here's your computer and you know they're trying to get me out of the white really because they've got stuff to do and I got hay run this maintenance for us I'm like I don't even know how to like answer the phone what do you mean run this maintenance for us like no no and I quote it is written so well that among could run it well I mean I can't be outdone by a monkey so I get on I pull on my computer and I'm looking at this

maintenance and this is really weird they're like four commands we're updating a database okay something called in my sequel cluster like I knew my sequel I'm not really sure what the word cluster meant but you can't really ask and they're done right so somebody had made the mistake of teaching me that you could pull up a terminal and you could broadcast across multiple terminals so me going why did they set two hours for this like for command maintenance I'm gonna really impress them so I log into each one of these nodes I broadcast across I run the updates and I click restart and there went the customers my sequel live database and what did I find out when my

phone started ringing this customer was a major bank in the UK and how did they find out that their environment was down Twitter was blowing up customers couldn't get to their money the reason that they had two hours allocated for this is this my sequel cluster hadn't been updated in over a year so there were quite a few updates that had to happen and we were down for a really really long time or maybe I just remember it being ours but the lesson that I learned in this is that I was on the right team and that's because you know what happened absolutely nothing happened to me my teammates stepped up and said we screwed up we should have explained this

to you somebody should have sat with you we did this they handled the customer calls and then about two weeks later when I was assigned to another maintenance for this customer and they understandably called and said we don't want L running our maintenance the team stood up for me and said you know this was not an elf fault she's the face of it but we made the mistakes we're gonna sit with her we're gonna train her and you don't get to pick and choose on a company like that and that's really kind of what brought us to come and show our vulnerability here and if you guys have seen this little penguin around this is really what I'm living my

life by the concept that it's okay to being you that it's okay to fell forward and you know what you don't have to please any of the neckbeards in this room because this journey is your own with that we'd be happy to answer any questions or what usually ends up happening is joining you in the hall and having you guys tell us about the time that you broke production thank you very much you have to keep growing I've heard that multiple times yeah them I am a vampire said hahaha I don't think I have gone into eMac twice and then having to Google how to get I will own that I've done it probably actually more

than twice I never remember

[Music]

there is what I'm sorry okay so my favorite distres are those who keep up to date up documentation like cop OS goon - is pretty good I have my issues with openSUSE they say that their documentation is up to date I can prove that it's not however most of most of the times what I end up doing is I actually end up installing the man pages if they're not available I'm a man page user that readability is easier for me I think you said you use info more that's one of the great things that we go with open source is you're not limited to the tools that you downloaded that distro with honestly I don't know if there is a

master list that tells you why or how they're going that way I think it really is down to the developers and what they want to include but I mean we open source for a reason right like you can get whatever tool you want to get I more than once have gone to Google and typed in man and gone to the man pages

we so um one of the linux distro one of the links one of the podcast we run is bsd now so yes I know those debates quite well and I'm saying you know what at this point in my career that's gatekeeping cuz I'm not there yet I'm just kind of playing with it and trying to figure it out and one day I will run as BSD distribution but I'm not there yet you know my daughter's back there she's writing a talk right now about Linux sorry I'm throwing you out there about Linux through a 9 year olds just and we sat down and talked to her about she's running her Chromebook you know what she's running Linux for all intents

and purposes for all she needs to know Cheers so both of our moms run Linux it's easier for me to troubleshoot it

sorry anyone else don't forget to get your stickers hi thank you guys [Applause]