6th Annual BSides Asheville Information Security Conference Live Stream

Live from the U.S. Cellular Center in downtown Asheville, North Carolina, this is the sixth annual B-Sides Asheville Security Conference live stream. We'll be getting started shortly with Jack Daniel on survival skills for hackers and security professionals. Then at 9.30, Paul Burbage on using stored cross-site scripting to beacon malware adversaries. Then at 10.30, Jason Gillum on the new normal, assessing modern applications in today's environments. Then at 1130, we will have Katie Murphy on Spoof Proof with DMARC, the Domain Message Authentication Reporting and Conformance Protocol. Then after lunch, we'll be returning at 140 p.m. with Michael Kernow on Cyber to Physical IT Convergence. At 2:40, we'll have a presentation by Eric Krohn, "What You Know, What You Have, and What

You Are: Multifactor Authentication in the Modern Age." And then finally, Wesley Smiley will be presenting on "I Can Build That Even If I'm Not Going To." So sit back, get ready, because we will soon begin.

Live from the U.S. Cellular Center in downtown Asheville, North Carolina, this is the sixth annual B-Sides Asheville Security Conference livestream. We'll be getting started shortly with Jack Daniel on survival skills for hackers and security professionals. Then at 9:30, Paul Burbage on using stored cross-site scripting to beacon malware adversaries. Then at 10:30, Jason Gillum on the new normal, assessing modern applications in today's environments. Then at 1130, we will have Katie Murphy on Spoof Proof with DMARC, the Domain Message Authentication Reporting and Conformance Protocol. Then after lunch, we'll be returning at 140 p.m. with Michael Kernow on Cyber to Physical IT Convergence. At 2:40, we'll have a presentation by Eric Krohn, "What You Know, What You Have, and What You Are: Multifactor Authentication in the

Modern Age." And then finally, Wesley Smiley will be presenting on "I Can Build That Even If I'm Not Going To." So sit back, get ready, because we will soon begin.

Live from the U.S. Cellular Center in downtown Asheville, North Carolina, this is the sixth annual B-Sides Asheville Security Conference livestream. We'll be getting started shortly with Jack Daniel on survival skills for hackers and security professionals. Then at 9:30, Paul Burbage on using stored cross-site scripting to beacon malware adversaries. Then at 10:30, Jason Gillum on the new normal, assessing modern applications in today's environments. Then at 1130, we will have Katie Murphy on Spoof Proof with DMARC, the Domain Message Authentication Reporting and Conformance Protocol. Then after lunch, we'll be returning at 140 p.m. with Michael Kernow on Cyber to Physical IT Convergence. At 2.40, we'll have a presentation by Eric Krohn, What You Know, What You Have, and What You Are,

Multifactor Authentication in the Modern Age. And then finally, Wesley Smiley will be presenting on I Can Build That, Even If I'm Not Going To. So sit back, get ready, because we will soon begin. ♪

♪♪ ♪♪ ♪♪

Up next, Jack Daniel with his opening keynote, Survival Skills for Hackers and Security Professionals. You. But first, presentation. I'm one of the folks that kicked off this thing 10 years ago. You are here. If you enjoyed the entertainment and cocktails and beer in town last night the way some of us did, maybe you need this little refresher. You're here in Asheville. But you're here and you're also part of all of this besides across the US and southern Canada that have happened. And I need to update this a little bit, but that's pretty close. You're part of this movement, which as it stands today, this is event number 523 since a few of us got together and

did a little one-off event in Las Vegas at the end of July in 2009. Events have been held in 157 cities in 46 countries since we launched something that we didn't know what it was. And As of right now, when I finished this last night, there were 39 more B-Sides scheduled for the rest of the year and several more coming along. Two people gave me dates overnight. So there are 41 more that will be on the calendar in the next few days. And then we'll end up with over 100 events this year. And if you look at this, yeah, Australia, New Zealand, that one's going to actually happen. Myanmar, what used to be called Burma. It's all over the world. And so I want

to talk about the founder circle. I don't want to make fun of any other conferences that do these sort of things, but you don't grow like this without having a growing foundation. And so whether, like me, this is your 80th or 81st B-Sides event or whether you've never been to an event before B-Sides or otherwise, you're part of this community, which is this community in the room, but also a global community of people who are interested in learning and sharing and being part of sharing what they know, learning and getting together and I don't know, I don't want to get too idealistic here, but it's kind of amazing. And this is One of the great things about actually being here in person is that there are actual people here.

So is this anyone's first time coming to a B-Sides? Cool. You said your first time? My name's Jack. - Michael, nice to meet you. - Nice to meet you. Hey, you know, I used to be technical. I'm not anymore, but I still know a little, and I know some people. I'm easy to find. I mean this absolutely. Reach out. If you get stuck anywhere, reach out. I'll probably know somebody that knows something if you need a connection. That's the most important thing you can do today. Seriously. You can make a difference in your career, their career, their life. Some of the people in the B-Sides community are closer than my family ever was. And some of the people in the B-Sides community have

amazing career success stories, project success stories. We don't share data well enough, but a lot of times, you know, our best stories are often under NDA. Right? But that doesn't mean you can't pick up the phone and it's like, "Hey, I work for a regional bank. You work for a regional bank. Remember B-Sides we met?" I can't tell you much, but I would kind of look for this thing in your network and maybe cut the wire to the ISP if you find it. Right? Right? You know, that's it. So anyway, the actual talk, survival skills for you and your team. By the way, this laptop and PowerPoint and Keynote and things and I, we spend so much time together that my digital

equipment heckles me. But the point is, yeah, let's go back to that. Come on.

If everyone in this room never suffers stress, nobody on your team suffers stress, nobody that you work with or for or works for you suffers stress, then you won't get a lot out of this. And that would be ideal. However, I've heard that sometimes our jobs are stressful. So why this topic? I have an amazing job. The job I have, I've been with Tenable eight years. This is part of what I do for Tenable, and you'll note I'm not talking about product or services, I'm talking about you and me. But there's still some stress. Right, come on. Hello, this is Denny, in-house psychologist for Norton Corp. Denny, I'm sick of typing. I want to quit,

but I don't know how I'd support myself if I did. Well, ask yourself this question. What is it that you really enjoy about typing? Are you still there? I don't like anything about typing. Every morning when I get ready for work, I start crying. What's wrong with me? You hate your work. It's normal. Have a good day. Uh... So there we've heard in-house psychologist and HR team. But life's more important than work, and there's more to it than work. At least, you know, there should be. So we need to be prepared. We need to be prepared for ourselves, friends, families, employees. And here's a quote from Dan Gere many years ago. Security is too wide to master, too deep to know, too fast to photograph. Let me emphasize

that Dan Gere said this, right? If Dan Gere thinks that, you can be forgiven for not being able to keep up with the pace of things. Speaking of stresses, several years ago, Joey asked about DFIR. I was starting to weigh on him. Do you work in DFIR? Have you ever done a case that bothered you? 25% said yes, 13% said yes, scared me, 22% no, and 40% said I drink a lot. That's a thing. So the threat landscape summarized in the immortal words of Bob Kimmel. Everybody must... So, you know, I mean, we all have breach fatigue. The press has breach fatigue. So where does that leave us? Well, especially if you're a defender, it leaves us here.

And, you know, that's the good days because some days we feel more like this. If... Okay, it's early, it's, you know, whatever. How about this? We'll do the cat. So how do we cope? And it's not just about coping. We don't want to just survive this. We want to thrive. We want to be, we'd like to enjoy life. That's a crazy idea, right? The goal is more than not to be lying there in the hospital bed listening to the machines that go bing. I want to enjoy life. So, the disclaimer, read it with me. Seek real expertise for real problems. I'm not an expert. Amateur guide at best. I occasionally get lost myself. I'll say this about

the professionals. Some of them are great. It's not only our industry where some of them are not. Something about stress and memory loss. Oh, yes, here's a thing that I've learned through life, and it's not just job stress. You get hit with walls of stress, right?

Divorce, marriage, babies, death in the family, work stress goes over the top, even if it doesn't reach some sort of critical portion. It has a profound impact on your memory. And every time you go through a profound stress cycle, your memory doesn't come back quite fully. The older you get, the less likely you are to recover all of your memory. I live by pen and paper. Yeah, you can take notes on your phone or your computer or whatever. It's been proven over and over and over, and I know it personally too. Scribbling stuff down on a piece of paper makes a difference. It would be great to not burn out your memory so that you don't have to write everything down and leave the post in

your office when you're standing there in your bedroom wondering why you walked in and you have to walk back. Forget that there's a post-it there 20 minutes later you look down and post it says what you were supposed to do So I've heard this could be frustrating Imposter syndrome. This is a cool thing. This industry is full of brilliant Motivated passionate people and a stunning number of us don't think we belong here me too says the guy in the funny outfit on stage Sometimes I'm a little funny I used to know stuff and This is crippling if it gets out of control. So, you know, coping skills, whatever. There's also this one. There are a few people I've heard in the industry

that are on the other end of the Dunning-Kruger spectrum. I wish they would shut up. They won't. The right people in this industry and in life don't have imposter syndrome. Let's just leave it there. So I mentioned three words. Exhaustion. Personal efficacy and cynicism. These are the three clinical buzzwords for clinical burnout. I want to stay away from them, but I want to talk a little bit about how they work because it doesn't just apply to the clinical burnout. Personal efficacy. Do you feel that you're effective at your job and productive? That doesn't mean you have to think that as an industry or even within your team or your company that you're winning a battle. but you're doing well and you're doing stuff you

feel that you can be proud of and you're satisfied with your work. That's really important because when you lose that, the other two things happen really fast and really hard. Exhaustion is not just physical exhaustion, but it's emotional exhaustion. Cynicism. So here's one of our challenges with cynicism. To be good in this industry, you need to be kind of skeptical. It's like, eh, eh. That's not going to solve all my problems no matter how many pizza boxes I put in the server rack. Somewhere somebody has to know what it is we do and why for this stuff to work. You need to be skeptical. And it's really easy to slide into cynicism. And what we've seen happen repeatedly is somebody feels like they're doing a good job, they have

a major breach, or their entire team gets dissolved or something that crushes them, and they go from feeling good and they're tired at the end of the day, but they're satisfied, and they feel like they're making a difference, at least a small difference, and then something pulls their faith in what they're doing at work out from under them. The other two collapse, and it just spirals. So those three pieces go together. A word about cynicism, because I can be pretty cynical at times too. I feel seen here, because this is straight up me. I'm the most cynical idealist you'll ever meet. or idealistic cynical person, whatever you want to say. So years ago, some of

us, almost 10 years ago, did a study on burnout. We did a survey, and then we did a more formal study, not up to academic rigor, but eh, One of the interesting things was half the people had no indications of burnout at all. They were just happy people. When I started looking at survival skills, I thought, "Hey, let's go back to that survey and let's look for patterns in the people who are happy. Let's see if there's an age range, a gender range. Let's see if any of the demographics or anything else tell us." I looked for patterns and got this. Absolutely nothing. There were no patterns. Some people just had good situations and have

good coping skills and didn't. So before I dig into that, you know, there are a lot of people often, some people often dismiss burnout. We tend to be well paid. We tend to have job security if, which is a lie by the way, you have to have the right skills and be willing to work cheap enough and where people want you, even though what you do is you go in and you sit in a cubicle or office and sit in front of a computer, often a corporate laptop, which could be anywhere on the damned planet with an internet connection, but they want you in the cubbyhole. So anyway, we're not getting shot at, usually. We

should be happy. And that works for some people, but it doesn't work for everyone. Let's see if anybody else agrees burnout's a thing. Washington Post, Association for Psychological Science. This was a talk based on a book that Alexander Michel gave a few years ago. Hears from USA Today. World Health Organization says it's a thing. I'm not going to stop and read all these things. Let's see. CDC has looked into the effects of workplace stress. That's the paper. That's the year. May of 1987. This is not a new problem. I'm not the first person to talk about this. We were one of the first groups in the InfoSec industry to do it. Here's one just quickly.

By the way, if... If anyone wants these slides, don't try to take notice. Just ping me. My email address will be up and I'll be around for the rest of the morning at least. I'm going to try to stay for the day. So you don't have to try to scribble these things down. But quickly, the personal and organizational impacts of burnout, alcohol abuse, health breakdowns, insomnia, because none of us wake up at 3:00 in the morning after getting to sleep at 1:00 with things torturing us, right? Marital problems, mental illness, organizational accidents, thefts, turnover, absenteeism, disability, Unpreparedness, lack of creativity, sick leave, job dissatisfaction, disloyalty, blah blah blah, right? Here's a bit that's kind of interesting. 1987, Centers for

Disease Control said, "Legal researchers are concluding that you can't, as a manager, ignore this anymore. It is a legal obligation to take the mental health of your employees seriously." Earlier this year at the RSA conference in San Francisco, my friend Josh Korman sat down with Dr. Christina Maslach. Dr. Maslach is the foremost, well, she's semi-retired now. She's a professor emeritus, but she's the one that came up with the clinical definitions of burnout. She's the one that rallied the psychology industry around the term burnout because it was pushed by the people she was interviewing. They didn't have a term for it, but when somebody said burnout, she started asking at the end of the interviews if that was it, and everybody was like, "That's

it." They had a little discussion. It's up online. I've grabbed a couple of clips out of it. So let's hear Dr. Maslach talking to Josh for a minute. So burnout, what we've discovered with all of this research over the years by many people around the globe, is that it is a response to chronic stressors, in this case in the workplace. It could be elsewhere in your life, but chronic, let me underline that. That means everyday stuff. It's not just a crisis. It's not just the emergency. It's the things that happen all the day that begin to wear you down, erode your soul, as people talk about it, take away what it was that you thought you were going to do. and turn life into not a good thing.

- To trivia a bit, also Dr. Maslach had a boyfriend whose name I forget and you probably do too, but if you've heard of the Stanford Prison Experiment, the professor that led the Stanford Prison Experiment was a victim of the mindset that he discovered in there himself. He didn't realize how bad it was. his girlfriend said, "You have to stop right now." They've been married ever since, but I'm not gonna say that, yeah, well, somebody was sleeping on the couch until he came to his senses, that's all I'll say. So she's had a concern for people's welfare for a long time. The other thing I really want to emphasize is that what a lot of the research is also showing

is that, you know, people say, well, is it the person, something about them that makes them burn out, or is it something about the situation, the job? And overwhelmingly, research is saying it's the job, it's the environment, it's the context in which you're working. So that's a key point. If you're really suffering work-related stress or you or people you know are getting into the burnout extreme of stress, it's the job. And coping skills become important, and sometimes you can't cope and you have to get out. We'll dig into that a little bit more. But here's one of her favorite ways of describing this. That's a cucumber. This is a pickle. What's the difference? The environment that it lives

in for how long it lives there. Now, if you like pickles, this is cool, but not all environments are conducive to tasty snacks. As a matter of fact, there are environments where the company thinks they're doing you nice things by providing you tasty snacks to try to compensate for a horrible environment. It doesn't work. Then you eat too much and have to go to the gym and you go see Cruella or whoever your trainer is and they make you suffer. She really prefers me to call her Kimberly. So moving on. Who wants coffee? It's morning. Here's one of my stories. So you're at a happy little coffee plant there in the mountains and it's like, you know, and then Juan Valdez comes and picks you. It's like, oh, this

is interesting. And the next thing you know, you're being shipped across the globe and put into a drying vat and then roasted, and then you're ground and subjected to hot water or possibly extreme pressure, extreme heat and boiling hot water. Does that sound like your work? Extreme pressure, extreme heat? Yeah. And then what comes out can be a little magical elixir or a bitter, nasty thing. And depending on your interest in bitterness, you can add creamers and sweeteners and flavorings and other things to it to make it a little more palatable. And that's sort of the coping skills, right? Your situation may yield something, and maybe you like the pressure of the environment, and that's cool, but maybe you need to add a little sugar and a little

cream or a little Frangelico or Baileys. Just a little bit. By the way, the steamer wand on those things makes great steamed milk, and I'm not saying that a little amaretto or frangelico makes a really nice evening drink. And so here's this burnout thing, but the whole point of what we're going to talk about is staying above the red line. We have all burdens. That's because we're alive. Being dead, burdens aren't your problem anymore. Coping skills, the goal is to stay above the red line. So let's talk... about ways to cope. And so I asked a few years ago, I asked a lot of, I asked people on Twitter and because Twitter is the right place. And so I got some amazing answers. Actually a thousand people

sent me answers from a couple of words to long emails and phone calls. I was amazed how many people took the time to share. Some of them were more insightful than others. This one, Yeah, so anyway, a few things came out really quickly in these conversations, which I hadn't really thought of until that point. There are big stresses and little stresses, and your reactions and coping skills matter. Because if your reaction to day-to-day petty nonsense is to change jobs every year because you have no coping skills, that's no good. But if you are really good at coping skills and allow yourself to stay in the same environment, even if your role advances for, you know, just to pick a number, 22 years,

10 months, and 17 days. I used to be in the car business. I'm a displaced auto mechanic. I stayed where I was long-- 22 years, 10 months, and 17-- I stayed about 22 years, 10 months, and 10 days too long. But anyway, if you have really good coping skills, what you'll do is you will apply all of these coping skills to an untenable situation, a horrible situation. And sometimes the right thing to do is to just make a leap. Sometimes making a leap is not the right thing to do. So, manic data, common answers, because, you know, Twitter, must be right. We got ones that you would expect, how do you deal with stress? Badly, not always well, not well, poorly, cry, rant. Into the real answers,

there's a whole bunch of stuff. Second most common answer I put under the bucket of do stuff, preferably outdoors. Maybe this is hitting the gym, maybe this is running, maybe it's camping, maybe it's fishing, maybe it's playing with your dog, maybe it's, you know, Playing with your cat, whatever, pets, family, whatever, especially outdoors. Get up and go for a walk. When I was working top tier support in a firewall company, the stress would get to a point because there were a couple of parts of things that I knew better than anybody else in our team or in the company. Open VPN used to be a thing I knew inside and out. Now I have to ask the internet for things I used to know. I would get really stressed

out. What I would do is I would get up and I would walk 100 yards. to Starbucks get coffee, which really calms you down, and then walk back. But at least I got out of the place, you know. But do something, preferably outdoors. Get outdoors. Music. Maybe something mellow and cheesy like this. Maybe you like something with a little more edge to it. I don't know what I'm talking about. I don't know what I'm talking about. Sonic's Strychnine from 1965, in case you're wondering. That's precursor to all punk and grunge. Yeah, if you're not familiar with the Sonics, almost all of their music sucks. It's horrible, horrible. They have a couple of decent songs and a couple of killer songs. But anyway, Strychnine, or whatever it is,

you know, that's... A lot of people, you know, you listen to music while you're working, whether it's, you know, Scandinavian death metal or whether it's some sort of transfer, that really helps. A lot of people like to create music, to play music. It's very cathartic. But music was one of the really common ones. And then some music kind of leads us into the next one. Where did that go? Come on. Yeah. Somewhere in there is computers. You can get wound up or not. I'm not going to get wound up. Somewhere in there is the Ramones. I want to be sedated. I bet it didn't make the copy. Drugs. This one came up. Not a lot,

but it was there. Legal drugs. People had seen therapists, psychologists, their doctors, and been on a variety of medications. More people than I expected were very candid about drugs. Drug use and not just marijuana all sorts of things trying to cope some of the people that really suffered used entheogens psychedelics occasionally to reset their mind and that's a whole whole other conversation if you're interested in it not necessarily because you want a trip but Michael Pollan's book, How to Change Your Mind, is a really interesting look at the way the human mind works and what psychedelics do to reset the mind and its value in addictive disorders and all sorts of other things. Amazing results with terminal cancer patients, as a matter of fact. Kind of gives

them this one trip, non-addictive, not promoting it, but it's a really interesting look at the way the mind functions. because that's a thing. Alcohol, this was the number one answer. When we first looked at this, it was also the number one answer. As I've looked at it more, more and more of us answer the same way, except at least now, progress, we kind of feel bad about it. I don't know how much progress that is. Friends and family.

this is kind of a challenge because they're often the source of the stress too so there's got to be a balance and if you've got if it's a one-way relationship if you were the one that always needs help you're going to find that eventually the people drift away if you're the one always giving help you may find yourself getting uh getting melted down under the burden of other people's stuff empathy is is rough um this is one i've dealt with disconnect and unplug. This is a big deal and it's hard for us. I mean, it's easy for me. I only have two laptops and two phones on me right now. Oh, and a watch that's

got a better, have you guys noticed this? The Apple watches have better radios than the current generation of iPhones. But I can quit any time. If you were more trusting, I would say close your eyes and come with me on A train trip through the western islands of Scotland. Or maybe just a drive down the bridge here, because that's a lot easier to get to from here than from Scotland. Maybe join me for a walk on the beach where I live. The beaches are close. Just kind of wander along the beach. Just get out.

And yes, of course, I'm bullshit because I should have been doing that. Instead, I thought this is a really good presentation material, so I whipped out my phone and recorded these sounds when I was doing those things. So I'm a hypocrite. I own it. Manage the way you go away and come back because I'm not going to pretend that work doesn't pile up while you're gone. It's actually why I use the Apple Watch. This is a... Message triage system. Don't care, don't care, don't care, delete, delete, don't care, don't care. Oh shit, that's the boss I have to pretend to be working. And it helps a lot for me. You gotta find the things that do it. More answers. Mindfulness, meditation, professional therapy. Man, we have

too much stigma. Talk to, I don't care who you talk to. If you're religious, there's somebody in the church synagogue, whatever to talk to. But talk to somebody who knows how to talk to people. Therapists, clinicians, doesn't have to be a psychologist or psychiatrist, but counselors, there are a lot of people that are good at listening. And a good one is just gonna let you tell your stories And unless you're in need of some real stuff, they'll do things like listen for 28 minutes of the 30-minute session and say, "Huh, why do you think you did that?" Or, "Huh, why do you think you feel that way?" And then for the next week, you're digesting this thing. You're like, "I never thought of that. Holy

crap." So professional help can be great. There's no stigma. When I... Yeah. being more productive through normal productivity tools. Be better at your job, more efficient. Pets, hobbies, video games, whatever. The video games can be tricky. My son works from home, desk all day long. Then he goes downstairs at the end of the work day, makes dinner, goes back upstairs, checks to make sure there's nothing on fire at the desk,

The gaming systems and the TV that's almost as big as this screen goes live and then he's watching anime and then flips to his video games for the rest of the night until he crawls into the bed, which by the way is right here, gets up in the morning, hits the bathroom, comes back, goes back to work, and this is his life, and then on the weekend he skips the work part. But video games, if they work for you, I don't recommend this one, but it works really well. Get married, have kids, get divorced. Have your mom die? Have your dad die? Have the woman you spent 40 years with and met in high school die just short of your 37th anniversary? That happened to me. Guess what? You,

as humans, are way more important to me than you were four years ago. Two and a half year cancer journey through clear cell ovarian cancer, saw some stuff, saw some people. I lost my wife two and a half years ago. And I thought I was pretty well prepared. I'd been talking about this stuff, and I actually was. I didn't completely crash and burn. But it's when we talked about it at work, Tenable was very supportive. Like, you know what, I'm going to shift my focus from technology completely to let's work with our technology, but I'm going to double down on the history of InfoSec, the project called Shoulders of InfoSec, some other projects like that

plus this. They're like, that makes sense. I don't recommend it because you don't always come back from that. But think about this, think about some tragedy that would straighten you right out and maybe debate whether or not that high pressure, high paying job is worth it compared to something else. Anyway, general common sense, don't face challenges alone, don't make your employees face challenges, I'm doing good on time, all right. Here's one that's counterintuitive, do more. It's like, that doesn't work. but it has to be stuff that you want to do that is satisfying and rewarding. It's gotta be. Volunteer, mentor, speak, teach, share help, teaching. You wanna learn a topic? Try to teach it. I

don't care how well you know something, somebody's gonna ask you a question that makes you have to dive deeper and deeper and deeper. Mentorship, this is brilliant. You want to see your own bullshit, become a mentor. Because you will have this one. You're headed down, as if people use phones, you're headed down a bad path at work. You can't do this, it's a dead end, you're going to regret it, you're going to close the laptop lid, and then you just put your head down on the desk because... you're doing the same damned thing. When somebody else does it, you've all seen this, you've struggled with a problem, somebody else walks by and you're like, why didn't you do, and they're like, ugh, right? And it happens back and

forth. Outside perspective, when you tell people things, You spot yourself. It's very good. Besides the fact it's rewarding and it's the right thing to do because there aren't enough of us who know what we're doing, we should take care of each other. Learn something new. Education is work, so you have to... Want to learn it there's got to be a reason for it or offer you know hope of career improvement Which you know goes back and that's a whole work thing you know the ancient line the two executives are talking about Training budget and one of them says what if we train everybody and they leave and the other one says what if we don't

and they stay? Little perspective on there by the way it's taken me 20 years to remember to point the clicker at that not that and Identity problem. This is a real issue in the hacker culture. You're not the iPhone hacker. You're not the drone hacker. You're not the car hacker. You're you. You may have a specific skill, a specific job. You are a person. People get wrapped up in that. What happens when somebody decides that your skill with embedded systems, whether they're drones or whether they're IP cameras or something, when they realize that's gonna be really handy for some IoT thing that they're rolling out or some IoT security program, and you no longer do that thing that you built a reputation on, you're still you, you still

have skills, don't confuse yourself for your skills. I like to ask this one. Do you have too many qualified and trained employees and is it easy to hire them when you need them? Yeah, so maybe we should take care of them. But before we get into that, you know what they say every time you get on an airplane right and the mask falls out of the thing and bend over and no um grab the mask and put your own mask on first before if you're not in a good place and you're trying to help other people you're probably going to accelerate your own problems because you're going to take on their burdens take care of yourself first but um dr moslock has six key issues that she likes

to talk about. Michael Leiter has some other ones. They overlap. They worked together years and years ago. There's some other folks. I'll have a bunch of resources later. She talks about these six things, and these are avoiding stress, avoiding getting into burnout. Workload. If your workload is too high, you get stressed. That's pretty straightforward, right? Autonomy and control. This is one of the most important things. Leiter has written about this extensively, Michael Leiter. The more autonomy you have, the better you feel about your work. The more control you have over how you do your job, the better you feel about your work. When you don't have those, things go south. The piece that I've found anecdotally that fits in there too is knowledge and understanding. If

any of you have ever been in a company that's gone public, you know you switch from logic to Wall Street logic. It's not the same thing. And in the ramp up to going public, you do things that don't make Wall Street logic, they make pre-IPO logic. If you've never been through that and you're doing dumb stuff or making people do dumb stuff, make sure you understand why you're doing it and make sure your team understands why you're doing what you're doing. Make them understand the nature of whatever it is. That's just one example. There are all sorts of things that are going on. So you have to do this thing. The thing has to be done, that's the job. But at least knowing where you or your employees

fit in the big picture helps a little bit. Doesn't solve the problem, but the more autonomy and control, the better people feel about their job, the less, the worse they feel about it, and that passes to the way they feel about the company. They leave the company, they trash the company, we go back to how easy it is to hire people. Social recognition, reward, it's not just financial. People that get rewarded socially. And that can be, you know, I drag you up on stage at sales kickoff or something. But more it can just be, hey, I know this was a killer week this week, but thanks. I appreciate your effort. That is huge. What's the community like? Put up with any bullying? Are there conflicts? Are

there strifes within the teams? That'll wear you down at work just like everywhere else in life. You want a good litmus test for whether or not you should be at a company. Is HR there to protect the company alone or are they there to protect the company and help the employees? Too many places, HR is just there to protect the company and I'll stop there but HR, there's been some really amazing statistics come out on things like employees that use whistleblower hotlines or people that complain about sexual or racial or religious harassment at work. People that complain, depending on what study, 50% to 80% are gone within a year. They don't have details on how many quit versus were pushed out, but they just

leave. That tells me that those companies that are that bad aren't places you want to work, which goes into fairness, values, meaning, purpose. What does the company stand for? Yeah, we're trying to make some money. I'm not opposed to money. I like to buy silly, shiny clothes and top hats and stuff. But there's got to be more to it. And in the computer security industry, those of us in vendor space, there are a lot of companies that are just out there to make a buck, and then there are companies that would like to make a buck and make things better. And the ones that would like to make a buck and make things better, those are often pretty good places to work, at least in my opinion. One of

the canaries in the coal mine there is sales engineers. The amount of sales pressure put on a sales engineer is a good way to gauge what the priorities of the company are. Their job title starts with sales, it's there somewhere, but the ones that they're just salespeople with some technical knowledge, those tend to be really ugly places to work. As a manager, provide feedback, continuously constructive criticism, manage workloads, employees need to be able to disconnect, maybe force it, offer education, But back to this, you're not a professional provider of care. You can be a friend, you can be a mentor, you can be a peer, don't take on burdens yourself. This is one that I'm guilty of

when I lost my wife and started to come back out of it. One of the things I did was I realized how many of my friends are suffering with depression, PTSD, CPTSD, and I started, you know, I should check in on him, I should check in on her, I should check in on him. Then I realized that two of my friends who had become a couple both battling depression and one had many suicidal ideations one of them had a suicidal evening and went offline. Luckily, it was a mutual friend of all of ours who lived in the same town, because of course it was a long distance, ran over and grabbed him and took him to the hospital. And they

just kind of hung out with him and it was all fine, but he was offline for a while. So the other one doesn't know anything other than suicidal. So then they started thinking, if that happens, I'm not going to wake up in the morning. And then I thought about... two suicides in the hacker community in one evening, and the cluster suicide effect. And I looked at the Excel spreadsheet I used to keep tabs on who I was keeping track of, and went to see a counselor the next day and said, "I'm doing this." And she said, "I am a trained professional with all sorts of degrees, and I am required by law to see another counselor every week because of the burden of this, and

you're doing it for fun? What the hell is wrong with you?" So a little bit is good. Don't let it kill you. And don't overdo it. And here's the thing. I'll tell you this if any of you have been there. That one time that you randomly ping somebody and a few days later they tell you that that may have saved their life because they were in the midst of suicidal ideation, it makes it really hard. It's not your job. Be a friend. But that can be crushing. That said, sometimes just listening. Let people vent. Speaking of alone, I'll skip through this because we're about out of time. Vivek Murthy, former U.S. Surgeon General, greatest pathology

he saw when he was practicing was loneliness. Half of all CEOs feel lonely. At work loneliness reduces task performance limits things blah blah blah. We're gonna see if that one sources. Oh, no, not sources of stress There's an annual report it's pop psychology, but it's good stuff and they change topics every year stress in America from APA Dr. Maslach Michael lighter Robert Sapolsky these folks have all done interesting things on stress and burnout really informative stuff and The clips were from RSA this year fireside chat with Dr. Maslach and Josh Korman. In the past few years, Southern Pride Security and Rally Security have had podcasts on it. Hacker mental health site, infosanity.org. Hacker Health, the organization that InfoSister has fired up recently, is another

good resource. And then it's behind a paywall now, but the September 2017 Harvard Business Review article on the loneliness epidemic. That said, if you want these slides, jdaniel@tenable.com. This is as commercial as I get. I am the wrong person to ask anything about Tenable. If you have any questions, comments, concerns, complaints, anything, ask me. Beautiful job, because what I get to do is say, hey, thanks for writing. I'm going to find you the answer. So, I mean, if you ever are frustrated, if you ever just can't get a response on something, if you're not satisfied with the response, if somebody like Ron and Franzi are amazing to work with, let me know. I'll let... Our

executive staff know that they're awesome, and that's appreciated, too. So, seriously, that, if you want these slides, kick me there. I'm on Twitter. InfoSecNoir has gone kind of quiet. That's my... hacker noir thing, podcast, which is if you're deep into the executive and finance and other parts of the industry, this is cool. We've talked to a bunch of CISOs, we've talked to a bunch of software security folks, and a bunch of venture capitalists. What's coming up next is some people that bootstrapped instead of taking venture capital. If you really don't understand how venture capital works, we did one with Kara Nortman, who usually doesn't get into security, but one of the first ones in her

portfolio years ago was a little company called AtStake. So she does know a little bit about the foundations of our industry. And she's actually just funded a friend of mine's seed round for a, as yet, not publicly announced company. So anyway, that's just, and they're long, they're long-form conversations, no sponsors. So that's that, that's me. And now the really important thing It's not the end. This is the beginning of the day. And so enjoy B-Sides. Thanks for being here. Thanks so much, Jack. We'll get our next talk started.

♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

Up next, Paul Burbage on using stored Cross-site scripting to beacon malware adversaries. Up next, Paul Burbage on using stored Cross-site scripting to beacon malware adversaries.

Up next, Paul Burbage on using stored Cross-site Script-8 to beacon malware adversaries Up next, Paul Burbage on using stored Cross-site Script-8 to beacon malware adversaries

Up next, Paul Burbage on using stored cross-site scripting to beacon malware adversaries. Good morning hackers, my name is Paul Burbage and today I'll be giving a talk on using stored cross-site scripting to beacon malware adversaries. I'm really excited to show you a project here that I've been working on for over three years. So let's go ahead and dive in. A little bit about myself, I'm a malware researcher. My passion in the field is taking a look at malware command and control families that utilize PHP and finding vulnerabilities within them. My hobbies include playing drums and music, of course. I was active duty Marine 2004 to 2008. Any devil dogs in here? Yup, a couple. For the rest

of you, it's not too late to enlist. You might actually get an all-expenses-paid trip to Iran here in a little bit. Feel free to reach out to me on the Twittersphere. I'm at Hexlex. We'll go over our agenda for today. I'll talk about what I consider a problem. It's far too easy for people, anyone really, with the intent to find a malware kit and use that to breach an organization. To understand the system, I'll go over what is stored cross-site scripting, just as a primer. Then I'll dive into the system itself. We'll talk about how we're able to inject stored cross-site scripting payloads into malware command and control, and how those beacons back from the adversary are received and recorded. We'll touch upon the

legal concerns that comes from this particular method, as well as some pretty interesting case studies from this data, and then finally wrap that up with conclusion and Q&A.

So what is the problem? It's far too easy for anyone to go out on the internet and acquire a malware kit. And these breaches are occurring, right? It seems like Krebs every week is publishing some article about how an organization is breached. In my opinion, current attribution is pretty one-sided. It's typically grouped by shared infrastructure to attribute that back to a particular threat actor, along with similar tools, techniques, and procedures, or from reverse engineered artifacts such as shared malware code. So we need a better method for attribution, and one way that we can do this is actually beacon the actors who are utilizing malware command and control.

I just wanted to talk briefly on what is stored cross-site scripting. Cross-site scripting in general is whenever a web app vulnerability presents itself to where an attacker can execute script. It's actually the most common vulnerability in web apps today. This is the same for legitimate web applications out on the internet, and even more so with malware command and control, because those particular systems might not be going through the rigorous testing and vulnerability analysis that, say, a legit software company would be going through. For stored cross-site scripting, this is when the actual payload is persistent. So typically the payload is stored within the database, but can also be stored in, say, a file or session information. This occurs when the unsanitized input is

stored and then later printed out without any type of script filters to that application user. So it's possible to inject HTML and JavaScript, but with this particular system we're solely doing image callbacks. And I'll show you why when we touch upon the legal concerns here in a minute. To understand how this works, I wanted to go over what is malware C2 and understand the two different interfaces that are typically seen as it relates to malware C2s that utilize a web server for their command and control. There's an interface where all the infected nodes communicate through. You can think of this as like an API. In the underground vernacular, most adversaries or people that are selling malware kits, they'll refer to

this as the gate. Again, this is just the interface that communicates with the malware command and control from the infected nodes. Then on the other interface, it's where the adversary actually logs in. They're able to authenticate and see all of the infected nodes and then issue commands to those particular computers that he has infected. And as the adversary logs in, they might be greeted with a pretty nice web application where they can, this particular one as an example is a form book malware. It's an info stealer. And as you can see, the admin, they might have the ability to list all of the different machines here. And then from there, they can choose those machines and then run commands on them like load additional malware, grab

usernames and passwords, depending on the capability of that particular malware family. So let's discuss the system itself. As I stated earlier, it consists of two parts. There's the beacon payload injection, and we have two methods that we're currently utilizing for that. It's the manual insertion method where we actually write a fake bot to check in with the C2. And then we also have a means of doing it automated through hooking within a sandbox. Then finally the beacon receiver which essentially just receives these image callbacks and then logs them all within the database for later review. By the way, the system that we're calling it, we're calling it MLBeacon. Let's go over the manual injection method. This requires researchers

to analyze the C2 source code. Most of these, by the way, they exist on a LAMP stack, so your Linux, Apache, MySQL, and PHP. You have to reverse engineer and find vulnerabilities not only in the C2 source code, but also the malware sample itself so that you can write a fake bot essentially to check in and take advantage of that cross-site scripting flaw. So the pros of this is that it's very precise and because of the preciseness of injecting that payload because of the known vulnerability, you have a very high callback rate. Cons, however, it's very time consuming, especially if this particular malware family is still under active development. The people or the authors behind that particular malware family can change how that

bot checks in, and then you obviously have to go back to the drawing board and reverse engineer that and understand the new method that the bots are checking in with the C2. Then we have an automated method. Show of hands, who's familiar with Cuckoo's Sandbox? Cool, quite a few. For those that aren't, it's just an open source project where you can It's written in Python, you submit samples to it and it'll detonate malware samples within a virtual machine environment. It's pretty cool. But what we did, we hacked the monitor for Cuckoo and we're able to hook functions as that malware sample might say, hey give me the machine name that I just infected. That might be one of the artifacts

that's submitted to the C2. So we're able to hook those Windows API calls like get username X and insert our beacon as that malware sample requests that information, that gets exfiltrated to the C2. So the pros of this, it's very scalable, going with Cuckoo, 'cause you can set up like a distributed sandbox system. And it's not affected by malware changes. So one of the families that I'll discuss here in a little bit called Kpot InfoStealer, This particular family was vulnerable a while to stored cross-site scripting. Although the authors behind it, it was under active development, although the actors behind it changed the C2 check-in usually about every month, it was still vulnerable with the sandbox method because although they were just changing

the encryption layer between the infected node and the C2, by hooking that Windows API call, still able to insert a beacon and then that was encrypted and then decrypted on their end for the C2. So it does cast a wider net. and it's more or less like a spray and pray method, and because of this, it's quite noisy. So worst case scenario, if a C2 is not vulnerable and you're doing this automated injection method, the request is just dropped by the C2 and it doesn't care about it. I'm sorry, that's the best method, or the best scenario for outcome. The worst scenario is an OPSEC issue where your beacon payload might actually be not rendered. So one

of the machine names might actually be the image source beacon callback. And the actor looking through those list of infected devices are like, what is this? So that obviously leads to OPSEC issues. Let's talk about the beacon receiver.

So as we're able to insert image callbacks into malware C2, it's just a benign image request that is being sent to this system. And what we've done is just set up Apache with some HD access tricks. So it might request whatever .gif. We're actually sending that to a PHP file where we're pulling out all of this information from that request. When I first started looking into this, I was really surprised how much information you're able to get out of just an image request. It really opened my eyes to what advertisement companies have on each and every one of us. So what do you get? You get the source IP address, which of course can be geolocated. You get the HTTP refer. This is probably the most

interesting field within the image beacon callback. Main reason being is that this is going to be the actual URL to where that actor is currently logged in, being the C2 URL. For manual injection methods, I also submit a UID. This is really cool, especially with some malware C2s, they might have several proxy layers between the gate and the actual admin interface. By submitting the UID parameter, I'm able to get that information back to match those particular infrastructures. So the idea being we're just enumerating more of their infrastructure, not only the C2s, but also the adversary's IP address, where they might be coming from. Some pretty crappy browser add-ons that claim to hide your location, they also include a

proxy request header. which reveals the true IP address. So it's pretty interesting to see those with people thinking that they're obfuscating their location. And then the user agent. Of course, all of these client headers can be forged, but if the user agent is true, that tells us quite a bit of information about our adversary. Everything from the architecture of their system, being x86 or 64-bit, whether or not they're coming from a mobile device, the operating system in use, and their preference in the web browser. The receiver logs all of this information and then just sends back a benign 1x1 pixel image. Why are we doing that? Because as long as that image callback is rendered properly, We don't want

to mess any of the formatting up within the malware C2. So by just submitting back a 1x1 pixel image, hopefully the actors are none the wiser that there's an actual image request in that particular field. And then I was talking to Jack's point earlier. I was talking to some colleagues about a month or two ago. I was telling them about this system. They were like, "Dude, why don't you set a persistent cookie in that image callback?" That's exactly what I implemented. We're seeing some really cool data. We're able to track as these guys are changing different C2s, we're able to associate or attribute that back to the same threat actor group. Just a couple days ago, I pulled

just one month of data all of the cookies and map those to the C2 host from the callbacks that I received. And this is a picture of the largest cluster. So 419 unique cookies which map to 134 C2 domain and IP addresses. What does this mean? You can essentially think of this as one threat actor group. and we've been able to attribute that to 134 command and control domains or IP addresses belonging to them. There's multiple different families too, which was pretty cool. So let's talk about the legality of this, some legal concerns. I actually reached out to the EFF, or I believe it's Electronic Frontier Foundation, if that's right. and wanted to get their take on this. The response back from them

was, "As long as you do not do code execution, "you're just using benign image callbacks "for the stored cross-site scripting, "then it should be fair game." By the way, if you ever need legal assistance with this type of stuff, the EFF are awesome to work with. So yeah, we're not doing any type of code execution. That would be like JavaScript. In the event that we were able to do JavaScript, Utilizing, say, the browser exploitation framework would be pretty cool. I'm not suggesting this by any means, but maybe you request webcam access and take a picture of this person behind their command and control. I wouldn't try that though. But we're only doing benign images. I wanted to keep

this ethical and legal, this particular system. And it's no worse than the advertisements. I bet if you pulled up some of the emails from your phone right now and looked at the raw source, there's going to be these image callbacks for tracking whether or not you opened that email and whatnot. We're not doing any type of authentication bypass, so these particular beacons that we're submitting, it's solely through the interface of typical traffic that would be occurring from the malware sandbox. We're not concerned at all. We don't submit this beacon on the admin interface side. And, of course, since we're not doing JavaScript, we're not able to steal cookies. Going back to no authentication bypass as well. In the event that we were,

again, we were doing JavaScript, we'd be able to grab that session authentication cookie and then log in as that admin. But we're not doing that here. Solely benign images. Talking about this with some other folks, I had one guy ask me, well, what about GDPR? I'm like, oh, fuck. Here it goes. Here's my take on it. This respects people's privacy. We're solely inserting beacons into malware command and control that people with malicious intent are logging into to administer. There's no way for these beacons to slip outside of the C2 and you would trip these beacons just with casual browsing of the internet. Furthermore, What's the level of expectation of privacy if you're conducting cyber crime? Dive into some of the case studies. So

BlueLive calls this group Air9, and I believe Talos calls them TA545. Again, I've been collecting this data for over three years, but back in January of 2018, these particular guys were using a Spambot malware family called Onliner to send to Canadian recipients an email attached with CanadaPost.zip. Particular ZIP file contain just a VB script, which happened to be yet another malware family that sold in Russian underground forums called the Ars VBS Loader. What it does is, as that victim is tricked into clicking on that on their system, It gathers some machine information that's submitted to the C2 and asks the C2, "Is there any additional malware that you would like me to load?" For these particular

campaigns, they were loading banking Trojans that were, again, targeting Canadian financial institutions' customers, as well as loading an additional info stealer called Zero Evil. So, received some nice beacons from this campaign, tracing that back to a dedicated server that was hosted in the UK. LE-friendly country, right? So, the cert guys up in Canada were interested in receiving that information. At the time, taking a look at that IP address on Shodan showed that it was a Windows server. And sure enough, the user agent confirmed that, Windows 10 64-bit, and asked also that these guys liked or preferred Firefox web browser. Second case study. This is an ongoing threat, actually, with the whole crypto coin wallet software craze that's been occurring. Some knuckleheads will

take a look at all the different wallet software that's available, completely just copy one of these websites, legit software company copy their website and stand that fake website up out on the internet. And then just from casual people searching for that software, they'll go to the fake website and download malware. In this particular campaign, back in September of 2018, they were deploying K-Pot, which in Russian is actually Krat, which is the Russian translated word for mole. It's an infostealer malware. This adversary IP beacon traced back to a VPS provider in Russia. And again, some pretty interesting artifacts there that they were utilizing Windows 8 with the Opera web browser. So this doesn't only work with commodity malware, leaked

malware kits or malware kits that are being sold or traded in the underground. This has also worked with advanced persistent threats, APT groups. Back in January of 2019, a piece of malware called Mole Rats came across my desk. I tossed that into the Hooked Sandbox. And again, I didn't have to reverse the malware command and control protocols to fuzz the C2, simply just deployed it, that beacon was submitted, and sure enough, within the next couple days or two, received some beacons back from this particular IP address that geolocated back to the Gaza Strip. And this malware family had been attributed, of course, to the Gaza cyber gang. One interesting artifact too from this is that they were using the strange user agent twos. And by

the way, this was one of the malware families that had multiple proxies between the infected computer and the actual admin interface or the actual malware C2. So we were able to enumerate the URL of the true malware admin interface as well as the strange user agent. and the login prompt featured down there at the bottom, you would not be able to access the C2, this admin interface, without that user agent set in the browser. So I thought that was pretty interesting. So that was kind of a hard-coded artifact. And of course, by searching that particular user agent across social media profiles, found a couple people on Facebook that may or may not be associated with this particular group. First look, would you

like to take a look at what the system actually looks like? Yeah! Big props to my friend Chris Cork who completely redesigned the front end recently. Just wanted to give him a shout out. Thanks, Cork. Can everyone say, "Thanks, Cork!" He's a bootstrap ninja, this one. Can you all see it okay? Cool. I mainly wanted to show this off just because of the dashboard of the stats page. Going over three years, been able to capture over 280,000 beacons or callbacks from adversaries. Unique source IP addresses going close to 23,000. And malware C2 instances that we've enumerated, well over 25,000. So it's been just a really fun project to work on over the years. And of course, we're able to enumerate more about the adversary's environment. What's

their operating system preference? What type of browser they prefer? More information like that. And even if the particular actor is utilizing some type of VPN to obfuscate their source IP address, just knowing what VPN provider they prefer, I think, is a pretty cool intelligence aspect too. The website itself, it just allows you to log in and you can search the entire data set here. Eventually, we would like to do more about the threat actor clustering, maybe come up with some silly names for the threat actor groups themselves. I think that largest cluster there, I'm wanting to call it the hip hop eponymous, maybe something silly along those lines. That's pretty much it right now.

So future work, definitely want to scale out and implement the distributable Cuckoo and be able to run not thousands but maybe millions of samples and see how many we actually get callbacks from. Again, if anyone has more of a data science background or just any ideas in general, I'd really like to understand the clustering and not to use like an RSA marketing term, but machine learning, if you have some ideas for that, I'd be really interested in talking to you. And of course, extend these hook sandboxes to other operating system malware, be it Linux, Mac, even Android. Right now we're mainly targeting Windows because that's obviously more prevalent, but with the exception of a couple Android families. But yeah, definitely

want to extend to other malware. And then finally, create an API to allow people to access this data. If you're in the SOC or work threat intelligence, should be able to query the system, say from like Maltego or toss the data into your SIM to correlate that. I'd really like to get some more ideas to grow this project organically. Even if everyone tells me, "You shouldn't be working on this," I'd probably still just work on it by myself in my basement. I think it's been a really fun project to work on. That being said, if you would like access to check out the data, you can hit up that short code URL. It doesn't go to a Rick Roll or anything. It goes to a Google form where you

can fill out that information. I just request a couple bits of information that I won't share, of course. I just want to verify that you're a real person and not a bad guy attempting to get access to the data is all. Good to go? And that's about it. Thanks for having me.

Thanks so much, Paul. Thanks.

♪ ♪ ♪ ♪ ♪ ♪ ♪ So so

so

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪♪ ♪♪ ♪♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

Up next, Jason Gillum presents the new normal, assessing modern applications in today's environments. Up next, Jason Gillum presents the new normal, assessing modern applications in today's environments.

Up next, Jason Gillum presents the new normal, assessing modern applications in today's environments. Up next, Jason Gillum presents the new normal, assessing modern applications in today's environments. You're live, sir. All right. Morning. Make sure your mic's not muted. How about now? Ooh, a little bit too loud. All right. Good morning, everybody. Hopefully you're enjoying the conference so far. Yeah? Some, yeah? Good. Had some good speakers so far this morning. So I guess I should start off with what I do as a... as Jack mentioned earlier this morning. And what I do primarily these days is manage a company or help manage a company. But when I have time, I do pen testing. I still do software development. I do a lot of conversing with other

companies that are getting into, I guess, trying to modernize their software development cycles. So I thought what I would do is put together just a few slides and talk a little bit about how things are changing. Because I've been developing software for decades. So I remember the old waterfall software development lifecycle. And now I'm looking at how software is being developed today and actually involved in some of it because we're doing some projects internally at Secure Ideas. and we're going through this whole agile thing and all that. And so I talked to a lot of InfoSec groups who are struggling with some of that transition, the various different parts as we're going to the cloud, as we're going to continuous integration, continuous development, how

do we plug InfoSec into that? So this talk is about what has changed in software development and how that impacts our role. is InfoSec experts. Okay. So, before I start, one thing, I usually like to poll the room a little bit. So, who in the room right now is involved in application security? Okay, so about a third of you. How about more on the operational side of things? A lot of you. Maybe deploy, any software developers? Okay, so there's a bunch. Okay. Well, a lot of them come from secure ideas. I'm a little posse right there in the middle. So, the traditional application architecture. Who remembers something that looked something like this? Yeah, okay, I see a few hands going up. So basically, this is how we used

to develop, anytime we wanted to put a web application out on the internet, which was really cool back in the dot-com boom especially, these things started popping up all over the place. You would have some kind of a, well you have your web servers because you had to have all your static assets, your images and such, served up by something else because the application server wasn't powerful enough to do that and also handle all of the logic. Then you had your application servers And you'd have redundancy built in, right? You're gonna have at least two. And then you have databases. Again, your DBA would say you needed two databases, so you'd have two of those. And then you have a load balancer in front of all of that. And

this is, you know, the basic template that was used all over the place. Some of them would be bigger. Sometimes you'd have this thing replicated across different data centers, depending on how important your application was. But that's basically how it happened. So these days things are very different, right? So it's this nebulous sort of, hey, we're going to put our stuff in AWS or Azure or Google Cloud. And you go to do that and you're not really sure exactly where everything is. It's moved around. You don't know if it's two servers or if it's, doing the application servering, or if it's 20, or even if there really is a server. Anyone hear the term serverless now? How does that work?

- Is this running in an internal process? - Hopefully there's a server somewhere. - You're running on someone else's server. - Exactly. From an information security perspective though, what exactly does that mean? How do I protect that? And so there's all these new questions that come up. From a services perspective, this is where everything is moving to services. If we look at, I'm going to point at AWS just because they're one of the most mature offerings out there with the most services available. This is a partial list and I know it's an eye chart, but it's basically about gluing things together. So this is what developers are doing these days. Instead of building all of their software from the ground up and putting that

on an application server, they're taking these pre-built services that are out there that do things and they glue them together basically. So in Amazon, they might have an EC2 instance, so that's basically a virtual server. And then they might plug in an RDS database to that. So they're talking to the database because it's a service It automatically has redundancy built in, does backups, you can make it bigger, make it smaller, it's flexible in many different ways. I recently learned that Amazon actually has a concept of a serverless database, go figure that, I don't know what I'm supposed to do with that, but that is a thing. And then they're going to plug it into other

things. So we need a place to put files. We're going to use an S3 bucket. And we need a place to just run some scripts and stuff. So we're going to use Lambdas for that. And then we have ElastiCache and CloudWatch for our logs. We're going to plug that in. Then we have different things that we've already put out there connecting to the other things that we've already put out there. So all these services, we have them glued together in various different ways to make your application. So this is a challenge for those of us in information security trying to figure out, okay, well, if we're trying to, both from the attacker side and the

defender side, if you're trying to defend against this, there's a lot of new things we need to understand. If you're trying to attack this, it's like, where do you start? Right? There's all this stuff, where do you start? So... Another thing that's another buzzword that's out there now is APIs or microservices. They kind of fit together. I'm sure there's some difference between them. I treat them as the same. Okay, so now there's lots of APIs. I know for Securidea, we've done tons of API testing just mostly in the past year because it's just bubbled. There's a whole lot of API development out there. People realize, hey, we need to figure out how to do this.

This is not... your mom and pop sort of application anymore. It's different. And we know that if we point static or dynamic analysis tools at a bunch of APIs or an application that's driven by APIs, they tend to struggle. They don't necessarily catch all the things or we don't even know if they are catching all the things. So that's something else. This file here, this is a partial swagger example file which is a definition of an API. And I'll get more to that later. So another thing that's really big, and how many people went to the training last night for containers? I know we had, there's quite a few of you. So the applications are being deployed in something called a container. So we've

taken virtualization to the next step, basically. We have virtual machines been around for a long time and that's that idea that we're going to take our entire operating system basically stand that up on top of a layer of virtualization, a hypervisor, and then we can stand up on physical hardware, we can actually stand up multiple virtual machines. And what's happening with containers inside the cloud is now we've abstracted further up and we've said, okay, well, instead of replicating all of that entire stack every time, what we're gonna do is there's things that are common between different applications or different instances of the same application. So we're going to share those resources within the same operating system. So you could have a virtual

machine and inside that virtual machine multiple containers. And the containers basically it's an encapsulation and definition of the application itself, just what makes up the application. So we have a bunch of technologies doing that sort of thing. And this is different too for us, again, in the security world. So our infrastructure is shifting underneath us. So we need to figure out how to put all this stuff together. It's all new stuff. It's all different for us. And these aren't just buzzwords. This stuff is not going away. As far as I can tell, it's not just because a couple of hipster developers think it's really cool. It's everybody's doing this. They're all moving towards all of

these new things. That doesn't make them bad. If we do the right things. So getting into what we can do about it. Looking at cloud infrastructure, so AWS or Google Cloud or Azure, those would be the three big ones. I know Oracle has a cloud too, there's a bunch of them. But one of the coolest things, and so I had somebody approach me, I think it was at DerbyCon a few years ago, when they were just first getting involved in AWS. And what they discovered was their developers would keep standing up machines left, right, and center. And so they were still trying to port scan their subnet inside of AWS for all of those machines. And of course, that doesn't really work very

well because the developers were standing up new instances every other hour it seemed. Things were just always changing. There's just so much change. But one of the awesome things about cloud is everything runs within an API of its own, so you can query it. You can ask for, hey, what are all the current instances that are running? So instead of spending a bunch of effort doing port scans, you can just say, hey, give me all the things. And AWS or Azure or Google will give you Basically a collection of all the things that you've asked for which is that's pretty awesome, right? So asset discovery enumeration very easy to do in the cloud if you

do it, right? You also have centralized identity and access management, right? That's as people as companies are moving towards cloud solutions. They're basically forced to agree on using one way to do all of their identity and access management within that environment and Which is pretty awesome because if you look at large enterprises in, I don't know, 10, 20 years ago, you'd have active directory, so like your users were usually managed one way, but then when it came down to servers and applications, That wasn't necessarily always the case. There was always this push towards single sign-on. The fact that we even had to implement a thing called single sign-on tells you that that's not where we started. We started with a very fragmented identity

and access management. And if you look at a lot of legacy systems, they're still that way. You'll still find a lot of fragmentation in there. But as companies decide, hey, we're gonna move all of this type of operation into a cloud environment, they're forced to agree on, okay, this is how we're gonna do identity access management going forward, which is awesome. It makes it a lot easier to manage. Next, availability. So I know we have multiple data centers, we can stand up multiple servers, but it becomes, it's a big hassle and it takes a lot of planning. Inside of cloud environments, so for those of you who've looked at your SISP before, you know the CIA triad, right? Confidentiality, integrity, and availability. The A

in that becomes super easy inside of cloud. It's very easy to stand up another one of whatever it is. And in some cases it's all automatically done for you. You know that Amazon doesn't put all of their stuff into just one data center. They have many different data centers, right? So you have a lot of that is just built in. Makes it easy. For APIs, microservices, they are meticulously defined. Okay, so we're looking at an application, and if any of you have done an application pen test on like older applications, older web applications, especially the ones that only run on certain versions of Internet Explorer, right? Anybody run into that? You don't know what you're going to find ahead

of time. You really have to poke at it and prod at it, and it behaves weirdly. And you don't know without doing a bunch of testing and fuzzing when you have certain inputs into a particular input field on there, how that's going to respond and what sort of behaviors are going to get out of the application, or what it's even expecting. So there's a lot of time trying to discover what the application does. By comparison, when we look at APIs, they're defined ahead of time. Like you can look at a swagger file, or a postman collection and you can see what all of the inputs are and what the expected outputs are from every single call. So it's very compartmentalized, I

guess, that way. Because they are all treated as most APIs, not all of them, but most of them you're going to encounter, they tend towards being basically a form of stateless remote functions. So they're isolated functions and they're stateless, so going from one call to the next, it doesn't necessarily remember what other calls were made. The reason why this is really useful for us from a testing perspective is this makes it very easy to build unit tests around an API. We know what the inputs are supposed to be, we know what the outputs are expected to be, we can just Narrow down a test to just those things because we can do unit tests that means we can do

unit security testing as well, right? So this is this is a new thing for us compared to trying to test a user interface when we're testing a function instead It's a much much easier. It's much better defined We look at containers so Similar to api's containers what goes into a container how it's configured. That's all defined There's a there's a file that basically tells you what the configuration is and So it makes it easier if we're plugged into that whole thing, it makes it a lot easier for us to monitor when that configuration changes, to do auditing of the configuration to begin with to make sure it's correct. So that's a good thing. And containers can be,

if built correctly, they can be more secure than not using containers. They can also be less secure, but as Corey and Mike showed last night in the container hacking class, but it is possible to make them basically use the container as an additional layer of security and isolation for an application, which is great. So we want that. Containers are, because it's in their nature, they're like a template. They're repeatable. So if a development team has built a particular container for an application and they're deploying that out there for their dev environment or their testing environment, then we should be able to say, hey, we want to do some security testing on that container. All right, well, here's the

definition. We stand up one just for us to play with. I don't know how many of you have done application pen testing and been in that situation where you're actually waiting on the dev team because... or the QA team because somebody else is using the environment that you're supposed to be testing. So that thing happens in traditional apps a lot. It should never happen once applications are inside of a container because it should be trivial to stand up another one. Okay, so I want to talk about a couple other things too. So traditional code promotion. So from the getting away from infrastructure except for the fact that we're still talking about the environments the code's going into. But I guess the path the code takes on

its way to production is shifting quite a bit. So we used to have, and you may have had a variation of this, but development environment, and then there's a system integration testing environment, and then a user acceptance testing environment, and then finally everything gets to production. You may have one called stage in there or something to... But all of the code would basically be moved from one environment to the next environment to the next environment and so on and so forth. And this would be a long laborious sort of thing. And usually when we get to those, especially those later ones, for those of us who worked at all on the operational side, that meant

late nights, weekends lost, deploying code, that sort of thing. So a lot of that's changing as we move towards this concept called DevOps. And this is mostly due to the idea of continuous integration, continuous deployment. So this is a very scary thing for us and when we first hear about it in the application security world, right? Because we're like, what do you mean? You're just going to write code, check it in, and it's going to automatically deploy to production? That sounds bad. Exactly, right. So there are definite advantages to it. So... The idea is we put our code together, we put it into the repository, it goes through its thing and it deploys out to the environment. One of the awesome things about

that though is there's this automatic feedback loop in it. So when you go to deploy something, if it breaks through one of the many tests that should be occurring, then there's an opportunity to fix it. For CI/CD, one of the main things that we are focused on is actually building functional code. And so we have ways of defining what is functional code. And one of those is, as I got getting back to that concept of, remember I said security unit testing, right? So you build unit tests around the code. so that they become part of that automated, we built something, it's functional. So you have to find it being functional as it's passing your set of tests. So this is what developers are doing. They're focused mostly

on functionality. What we need to do is make sure that they're also focused on the security part of the functionality. And as long as we do that, if you think about it, a lot of this stuff that we worry about could actually be incorporated in part of their automation, right? So a lot of the security testing that we might, in a traditional solution, we're waiting until the code's ready to be in a certain environment. But if we're breaking things down to a unit test, down to testing individual pieces of the software, then it becomes part of the automation, which is awesome. Now we have, we also have... static analysis and dynamic analysis testing tools. And I know there's a bunch of variations of those as well. So

those things, trying to fit those into that pipeline. So you have a continuous integration, continuous deployment pipeline of code that's starting off in the developer's hands. They submit that into a Git repository and then somehow that code's supposed to make its way to an environment In a traditional, especially in an enterprise environment, somewhere along the way, we're hoping to be able to run these tools. Every big organization has their tools. They want static analysis and dynamic analysis tools to run. But that's not really practical. CI runs your tools. Well, sort of. So some of the problems we have with SAS and DAST tools, they tend to interrupt the dev cycles. Those things never run as smoothly as we hope. The

vendors are always going to say-- and I'm not harping on the vendors-- but they will always say, yeah, no problem. You can integrate this into your CI/CD, and it'll just run. But they take a lot of care in feeding. right they you'll you'll go to run the tool and it like i don't know won't be able to log in or something like this something will go wrong um they they tend to be pricey a lot of them right um they're they i already mentioned care and feeding um the other problem with them is they tend to produce a lot of false positives so if you're trying to automate a system but every time you run

one of your tests you get a a log full of false positives that's not a good thing right because then somebody has to sift through those and make sure they're okay get rid of the false positives make and there's also false negatives in there too so when you get to a pure automation solution, it's 100% automated, there are going to be some things, no matter what, there are going to be some things that require a person's understanding of the context of the application of the inputs of whatever it is in order to be able to discover them. So people are always, in that sense, at least for the foreseeable future, people are always going to

be smarter than machines because those scanning tools, they're only as good as their program. They can only find the things they've been taught to find. And the bigger problem with this is, if you go back to when I was showing the picture of AWS and all the different services that are out there, how do you get these tools to run with that sort of solution? They don't really understand all that stuff that's glued together. You might be able to get them to run against parts of it, but now you're getting into a point where things get really tricky again. Do we even really need them? I think there's still a place for them. I mean,

we're still going to always have legacy apps, so they'll probably still need those tools. I think that parts of applications, depending on how they're built, they'll still benefit from having static and dynamic analysis on them. But I also think there's other tools out there that we need to be paying attention to. Developers already know about most of these, but they just don't necessarily think of them as security tools. And maybe we don't either, but we should be. So there's a lot of plugins for the development environments that are quality control type of plugins, right? They tell the developer when they're coding something using the wrong kind of styling, right? Their code style is incorrect. Or they're using a convention that's that's incorrect or their

complexity of their application is too high. So there's a lot of things that those plugins can check. Dependency checkers is another one. GitHub actually has something built into it now. So sometimes if you have anybody who's got a GitHub project out there, you might start seeing these alerts showing up that are saying, "Hey, you're referencing this library that we know is outdated." So that's kind of cool that that's on there. Code reviews. So this is something that we should be doing more of, I think, as application security people. And especially with the direction that application development is going, code reviews are becoming more common because you have this... Almost everybody's using GitHub or Bitbucket or some similar tool, right? They're using... a centralized

repository for their code that has a concept of a pull request. Everyone heard of a pull request? Yeah? Use it daily, yeah. The nice thing about pull requests, if you're on the security team, you need to get yourself integrated into the I need to review when a pull request comes through. That way, every time there's somebody submitting code that's going to become part of the code base, you have an opportunity to look at what exactly has changed on that. You don't have to review everything every time. But you can quickly look through and look at the comments and say, oh yeah, they changed something with the login. Oh, I definitely need to look at that. And then you can actually look through the code and see what

they did. Why are they using MD5 for that? That doesn't make sense. So you have an opportunity to look at that stuff as the code's being checked in and be part of the approval process on it as well. Another thing that we are finding, I do training classes for developers and security savvy developers and quality assurance is, I think it's on the rise. So I still find a lot of developers, a lot of quality assurance folks who, they're kind of new to a lot of the concepts, but I also find more proportionally I think that They've either taken classes or they're a little bit more focused on security than they were previously, which is good to

see. It's a good trend. That is, I mean, it's anecdotal, but I wish I had a nice little graph to show you, but I don't. And I think one thing I'd like to just narrow in on a little bit is the QA. The Quality Assurance Group is one that we kind of skip over. We don't really look at that very closely at all, but Security testing, in my perspective, for application security testing, is really, it's kind of a specialized form of quality assurance. When you find a security flaw, it's often very closely related to a quality flaw of some type. And there are a lot of things that your QA team, who already know how to write their scripts for

testing the application, they have whatever tools they're using, They can be taught what to look for from a security perspective as well, which is a good thing. What about pen tests? Do we still need those? So I work for a penetration testing company. I'm about to say, well, maybe yes, maybe no. I think right now we still do, but as software evolves, I think the way that we do them is going to change. It has to change, right? They're manual. Pentests are manual. They're time consuming. You're still going to always have the, for compliance reasons, you need to do an annual pentest. Okay, so that's still going to happen. But I'm talking about specifically application pentesting here, not your entire environment, although that's probably also changing as

we move to the cloud. But we do have this idea, again, getting down to the functional level testing that we could potentially do. We have, is it really going to be like the toll gate before you go to production? Not really in the same way. Let me jump on to the next couple of slides here and this will make more sense. So the traditional, if we look at the traditional, and it's a little bit hard to see there, but if we look at the sort of traditional waterfall, software development, life cycle, it's these steps, right? Everybody, most of you, there's a few younger people in the audience, but most of you, probably have, I imagine, seen something like this before in a development cycle, right?

So we start off with requirements. We define our requirements, okay? And then from the security side, what are we going to do once we have our requirements? We want to review the requirements and make sure that we also incorporate security requirements in there. Next, we move on to the design. So our requirements are perfect, right? Because they're always perfect the first time. And then we move on to design. and the architects design this great, wonderful application. And then once the design's done, what happens is the security team should review the design. So again, after they review the design. And they say, "No, you gotta change that. "You gotta change this thing over here. "Go back, fix the design." And then the design is perfect, and then we move

on to implementation. And it kind of goes through step by step, sort of the same thing every time. And our role, traditional role in the waterfall SDLC, is to be a roadblock, right? So once we get so far down into one of those buckets that were there, we say, "No, wait, stop, no, you can't go any further, you need to go through us first." And that's traditionally how things happen in most organizations. But that's changing. So as we move towards agile, so who's working with an agile team or one that's pretending they are? Okay. I say pretend. There are a lot of, and this is a pet peeve of mine, I've run into a lot of organizations say, well, we're moving to

agile. We're kind of doing a hybrid thing. I just want to point something out here. So this looks nothing like this. What's inside the boxes is similar, but this one's a circle. Okay? And the important thing here is to recognize is that in Agile, we're not building the end software any faster. It might even take longer, I don't know. But what we're doing is we're acknowledging the fact that it's not going to be perfect the first time. Okay? So we're going to build some requirements or stories. Then we're going to move on. We're going to design just that stuff. And we're working within a sprint, it's what it's called, right? So, there's a lot of other concepts around that, but basically the idea is

we're saying, "Hey, we know we're not perfect. We know that we can... We're going to make mistakes." So what happens is, as we go through the cycle, At each step inside of this, we're giving feedback back up to the evaluation prioritization. Let me just point to that very quickly, right? Okay. So, evaluation prioritization so that for the next sprint we can fix stuff that we know we didn't get right the first time, or the second time, or the third time, you know, depending on which sprint we are, the hundredth time. Initially, this seems a little bit scary for a lot of us in application security because they're just moving on and they're not really paying any attention to us and that

roadblock thing just doesn't seem to work. Because if you start telling a development team that's actually doing Agile that wait a second, once you do your deployment, we need to do a pen test on that before you can move on, they're gonna go, no, we're going on to the next one. And that's inevitably what happens. So if you try to butt heads against that, it's not going to work. The art of blocking has changed. So we need to become much more part of that development process and get ourselves embedded in it if we're going to have an impact on the security of the software that's being produced. So this here, this is a thing, it's actually called the Manifesto for Agile Software Development, and

you can look it up online. There's more to it than this. But basically the idea here is the things on the... It's your left, sorry. The things on your left are valued higher than the things on the right. That doesn't mean they don't do those other things, but they carry more value to the Agile development team. So we look over here, we have individuals and interactions over processes and tools. So yes, this means that we actually have to be social. We can't just be sitting in a corner somewhere and never talk to a person. That's not going to work. That's not going to get you into this. This is one of the best things that the Manifesto for Agile Software Development has done for developers is make them, force

them to talk to each other. So they stop working inside of a little silo. Working software over comprehensive documentation. That one sounds scary. So again, they'll actually, and I've heard this from dev teams, "Oh, we don't do any documentation anymore." Okay. Now, again, things on the left are favored over the things on the right. That doesn't mean you don't do any of the things on the right. So if they're actually saying, "Well, we don't do any documentation," well, they need to fix that. The working software part, what you need to do there, and I'll show that in the next slide, what you need to do there from an application security perspective is we define part of what is working software as working securely. If

it's not working securely, then it's not working correctly. So that's the paradigm shift that we need to affect on the development teams. Customer collaboration over contract negotiation. That one, one way of looking at that, I think, is to say that security or application security, because we have the interests of the organization's security in mind, that we are one of the customers of whatever they're building. So we need to be part of those discussions. They do, one thing that they do in Agile development is very common practice, is this thing called Scrum, and it usually takes the form of a daily meeting. I don't think the expectation is that we're going to be able to make it to that daily meeting. But they also do something

called sprint planning, which happens at the beginning or end, I guess between each sprint. And that is something that we should start getting involved in if you aren't already. And that's where you get that customer collaboration. The last one, responding to change over following a plan. So when I read this, I got really excited because those of us who have been dealing with security for a while, that's what we're used to. We respond to stuff all the time and that's instant response, right? So when stuff changes on us, we know what to do. We know how to react. We try to follow a plan and still gonna do the things on the right, but we

wanna be able to respond to change. So, something new comes out, a new vulnerability of some type, we want to be able to say, "Hey, this new thing's out here, what can we do to fix it? What can we do to address it?" So our role, and really, this slide, the main thing on this one is, all of the dotted lines come back to this risk ranking and evaluation. Like, that is something that actually happens as part of this. Okay, so we want to make sure that that's happening. So at each part of this cycle, we start finding places where we can inject our expertise, where we can help collaboratively help the development team build secure software. Scalability issue. I know some of you are thinking, yeah,

but we only have one AppSec person on our team and we have 100 developers. So this is a problem. So we need to come up with solutions for this. And I don't have all the answers here. I think it depends on the organization exactly what works best. I think that from an application security perspective, we need to take on more of an advisory role. So we need our developers and our quality assurance people to be smarter. about security. They need to understand security terminology, they need to understand best practices. If they have not looked at OWASP yet, they definitely need to be doing that. Right? Everyone heard of OWASP? Some of you. Hopefully everybody who deals with applications heard of OWASP. And it's not just the

OWASP top 10, or at least not the top 10 vulnerabilities. That's one of the biggest mistakes we as an industry make in the application security space is to just reference the OWASP top 10 because we know that the OWASP top 10, that's just a list. It's like the David Letterman of web app vulnerabilities, right? It's the top 10. But there's a lot of other things that they need to look at as well. And for developers, there's actually a better top 10 list. In OWASP, there's a top 10 proactive controls list. If you haven't looked at that and you're dealing with developers, look at it. It basically gives them instructions on what they should be doing right instead of these are the things that could go wrong. So we

need more of that. So training, get them involved. I think just getting involved in the discussion, in the sprint planning, for example, I think those types of things will help the development team start focusing on that more. We need to take on an advisory role. So some stuff to think about. First of all, none of this is going away. These aren't just buzzwords. It's not just the latest thing. The world is basically moving towards cloud computing. We're already there. A lot of organizations are already using cloud computing for a lot of what they're doing. Some organizations are still just gradually moving there or just looking at it, but it's happening. It's happening now. So if you haven't looked into cloud computing yourself, I strongly recommend that

you do. At least to understand what's going on there. Continuous integration, continuous development, DevOps, some of you may have heard the term DevSecOps. I don't really like that term because I think that security always should be part of operations. But that's a thing too. So you have a role that there are people who their sole responsibility is to manage the whole application into usually cloud environments. Agile's also not going away. We're gonna probably move more in that direction. The testing is changing. So there's a lot of what's there leads itself to better automation. So although there are still static analysis tools, dynamic analysis tools out there that can do parts of those jobs, we need to be thinking about what other things can we do in addition to

that. I'm not saying get rid of those, but I'm saying that as we move towards APIs, for example, so now we have a unit of work that is very well defined, we should be building tests around that. We should be building security unit tests around each API that's out there so that anytime anyone makes a code change on that, it can go back through its unit tests. So that's potentially a whole new skill set. that hardly any of us really have really. We might have to look back at QA or development teams to help figure out how we're supposed to do that. Like what's the best way to test your API? I don't know offhand of any tools that are specifically security

tools for that. And our role really has changed or is in the process of changing. I think it's a good thing. But I also think if anybody who's in an environment where the organization is moving towards cloud, moving towards agile, CI/CD, and if you just kind of dig your heels in and try to keep doing what you've always done and be a blocker, that you're in for some awakening there. That typically does not work. That's basically all I have. So thank you very much for listening. Hopefully everybody learned a little bit. I enjoy talking about this. So if you want to continue this discussion afterwards, I will be around the rest of the day. Thank

you. Up next Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC.

Up next, Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC.

Up next, Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC.

Up next, Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC. Up next, Katie Murphy presents Spoof Proof with DMARC. You are live. You're live. Back of the room, you can hear me? 49th Security Division, you can hear me? Where'd they go? Well, if you weren't at CarolinaCon, you missed out. And hello to my friends in domestic surveillance. I forgot to do a slide about who I am, but my name's Katie Murphy. I work at Credit Karma, but I'm not here to plug them. And I do a lot of stuff around their email security. I did my first email, had an investigation when I was 12 years old and my friend had an account takeover. Those were the days,

man. But I'm here today to talk to you about DMARC. So basically we're going to talk about why you should care. What DMARC is and more importantly what it is not because email has a lot of terrible things about it and DMARC isn't going to solve all of them. What is an email? Like what's the anatomy? A few technologies that have previously been used to provide some form of like email validation and assurance SPF and DKIM. Why we need DMARC like why these previous security technologies are not enough. and then some fun with a part of the DMARC spec called DMARC Reports. So actually, first off, anybody here actually do email security for their company in charge of that? Cool, is that both in and outbound? Maybe? So this

is basically outbound only. Why should you care? I actually haven't updated the dollar amount on this slide in a couple years, but I think we're up to like, business email compromise is like a $12 billion industry now. Email is your front door. We tend to think of it as like this very simple thing, like, oh, maybe we just do phish me training and it'll be fine, but no, and it's very expensive to be wrong. And that's just your own company, right? If you're talking about your users or your customers, Then like once they lose that trust, I can't even put a dollar amount for you on what it costs you to lose that trust. And

then of course like actual partners that you work with or like other business relationships. And if you actually want some situational intelligence on this because previously you've been completely blind, DMARC gets you reports on what's going on with people trying to spoof your domain. Or internal shadow IT, which is sort of like internal people spoofing your domain because they didn't go through your purchasing or compliance process. So what actually is DMARC? Domain-based message authentication reporting and conformance. It's created to combat domain spoofing, so specifically masquerading as your domain. It allows you to discover shadow IT and spoofers with reports that it sends you, as we'll get into later, and you can actually keep them out of inboxes with policy, which is

something none of the other technologies have offered previously. So what is it not? This has nothing to do with encryption, confidentiality, or anything else. There are some technologies about that, but for the most part, email is a postcard, and TLS downgrade attacks are the norm. Expect that unless you are using an end-to-end encryption technology, everyone can read your emails. It doesn't handle nickname attacks or the display name in the body. Literally nothing handles that right now. If anybody here is a member of MOG or otherwise on an IETF mailing list for RFC 5822, can we all agree that we need to drop display names because they are the actual devil? And this is all based on DNS. Honestly, the best video I've ever seen for explaining the implications

of our trust in DNS is "A Cat Explains DNS." And it's a raver cat getting drive-through in their car explaining everything that we depend on for DNS, and it's amazing. And you should read up on DNSSEC, but lol, that might never really be implemented fully. So what is an email? There's two parts of an email. There's the body and the envelope. The message body is the part that you actually see when you're using your Gmail, your Thunderbird, like whatever you've downloaded if you're not going in and saying, you know, view original. And the envelope is the part that, you know, as the servers are moving it around across the internet, they're actually looking at and

like attaching things to. And for the most part, we consider this like too intimidating for users, so we hide it. Oh, and if you could tell, there's a from, both like an identity based on a domain, an IP address that it came from, there's somebody that it's going to. It could also be CCs and many other pieces of information in here. So the first and earliest technology or standard that we tried to implement to give some kind of assurance about mail was coming from was called the sender policy framework. So basically it's a text record at your domain or subdomain and it says who is allowed to send mail as my domain and by who

I mean servers, not identities, at a given domain. So back in the day that was IP addresses, or you could list your A or your MX record and reference that for your domain. But nowadays everybody uses SAS, so there's also a way to include another DNS address and it can look that up. There's actually a limit of 10 DNS lookups, so as your company gets more and more SaaS, this is going to get harder to flatten. You might need to use some wacky stuff from Section 7 of the RFC. You can talk to me about that later because it's complicated. And theoretically in SPF you can choose what to do or what to declare about servers that are sending mail that you didn't explicitly allow. So

you can either soft fail or hard fail, but it doesn't actually stop anything. This is all just information that the receiving mail server can use to inform whether or not it thinks this is spam or legitimate. But many legitimate mail senders actually still fail SPF for reasons. So if you're depending only on this, your mileage may vary and it probably won't get you super far. What does an SPF record look like? Here's an example from my company's old SPF record. So we've got one IP address that's coming from our data center, so when we're sending things directly from our rack, that's what it looks like. And then we have some SAS. We use Mailgun, we use NDS for support, and then we have a soft fail at the

end with a little tilde, so that's sort of conservative, although as I said, it doesn't really do anything. The second one would be appropriate for a non-sending domain because it says absolutely nothing is okay and you should hard fail everything. We will get later when talking about typosquatting domains to why this is useful or important. Here's a little animation. Do you have a question? A heckle? Oh, cool, thanks. Yeah, I should have done more cat memes, but instead there's just little moving icons. This is Alice. Oh, sorry, no, that's Lara sending an email to Barb. And this is the envelope. So that receiving Gmail server goes, "Oh, okay, so it's coming from legit.com. "Let me look up the SPF record of legit.com. "Okay, it

says it's allowed to send from 1234. "Where is this from? "1234. "Cool, let's give Lara that message. "This is Evil Eve." Evil Eve is claiming to be Lara at legit.com, but sending from 6666. Gmail checks again and goes, "That's not allowed." And possibly Postini Special Sauce drops it. But we don't 100% know because it's Special Sauce. And unless you're depending on DMARC, inbox placement is voodoo. DKIM is our second technology. Oh, sorry, did anybody have questions about SPF? No? Okay. It'll be clear as mud very soon. DKIM, so what does DKIM actually say? It's a testing that nothing was changed between the sending server and the receiver. So this is essentially, once again, it's not saying anything about a user identity on a domain. It's saying something about the

actual server that it left from and a public-private key pair there. So it will host its public key somewhere, and it'll tell you where in the header. The subdomain's called a selector. And then the sender hashes the message headers. It will tell you which headers it chooses. There's some stuff recommended in the RFC. And the entire body hashes them separately and then signs those hashes with this private key, which it should not lose control of. Then it attaches the signature and a list of things like these are the headers I signed, these are the hashes I arrived at, this is where you can go get my public key. Sticks all that in the header. Then

the receiver gets it and it looks up the public key. It goes, okay, I look here, I get it, I check those signatures, check that hash, and then I hash the same data and I compare it. And if those hashes haven't changed, then that information hasn't changed. Hooray, the message has not been tampered with. Once again, that doesn't mean nobody's looked at it. It just means nobody's changed those specific things about it. I couldn't really figure out a cute animation, so here's just a flow chart. So the top part happens on the sender side, they're like, "Alright, I'm gonna hash this, I'm gonna sign it and slap it in there." If anything happens on the

right side that changes it, such as somebody's got some kind of fancy proof point gateway or something like this, then there will not be a nice little green match at the end. But the receiver goes, "Okay, let me make sure that works, let me hash it, does it work? Yay, it works, it was not tampered with." So what does that look like? Sorry, at the time I thought green on green was smart. That was silly. So here's a lot of the information about it. So you use A as the hashing algorithm. It's DKIM version one. S, that long terrible string is like the actual type of selector name that Amazon SES gives you, I think.

The domain is creditkarma.com. Oh yeah, the identity. We won't get into that, but it's like how granular of an identity we're looking at here. H is the header fields that it signed, and then BH is the hash. So in this case, it was signed by CreditKarma.com, or it was signed by, rather, a server that we've already set up to allow itself to sign as CreditKarma.com, which is basically a validation you'd go through in that particular tool. So you go in SES and you go, yes, custom domain alignment, my domain. The receiver is going to check for the public key at that long terrible string dot underscore domain key dot credit karma dot com because all

of these selectors are subdomains of the subdomain underscore domain key because that's what it stands for. What does the record look like? It looks like a public key. So, as I mentioned, some things along the way can break DKIM. This is actually not ratified yet or whatever. You can still get on the discussion list at the IETF. Basically, they call it indirect mail flow, can break different things about the signature. Mailing loads can do this too, different gateways in and outbound, stuff like that. So ARC basically allows this same process to occur with every hop along the way to say, yes, I validated it, that's fine, and now I'm going to hash these things and sign it and move on. So the last receiver just verifies every step along

the way. Any questions about DKIM or ARC? Yeah. So is DKIM really just giving you data integrity for the entire message or for the header? Both. Oh, sorry. The question was if DKIM is giving you assurance of no changes for the header, a set of headers, or the message body, or both, it does both. It is in the spec that you have to check both separately. But which headers it's giving you assurance that haven't been changed are specified in the H field. So it'll like, there's like this whole list of like how to canonicalize them and how to like do all this stuff. And you can ask me later about double signing headers. But yeah, so for example, if there's like a

header that wasn't here that hasn't been signed, then like that could have been modified. DKIM doesn't provide assurance that that wasn't modified. We good? Okay. So, oh, hey. Oh, BH is the hash. Yeah. So something I didn't show in the first diagram, because I was running out of space, is that there's actually two froms going on. So as I said, there's the envelope and the message body. There is a from in both of those things, which is super wild. So the from in the envelope is used by the mail servers that are actually using SMTP and is handled that way. But that's not what your users are actually seeing. They're seeing the one in the message body. So this is why

DMARC became a thing. They were like, that's weird. That's weird that we're not checking on that one. So we're using a concept called alignment. Do the froms align? So what's an example of when it's not aligned and what spoofing could look like? Once again, sorry, green on green. So in this particular case, the return path is one of the names of the envelope from. So once again, it's Evil Eve who owns evil.com. She got into the internet early. So And then you'll see along the way if you've looked at email headers before the authentication results that every receiver along the way slaps in there. So Evil Eve is using Mailgun, just signing as Mailgun. So

the Deacon passed. Mailgun.org, yeah, sure, wasn't modified. SPF passed, so, you know, evil.com did designate 666 as a permitted sender. Cool. And the reply to is evil.com, so that if it bounces, she'll know. But the from in the body turns out to be from yourcompany.com. From your boss to you, the security peon. Hey, hey, you want to forward me those vulnerability assessments for our whole company? That'd be awesome. So, oh yeah, highlighting it little by little. That's the return path, and sure enough, I didn't do a dig here, but evil.com says, yeah, that's cool. Everything passed, none of those things were changed. Awesome. And that didn't include the hash, doesn't matter. Oops, looks legit. Not great. So

this is what we mean by misalignment. It passes both evil.com and mailgun.org but it came through looking to you like yourcompany.com because you probably didn't click show original because if you did that for every email you would be exhausted. So, that's the entire part of DMARC alignment. Basically we call it aligned if the envelope from domain and the body from domain are the same. So, what happens if they don't match? This is the part about DMARC allows you to set a policy of what do you do with this message. So unlike if DKIM didn't check out or if SPF failed, you get to actually do something if you are Alice at Foo Bar. So if you

are the owner of Foo.Bar in the message body and you have a published DMARC policy, it will follow it when it sees something like this and it'll tattle on Eve at Evil. Got another animation for you. Customer and Legit Co. So Evil Eve decides that she wants to spoof Legit Co. to your customer. But, yeah, password reset, totally. Reset your password to my company and I will collect it. And then customer goes, "Okay, well, Legitco, "do you have a DMARC record?" And the DNS server at Legitco goes, "Yeah, I totally do, "and the policy is reject, "and you can tattle on it at aggregate reports at Legitco." Customer goes, "Okay, well, I guess I'm just gonna drop that then "and not tell Evil Eve, "but

I am gonna tell you, Legitco, "DMARC failed, and this is the information about it." So some poor threat intelligence analyst at Legitco is like, "Oh, goddammit, it's Eve again." I see her every day. So what are these reports of this tattling that I'm talking about? So there are two kinds specified, aggregate reports and failure reports. Aggregate reports come in on like once a day from every MTA that received a message that was claiming to be in the body from your domain. Whether it passed or whether it failed, all of it tells you about all of it. And it tells you useful information such as the hello IP or LOIP which is from the actual last server that connected, the SMTP from domain because those don't even have

to be from the same thing. They could have been doing a backscatter attack on somebody else and you could have been an unwitting vector. DKIM domain and selector, things like this. There is also specified something called failure reports or forensic reports, but after using those for a short period of time, we realized they were a terrible idea because one is sent in real time for each failed message. So that sucks and can be another way to attack or to DOS like another mail receiver. But depending on what email was sent or what was in it, If you were being silly and you were not encrypting mail that might have had some form of PII, either

by the American standard or the GDPR standard, then like now you've already passed that over the internet once as an unencrypted postcard, probably with TLS downgrades, and now you're passing it back again and storing it on your servers. Shit, we don't want that. So US companies don't send these, but I did leave a failure receiver up for a while and it turns out a ton of Chinese companies were still doing this. But you don't want the liability, so you should just not specify a place to send failure reports. So what does an aggregate report look like? Well, actually, it's going to come in as an email sent to you once a day with an XML

attachment. Fantastic. What do the XML reports look like? So here's an example of a single record. All of them will be a set of records. So here's just one email was sent to this person. or from crimson.ua.edu, it turns out a lot of universities seem to have improperly configured inbound gateways, so things will look like they fail when they come back to you. But in this case, the DKIM passed, the SPF failed because they have something weird in their inbound gateway. And we know this was for the domainsavings.creditkarma.com, so if I'm looking up my configuration at Mailgun or something, I know which one it is. And I know it's coming from crimson.ua. Any questions on those reports? Yeah. - So you mentioned that the

failure reports would be used to DDoS. What's the amplification? - It wouldn't be amplified. It would be one to one 'cause it's coming in for each failed message. Yeah, honestly, probably something like a backscatter attack is gonna be way more effective. So, how to deploy DMARC in five easy steps. I did not call this easy, by the way. DMARC.org called this easy. It can be a real pain. So, first, you should know what SPF and DKIM are and just try to get them going because you're going to have to do that anyway. and try to get them as aligned as you can. But then go ahead and publish a record that doesn't have a policy and just request reports and

have something in place to deal with those reports. Unfortunately, as of right now, there's no open source project that'll handle this for you. So you would have to set up some service that'll suck in messages from the aggregate report receiver inbox and parse through XML. Fantastic. Right, right, that too. Yeah, yeah, it's kind of a nightmare. I believe Nick from DuMartian is here to tell you about their free service for that. Ha, ha, ha. Yes, being caught out in the back. And then basically you have to analyze that data, which is why you have to have something that's unzipping all of these things. You can't do it as an individual. It's awful. Don't do it. Use a tool. And then basically as

you actually get everything aligning and you stop seeing failures, then you can start ramping up to various policies. Quarantine moves things that are... Or tells... receivers that if something does not align, put it in their spam folder and reject means just completely silently drop the message. If you have customers at all that are not using perfect mail receivers, I would basically never get to reject. For example, we have a lot of customers that are university students working on their credit and as I just already mentioned, a lot of university mail receivers are not configured correctly. So if we silently dropped everything, that would be a terrible customer experience. So, what can you do with these

DMARC records as you're configuring them to set a policy or slowly ramp up? The first one, which if you want to take a picture, this is what you should be doing pretty much to start, is you're doing version DMARC 1, cool. Policy is none. And RUA is where you send the aggregate reports, so whatever inbox you have configured to do that. So, do try this at home. And then the one happening on the bottom is you think you've got some stuff going, you think you can start to ramp it up. So the policy is now quarantined, so you can just tell people to check their spam if something went wrong. You can roll it out

by percentage, so this is do it at 20%. And then if you're feeling brave and you also want to accept failure reports, this has an address to accept failure reports. So, as I mentioned, this is all about protecting your domain. So this is only domains that you actually control the DNS records for, right? So this means that if somebody owns a typosquatting version of your domain, they can totally still abuse that. So what you need is an awesome fraud team that is working on buying all of the permutations of your domain that could be abused that look similar. We have like credit K-A-R-R-N-A, looks a lot like Credit Karma, things like that, about a thousand variations on them. So then you mark them all as non-sending, the

return of the SPF hard fail record, and then you can just immediately set this up because you know from the jump nothing should be sending from here. Go ahead and set the policy to reject. And you probably do want to collect those failure reports if you can because you know you're not sending them. Can you operationalize reports? So technically, this gives you indicators from the Hello IP, various domains that are being used to potentially try to abuse your domain. You can try to pick that up. Ideally, if you had some kind of orchestration layer that's just sucking that in and feeding it into your custom TI feed, you could do that. If you want to report stuff back to something like SpamCop or Fishtank, you can go ahead

and do that and be a good netizen. Or compare various, if you're trying to catch shadow IT, you can compare failing services that are probably just from internal people that didn't go through the normal third party review process, compare that to their DNS history and be like, oh, that's Bob from accounting. He just went and bought bill.com and didn't tell us. which is cool. But they'll probably already, if you have people in IT that you talk to, which you absolutely should, they should already know when Bob shows up and goes, "Hey, why are my bill.com emails going to spam?" They're like, "You didn't talk to Katie." Personally though, in terms of the custom TI layer, I have not found this to be any more useful than WAF stuff, 'cause

it's like, well you could just pivot through any open IP or domain out there, so it's not super great, but you could sort of use it to enrich stuff hypothetically. It's much richer and more useful when you're talking about it in the shadow IT context, to be honest. So what's maintenance mode? Like I said, basically IT needs to know about this because that's gonna be your most useful application. You need to have people that are on your production mail send team, so both the system administrators and also your performance marketing team or whatever they call themselves. Make friends, bring donuts. And then keep an eye on the total DNS lookups in your SPF record because

as I said, if you exceed the 10 DNS lookup limit, your SPF is gonna start failing. And now if one of your aligned identifiers starts failing, you have a risk of like more things either going to quarantine or reject according to whatever you have set up. So as you're adding more SaaS services, you need to actually test that. So another great reason to let IT know and/or have an additional test domain set up for all of these things. So, the main takeaway: you should definitely start collecting reports now if you have a way to parse them. Do not expect, as I said, to do this manually in your own inbox. That would be terrible. And

go buy as many cousin domains as you can and go ahead and post that because even if you don't use it for threat intel, it'll absolutely protect your customers and partners from receiving anything spoofed on those other domains. And then here's just like a stack of hundreds of pages of RFCs that I read that you should read, but this talk is probably enough to get you going. And then there's some sort of other additional things like abuse reporting format. I don't know that anybody even does that. Like email feedback reports, other issues with DMARC. And then DNS twist is actually just a helpful little tool that you can use when you're looking to create those permutations on your domain for typosquatting that you want to buy up. So if

you want to like generate a list for fraud to buy then just be like, here you go, here's like, you know, a couple hundred that I came up with. Small plug, the AppSec team is hiring. We have offices in Charlotte. Anybody have any more questions? I've set up DMARC on my own personal server, and I'm also running mailing lists. So it is kind of a pinpoint. So where can I find out more about ARC? I don't know if they're at the security level that they'd be integrated in mailing lists or not. - So they're not like a fully ratified RFC. You can go to arc-spec.org and you can also be on the IETF mailing list for that. I'm on it, there hasn't been

like activity in a minute. I'm not sure where we are with that though. We should probably check back in. - How do you deal with the alignment issues on various services? - So it depends because like there's, there are some services that like, I guess there's just a ton of people that are just like, yeah, I'll spin up this goofy service that can like send mail as you that like might not be able to do DKIM custom aligning with your domain. And that could be a problem because as I said, SPF is like pretty brittle depending on like the inbound receivers. So honestly, that kind of has to become part of your review process for buying new software. It is entirely possible that somebody on your marketing team

will want to buy software that's kind of a piece of shit. And you're gonna have to say no, because otherwise things are gonna wind up in spam. - I think with SES or one of them, it was signing it for DKIM under its domain, but then that's not gonna be aligned with the regular sending domains, so now you've broken your alignment. So SES, Mailgun, and some of the other major ones actually do allow you, you just have to go in the settings and say, like, I would like you to sign this for my custom domain. And then they'll, like, require you to do some kind of, like, validation step where you, like, receive an email

or generally to, like, prove that you own a certain domain. They'll have you, like, host a text record at your domain that they can go check on. Yeah. Yeah, yeah, things like that. Basically, like, prove that you are the owner of this domain, and then, like, yeah, we can do stuff for you. But generally, you, like, already had to do that to even, like, sign up to send in the body as that if they're being responsible. But again, totally varies by mail service. I actually have that exact problem. And on DKIMP, one of the things that I'm using to be... Yeah, also I've been bad and not been like I'm repeating back the questions, but I will repeat back the BIMI thing. So there's this

interesting other standard, I don't know if that's been ratified yet, where various mail senders, or mail receivers rather, are trying to encourage you to do DMARC alignment. And I think you can't just be in reporting mode, you have to actually have some kind of enforcement, right? Where, again, they'll let you add a specific subdomain that, I think it's like a dot, like, underscore bimmy dot, like, yourdomain.com if they find, like, oh, this message came in and the policy is at least quarantine and it passes, then, like, in a little avatar spot, we'll put whatever little icon that, like, you happen to have hosted it, like the BIMI domain or something to try to be like, look, it'll make it look more legit, more people

will open your email or something. But that can also totally be abused, so I also have problems with that, same as my problems with the nickname, catch me at lunch. I hate email. I talk about email, I hate email. Any more questions? Cool, thanks guys, go forth and authenticate. - Katie, so we've got some time before lunch, really tasty, Can I put this down? Or is this like live?

Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence.

Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence.

Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence.

Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence.

Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Up next, Michael Kernow presents on Cyber to Physical IT Convergence. Yeah, hold on a second here. People are pointing out it's a little bit small. Here, we can roll with it because I'm going to be describing what's going on. Anyways, hey, how's it going everyone? My name is Mike Kernow. I made a few changes to the presentation. We're actually going to be talking about the granular history of Unshielded Twisted Pair. I'm kidding. I actually want to be walking back next year, so we're not going to do that. It's kind of boring. But what I am here to talk about is cyber to physical IT convergence. I just See I

kind of feel like I have to flood this with memes and gifts because I realize I have the after lunch position Everyone's stomach is full people start getting the itis that forces me to be much more interesting in the presentation so before I descend into my standard demeanor of delinquent, casual brevity, I'd like to take a moment to say it's really awesome being here. This is my first B-Sides event ever, and now that I'm here, it's definitely not going to be my last. So yeah, really great job everybody who came before me. Also makes my job hard to get up to standard. But yeah, so who am I? A little bit about who I am. Personally, I spend a lot of my free time doing software development,

creating open source tools, helping people with open source tools on GitHub or Bitbucket. You wouldn't believe it, but I actually do a lot of public speaking. I spoke at three conferences this year, a couple panels, and I've been doing security meetups for the past three years. I spend a lot of time exercising, however, that doesn't help me on these stairs. I play a lot of Xbox. Halo 5 really so if you play Xbox one hit me up talk with the boss I'm an avid football fan I kind of feel like I'm gonna lose like 90% of you when I tell you what team or I could just sit back and collect all the adulation I Don't know I didn't expect this so in the profession

the professional side Cybersecurity instructor for the Coder School, which is an after school initiative that mainly teaches kids how to program, but I run the security class there. I'm there for about almost three years in October. Just a quick screen capture gif of the website, check it out in your free time. So I wear a few hats, right? So I'm also the co-owner and chief information officer of a financial analytics company called RapidCap. And what RapidCat does, oh, so that's the most professional you're ever going to see me on camera. What RapidCat does is it's a platform that aims to kind of decrease that intellectual barrier of entry for people that are trading in the Forex binary market. And last but not least, I'm the lead security operations

center architect for Block Harbor Cybersecurity. And so what Block Harbor is, We're a cybersecurity firm based out of Detroit, Michigan, and we have a pretty firm foothold in the cyber-physical space, mainly automotive. And so what we do is we actually help secure the cyber-physical systems that these major automotive OEMs out there use. And I'm also the former head of cybersecurity for an international MSP called SAM Analytics. Also former PSYOP in the Army. The same guy, just without the scarf. So security acumen, just kind of a little bit about my security experience. I've done Red Team side, right? Like we've all done pen testing in some form or another. So pretty much everything that ranges from a standard network to web application, mobile application for things

like everything from IAM management platforms, banking platforms, hotel financing, really any, not everything, but a lot of things. I've done blue side too, so I eventually got into the engineering side, specifically developing a custom in-house SOC utilizing Elastic Stack and then being able to integrate that with other in-house tools, creating a pretty good functioning SOC from that. And so some of the positions I've held in that space is engineer, analyst, and team lead. So right now I actually lead a team of two analysts. And then right now I'm lead architect. So what that means is being able to, like basically what I do for Black Harbor is I handle all, I create an architecture for how to set people up with our SOC and then I turn that

architecture into working infrastructure. So all the plumbing, all the connecting, all that stuff is what I do. And where I make my bones at nowadays is industrial control systems, cyber physical devices, So I got my start actually. The first job I ever had with cyber physical assessment was a piece of major metropolitan public transport. Right? My first time ever. But I rolled with it and actually turned out really well. And so that kind of lit the fire, you know, the passion of cyber to physical security. So I also do a lot of research in industrial control system and operational technology security integration. basically how to secure cyber-physical systems. I'm using a lot of repetitive phrases, I'm sorry. Or not. And automated tank gauges, so

gas stations, they use automated tank gauges. Plug this stuff right up to the internet for some reason. So really quickly, just want to go over what I won't talk about. Can you guys see that? Good, okay. 'Cause I don't want you to read it. It's literally... It's literally just a list of very minute granular regulations and compliances that if you're even fathoming putting your stuff on the internet, you should have all these covered. Yeah, so what I am going to talk about is industry 4.0. what it is and what the problem with it is and some proposed prospective solutions. So basically, how many of you guys in here have worked with industrial control systems, PLC, SCADA, IoT? Okay, so a couple of you guys don't chastise

me because I will be covering notional concepts, just trying to build a secure mindset with this. So if somebody says, "Hey, that network architecture is not gonna work," cool. The idea with what I'm going to show you is basically just trying to get that mindset there. And now I'm going to do some demos. So what some vulnerable nodes look like in the wild. Really we're just going to be doing a little bit of showdanning. We're going to rage against the machine by just screwing up a PLC. Then that's what this is here. This is my PLC and my automated tank gauge and my MQTT IoT broker. We're going to get into MQTT broker Mischief. MQTT is a device-to-vice protocol that a lot of IoT solutions use, and

so we're going to just mess with a sensor or two. They're not automated tank gauges, so we're going to be passing gas tanks. I got a setup. Oh, yeah, if anybody has any questions about anything, either just get at me at the end of the slide or after the presentation is fine. So to build a little bit of historical context as to what Industry 4.0 is, we got to understand what Industry 1, 2, and 3 are. And so I'm not going to read every single thing here, but pretty much mechanization kind of helped us pivot our infrastructure and economy from agriculture to building ship. So then from there we have a steam engine. So with the steam engine we subsequently created ways to massively extract coal

really quick. We're also able to accelerate the economic material and human driven stuff. So like people taking trips, transporting things that expire like meat or something. And then this actually is like the original creation of like, it's helped build the blueprints of how we operate industry today. Industry 2.0 started late 1800s, so new energy, gas, oil, electricity, combustion engine. So the combustion engine means that the steel industry had to grow rapidly. And after that happened, we have new communication formats and we have new transportation. Industry 3.0 started 1969-ish. So we have nuclear energy, we have nuclear energy. So we have new electronics, so transistor, microprocessor, all that stuff. And we have automation through robots and programmable logic controllers. And so Industry 4.0 is now, there's a whole litany of

crap here. So virtual augmented reality, additive manufacturing. Additive manufacturing is actually really huge, obviously, like 3D printing. Normally manufacturing comes from the fact that we take raw materials and then whittle away at them until they become the form we want them to, but with additive manufacturing you're just creating from nothing, which is awesome. Internet of things, shits everywhere. So I don't want to read all that. So big data, the ability to process large amounts of data, create predictive analytics and machine learning by byproduct, being able to use big data, cloud computing. Advanced simulation to be able to simulate things that are conducive to cross control, things like that. Autonomous robots that can think, put thing, please disregard

that spelling error, that can thing, act and react. Universal integration is basically the communications between different systems that's done in lots of different ways. and cybersecurity because for us to advance as a culture and as a people and as the human race, we need to be able to keep industry 4.0 going so we can do industry 5.0 at some point. Stuff doesn't work if it's not secure, right? So all of us in this room have real big shoes to fill. So the problem is I see it from a professional standpoint. There's a large influx of companies that want to be industry relevant. So what happens, I got ahead of myself, what happens is they want to meet the needs for

communications management analysis, right? So companies that utilize cyber-physical technology will put their stuff on the internet haphazardly, not trying to be disingenuous, just trying to meet a need and increase convenience. So here we just have like a sample cloud dashboard for a SCADA system. SCADA is a supervisory control and data acquisition. It's kind of like the management for different PLCs under it. It's a little more complicated than that, but that's the easiest way to explain it. So here we just have like a quick dashboard. No product plugs or anything. It's more so just to kind of show what some of these cloud dashboards look like. They want to be able to manage these things remotely, right? And so here we have like an IoT, you know,

smart home type management device, light switch and temperature and all that. Here we have like an ATG monitor. Does pretty much the same thing. Monitors gas tanks and tells you like, you know, how much of what you have. And so the way that I see this process go when I talk with people and work with people, starts off, organization puts their PLC on the internet, right? is this IoT convergence. Then the organization drops the ball on cybersecurity. So the ball represents the security and the organization is the guy that dropped the ball. And then they fall flat on their face because something really bad happens, right? So I just want to say what's up to some scary

stats. So I did this was done a few weeks ago the numbers I got, so they might be a little bit more now. So ICS-OT devices that I found in the wild, 21,000 Modbus devices. Modbus is the most prolifically used protocol for either PLCs or other cyber-physical devices. You see a crap ton of it. Honestly, you know, some of these are honeypots, but they're not all honeypots. Siemens S7, DNP3, devices that use a DNP3 protocol. Tritium devices, 35,000. BACnet, 19,000. I'm just going through stats here. Then we have IoT MQTT brokers. Here's a quick screenshot from a couple weeks ago here. That's a pretty scary number because the MQTT broker, what it is, it's not the IoT

device itself, but it is the catalyst of communication between smart devices. And that's why it's scary to me. Automated tank gauges. A good amount. Not high comparatively, but still enough. Oh yeah, demo time. So let's do some demos. So the demos I have set up, I already went through them. I've got a Modbus PLC lab, I've got an IoT MQTT broker, and I've got an automated tank gauge demo. oh whoops i this is a video just in case something happens with a demo so pretty much i did some testing with my hotspot here and uh when i when i tested it in this room things got kind of weird and choppy so if that happens i'm just gonna remote um oh yeah i'm forgetting the whole showdown thing here

we go so uh real quick like here we're just gonna take a look at um what Shodan sees on the internets. And we can see there's a bunch of stuff here. I don't like this because I'm not paying for this crap and it's got like two pages I can go to. So instead what I do is go to maps. Granted this loads well. This is kind of hit or miss. Like sometimes I can pull these up just fine and sometimes it kind of bugs out. So really quickly let's take a look and see if there's any devices. Nashville. There we go. I almost forgot where Asheville was. Which is terrible because I come here like once a year for

work. Let's just take a look and see what that is. Okay, so this is a PLC that's open or this is a Modbus device that's open. The scan that Shodan uses did not really produce any good results. We just have one line with an empty value. So let's go here to Silva. There's another one.

I am looking for something that has more than one line to that. I am just going to go back to the... that didn't work. Okay, so here we go. So right here we can see in Poland we have a device that looks like it has three... it's got three different IDs, so these could be for different functionality. We're getting some stuff, oh, we can also see illegal function. So when you see that in the wild, that means that we actually can't access this. But don't try this though, because there's like this legality thing. So here we go. These right here, because it doesn't come back with the slave ID data, legal function error, we should be able to interact with these. So

at least be able to like read the holding register address values or read the coils in them. In a PLC, the coils contain the program logic and the holding register address values are values that can be anything. They can be the weight of something, the height of something, the temperature of something. Whatever is controlling those values mean something, even though they're not always really human readable. You have to be on the inside to really know what that means. So enough of that. Really quick here, we can see we have a few MQTT brokers that are online. Most likely this MQTT broker is running Mosquito. And the reason I say that is because, so these are called topics, right? And so at the beginning of a topic,

we have the dollar sign sis. Mosquito is a tool that does do that. But if you scroll down, we can see ActiveMQ is also another MQTT broker. So it normally starts with something indicative to what the device is. So can everybody see this okay too? I haven't even been thinking about that. Okay, here we go. All right, can everybody see that? I'm not sure. Yeah, it looks fine to me on the screens. I don't know. Here we go. So this is most likely a Mosquito MQTT broker. This says right here it's ActiveMQ, which is just another tool. It's another MQTT broker you can set up. I can't recall if this one's open source or not because I've

only been testing with Mosquito and not with ActiveMQ. So here we have gas tanks. And here we can see a lot of, scary enough, but a lot of local gas stations have these automated tank gauges that they stick right in the internet, again, because they probably sign up with some sort of service, a cloud dashboard or something. It's like, "Hey, open up this port on your network device and just let us hook in." Like, "Oh, cool, I can see all my stuff, but so can other people." Sarah, what's that? - Just enable UBHP. - Yeah. So here we can see, so pretty much what this is is a telnet session. And so you can look up vendor commands for

this. Just type in automated tank gauges and you can find a couple brands. There's one that sticks out more than most that's pretty highly used. But we can see right here, this command right here is actually entered via telnet. The 201 command, that's to list your inventory, what you have. There's a couple of commands you can do, like the one to restart the tank gauge is pretty cool. That means that people can't pump gas. I like that one. So... All right, so that happened right as I was doing my terminal here, so the demo guides have answered me. I really wanna try and make this a bit bigger here, but if it takes more than a couple minutes, I'm just gonna try

and explain really well what's going on. Don't know if that worked. Does it? There we go. Yeah, I've had this laptop for so long I didn't even know I could do it. See, you learn new things at conferences. I gotta make sure these are the same size. Yeah, they look... Okay, that's terrible. Alright, cool. So now we can see that. So we're gonna be working with a tool that I built to actually help me with my very first SCADA security assessment. It's called RapidFire. The reason I called it rapid fire was because I needed to address a need to test multiple holding registers addresses at once. The socket tool that I was using can only handle one of

these so I had to create a small Python script that would just automate given a list of addresses. And I had turned it into a tool because of free time. So really quickly we're gonna kind of go through these. We're gonna use the Modbus module first. And we're going to use this mod Modbus pen test framework here. Cool. So I stuff doer because it does stuff. Show modules. We can see a couple things here. When you find a Modbus device that you want to mess with, you need to get a couple things. Get a UID.

And then, okay, our UID is one. So there's tools a little bit buggy as far as going back to the beginning, but that's fine for now. So now we have our UID of one, so put that in the back of your brain bank thing. Now we want to identify the functions. We want to identify in what ways can we interact with this device. I'm starting to think thunder's a good sign now because everything's working. I forgot. Okay, so set our target again to that and then set UID to one and then so we're printing out the function codes. So read coils function one is supported that means that I can send a request to say hey give me your

coils or tell me what they are and it's really just binary zeros and ones. Then we have read multiple holding registers, so that's really important because that allows us to just kind of identify which holding register values have addresses in them, or have values in those addresses, so you can kind of pinpoint your attacks if you want to. Or you can do what we're gonna do and just rewrite all of them. Again, I have to repetitively start this just... That's fine. So anyways, we've seen that a few functions are there. We can read coils, we can read holding register address values, we can write to multiple holding address values, and we can write to multiple coils. Two things we need to make some SCADA sysadmin's day.

I'm being sarcastic, they wouldn't like that. So we're gonna go ahead, quick like here. So these two terminals right here, these are going to be listing the coils, right? So your zeros and ones. And these are going to be, this is going to be listing the holding register address values. Again, these could be anything from the outside in, we have no idea. So what's going to happen is I'm going to go ahead, do a thing. Then our start address, we're going to start at one. So we put zero, enter. And then if, what's it getting, let's stop at 20, 20, 28. So then we just enter what we want to do. Let's just do this. All right, now watch these. They're all different. They're

all 69. So, go back. There are ways too where you can actually, you can just pinpoint them if you use a smaller range. That's why reading what these are helps so that you're just not rewriting all of them if you want to be a little bit more stealthy, but just for example's sake we are doing that. And then so we're gonna do the same thing with the coils we're doing here. So we're just gonna break the logic of that controller so that like, I don't know, some machine fails and drops and hits someone. So we're gonna start here and so we're gonna put a zero or one in. Because there's more zeros, I'm just gonna put a one in. Whoa, look at

that. Ones. So that's the Modbus. Next we're gonna do the MQTT. So basically what we're gonna do, the way I wrote this module is a little screwy. It utilizes two Python subscripts, so I gotta enter the address a second time. So if you're wondering to yourself why the hell is he doing that, That's why. So you can see here I've got MQTT setup. You can see sensors, sensor 3542. This is a light switch, right? Notionally, it's a light switch. Huh? I might need to go bigger. Hold on. Okay. Okay, so... Yeah, thank you though for saying that. I would have kept going. So yeah, this is notionally a light switch. So you have on/off function. So we're gonna

do it on here. We're gonna write a string value to it. It only accepts on or off. So if it's, oh wait, hold on. I need to first subscribe to the topic so I can write to it. Then we have to, spelling is off, I guess apparently I'm gonna public this topic. I made some last minute changes before doing this today. So we're gonna write to control, control is just the default, and we're gonna write on. So before it was off, it should be on. So if you get message received on, that means that it worked but you wanna double check it. Oh Jesus Christ. I forgot how big I made that. So we're just really

quickly just kind of parsing through the output. Oh, hey, look at that. We're on. So that's that. I mean, this is not, this really isn't super impressive for the most part. It's, the reason I show this is because people put this crap on the internet. And these are just some things you can do to it. Next we're going to get into automated tank gauges. If we found that somewhere, and that light switch was hooked up to... If somebody has it hooked up in that way, yeah. And you'd be surprised. Like, you just go to Shodan, you'll see loads of MPTT brokers on there. And if you can get it to print out its topics, chances are you can probably subscribe

and publish to those. And you'll know if it worked because you get that message received. Depending on, like, most of the brokers follow kind of a simple, similar-ish schema almost, so you should get some kind of response message back saying that it worked. So from here, let's do automated tank gauges. 10000 and 1 is default port, however, it's also default port for a specific Ubiquiti network device. So you want to make sure that when you're looking at these, you can tell it into it and running the 201 hundred command, you can just print out of that the inventory or just go to show it and it does it for you. Because show it in basically takes it says, oh,

you're running 10000 one. I guess I better tell it into you and run this command. So now that we're in, So one thing too is that this tool I made, I'm running it in Mac, but I don't have a good X handler set up and for some reason X Quartz doesn't like to play nice with this. So if you're running this on Linux, you get different terminal windows that pop up with, like if you're running the Modbus module, then you would get windows that pop up and actually show you the addresses and the coils instead of having to kind of hop skip back and forth. In this case, you'd actually get a terminal pop-up that

tells you the vendor commands. What we're going to do here, we're just going to do basic 201 command. I can't remember if it was control or FNA. I forgot to put the I. When you're telemeting into this, you gotta do Control + A and then I, and then you can do 201, 00, and then it should print out the, let me do a double take and make sure that's, oh, my bad, it actually timed out. Don't worry, all is not lost. If this doesn't work, that's fine, I got a video. Okay, that's not working. So I'm gonna just pop this open real quick, and just kinda show you the tail end of it. All right,

that's MQT sensor. Should be big enough, or not, that's pretty small actually. But can everybody make that out? It just says ATG tester, it's the same thing I just had. The address, localhost 0.0.0.0. Yeah? Zoom in. I'm not sure how to zoom in on a PowerPoint video, but yeah. Anywhoos, yeah. So pretty much when you're using it in Linux, it does have these little terminal windows that pop up and gives you the whole menu for it. And what we have here is we just did, or we're about to do, we have the inventory here. That same command I was running, something happened here where the module timed out. I don't want to take the time

to do that, so it'll be time consuming. But then what we can do too, We set up the command 001 restarts the device, right? So that can be kind of a crappy thing if you have a gas station that has to shut down operations because some dick decided to turn it into it. A lot of these you can interact with because you have, for a lot of the cloud dashboard management platforms don't have like a VPN or some kind of crazy routing setup where you can do that. Pretty easy to access. Anywhoos. Yeah, so there's two types of people in the world of cyber to physical. This might not be concrete, but in my experience it's the kind of people I meet. First

one, we're gonna get an intro into the adventures of, let's just put it on the internet, bro. That's the guy in your organization who's just like, let's just put this shit on the internet. Again, not being disingenuous, like I'm sure they're really trying to help out or make things more convenient. However, it doesn't really work. I already gave it away, but his superpower is putting shit on the internet. That's one type. And then on the flip side of that coin, you have his offspring, got to keep them separated. We have the air gap type, right? Air gap... error gapping and a security by obscurity are the two of the main security tools in most SCADA admins toolset. So the error gap is pretty much just, you

know, just keeping off the internet at all, right? The only problem with that is that it's not sustainable at all. If you want to achieve that, you know, industry relevance, You've got to find a way to play nice with the IT side to, you know, constitute these connections and use these dashboards and do all your cool stuff off-site. No person really wants to have to go on-site every day just to, like, you know, get the stats of something obscure. So how do we securely bridge the air gap? Let's get into some fun. Dementals. I'm glad you guys got it. So here, cybersecurity rule number one, you're not secure. Even if you're secure, you're not secure. Don't think you're secure. Only

because the mindset, that little bit of paranoia helps. I don't have to tell you guys about that. So I got this gif of a guy poking holes into a water bottle. I feel it fits somehow. Then there's Rocco Bodi of Mega64 and Best Buy hacking everything. That's pretty much how it works from what I know. So in order to kind of really cement this, you know, Whenever you're working with trying to bridge the air gap, always assume that these little things can go wrong. I have this growing text here. You can reflect while that's loading. It's going to be my drink break. It gets pretty big. How do we securely bridge the air gap? We have

your compulsory security methods here. employing proper encryption, your AAA, routine vulnerability checks, deploy active monitoring, you should do that for sure. So like have a SIM set up or have somebody set up the SOC for you and integrate you. Pointing to myself because that's what I do. And so continued architecture. So the things I'm going to get into from here is like a little lightweight threat modeling and some notional network concepts here. So this icon right here, this is going to represent the SCADA device itself, and this right here is just going to be the cloud-based admin management dashboard doohickey, right? So whatever you're plugging into. And this is just a quick setup. You have monitor points. Your PLC is controlling those, which then report to

your SCADA. And that goes to your cloud analytics. That's the idea here. So what are we building? SCADA architecture. What can go wrong? Assume everything can go wrong. In this instance, we're doing the... is SCADA and PLC. And so what that means is incoming communications of something besides what's meant for it, because it's like so rigid and strict in what it can accept, things can air out and break pretty easily if you just send it random shit. So right here we just have a quick trust barrier. Obviously everything that goes out is cool, but you don't want shit getting in, right? So what are we gonna do about it? You wanna protect your SCADA network from unsolicited incoming traffic. One way of doing this is by

employing... Oh yeah, don't do this. Yeah, don't be that guy. One way of doing this is by utilizing a piece of technology called a unidirectional gateway. These exist in hardware form, One-way data diode that just allows traffic out but not in. They can be reconfigured to let traffic in in whatever periods that people need to come in and do maintenance or whatever. They also exist in software format, though those are hard to find. But there is a tool called HairGap which at least allows for one-way network traffic going to a receiver, but you do that for logs or something, right? So yeah, that's that's one way of doing it and then the last part do

we do a good enough job? Just have somebody test see if they can get in right. There's a very very simple concept of You know if you're employing a unidirectional gateway the easiest task is to try and send shit to it and see if it gets in if it doesn't then you're good if if it does then you got to fix your stuff, so I'm this the architect from the matrix and So, oh yeah, so this is the marker where I get into some, these are the notional things that, this just kind of build an idea of some ways you can set up secure architecture, depending on the constituents of your network. So it might not exactly work for you specifically, but this is, again, build

that mindset. So right here you see I've got, the router, the switch here, and then we've got two different subnets. One is the operational technology network, which you want to keep separate from the IT network, because if the IT network gets infected with something, you get phobos, or you get anything like that, then hey, guess what? That might propagate to your SCADA device. A SCADA device is normally a pretty big, hefty, powerful Windows machine, or is running on a Windows machine normally. You can get things like OpenSCADA and SCADA VR and stuff like that. So you want to keep those two networks separate. An example of what to do is have a relay server which can then forward the traffic over a VPN utilizing some kind of

network trunking mechanism done to switch. Another way of doing this is having an OT network separate, having your independently synced into network that will really just hold a secondary relay server to send out. Again, this isn't the QLB, just some ideas. And your OT network's separate. And I have a fridge in here too because IoT. Next, this is pretty much, so I'm gonna reference this real quick. But, so where this green arrow is, imagine to the left is here, okay? So everything that's this green arrow here is this green arrow pretty much. So here we can see, Here we can see that we have three different subnets here. We have 172.168.3.0/24 notation here. We've got

172.168.4.0. And we've got 172.168.5.0. And it just arbitrarily assigned IPs right here. We've got 172.168.3.9.0. That's the relay server that's right here. And it hits a network trunk utilizing the switch's functionality and then sending that over open VPN from one relay server to another. That then sends to some kind of proxy, which can then send out to the internet. And then again, the proxy isn't super necessary, but I like the idea that if it gets sent out to the internet, if something comes back, you could then utilize some kind of reverse proxy functionality and send that to a honeypot, which can then collect information, send it to a network sensor here. Got this little bro box right here, your network intrusion detection system. So basically

we're just collecting intelligence on the network and on what people are trying to do to your security device here. And then right here also, I like having nids on the IT side because we can then collect network traffic utilizing a tap or span port, be it like Snort or Suricata or something, and then sending that up to a security incident event management platform. So that's one way. I like this model. It's kind of colorful and semi-complex, but the idea is It's got to be a little bit complex because the idea is that you don't want stuff coming back from here and hitting any of this, this or anything on the OT network at all. So really quickly, I'm going to go through a

potential attack scenario that utilizes a disgruntled employee as an attack vector because that can happen, does happen, can happen, shouldn't happen. So let's say that this is your prospective OT network and you have monitoring, you have visibility and insight into the SCADA, you're monitoring the SCADA, but you're not monitoring the other choke points like the unidirectional gateway or your network device facing out to the internet. And you have... employee here, he's probably been asking for a raise like 20 times and you keep denying him. He's getting bitter and angry. He turns into a criminal. And so what he does, comes into work one day with a flash drive, decides he's gonna plug it right into that damn

SCADA device. And so what happens from there, he puts a virus on there, right? Basically turns this into whatever he wants. And what happens is it starts making out bond requests to something malicious, right? CNC server or something like that. And it comes back because he probably also did something to this unidirectional gateway, or if one even exists, to allow that returning communication back. So basically, because you didn't give this guy a raise, he just turned your OT network's huge, powerful device into a Bitcoin miner. All because somebody couldn't spare a few bucks. So... And the reason why you wouldn't really be able to see that is because in this example you don't have visibility in here. So one good thing to do is that if you have a

SCADA network, make sure you're monitoring all your choke points. So from PLC to SCADA, normally your SCADA device will output some kind of logs. You can then aggregate, do syslogng or nxlog stanzas, send to a collector server somewhere. And then being able to use the unidirectional gateway and the network device, send those logs as well, or if you're monitoring over TAPS band port. So make sure you do all your monitoring. This is a gift from hackers. This is a gift from hackers. This is a gift from Matrix. And this is just a very tropey, beepy, blinky dashboard. Don't be scurred, be secured, for those of y'all who are watching Practical Jokers. Credit. We're closing up here. I'm kind

of getting low on time. So that's all the ICS research that I did. The tools I created. The only thing I didn't really make was the Modbus lab. That's called Modbus Palette. You can Google it. You just run it with JavaJar. That's me at the North Carolina Modern Art Museum, staring at the dangly shit, man. And for the IoT MQTT stuff, stevesinternetguy.com helped get me started. He has a lot of really cool projects, test sensors, things like that you can do. And the automated tank gauge, Eric Zhang, I hope I'm saying that right, aka Zero Day. So back when these things were starting to be discovered, he was one of the few original folks that went and said, "Oh, this shit's bad. "People need

to know about it." So yeah, he is how I got into that, so yeah. Done. So I do have the tool rapid fire that I made. It's on Bitbucket. It's yours to use. I also need help with it. So if you want to get, just email me with anything that you want or need and I'd have to provision the access for this for you. Or if you just want to tell me this presentation sucks, you can email me too. But yeah, that is it. Any questions? I think we have maybe two minutes left. Cool, I've effectively confused everyone. Okay, so you're saying, can you repeat that? I heard Modbus and Gas Station. There's a couple of open source honeypots

you can try. They normally start off with... By default, they have the 201 function, the inventory function, they have that there. For anything else, you'd have to, it's an easy Python script, right? So you'd have to implement that yourself. But yeah, so, compot, I think, is one. There's compot and there's tank fetch, I think. So you just Google one of those and you should find what you're looking for. They're pretty neat. Anything else? Cool. All right, well, hey, thanks for having me.

Up next, Eric Krohn presents What You Know, What You Have, and What You Are, Multifactor Authentication in the Modern Age. Up next, Eric Krohn presents What You Know, What You Have, and What You Are, Multifactor Authentication in the Modern Age. Up next, Eric Krohn presents

Up next, Eric Krohn presents What You Know, What You Have, and What You Are, Multifactor Authentication in the Modern Age. Okay, well, I say let's go ahead and get started whenever you're ready. Okay, well, that makes it easy. All right, well, first of all, thank you all for coming. Who's enjoying the show so far? Right? The conference? Okay. And I saw this morning with Jack when he was asking, there were quite a few people here that were first-time B-sides. How many of you are still here that were first-time? Okay, awesome. Awesome, cool. I've done a lot of these B-Sides conferences. I love them. These are fantastic conferences. It's a wonderful way to do something and learn

some things pretty inexpensively as opposed to some of the other fun stuff that's out there that costs quite a bit, which is why I'm so passionate about it. One of my other passions, though, is MFA, multi-factor authentication. It's not mother... it's multi-factor authentication. It's something that we're missing a lot in these day and age, but it's starting to grow pretty good. A little bit about my background. First of all, any CISSP's in here? Okay, we got a couple of y'all. A few years ago, there was an email that went out that said your annual CPEs are going up to 40 per year. Does anyone remember that? Yeah, my name's at the bottom of that, okay?

So don't throw things. All right, I've been hit by more than one chair over that one. I was the director of member relations and services at ISC Squared for a couple of years. What matters here is I spent about 10 years of my career, which started in the 1990s, back in bulletin boards and Fidonet backbones, if any of you all remember that, right? Okay, all right, these are my people. So I've been doing it for a while, but I spent about... I spent about 10 years with Department of Defense as a contractor for the US Army. So I was active duty Navy, then I went Army as a contractor, ended up the security manager at

the second regional cyber center Western Hemisphere, which sounds super impressive, really sucks when you have to answer the phone like that every time. But we were the organization, we rolled out the active directory for all of North America, for all the Army post camps and stations. It used to be called the CTNOSC at the time. We also did the network infrastructure for all of North America between the Army stuff. So we did a bit of time with DoD. And if any of you were DoD or had DoD, been around DoD, we had these wonderful thing called common access cards, which we'll talk about a little bit more, which was actually a pretty good multi-factor authentication

piece when they worked. But we'll talk about that a little bit more. But that's kind of my background. I've been doing this for a while. Like I said, I have been in the healthcare industry, manufacturing industry. I've been through a lot of different things over years. And basically, you know, we see these issues that pop up. And this is one that, again, I'm very, very passionate about. The company I work for is called Nobifor. We're out of Tampa, Florida. I brought my weather with us. This is afternoon Tampa weather right here outside. But yeah, we do security awareness training and simulated phishing stuff, so really not much to do with what we're talking about here,

but I do happen to work for one of those great companies that is really all about security overall. We know that humans mess up, and so... Anyways, that's where I'm from. So what we're going to talk about today, damage done through credential theft, hardware and software tokens, PID certificates, etc., etc. Now, I will tell you this. I do not pretend to be an architectural know-it-all when it comes to this stuff. The stuff I'm putting out here today is from my experience, my knowledge, the things that I've learned, the things that I've seen. If you guys have information on some of this stuff, please feel free to put it out. I like it to be interactive,

but don't think that I'm not open for discussion. There's one point, you'll see the slide. I'm willing to fight over it, you'll see that later. So damage being done through credential theft. Now one of the biggest things I see is people don't realize how significant the issues are out there and why we're bothering with multi-factor authentication. Credential theft is a big deal. Unfortunately, we've gotten kind of used to it. We've all seen this one before. This is the world around us. How many of us hear a breach happened and we just kind of roll our eyes? Okay, yep, again, breach fatigue, yay. We're just living in this world now where credentials are getting dropped all over the place. How many of you heard about collection number one? 773

million passwords and usernames. Then there was collection 2, 3, 4, 5. There's actually like five of them. But that made a big deal, 773 million. What was interesting in that is of that 773 million, there were only 22 million unique passwords. What's that tell you, right? We're a reuse culture. But that was all brought together from all of these different breaches that have been going on, put together, and now they're using it for some stuff. So we've gotten too used to that. The other thing is, organizationally, we're not doing our part to protect accounts. So 74% of you in the survey here said their breach involved privilege access credential abuse. Only 48% had a password vault, and

21% had multi-factor authentication. So of the breaches, 74% said it had to do with credential abuse, but they're not doing much about it, right? Anyone here use password vaults? We're going to talk a little bit about that. Yay. Okay, good. Maybe we're making some progress here. I still see a lot of people at these talks that go, what's that? So this was disproportionately on the positive side. I like that. And we'll talk about password vaults here in a little bit. Exactly, exactly. So what's happening with these things though is they're using these passwords for things like credential stuffing. And what credential stuffing is is taking these lists of usernames and passwords and then trying them

on other websites in an automated fashion. And that's where so many of these account things happen. That's where so many of these people that go, "Oh, my account was breached. My Amazon account was hacked." Well, it's because you use that on every other website that you've ever logged into. Most of the time, it's just an automated process of trying that somewhere else. Over and over again, we see this sort of thing going on. That's why the attackers are doing a lot of credential phishing, too. If they're not pulling them off of these other breaches, they will get you to put in your credentials, and then not only will they take over that account, but they're

going to turn around and try other accounts as well. It's just working for them. So we have to understand that what we're doing with just a username and password just isn't flying anymore. So when we look at some of the things that have happened, Samsung Hancock Health, they got hit with the Samsung ransomware. This was done by a vendor who gave up credentials and then The SAMHSA group used that to hit the remote access to Hancock Health, cost them like 50 grand in the ransom that they paid for their data because they couldn't restore their backups fast enough, which is unfortunate. But business email compromise attacks are the other things that happen to this. So somebody basically gets a hold of credentials, let's say in

a real estate office, and they sit there and they watch the emails come and go. And then when it's time to transfer money, they'll either hit them beforehand and say, this is where you're going to send it, because they know that that's the day the money is getting transferred. Or they'll follow up and say, hey, messed up, here's the account it actually needs to go to. And the money gets transferred somewhere else. We're seeing this in escrow, we're seeing it in commercial and personal real estate down payments. Okay? Earlier, actually it was last year, I got to brief a bunch of the House subcommittees in DC and one of them, it was terrorism and finance.

And it was, there was a guy there, he wouldn't give us a card, you know, very menacing looking. At the end of the discussion, he pulled me aside and he was with Secret Service, okay. And they do the finance piece, definitely. But he said, "Man, I just worked a case like this for an organization, it was a real estate, it was a title agency, I think, they ended up transferring $500,000 sideways to a bank. It was actually HSBC. They were fortunate they were able to put a lock on it because HSBC has been under a lot of heat for that. But the attacker still got $40,000 before they stopped anything. And all they did is

they sat in the account and watched the stuff go by. And then when it was time, they said, oh, send it to this account number. There was just one in Florida, personal real estate, lost $77,000 down payment for their home. So this kind of stuff is going on. It almost always starts, though, with somebody giving up their credentials and somebody getting into that account and just hanging out for a while. And then the bad stuff really starts to happen. Now, we are part of the problem, too. as security people, right? We tell users to make this unique million character password, special characters, numbers, mix up the, you know, never reuse it, and then we chastise them when they write it down, okay? What do we expect from them? We've

put all of these rules in place. Hell, we can't do it ourselves. And yet we're going to chastise the users when they go and write it down. We're going to make fun of them for that. But part of that is we're not giving them the tools to keep it secure enough, and we're having these huge expectations on them. Now it's true, of course, that longer passwords are harder to crack. But really, how many people are truly cracking passwords these days in order to do that? They're reusing them. That's really what's happening, right? So it's a matter of focusing on where the real issues are and how can we make this more secure? The problem is not only these days are people writing them down, but they're putting them in

the notes in their phones, right? We know this. They're putting them in Notepad or Excel, okay? You know, I see every once in a while on Twitter, somebody pops up something, they see a picture of the password book in the bookstore, okay, and somebody sees that and they post it up and they're like, oh yeah, look at this. But I'm going to tell you something right now. For older people, for elderly people, it is far more secure for them to write that down in that book and put it in their safe or lock it in a drawer or keep it off of their computer than it is for them to type it on the computer.

We want to laugh about that, we want to joke about that, and we want to make fun of people for writing down the passwords like that, but I'm telling you, it's more secure for them to do that than a lot of the practices we see in the digital world. So we got to think about that from a little bit different angle, because we've been making fun of people for doing things that are actually a little bit better. So what do we do about some of this, right? Well, for one thing, multi-factor authentication will eliminate a lot of issues. It does. I always think it's better to give up or better to avoid giving up the

credentials in the first place. Does anyone disagree with that, right? It's always better not to give it up. But when they do, two-factor or multi-factor will definitely help to reduce the damage that's being done, okay? Here's the thing, though. Not foolproof. I hear it. I get it. Most of the time or a lot of the times when I do talks like this or I talk about it in groups, I get the whole, well, people can bypass that. They can do this, they can do that. And we'll talk about some of those sorts of things. But really, it takes more effort most of the time. The effort's not made unless it's something important. So it's not

foolproof, but it's still far more secure than the single factor. And here's the big one for me. If you're not protecting your critical accounts with some form of MFA, you're heading down a very ugly road fast. And yeah, those are outhouses, and that's a convertible, folks. Okay? This is something we need to understand. We have to be doing our critical accounts with some sort of additional tax. And a lot of times we don't. So we're asking for trouble when we do that. Now, talking about these things. Well, it's interesting to me sometimes people forget what factors are. We've heard it. If we've taken the CISSP exam, we've probably been through it. But we tend to forget

what really makes up a factor. And so I want to do a little quick refresher on this. When we talk about factors, we're talking about a combination of something you know, something you are, something you have. For example, a password is something you know. A fingerprint is something you are. And a key card is something you have. So just a little refresher there. Multi-factor is when you combine more than one different factor. And this is where people are still getting tripped up these days and I have to remind them. A password and a PIN is still a single factor. It's a two-step, but it's still a single factor. So when I talk about multi-factor, when

I talk about two-factor, I'm talking about two different kinds of factors. Now, I've seen three-factor in play where I've really used it one time in my life. And that was in a room that we had in the military. It was a low classified area that we went to. We had to have our CAT card. We plug it in. There's a PIN number that we had to type to unlock the certs on the CAT card, and I had to do a biometric fingerprint. That is three distinct different factors, but not a lot of places have I ever seen that they actually have three distinct factors. So something to think about when you're talking about factors is

they have to be different. Now, HOTP versus TOTP, words, words, words. Okay, there's a bunch of them up there. I can make these slides available to you if you want to afterwards if you want to read some of this stuff. But basically, HOTP is HMAC-based message authentication code. Before we get too carried away on reading all that kind of stuff, basically the idea behind HOTP is there is something that increments a counter somewhere. Whether it's on a key, whether it's on the back end, a counter is incremented. usually with a successful login, and that generates another one-time password, okay? As opposed to TOTP, which is time-based. Everyone's had those little RSA keychains in their lives, right? Everyone's had their

Google authenticator out, right? And every 60 seconds that thing changes, right? Is there a bigger tension builder in this world than when it starts flashing red or starts flashing, right? You're about to change and you're trying to key it in. You're like, "Oh, good God." And all you really got to do is wait a couple seconds. But it's a huge source of anxiety for me, not going to lie. But the key is those are constantly cycling. Nothing has to be incremented other than the clock, right? The problem is though you have to keep time synced between devices and that can be a problem, right? Whereas the HOTP ones generally have that counter somewhere. So the pros are though, although you don't

have that anxiety and stuff, an HOTP password could be sitting there valid for weeks or longer if you don't successfully authenticate. So just a couple of different things to think about when it comes to that, just so we understand. Now, FIDO2 and WebAuthn, this is a new standard that's coming out. It's just getting going. Windows is actually supporting it through Windows Hello these days, which terrifies me that Windows can be treated as a FIDO keychain. The idea here though is to honestly get rid of passwords. basically to eliminate the use of passwords. What I have in my hand right here is a FIDO2 key. Now the only thing I've really found that it works on is some Microsoft accounts and Dropbox supports FIDO2 at this time. What we have

to understand though is most of these keys, if you just have a key that is not a multi-factor authentication, that is a key, this is your username and your password, you plug it in anywhere you are that person. So like this one here actually has a biometric fingerprint reader. That at least gives me some protection on that, but I see people talking about keys that all you do is plug it in and now you have access to all their websites. Well, that's a YubiKey, right? That's a FIDO U2F YubiKey though, right? So FIDO 2 and FIDO U2F are a little bit different things. When we're talking about this FIDO 2, That's the problem though, is

once you have it, you have it. So make sure if you're going to do that, don't confuse it with things like UB keys, but know that this can be a way that while we can eliminate passwords, if you own the token, you own that person. So when it comes to multi-factor, key ones that we know, SMS-based, right? Has anyone here not gotten a text message with a code? This is stuff that our grandparents get because they bank. It is the most common type of multi-factor in use. Here's the key. It is one of the easiest to get people to embrace. This brings me to my biggest point. If you're looking at rolling out multi-factor authentication in your organization, you have to understand that one of your

biggest issues is going to be getting people on board with it. We have to understand why. So imagine it from a user standpoint. For years, I have logged in, I have gone to my Gmail thing, whatever, I put my username, password in, I read my email. Life is good. Now you want me to put in my username, put in my password, wait to get a code, mistype that two or three times, screw it all up, finally get in to read my... Why am I doing these extra steps for the same result? You're asking people to take extra steps for the same result. This is going to be the biggest challenge that you have when it comes to rolling out multi-factor authentication. And this is why I

kind of put these things in the order that I do. Text messages, very simple. It's something we're all familiar with, people are used to doing. These days, pretty much everybody has a phone on them, right? So that's not really a hardship. Not only that, on smartphones, most of the time you do your login, the thing rolls in, you get it on your notifications, you don't even have to unlock your phone to do it. Most of us, unless you're more secure than me. The point is, it's pretty easy to do, right? You get that code, you type it in, and you go. Remember that when it comes to getting people on board. It is the easiest

to get people involved with. What's that? - System seven on or with the state. - Yeah, there's, we'll talk about some of that, yeah. But here's the deal. It's better than nothing, right? It is better than not having a factor at all. That's the key. So there are other issues too though, like this. I was doing some work, I did a demo on a tool called Modlishka. which basically you can get it on GitHub. It's a man-in-the-middle attack for this sort of thing. And basically the way I was using it is log into a Gmail account, you put in your factor, it forwards all that stuff and then steals your session. Okay, so pretty slick

how that works. But one of the things I noticed when I was troubleshooting it was if I did not get, I was having problems with the tool when I was troubleshooting, and it would not reset my code. Look at this. this was about 15 minutes worth, my SMS code was the same every single time. It wasn't resetting the code. So it's not perfect. There are ways around it, there are issues with it. I would hope to see that this resets much quicker than it does, but it was relying on me successfully logging in with this and my second factor in order to kick off the next code, which could be a problem. You could have somebody grabbing that code, you could be doing

all kinds of stuff while you're trying to figure out why it's not working, they're logging in on the back end. So, not perfect, but then there's also the application-based 2FA. How many here use like a Google Authenticator or something like that? Right, okay, awesome. So you're familiar with that, doing the time countdown. Now here's the thing about that, if you're looking at rolling that out, that's going to add another step to people. And while it's similar to SMS and the fact that people have to have their phones, and everyone but my father-in-law, honest to God these days, has a smartphone. My father-in-law still has a flip phone. I don't know how the hell he gets them. I don't know where he gets them from. Okay, but basically everyone else

has the capability to be able to run something like this. But you have to log into your phone. You have to open the application. You're adding some more layers there, which makes it more difficult for people. It's even adding a little bit more for people. This is where I want to see most organizations get to is to use some sort of an authenticator like this because it is the best. Now, it's not perfect. Again, you can man in the middle stuff like this. It's not an end-all be-all. It certainly isn't. But in my opinion, for most places, It's the area that gives you the best of both worlds, if you will. But you're still gonna

have to fight people. And sometimes what you wanna do is if you can, roll out something with SMS and then move them onto this. That way they get a little feel, their foot in the door, and then move them into something like this. Last but not least are these little beauties. I am not a YubiKey spokesperson, but I love these things, man. I really do. Hardware-based factors to me are fantastic. This includes smart cards. This includes all of those sorts of things as well. I'll tell you right now, in the military, we had the CAT cards. They had the certificate, the PIV certificate, your smart card credential on them. My biggest issue with those was they failed a lot. At about 250, 300 people in the

organization, we used the smart card for our standard login. We used a smart card for our elevated privilege account. We used a smart card on the SIPR net for our login. We used a smart card on the SIPR net for our privileged account. Everybody had three, four, sometimes even more smart cards. I would replace those things probably, I think I was at one point doing about four or five a week. that I was having to replace. The failure rate was extremely high on them. Now, they gave us some really cool stuff, which I'll talk about, but they failed a lot. Now, I will say all of these UV keys right here are mine. The one

on the left, the Neo, I've had for about four years. It's been through the washer, the dryer, teenage children. It has survived everything next to a nuclear holocaust. They are very, very resilient these days, but damn, they're expensive. and they're a lot of work to provision, and not everything supports them. Also, imagine you're a, I don't know, intelligent, good-looking, humble speaker that goes and catches a flight somewhere and accidentally takes his wife's keys instead of his because he takes a wife's car and then needs to get in their email. You know, this is a friend of mine, close friend. That can be a giant pain in the butt. So the problem with these is... They're really secure, but incredibly inconvenient where forgotten or damaged.

And you are asking somebody to do a lot of things with these. Now, they're cool. You get to that login screen. You plug them into the little USB thing. You touch the little disk. Bam, it generates a password. This is HOTP, so then it increments it for the next time. There's actually counters on there to make sure you can't spoof the keys. There's some cool stuff that happens in there with the basic OTP there. But they're really inconvenient if you forget them or lose them. That's why I have some spares. But what people will oftentimes fall back to is making a lower security fallback, which is now your most secure of the accounts. Yeah. Now the flip side to that is, when you enroll a lot of

these, they'll give you a chance to print a group of one-time passwords, usually four, five, 10, something like that. Print them, put them in your safe, put them in your password book you got that people made fun of you with, right? Keep those on hand like that so you can maybe call home and say, "Hey honey, I did something stupid. "I needed my email. "You wanna give me that code?" Again, close friend, very close friend. - Did you see Google's security blog had some stats they published and this device had 100% success rate blocking whatever attack profile they were saying of all their multi-captures. This was the one that was perfect in that case. - Okay, we're gonna

get into this. So Google, the marketing geniuses, posted a story that Krebs picked up and Krebs said it eliminated phishing in Google for a year. No, it did not. Now coincidentally, about a week later, Google says, oh, and we're selling one. Okay. There are limitations to what MFA is gonna stop when it comes to phishing. Phishing is not just credential theft or logging in like that. There's business email compromise. In other words, there is a wire transfer fraud. There are scams like that. It does nothing for those. I actually have an ethical hacker network column on that I wrote last week. That story was almost pure fluff, okay? The point is, yes, it can eliminate account takeover, but that's one specific type of phishing attack. But

it did not stop all of the phishing attacks. It's kind of a soapbox sort of thing for me on that one. But it's Google's marketing, which is fantastic. We have to admit, it's fantastic. Now, when it comes to 2FA2, even with the hardware ones, we have to imagine attacks against them, they happen. Most of the time, it captures on pulling the session or the additional factor. In other words, they make the phone calls, says, "Hey, this is your bank. We're doing a new security thing. We want you to beta test it for us. Need you to go to this link and put in your username and password." Cool. Okay. Did that work? Now, you should

have gotten a code. Put that in too. "Hey, thanks for being a part of the testing. Okay, and now they've captured your session. Beautiful, fantastic. There's attacks like that that are on this side, right? So we do have to watch on that. Usually it's going after the session or the additional factor. They'll get the number and if it doesn't reset like that or they do kind of like what I did there, they turn around and enter that number for you. Now, The thing is for TOTP, capturing the generated code and reusing it is tougher because it's a limited lifetime. There are SMS attacks performed by fraudulent port outs and against SS7, like you said. But

honestly, phishing remains the easiest. So if you're in a super high targeted sort of thing where somebody's gonna put this kind of effort into it, those are definitely things you have to concern yourself about. But if you're Bob's Donut Shop, probably not going to go through the trouble of trying to do that for you. Probably not. So it's a matter of looking where our wrists are. Now, we've also got to think about this. Speaking of the Google keys, this just came out. Disclosed Bluetooth flaw on Titan security keys. Now they had to recall it. Don't forget the communication protocols. I actually talked to the guys at Ubico about this a while back. I think actually

it was RSA. I was talking to them about some certain things. And I said, why don't you do something like this? And they said, we don't trust Bluetooth. Brilliant. Good on them. Yay. Google did. Now they're having to recall all the keys. So remember your communication protocols that are involved in this. Much like anything else, we don't break encryption. We break key handling. We break protocols. We break communication strings. We break that, but not the actual encryption. Same thing happens on these things. There was actually just one here against my little fanboy group. Peace out to my homies. They just had an issue with how their FIPs their FIPS keys work. Now this is a limited type of key. It's a very

specific type of key. It has to say FIPS on it for it to work. But the problem was with the random number generator is not random enough. Who here has ever heard of that vulnerability in cryptography, right? Okay, and it's like there's a certain process like if you plug it in while the machine's booting up and you're standing on one foot or I don't remember, you know, you got to cross your eyes or something. But it's enough that they recalled all of these keys. You do want to keep an eye on them and they're not invulnerable. We have to understand that. We've actually seen hacks where YubiKeys can be turned into, because it works very

much the same way, a human interface device, right? So you turn it into basically a rubber ducky. Kind of fun. I think it may be more expensive than a rubber ducky in some cases, but there are vulnerabilities with these sorts of things as well. So nothing is perfect. But let me ask you, how many here have a security control in their organization that is 100% perfect? Yeah, that's what I thought. No hands. Thank you for not raising a hand, okay? This is something that we have to understand is nothing is 100% perfect. We're doing our best to limit at the levels that we need. So, PIV certs, smart card logins. Again, big fan of this because there's some things you can do that a lot of

people don't think about with this. PIV credential, it's kind of the, I don't know, it's a government sort of thing. This is a smart card here. Basically, the PIV is a pin-protected PKI key pair. that can be used for authentication, encryption, digital signing. Something we did with these things that I just absolutely loved and hated sometimes was the ability to digitally sign and encrypt emails within our email system in the Army. It was fantastic until it didn't work. And then you have to go do a bunch of stuff because all of your emails are encrypted to the card that just died, right? But it was fantastic. Just the ability to be able to digitally sign an email using an Active Directory

certificate was phenomenal. This was back in like 2005. We were digitally signing stuff like this. So now somebody tries to send me something and says, "Hey, this is Colonel so-and-so." I can tell if it's Colonel so-and-so or not. because it's digitally signed. Now here's the thing, if you are in a reasonably current Windows Active Directory environment, you have native smart card login support. You can build these certificates and it's built into Windows. It doesn't really cost you anything but the medium on which you're gonna put this stuff. So, there's a couple ways to do it. One is the cards like that you can get them on Amazon. Now, there is a cost to that. You have to pay for the cards, you have to pay for the

readers. - - Well, YubiKey does have one. This is actually off of my YubiKey. The thing is, the YubiKeys, like the Neos and some of those, they tend to be on the more expensive side for the ones that'll take PIB credentials that aren't just a simple U2F sort of thing. So you pay for those on the front end, okay? But you don't have to hav