Fighting The Third Party Risk Monster

uh hello everybody good afternoon i hope you had a good lunch appreciate you guys sticking around with me uh after lunch uh we'll be talking today about third-party risk and what a monster has become and how we can get that under control um first though uh let's just uh take a look at the pieces first we'll kind of do some backstory uh how we got here uh laid that lay the landscape and the scenario um by the way uh uh apologies to people who aren't uh role playing in d d geeks but there will be lots of references throughout there i think i had that in my in my uh abstract so fair warning um

then we're going to look at uh what kind of adventures we have in trying to wrangle this beast and finally let's level up with some best practices for third-party risk management and how you can get it under control so uh quick uh shameless personal plug who am i uh tony howlett i've been around infosec for over 20 years and uh in i.t for longer than that um i've worked as uh in various roles cto see so before we had csos we just had ctos uh and done startups uh other companies and i've worked in uh pretty much every regulated industry uh telecom finance government and healthcare i currently hang my hat at securelink we're a vendor privilege

access management software company and in my spare time i like to sail i like to golf once in a while and play the drums so you see there's some my boring uh professional headshot uh me doing something more cool which is sailing that's actually an america's cup boat that i'm sailing and uh my proudest moment was when my band got to play defcon 26 in 2018. uh that was super cool if uh you caught us there doing our live karaoke gig um so enough about me let's roll up a character and get this show on the road

so how did we get here um well outsourcing is a big part of it and outsourcing is great uh it brings us a lot of benefits um and we kind of eased into this situation we started outsourcing non-core functions right uh i mean if you look back 10 or 15 years ago you know you really didn't if you outsourced anything maybe it was accounting or something like that or sales management but as time went on uh we outsourced more and more functions to the point where now many companies actually outsource significant parts of the it portion of their core business um you know a lot of banks now uh they use a a sas platform for their entire bank

function basically bank in a box um a lot of hospitals are using technology vendors for their key you know it's not just a doctor with a chart anymore they use a lot of technology so they depend on those vendors it pretty much covers the entire um it spectrum everything from your hardware to your software uh and even your infrastructure where it used to be you felt like you had to own the servers on the on the colo we've slowly extracted that to the point where a lot of that is run by other companies um and you know it's like i said there's a lot of good reasons for that and we like to have people do

the jobs that we don't really want to do um a lot of talented i.t people don't want to you know maintain a mail server or maintain a website and do all these you know look through logs spend all their time especially if you're you're um you know a single staff or a small staff so you like to have that as a npc non-player character to throw out there to do those boring things open up that door uh for me take that danger sorry sorry sorry to pause you tony real fast um i believe your screen share is not showing right now sorry about that everyone thanks for your patience we'll move on but anyways uh our npcs can be useful our vendors

are third parties but they can uh what this has caused this massive outsourcing move is this tsunami of third parties with the average enterprise having about 67 vendors with some sort of privileged access within your systems and networks you may have a few more you may have a few less but this is what the average enterprise has on the flip side of that the average technology vendor has 238 customers that they service and that creates a a motivation for hackers to go after these guys and we'll talk about that in a bit but you can imagine why hack one enterprise when you can hack one company and get hundreds of enterprises possibly and this ends up with a lot of vendor

reps wandering around on your network hopefully they're not wandering around they're doing the jobs they were meant to be but maybe they get bored maybe they're malicious or maybe a hacker's taken over their account they might have privileged accounts maybe they're accessing it through some simple form of access like a vpn with not a lot of controls on it they might be sharing credentials maybe you've given them one login for the entire company so they don't have to be uh you don't have to be putting different reps in and now they've got that on a whiteboard in their in their main office maybe like in the conference room by the window um and as i was saying they're not just

into your back end systems or your website anymore they might be running critical systems databases things like that uh they might contain sensitive data possibly regulated data that you're supposed to protect so that's a lot of folks wandering around maybe not i'm always as skilled as you might always think they are again it's great to have this help but and hopefully some of these folks are skilled uh maybe more than than your companies that's why you've hired them but they might also hire some new folks and be training them uh and a newbie rep can get you in a lot of trouble there's your first level character joining your party uh and they can get

you in a lot of trouble by uh alerting the monster knocking down the wrong door or letting a hacker in that door and this perfect storm really has resulted in a lot of major breaches that are related to some third party that come in via a vendor or a third party um this is just a short list but um target was kind of the uh er large breach back a while ago uh you all probably heard about it when it happened uh hackers got in it was a christmas time and stole about 41 million credit card records it was a big deal the company lost a ton of stock value in sales and most of the c-suite ended up getting

cleared out what you may not know is this was caused by a lowly hvac vendor a company that was really supposed to only work on cooling systems was in a project management system and hackers took over or got into that vendor got into the network it wasn't properly segmented so they were able to jump onto payment systems and get get those credit cards a little more recently uh this last year lab corp and quest diagnostics probably uh have gotten a test from these companies these are the largest companies that do medical tests if you've got any kind of blood work chances are you visited one of these companies for it they were hacked 20 million plus records

uh and this included their billing information of their customers and services rendered so if you can imagine um you know what kind of test you were getting maybe it was an aids test or a drug test or a cancer test very sensitive uh that phi information that if you're in health care you have to protect and the way they got hacked is through a collections agency again pretty low tech seeming uh not something you think of as being a critical vector but they did have all that information they got hacked therefore those two companies plus several others got hacked capital one was also in the news last year they had a online application form and it got hacked 100 million

credit card applications got stolen and again imagine what's on that it's pretty much your entire financial history so your social security number how much you make where you work et cetera et cetera very valuable information to identity thieves and this happened through a broken aws application firewall so um again the fingers are being pointed and capital one says it's aws's problem adbs of course using their shared responsibility model says no that's your responsibility and that legal fight will go on for a while but capital one is the one in the news capital one is the one getting sued for all those records uh getting breached um another one uh this myfitnesspal.com app and you may not recognize that particular

name but they run a lot of these uh track my bike ride track my run type of apps they have a bunch of them and they got hacked their customers emails usernames and passwords were stolen and if you think about what's stored on these phi personal health information so it's going to be maybe your heart rate how much you weigh pretty sensitive information um and that was 150 million records like that's half the country in terms of numbers and the way these hackers got in is through a security vulnerability in an acquired business unit so um the lesson here is if you're acquiring companies or maybe being acquired you definitely need to do your due diligence

because you don't want to acquire their vulnerabilities and that happens quite a bit and my final example is also last year this was um what i call a mass ransomware attack where uh they launched attacks on 22 separate small texas cities uh and this affected their municipal sort of management systems their law enforcement systems 9-1-1 payment utilities or utility payments um all the core things that that a small government does was shut down and demanded ransom uh they launched it simultaneously so that it would be very hard uh for um these folks to to react and it would draw a lot of resources in fact the um the governor actually declared a small state of emergency because the

texas rangers security cyber security unit had to come in and help um we don't really know how many records were possibly stolen but thousands of citizens obviously were impacted if you needed to call 9-1-1 or needed to access a property record you were out of luck maybe you're closing on a home uh pretty pretty massive several more have happened like this in louisiana and several other states and this came in through a managed service provider they all used jointly happened to be a company that that ran i.t departments for small uh police offices so if you're in a really small town and you just have a couple officers maybe they're just out there giving tickets at the speed trap

but uh you don't have an i.t guy and they would run the i.t for you well turns out they weren't using mfa to get into those remote systems it got hacked and they got into all the other city systems as well so that was pretty bad and again this is just a representative example there are hundreds and thousands of more uh out there that you can look at here's an example i think that i'd like to bring up to identify a emerging vector uh hacked by fish well i mean real fish uh this casino had a large fish tank like you see behind their their uh check-in desk sometimes and uh of course i doubt they were thinking

this is a cyber risk when they when they put that in right one of the fish gonna jump out of the of the tank or whatever but it did have an ip sensor in there ip based sensor that would check the temperature and let the servicing company know when uh when that needed to be serviced well they got hacked they got into that got access to the network jumped over onto the payment systems and of course got the valuable stuff uh again i doubt the it people ever thought or worried about this little fish tank may not even know about it for that matter so that just gives you a feel for this world of iot of internet

of things and how much of a threat it's becoming i think it's approaching the threat of the servers and workstations because we just have so many of these devices y'all may have heard of the i don't have a slide on it but the ripple 20 uh or 2.0 however you say it announcement about two weeks ago they found a massive vulnerability actually 19 of them in this ip stack built by a company called trek you've probably never heard of them because they build tcpip stacks for iot devices like webcams and things like that uh and it literally affects hundreds of millions of devices we don't really know the full number because we don't know who all used this this um

[Music] uh stack and even there's companies that use equipment that use equipment that used it so uh we'll talk later about the supply chain problem but um we don't really even know the full scope and this is going to be a problem for years and years trying to get just this one issue resolved so i mentioned earlier uh the attraction of hacking a vendor or someone that works with a bunch of customers and has access to their i.t and uh there has been a concentrated concerted targeting of vendors and managed service providers because the hackers really see this as a force multiplier right again i can hack one company and get out possibly get access to hundreds of

networks probably some some big people too because a lot of these technology vendors are smaller than their customers so i might be a little no-name uh a hvac vendor but you've never heard of them uh but they got into target and that's the kind of thing they like and these managed service providers provide services across a wide spectrum i mean it services is probably the most common but security web hosting like your wp engine and wix uh networking services mdm uh security monitoring companies like alert logic and secure pipe uh data storage a la the uh amazon s3 and so forth uh and you know the big big uh state actors the apts as we call

them advanced persistent threats are using this extensively china uh has hacked into at least 45 msps and tech companies and they have actually specially designed tools to go after these these type of vendors uh and again that that texas uh city ransomware attack was packed through an msp here's uh just a specific example now during covid they've even focused tighter on healthcare msps uh nice guys that they are um this quampears malware uh targets uh healthcare msps uh and you know they're not completely heartless though some of these ransomware folks have been offering discounts if you get uh your healthcare organization gets what nice guys i mean i really appreciate them giving a discount to

uh our hospitals but yeah it's it's pretty bad and they they um i'm being facetious of course they take advantage of uh these situations and will kick you when you're down and you know uh you may think that uh well i've got all these contract clauses and they're going to be liable for my breach and i've locked them up tight in the contract phase that's great you should definitely do that but in a large breach like some of these uh your your third part of your small vendor is probably going to go out of business they're not going to be able to shoulder um the the costs and and the lawsuits that you're going to endure so

in the case of the quest diagnostics and lab corp hack the company that caused it they went bankrupt right away so that's going to be on labcorp diagnost diagnostics and those other companies to shoulder the costs and the burdens and uh i spoke earlier about the vendor of your vendor's vendor and this is called nth party risk so we're not just talking about the third party we're talking about uh the fourth party in the fifth party um you can see that kind of webbing out from there so example you might have a vendor that does something for you they host some system you use but they actually store that on aws or some other colo provider et cetera or

they might use some tool or something that's another company that stores your data in some place you're not aware of and again it only takes a breach of one of those people in the chain possibly for the bad actors to work their way through um so you may end up feeling kind of like this fellow here uh surrounded by a thousand orcs and getting attacked and and really surrounded on all sides by the way extra credit uh for uh knowing who this where this picture comes from you can put that in the chat if you want uh but um that is how you might feel when you're trying to manage this like this massive web of third parties

so what we have is a bunch of different kinds of risks uh that these third parties bring um the operational risk right so these hospitals and uh some of these power plants and manufacturing companies uh ransomware or any company really takes them down that's time and money right um especially when you're talking about key infrastructure and healthcare where it could be actually a life critical so uh you don't want to be down in those situations and that's what the hackers know and that's now why they're going after on using ransomware more and more as the primary attack obviously security risk when you have all these users uh connecting into your network and systems hundreds maybe thousands because

if you have dozens or hundreds of vendors they're going to have more than one rep and you might have even thousands of users wandering around your network doing different things uh the money uh for these types of breaches is significant uh the average uh cost is up six point four percent over last year to three point eight six million that's average even the smallest breaches i can tell you every breach i listed in my in my table there is is a tens of millions if not hundreds of millions of breach and some of these are now reaching into the billions when you talk about the cost of the lawsuits and the regulatory fines and if you're in a regulated industry

you're also looking at a regulatory risk so not only might you be hit with money fines and lawsuits but you might get shut down or face operational uh limitations and so forth like with banks you have your charter uh you can get sighted and hospitals the same thing so it's not just money it's it's uh uh the regulators can come in and shut you down and even find you criminally liable if you're uh you know if you're criminally negligent so all right we know the problem is big let's sit down let's form a party let's uh let's solve this problem let's go after it uh what are we looking at what are the challenges well it turns out there's a bunch of

them if it was easy it would be it would have been fixed a long time ago so here are some of the barriers to a good solution to this third party problem um well first of all money and resources uh a lot of i.t and security departments still don't have third-party management third-party risk management budgets you have to to make it up elsewhere or find it somewhere or do it in house you might be using the wrong tools for the solution uh maybe you're just using the same vpn that your your employees use i'll talk about why that's a bad idea in a bit but it's not necessarily designed for um letting people outside your company uh

they who aren't employees into it or uh each vendor has their own tool so they're then stuck with uh having to manage learn and support there are different uh support platforms there's no standard you got vendor managers uh not you know all over the place both in physically separate and logic logical different divisions uh so you've got all these folks who are managing needing to get people in uh that aren't necessarily part of uit department or even in your at your uh physical location and finally once you once you do come up if you do come up with a technological solution or a process or policy you've got to get the vendors to buy into it not so hard if you're a big

company and they're a small company but if you're us if they're the vendor is a ge medical or someone uh it's going to be pretty hard to to move them off of what they're using uh into whatever you want to have them do so you end up with this the sort of teeter-totter of needs where as you increase security the efficiency of those vendors go down and you might get complaints from internal uh stakeholders saying hey i can't get this app up and running um they can't get into services i'm sure you've all heard that we need to give them you know wide open access get rid of the firewall rules we just got to have this

thing up and then also you have your vendors who um you know they're typically graded uh and paid on on things like time to resolution and they have slas they have to meet security is not always one of them so um and you can build that in this in the contract phase uh but it's not always in there so uh as their requirements get met some of yours drop off and that's before we get to compliance right we get the real security done but then we got to satisfy the auditors and it really matters to pretty much everyone now 85 percent of companies uh state that they have to get third parties to comply with their policies for part of

their regulations and if yesterday or a year or two ago you were in a regulated industry you most likely are now because of the privacy laws right the gdpr for eu and now we have the ccpa in california so you may have been able to brush off gdpr concerns oh we don't do business in europe etc but everyone probably has does business with a california company or california customers that means you do have to comply so this adds up to a lot of time the average organization spends 17 000 uh personnel hours per year complying with various regulations audits etc if you're in the finance industry you know that you're getting auditors different auditors in and out almost every week

i saw that when i was in that side of the business so this adds up to about nine ftes which we'd all love to have in rit or security department um to do other things to do more productive things than just pull paperwork but that's the reality we have to do it so another problem we know the challenges let's let's see about leveling up here and how can we leverage uh some best practices for dealing with this risk well first of all there's a lot of regulations and your your organization may fall under one or many of these uh so you might have a health care organization where you have hipaa and high tech but you also do credit

cards you got pci maybe you're uh you do gdpr uh you probably have to do ccpa for california folks so there's a lot of regulations which have different stipulations different levels of of uh stringent uh things so you know if you try to tackle them one at a time you might end up like this where you're arguing and i know we need to do this for pci i'm going to do that for gdpr um but you know you can really boil most of these down to three things uh you need to identify on authenticate properly you need to control access and you need to record an audit much of this is similar to what you want to do

with your employees but there's some important distinctions i'm going to talk about those so let's dive in and see what kind of best practices uh and uh spells if you will that you can cast it to to get control of these vendors um identifying your vendors this is the this is the most important part there's your identity identify spell if you're familiar with the dnd um problem is that most companies don't even know how many vendors they have coming in 37 percent aren't even sure not only who their vendors are but how many they have and here's the thing uh what i've seen is most companies who are sure of it are wrong uh they might say we have 10

vendors or we have 20 vendors and come to find out they have 100 and it's almost universal i've never seen someone have fewer vendors they always have more because you've got sas vendors you've got shadow i.t you've got a lot of things going on that you didn't have 10 years ago but the best practice is to move towards doing a comprehensive list of vendors um and uh you know i recommend starting literally with the general ledger uh and seeing who you pay that's not going to get all of them by the way because you might have people writing off things on credit cards on their expense report like even aws fees infrastructure on a credit

card it's very hard to track down if they don't report it um but you want to end up with who they are what are they doing for you and they might be doing multiple things if it's a large company and what are their access needs in other words is it uh remote access is it on site access is it privileged access uh so on and so forth and and you know this isn't something that's a one and done it evolves obviously you pick up new vendors things change um again i've never seen anyone on the first round get everybody but as you refine it as you refine your process and get policies and procedures into place

you can start to get closer to what you believe is the true number the actual number of vendors another thing that that i recommend as a best practice is is not trying to manage these folks like your regular users and active directory i know a lot of folks do this uh because it's there it's already a process but you end up with something like this where you have you know vendor acme sean vendor acme max uh it gets really unruly after even a couple of vendors um it gets out of date and uh you know after some point you have more vendors and employees in your a.d or again the naming conventions break down so uh it may work for a while but

it doesn't really scale well and there's some other issues with that too and i'll talk about that in a minute uh generic accounts if you get anything out of this presentation this is probably the the gem um you know having those those shared accounts those acme corp uh you know you don't want to sit there and have to manage all their people coming and going so you give them acme corp and again that gets posted on a whiteboard shared on emails or sticky notes and that that is passed around uh you know it's just not great and it'll also take you out of compliance with a lot of regulations especially pci and things what you want to move towards

is identifying every individual who needs access to your systems you need a process that creates these accounts efficiently gives them the least privilege they need and then then off boards them when they're terminated as quickly as possible that's again that's the gold standard that's what you want to move towards mfa multi-factor authentication this is probably number two as far as the most important things in this and this is becoming a standard now uh in regulatory frameworks so again pci and a number of things he just will require this uh so if you haven't moved towards mfa at least for privileged accounts highly highly recommend it it's a great uh way to kill a uh it's a great thing in

the kill chain and if you're gonna do it try to use a standard like totp so if the vendor uses uh authy or google authenticator whatever they use it's gonna work with your system and uh because yeah different vendors are going to have different ways of doing it so again the more standard based you can the better okay we've got them in we know they're who they are and so forth but let's make sure that they're still employed there right we talked about the uh how we're going to verify them so how we're going to get them into the system that onboarding process is very important and then i ideally automated and then how do we

uh off-board them how do we know when that person's quit because we're not in their hr department right we uh we need to know and get them out of our system as soon as possible all right number two is uh control um you see there the control weather spell i wish we could control weather especially in texas uh oh if only there were controlled vendor spell but here's the best practices to get you to that um so again i talked about up front a lot of this is happens before the vendor becomes a vendor when they're still being evaluated the more you can get into their contracts or the more you can uncover why you're evaluating them the better

um you know what access method is going to be used require them to use yours or at least know what it is and and approve it um require those individual accounts you're not going to give them a generic account require mfa um require them to sync up with you until when when reps are terminated um require them to to notify you if they have an incident and and so and also if they are putting your data if they're outsourcing if they have downstream vendors that's handling your data they need to let you know all this is stuff that if you do it in the contract you have recourse if you try to do it after the fact it's

a negotiation and they don't really have to say yes um you also uh again once you onboard these folks you want to you know you want to have a paper trail or at least a documentation trail to know uh you know who approved this what department what's the application owner and then who approved it who set it up and then they have access so you want to be able to unwind that that process and roll it backwards so if there's a problem you know where it came from uh again uh you want to make sure this vendor is actually secure uh are they using current and managed uh endpoint protection uh especially if they have reps that

work at home are they using you know free antivirus or any antivirus at all are they using the latest and greatest encryption so forth are they basically meeting the same standards that you need to meet and can they prove it i recommend especially if you have a lot of vendors that when you're doing your risk assessments you tier them because not all vendors are the same your janitorial vendor carries some risk with them it's mainly a physical risk um but they may aren't the same as necessarily as an administrative company managing your database so put them into tiers and depending on what they have access to and what level of access they have and this will allow you to

develop different systems and different levels of controls for different tiers of vendors least privilege we practice this with our employees hopefully so we should definitely be practicing it with our vendors and third parties all the more so um you really really really don't want to just throw your net vendors on the network with a vpn that's unsegmented um that is the recipe for disaster it's been the the main technique for a lot of these hacks uh if you're just handing them a vpn uh like your employees you know i always like to say a vpn is like an ethernet plug that's extended you just let them plug right into your network and hopefully they'll work on

the servers that you've you allow them to but nothing is really to stop them from scanning the network uh leapfrogging onto other systems this is pretty much a element of almost every successful deep hack so um make sure that they have least privileged access both from a network standpoint from a host standpoint even from application ports on specific server standpoint

control so i talked about earlier you've got these application vendors and you've got your it people and your security people and uh if the it people are just being handed uh they need to get access to an application that they don't really understand right they're not the expert they're just the administrator of access they will tend to to do a less granular because they don't know and they don't want that application vendor coming back to them or manager or vendors saying hey we don't have access got to troubleshoot this so they might be have a tendency to hand out more uh privileged or super easy access than they than they might normally if you let the

application owners delegate delegate down to them they tend to understand this users just needs view only and so forth they understand the application

so this is a fairly new innovation here where there's technology it's called credential vaulting that allows you to not have to hand out a password and a login to to a user to get privileged access really all they get is access to a lobby and they can check out that privileged access for the time they need it then check it back in they never actually have the login privileged login and that's pretty powerful because it uh they if that again that vendor rep writes it down and leaves the company they can't necessarily try to come in from a different angle um and uh also single sign-on things like this where you can turn off their access with a single

switch and if you can again push that that authentication down to the vendor into their ad usually companies when they terminate someone that's the first thing to go is their network access and if it's federated down to them then it's going to turn it off pretty much immediately you don't have to have a sync up process it just goes away and they take it out of their directory service so final piece of this three-legged stool is is auditing and reviewing um there's not such a spell in d d so i'm using the scrying spell but you want to be able to keep an eye on things once you've got them in you've given them least privilege now let's make sure

they are actually doing what they're supposed to be doing um so a basic audit you know this might be like a vpn log is gonna you'll know like destination and source ips maybe the username uh maybe a session start and stop time doesn't really tell you much uh other than someone connected and maybe did something but you want to get more granular with your your uh third-party logs you want to know who is the authorizer for this access why are they connecting is this regular course of business this is what they do every day or is this an emergent uh you know they're trying to fix some some problem is there a ticket number in a

case system that you have so you can go to that and see what's going on um and ideally this is the gold the holy grail of vendor audit would be to have your keystroke logs and your video capture of any graphical sessions you probably wouldn't do this for your internal people it's just too many but with vendors again that just gives you an extra level of control to detect any problem uh while it's happening before it happens or worst case if something has happened you can sort of uh put humpty dumpty back together again by backing up and seeing what they actually did forensically

so um if you're looking and you're reviewing audit logs um if you have them in a bunch of different places that it gets really difficult to to put the the puzzle together and if you've ever done this if you've ever tried to pull wrap logs firewall logs and then windows event logs and so forth it can be a real hard thing to get them all synced up and see what's actually happening so having a single sort of source of truth for your third-party access and an activity makes it uh a lot easier to see the forest for the trees you can kind of see what's going on see if it's benign or uh you know if

it's not you can investigate further and if you're going to keep all this great audit and log data you got to look at it so a lot of folks only look at their logs when there's a problem when they think there's an issue and that's probably too late you want to have a regular review process you want to have you know automatic notifications when certain things happen tripwires if you will and things like that so you don't have to uh go to your logs after the things already happened and at that point it's too late this is the reason why over half of breaches are discovered by outside sources so you get an email or call from the fbi or

some friendly white hat hacker or something saying hey find your data in this paste bin do you want it and at that point uh you panic so don't you know the way to get around that is to really look through your log data have a review process so maybe you can catch that before before it becomes an incident so uh kind of at the end of my deck here to summarize though the third party problem is is a clear and present danger uh it's becoming bigger and bigger as we have more folks on our networks than in our systems who aren't employees and you know uh there's a lot of challenges to it there are ways to

deal with these challenges and at the bottom line is really third party access shouldn't be treated as regular internal employee access you have to treat them differently and have different controls so there's my information uh if if you want to email me any questions later or get a hold of me about anything

you