A Journey To Zero Trust

the next talk presented by ground one two three four is journeyed to zero trust joshua danielson and brittany little delete garzarda hello my name is brittany little i'm from copart and we are going to give you all some insight into our journey to zero trust so our agenda for the day we're going to start with a quick intro of the team then we're going to go into what xero trust is what it means to us specifically we're going to discuss our solutions that we've come up with and that we ended up choosing and provide some insight into some of the lessons we've learned so far that we think are key to having a successful deployment and we're going to

close with how we plan to expand our zero trust journey and then we will give some closing points so i focus more on the product management here at copart and my primary focus is around identifying security risks developing strategic implementation plans for security products and then communicating with key stakeholders so hi everyone myself dileep gorazada head of absence co-part with my background in security automation and engineering allowed me to build and deploy solutions for wide range of use cases at copper and josh danielson i'm chief information security officer here at copart uh being here for quite a few years i've had the opportunity to work with a lot of great people and do some do some great things here uh so diving

into the next slide uh some caveats before we jump in uh what is zero trust well there's no one-size-fits-solution for everyone anyone who spends some time in the space know there's a lot of controversy and debates of what zero trust even means um the industry still lacks some standardization with the sa sase to zero trust networking access and otherwise too so we've for the purpose of this conversation we've structured it in a few different uh categories of use cases and also solutions to help us have a good talk and hopefully have some value to some other people understand what's a good use case for your programs too um a lot of organizations i know focus on

replacing uh their vpn products and solutions with the xero trust or an sae based type of solution uh for this purposes we're actually not going to focus too much on vpns as a lot of our applications and resources actually on the cloud so we'll talk a little bit about those use cases but i know that the primary use case that a lot of organizations function on that's not our primary use case here um also the last piece i know the traditional uh industry standard may be even focused around pattern matching and different track traffic patterns um that's not a case what we'll be covering here so with that we'll dive in and talk a little bit more

yep so same question that josh has asked what is zero trust and there's a couple different definitions floating around and we went ahead and just pulled one from cloudflare that says zerotrust security is an i.t security model that requires strict identity verification for every person and device trying to access resources on a private network regardless of whether they are sitting within or outside of the network perimeter so again this is just one of the general definitions that's out there but this could mean zero trust could still mean different things to different organizations depending on the situation use cases that you've defined so really just have to continue to ask questions like are we specifically talking about corporate

owned workstations that are connecting to cloud apps or on-prem apps are we referring to doing device authentication of these or are we also looking at the doing a device health check in addition are we including personal mobile devices or personal workstations and then also again are we going to be doing device authentication and device health of those mobile devices as well so really just continuing to ask these questions and and that's what we did to ultimately help us define what xero trust was to copart and so we really started evaluating what we already had in place today and we use an idp to do things like sso and mfa to evaluate identity but the next step

for us in xero trust was specifically focusing on device authentication and device health and we want to ensure that all corporate owned workstations that connect to our email and internal apps are authenticated with a xero trust digital certificate that includes mfa as well and then additionally each of the workstations must pass a device health check so this is really meaning they they must have things like antivirus network security and encryption enabled in the future we do want to add additional security requirements on the health side but this is the baseline that we started with if both of these things are fulfilled on a corporate owned device we will the the device will gain access to our specific resources

if there is no zero trust agent installed or the device has a low trust score meaning it's missing crowdstrike or umbrella or something like that then they're not going to gain access to our co-part resources again this is just what we've done to define what zero trust means to us now i'm going to pass it off to delete so he can talk a little bit more about how we define our access use cases thank you brittany so to define our use cases we need to identify the services or apps we want to protect and also the consumers of these services say an user or an application service itself based on who is accessing what we have

categorized zero trust use cases as we love if you see one of these cases user to cloud consider if your company is using a lot of software as a service applications like salesforce slag zoom office 365 etc and in these scenarios are you utilizing any mobile device management solutions to handle workstations or endpoints and then the second use case is around user to on-prem services in this scenario consider if your company is trying to protect internal tcp services like rdp ssh or https and typical answers include network access control solutions or basically vpns that are out there that can help you provide these remote access especially during this fandom and then the third use case is machine to machine

in a typical virtual machine deployments based deployments production application environment allows all traffic from one server to the other or may not be restricted by a network so our usage should be restricted only to add traffic from a server or container but in a container-based deployments we use calico policies or service mesh architecture to provide network level access or application based access consider if there is any orchestration tool is needed say docker or kubernetes for managing these machine to machine access so once we know our use cases we have below zero trust solution categories in the market so in a typical network-based solution type an employee accessing office 65 is not directly allowed as you can see an idp is required for

the employee to access these cloud solution and the xero trust is the solution that is helping the employee to provide device heart device help and device search for the employee to access these cloud solutions and in this scenario endpoint agent is not needed on the network based deployments employee accessing office 65 is denied and you require an idp to access it and our zero trust solution is actually integrated with idp to allow the process flow or access flow to this application in this scenario the idp is directly tied to zero trust solutions so only traffic coming from zero trust gateway is allowed to be accessed in this scenario employee have an endpoint agent deployed on each device

and all the traffic field goes through that endpoint agent and routes through the zero trust gateway before it reaches to the actual office 365. and then the third one is search based deployments where employees still needs to go through idp to access these office 365 but in this scenario these idps are directly tied to these zero trust solutions to allow continuous checks on device health and trust in this scenario employee provides this certificate and health through the agent installed on the end point and it continuously monitors through the gateway and then the gateway is going to work with idp to allow or not allow the access let's dive more into these solution types and see

some of our findings in these scenarios on the network-based type of deployments it's the most traditional definition of zero trash like josh mentioned this just provides the bare metal zero thrust solution here and it's an easiest or quickest to deploy as they integrate with mobile device management solutions so it's a plug-in needed to be added to web browser since it's a easy deployment in that part of things in this scenario device health is having a least priority as you require a different source to provide that and it might be the easiest way to replace your vpn as well let's move on to the next one which is the net based endpoint based on the network side of

things deployment this is this is similar to a vpn based solution and as you can see are talked about in the access flow we need a gateway in here and the gateway should be restricting the traffic to these idp or the actual applications so in that in that scope we require the applications to support ib restrictions and in this type of solution deployments consider if any privacy issues are encountered up front uh as as example the gateway might be looking at traffic if you're not properly configured the end point might be looking at all the traffic irrespective of corporate traffic or your personal traffic and what we have seen so far is these solutions require

the app to run all the time and it might include running on your mobile devices which might incur a lot of battery life etc and then the third one is search based deployments where in the end point you have an agent installed that provides the device and device health this uh apps or this type of deployments basically require digital certificate to be presented whenever you try to access any sas application for device authentication and also the agent must be compliant with with the device health policies like brittany mentioned before and this requires integration for cloud-based resources and probably less concerned less privacy concern as compared to the network based as it might not need to

look at all the traffic that goes through the application and it shouldn't be passing through the gateway let's look at the solutions and the gaps we have found across the different types of products which we have evaluated during our poc of these solutions one one type of the product is basically these are like cost is uh very affordable to the companies and leadership uh is very like good and in the sense of management and also tool development kind of things but the problem we have faced is the product maturity they only solve certain use cases not not all use cases that we might be looking at and also the problem with the product maturity being

there they have defined uh updates that are already being managed and for the product two uh type of uh this is another type of solution probably around the network based you can see the it's a new solutions that are out in the market and it's been very successful in in being utilized by many companies and the main problems we have seen so far with them is employee experience around mobile apps as i mentioned this could involve the kind of vpn deployment on those devices and continuously be on which might which might consume battery life of those devices and and also on the product three we you can see it's similar to the similar to the network page it's an

endpoint based it has the same upside around the market penetration and it has provided a lot of value for device health and trust compared to the other solutions as well and most of these solutions have kept in mind are being managed by multiple companies have helped them shape the product very much defined for the new use cases that are coming up for example continuous device health and trust and the major disadvantages of constant disease considering that like trust coverage or health coverage across different types of devices and also different types of operating systems say linux windows and mobile devices employee experience in deploying these apps as well as registration and all those have been have been a difficult task for them

and finally we come back to talk about who did we choose and on based on what factors did we decide that we we have chosen cert page deployment for endpoints because of aligning support for technologies in use for example most of these solutions require to be deployed by azure or basically sccm for windows deployments and also some of these solutions mobile device management solutions to deploy the software across different platforms say windows or mac environments depending on what type of endpoints do you have and flexible device health policies which which means like brittany mentioned before this could be anywhere ranging from do you have your uh endpoint antivirus solution running do you have your network

network um security solution running on your device or configured with that or do you have data encryption in place for different types of platforms say mac os or windows or android to support the use cases like personal mobile devices to be secured as well and uh the next one is co-party savion's sas applications as as josh mentioned before most of our use cases is around protecting our sas apps and some of our in-house build solutions that are tied with tied with our idp solutions to provide sso and mfa and endpoint or cert based solution directly integrating with idp solutions have helped us to even cover most of our http based apps within our company or

internal apps and search based solution actually helping us to provide device device trust in continuous basis and also being monitored continuously so some of these solutions ideally come up with managing these device certificates as well so that has helped us in in deploying these software solutions within our environment and this typically so these solutions do not typically replace vpn in the sense they don't work on the network page layer so you don't need to change the underlying protocols we have already and endpoint search base these mostly run on the application layer in the sense all these are https traffic and being and being monitored for any anomalies or basically blocking the traffic based on your device health and trust

finally the vendor had strong technical leadership team and vision working with our leadership teams we were able to meet co-part use cases and being flexible in deploying these solutions to us let me pass it on to brittany for some of our deployment cabinets yep instantly so before we talk a little bit more in detail about how we actually deployed this and some of the some of the roadblocks that we ran into um i want to give a little bit of detail about our environment specifically so you can have a better understanding at copart we have about 8 000 employees located in 12 different countries about 1500 to 2000 employees are considered corporate employees but the remaining

large portion do work in our yards and are more considered to not be technical employees blue collar workers so um really having to pay attention to anything that requires a lot of technical knowledge in terms of enrollment and things like that we had to be careful about about those things but now as we walk through as we worked through this deployment we ran across many lessons learned and really want to highlight some key items that we think are useful for a successful deployment so our first one is getting leadership buy-in from the initial purchasing of a product to the technical side to even the end user experience the leadership buy-in was needed at all ends of the spectrum

to help us push the deployment because we had to rely heavily on different teams to help us do things like creating packages and integrating with different systems and trying to push employees to help us volunteer for pilot testing and getting that feedback and ensuring that you know we've tested for all of issues that could occur with so many of these different groups being involved it was so important for the overall zero trust strategy to be prioritized from top down and ensure that that buy-in is coming from the leadership first so for the second one um another thing we really wanted to pay attention to was privacy transparency so early on we started to see areas where

employees were getting concerned with the information being tracked by the product being installed on more so their mobile devices but sometimes on the desktops as well so in addition to the much needed leadership buy-in that we just spoke about the the buy-in and support from legal in hr was was even more important because we realized we needed to be very transparent with the employees about the information that the app was tracking and we did this by getting privacy statements in agreements with the product signed and ensuring that these agreements between us and the product were provided to the employee in the initial introduction of the enrollment phase and anytime we started to involve the employee we

ensured that you know we were able to make them feel as comfortable as possible and from a privacy perspective so for some use cases if people had more questions then we even looked to have demos available for some employees that allowed them to see the backend data and logs that we would see in the console like hey this is this is the information we're seeing this is the information we're pulling that way we can just be as transparent as possible and making them feel as comfortable as possible number three kind of ties directly into number two saving the difficult people for last was something that we realized could help us push forward and show progress in the

deployment every company has some group of employees that either have legitimate concerns or simply don't want to participate or have extra privacy concerns and so we just thought that focusing on the larger majority that were willing to participate and willing to help us push along in the deployment would be the best approach and then once we enroll and get about 98 to 99 of the company completed um you know we roll back around to those difficult people last and we say hey you know again making the film more comfortable and saying we have enrolled this many people and everyone is supporting this so um now what else can we do to to get you on board with this

so the next one is addressing the hard topics so really you can tell um obviously we're trying to solve challenges of device authentication and device health across the organization with our strategy um but we're also looking to be able to provide better solutions for things like byod and securing personal devices device use cases as well so addressing that this will be a challenge that we'll run into but also one that with this grander strategy we'll be able to solve and provide a better solution for

so for number five the upfront effort to ensure easy enrollment is well worth it um i've mentioned user experience a couple different times and this really became our main priority the deployment really can't be successful when employees are unwilling to work with you or you're not making it easy for them so to be honest when we first started the deployment of our xero trust solution the end user experience was not as high on our priority list as it should have been originally the enrollment process uh for xero trust on workstations and mobile totaled about 12 to 15 enrollment steps for the employee which is obviously causing a hassle to not only them um that all the people that are

enrolling and having to follow these instructions but also our security team and other teams that are helping us you know the support teams and anyone else that's receiving calls um there were times where we were having to handle tens and twenties of individual calls with questions and people getting stuck in the enrollment and they couldn't figure out what their next step was even with the very detailed instructions that you know we took feedback and we edited over and over there was just too many steps in the enrollment process and so really we had to take a step back as a security team and say okay how can we um take this feedback and fix this process

how can we suppress as much of these steps that it takes so what we did was our team worked with some other internal teams of copart and then we also worked with the product engineers and we were able to ensure that the workstation enrollment is actually now zero touch it doesn't require any engagement from the employee it just triggers off of the octa login and then the mobile enrollment is actually down to two to three steps so really this this effort on the back end has by far been the most impactful change that we've seen um to our zero trust deployment so far and then lastly if you haven't already noticed the main component of each of the

bullets that i've spoken about before is communication and coordination it from scheduling testing identifying individual users and device names per department per country coordinating and collaborating for all the things needed to make this a successful deployment cannot be undervalued and it has to be uh it has to be really important to to the team for a successful deployment so i'm going to pass off to delete so he can give some insight into what we are planning to do next as you can see we have tackled the use cases of user to cloud but the coming up what we are looking for is user accessing any of our internal services which could be advanced remote access

for engineers which is not a typical vpn but we like to see device health and trust at the same time while they're accessing ssh or rdp services to our remote systems and then the other one that we are trying to tackle is machine-to-machine deployment since we are we have identified or addressed all the concerns within our internal architecture as well as the internal teams we have identified what is our next step to deploy zero trust within our machine to machine deployment so that we can only ensure app traffic has been has been uh locked between these two servers and now the traffic is allowed to be flow

let me pass it back to britain yep so as we mentioned a couple of different times we've we've needed a lot of help from others within our team that are not here presenting today and also external teams within copart but we did want to highlight a few other members on our team that continue to play a key role in our deployment so just want to give thanks there and then uh as we close out really we have a couple key takeaways and one of those is being transparent really at all levels of leadership teams that are involved in the deployment and employees being transparent is is the biggest piece and so we really want if you take

anything away take this one and also the next one is know your use cases define what zero trust means to you and your organization because it can vary and there probably isn't going to be a product that's able to solve all of your use cases that you came up with because many of the solutions are still maturing so the more you know your use cases the better you can prioritize them and the better you're going to be able to find a solution that's going to work for you and this maturing industry so that's all we have for you and thank you hey it's uh jeremy and baby doll here and we're with uh jay dileep and brittany and

you just listen to them talk about uh their journey to zero trust uh the first question is what was the main goal of your xero trust deployment and do you feel that you've achieved it yeah um so because everyone always has their own definition of zero trust that's why we try to emphasize that so much at the very beginning our goal was to ensure that every device that has some type of authentication in this case for us is having a digital certificate and also passes um a device health check as well too against all of our resources primarily of course it comes on the cloud side of things for an octa customer i think that

was probably pretty obvious throughout the talk and then also for on-prem resources as well uh we're still in the middle of the phase rollout right now this is something that's definitely not done within a few months period across around 8 000 different assets uh or a good-sized organization s p 500 fortune fortune 1000 so we have to be very mindful of different kind of customer bases different countries and what not to and time schedules too so oh yeah absolutely okay next question is how do you deploy and manage your certificates to make sure access is limited to just the devices that you've allowed or do you allow logins from any device so the specific solution we end up going

with does the certificate management fully transparent to us so it's not an issue at all it's just pretty much saying the agent is kind of a proxy for it a lot of other solutions even some very mature solutions don't even do that piece for you at all though um there's really prominent series c kind of company uh that we really like a lot that actually doesn't do any of it for you you have to maintain your own pki your whole certificate store as well too so but we've also heard a lot of other customers are able to do different things they key off like maybe if it's just part of your domain or some type of

ssi something of some other uh signature that they want to be okay but obviously this certificate is always nice to have you can okay so the uh follow up question to that is see you loading agents on every endpoint correct and that maintains it yeah okay great looks like we have one more question coming in but it uh they're still typing so we'll wait for him

while we're waiting for the next question um oh wait we just got we just got the question never mind no more uh donated what do you do about printers and uh devices with no agent capabilities that's a good question um delete i can take this one if you want did you want to take a chance to shake a shot at it though you can go ahead josh okay uh so within our context we're really focusing on the data itself so we're really focusing on the data the actual repository so if any access that's any access point or any endpoint that's going to be accessing the data or the email or database or something along those lines

so iot and our environment i think we're really fortunate we're going to have a lot of need for a lot of these other products that that really focus on we have printers obviously but we're not a huge iot shop so we really focus on the data scopes printers for the most part are having a lot of one-off issues so uh it hasn't been a core part of our definition okay great and what was the net cost decrease or increase what was the main selling point and what are some other points um i think the net cost increase or decrease i don't know what i don't know what the baseline is due like it was comparison

right now before we had nothing for zero trust right zero trust was a net new line item on our budget though it wasn't replacing anything else although i guess someone could see it supplementing something else like an mdm but we don't see it as a we saw it as a separate line i'm not sure if that uh answers the question though no i think that's a good answer uh perhaps a more um targeted question would be like what did you think the total cost of deployment was um okay so the tool itself it depends on what stage you get a company for sure if you get a series a and you think you're going to be

valuable uh to be kind of like a design partner and give them a lot of constructive feedback for our case i think with the who we decide to partner with we're in a different demographic geographically and we also have a lot of different value-add because we're into retail vertical we had a lot more funny use cases uh or actually funny but they're different and challenging to an area that they saw was a market they want to be able to tackle so in retail we got one-to-many issues we got one device and many employees that was a nuanced piece i think we had an opportunity to be able to work with this specific vendor we work with to

deal add some value okay excellent it looks like we have one more question coming in we'll wait for them to type that why so many trust issues i mean we had started on our zero trust journey like two years ago or two and a half years ago something like that too so i think we got really fortunate that uh we're a little bit ahead of the game in that kind of way of course now everyone's talking buy it in dividend administration wall street journal everyone's talking about zero trust right so we really got fortunate that we were a little bit ahead of the game on this piece excellent okay it looks like um the last

question is would you be willing to provide more specifics and product information offline in our discord channel oh yeah absolutely we're not doing it to hide any information we just don't want to make it a product shaming bashing or promoting kind of situation so that's the reason why we're kind of vague uh not at all for any other reason though but yeah absolutely well we're on talk shop all right that's excellent all right well that's going to wrap up our q a session if you have any more questions for brittany delete or jay i encourage you to join our discord channel we'll be in the ground one two three four channel all right thanks everybody

thanks everyone