WebAuthn, Yubikeys, and You: What we wish we knew before rolling out WebAuthn for internal use

this is to talk about rolling out webuthen newbie keys to everybody at Discord for corporate authentication um but first a little bit about me hi I'm Alex I work on platform security at Discord a lot of infrastructure a lot of I.T stuff before that Healthcare technology uh platform is a service company called Alto with an app Sarah did electrical engineering back end full stack and that brought me to security over the last decade or so and I want to talk a little about Discord security we have a pretty uh awesome security team nowadays our real intention is to make a secure way the easy way we want to come meet our users where they work

means we want developers people who are Engineers to come talk to us about the tools that they're using we know how to work on those tools we make PRS the same way they do we always want the right solution we don't just want the easy one and that's kind of hard in security right like it's really easy for us to come up with rules it's hard for us to come up with things that actually make people's lives better and move the security needle forward was it mean to actually enable developers enabling our users supporting BYOD I think if you're a security professional you probably have had challenges around BYOD within your org if you haven't had that you're either

very fortunate or you just don't know about it yet with 400 plus people like we have an engine product we absolutely have that as a challenge we have desktops we have MacBooks and many more in many many countries makes it challenging to do something like say ship UB keys to all of them so if you want something that's better security and better user experience what do we want security feature we want more MFA props we want more multi-factor authentication prompts things like if you're accessing a production uh piece of infrastructure you know accessing a Production Service uh you're accessing a sensitive application like your hris system or doing the role escalations I want to have some sort of check be like hey are

you actually who you say you are a presence check in there and you know if we just start saying more MFA props all the time the misery index creeps up so how do we do that while sustaining the misery index where it's at or ideally Bringing Down the misery index so emergency to web authen but first we're going to talk about MFA and kind of the state of MFA and why web authent is what we chose so why MFA at all starting from the beginning password reuse is incredibly incredibly uncommon um you've probably seen people successfully or successfully do credential stuffing attacks you know having breaches from things like have it been pounds and try to use that on

multiple websites some and within your org beyond that phishing attacks are incredibly prevalent uh we all have people who have been fished we've all been fished anyone in this room can be fished whether you think you can or cannot you absolutely can be and the goal of MFA is to raise the cost to attackers it's not going to make it impossible but it's making it harder for somebody to take your credentials and log into a different website nobody really uses MFA the stock is really focused on corporate users and this is for users of Twitter Circa I think July 2021 is when this report last was pulled but for all types of MFA across Twitter 2.6 of accounts had any

type of MFA at all the changeover from a July to December of that period was plus 6.3 percent so really not growing a whole lot and I think this is pretty similar but we see at other places really single digit percentages of MFA obviously you have more levers you can push with corporate MFA but it's something that users are telling us that they don't really like using because it's very high friction so there's a whole Spectrum right you have a lot of options of MFA you have um no MFA right we have one factor we just have passwords we have SMS and email we have totp probably the one that a lot of us are pushing out in orgs and

tftp or push notifications something that we probably have in a lot of places we have web authen which I promised we would come back to and then perfect security is you live on a Mountaintop you uh three computer out the window and you never log on the internet again but you can't get away with that while running a business so we go to the next best option so what do we do web authentic the primer on a better MFA so what is this what about that is a web authentication API so the auth n stands for it's an open standard leveraging public key cartography so you in general have a public key that you advertise out

and a private key you keep on your device talk through the registration flow in a minute why is this thing so good like why are people talking about this why am I talking about this why did I spend too long working on this uh it's because it's really strong right leveraging public key cartography is something that we've used for a lot of problems in the past let's bring it to this domain it's scoped you have it scoped to a certain website you know you do it for for instance discordapp.com you don't have to do it to Discord weird a that looks like an idpu.com let's go up to a certain domain and to test it

you're tested from the Hardware security module on your device so these are kind of put on the principles let's talk through what this registration flow looks like so you have a device in this case it's a cell phone but there's a wide plethora of devices all of them valid you register by saying I want to create a new account with the site the site says sure generate me a public key you donate the private key as well so the key pair is generated we send the public key off to The Ether and the private key is only ever kept on the device and you've registered a website and just as a shout out webauthen.guide is a great website

it's where all these graphs are pulled from so you have registration now you have the handshake when you're actually trying to authenticate you have a device come and say hey I want to sign in so the device says please or sorry the service says please sign this data because payload so I know that it's really you so I can do that at a station we're talking about so on the device you come back you say cool here's a Blog I want to sign up my private key and then you pass that back up saying here's a signature it shows like no one can be tempered in transit and the website says let's take that public key let's verify the signature

and great it checks out you can sign in so how do we use web authent I talked a little about logistics it's going to be the second half of this talk but um the spoilers you probably already have things that could act as a web weapon authenticator you may already be using it under the hood even if you don't know that it's kind of two big categories there's external and roaming authenticators Yuba Keys Titan security Keys other things like that and then there are internal or platform authenticators you know MacBook Touch ID face ID on iPhones Android fingerprint unlock Windows hello things that are inherent to the platform whereas the roaming key is something that you plug

in walk around with it can roam with you you don't need to be tied to your laptop for that one to work so it's a little bit of a breakdown of ATL account takeover prevention rates by type this is from Google security blog which is excellent comparing on device prompts so push notification SMS code security key and then a few other types of MFA and you can see that the efficacy of security Keys is really really good which is awesome this is even for targeted attacks which are the yellow bars I'm colorblind I hope they're yellow the bottom and you can see that they have 100 for all the cases that they've been auditing which is really

great so by the way Google research has made it really easy to make this case internally I'll quote that more later but just shout out thank you for making it easy to make this sell so more Google research here we've had no reported or confirmed account takeover since inline security keys at Google and that is Google I imagine people are trying to hack Google people who work there and they said they haven't seen anybody get compromised in this way it's really cool to be able to take one of these threats off the table because we have enough keeping all of us up at night so then what about that at Discord I work at Discord this is how I make the

case for better corporate MFA for Discord and hopefully can help all you make the case if you're not already using web but then internally so security is not an island we've hired a lot of talks about this this weekend people need to come to terms with us and US agree to them like how important a thing is and how much is it worth disrupting workflows ideally like I said we can move the security bar up without blowing up people's workloads without increasing that misery index ideally stakeholders go up at misery index either stays flat or goes down so we had a lot of stakeholders to work with I.T Engineers CX people who were both ftes within the company as well as

people who work outside of the company all of whom work together to make Discord great so the mission here was that we had to show that this was safer than the status quo it's in better MFA more effective I want to show that it was easier too this wouldn't just be incurring that misery index on there we also want to show that it was cost effective you know you're talking about buying new Hardware in a lot of these cases so it's going to cost some money it's going to cause some disruption how do you show that's worthwhile to do this so safer again shout out to Google he made my job here really easy we've had

no confirmed uh no reported or confirmed account takeover since employing security keys at Google that's big right so really easy to make that case saying this is the de facto standard people have been moving toward and it's not just Google there's also great research coinbase security blog also has some great uh talking about rolling out this internally as well easier to use so you can all see here it's kind of a jumbled mess of all the different Topp authenticators and I'm not knocking to be on authentication whatsoever I use it all the time but you know you're getting your phone out of your pocket you're logging in I use authy and I have to use my

fingerprint ID twice to log into that people feel the urgency as well of the time-based nature of this too and something that just adds friction every time you add this prompt if I want to say do this every time you're pushing a commit or every time you're logging into a server it's going to make people get their phone in and out of their pocket unless you have a cell phone holster and if you do it's still going to increase the misery index quite a bit so I think it's easier to use web out then and we're hearing that from our you know users at the end of the day as well the most common experience at discordant

is 80 of people are working on MacBook Pros and they register the Touch ID on the device and you just tap it when you get the prompt for other devices we have Windows hello you also have ubiques of a wide variety I'll talk more about what we lean toward later on but you don't have to fish uh phones in your pocket and if you're fishing resistant with web authen MFA as well the friction is really low and which means you can say we can start playing with MFA prompts and adding them all over the place without increasing that misery index is it cost effective that's the next big one right uh two factors per user is

about you know 30 to 75 per person depending upon the kind of prompt that you're doing the kind of uh you know keys that you're rolling out you also have to worry about an increased burden on it distribute and service like we have to talk about shipping this to every new hire everybody who comes in Discord works from everywhere so we have an office most people never come into it so how do we enable that how do we enable people in countries all around the world without you know ruining it's life because they are great it really helped me roll this project and they would not be as happy about it if I you know took up all their time servicing

lockout requests for instance if you're comparing this not to an action right you're comparing this to the cost of an action the cost of an action on average from 2022 report from IBM a successful phishing attack at a company is about five million dollars 4.9 million dollars I think we can all think through like whoever gets fish depending upon who that person is and what kind of standing access they have it can probably go a lot higher than that too so again shout out to research from Google this is talking about the support incident so the burden to I.T you can see here that the support incidence of lockouts precipitously dropped as people adopt Hardware security Keys which is

awesome to see so you can see over time the support rate from OTP drops and for Hardware security Keys it's flies they rolled it out throughout the org for the deployment we found the increased user productivity and the decreased support costs were worth the increased Hardware cost so that's big it really helped make the case this way not only are you decreasing the burden in I.T but you're also saying the decree increases your productivity from being locked out less often less support requests means that people are getting far fewer lockouts than they were before so it's not just that cost in I.T it's the cost amortized across your whole organization it's big cost savings everyone gets Yuba Keys some people get

you two UB keys somebody gets a Mac like a touch ID keyboard something like that we have a wide spectrum of devices that we support here but it's cost effective and we can make exceptions like that because really savings are worth at the end of the day so we sell people but what's the plan we sold them on step one get everyone asterisks and to OCTA and get every asterisk app into octave anybody here who's been early on Security Programs knows that time is a flat circle you see a lot of the same themes you get people behind MFA or behind SSO rather it's the first thing that you do for us we did that with OCTA

something we started the ball rolling on prior to starting this project step two get two plus authenticators to everybody a platform in a roaming is what we decided upon but it could really be any of those some groups we wanted to go with uh three for redundancy so execs or people who are super admins in Octa G Suite super admins people have high privileged accounts that are hard to service if they get locked out or have a lot of risk we got multiple authenticators for those because again you're talking a cost of like thirty dollars for an authenticator fifty dollars for an authenticator much lower than the uh the risk of compromise or the risk of lockout in these cases

step three uh we've heard this a few times this weekend in other talks but it's important to start testing on us so when we laid out the strategy if we wanted to start rolling this out too gradually the first step is security if this is a bad experience we want to feel that pain first and when we come to other users we want to say hey we've been dealing with this for a month and it's actually better than it was before it builds trust with your users if you're going through this sort of stuff and not just imposing it upon them and saying we'll get to it later so we did security infrastructure it execs

engineering all ftes and then everybody again Discord has people all over the world in all different capacities working for us that all help make Discord great we want to make sure we encompassed all of them reduce the risk across the board from there we had an exception group in octave so this is kind of octa-specific whatever your SSO provider is probably has similar functionality but for us we had an authentication policy that said everybody needs to get password and web authent prompt to log in or webathon prompts on these certain cases except for people in this exception group so it's allow us to say okay we can opt in the whole category but if this one

person infrastructure is on paternity leave or whatever else we can opt them out so which is awesome to have you can say like these long tail cases you know someone's ubiki got lost in the mail it was stolen whatever that's fine add them to the exception group but follow up on them judiciously and then over time you see you shrink the number of people in the exception group but you keep it around in case there are lockouts or loss authenticators there's always cases like this where somebody gets stuck in some sort of exception case so it gives you that flexibility we had a lot of moving pieces here but we had enough uh you know kind of back doors if we needed

to to get people into a working condition while we roll This Out so we had some tools to work with here right I talked earlier about platform versus roaming authenticators the first most common thing was that people had MacBooks already they had Touch ID on there so this is something people already knew how to use people are often using this to log into their devices it's a really familiar interface which is nice but past that you can also have Windows hello other devices like that you can also have web compatible keyboards you know some people keep their laptops in docked mode we're pretty lacks about like you can use any Factor as long as a

web authen factor and we will probably support you we want to meet your users where they are because we're asking for a change of behavior so it's important to come to them has that we want a roaming authenticator so it's important to have two plus factors to decrease that lockout we talked about earlier so we chose to use the UB key security key C NFC uh it's kind of a mouthful it's this blue key about 30 bucks a pop and one of the really cool things about it is you can bootstrap a new phone using NFC so if you have a phone you want to use it as an OCTA authenticator itself with Web Bot then you can tap on the back with

NFC and use it that way one of the other things sort of like a ux philosophy here you don't want both your factors to often travel together you know you can have the low profile 5c Nanos that live inside the laptops but if the Key C and A C is Big it's probably not going to travel with laptops because it sticks out like this and so we want them to live in separate locations that was my theory of the case so it is 2fa for us it's 2fa you can use these for uh your single factor for pass keys I'm not going to touch on that too much today but very exciting highly recommend reading more so we have a plan

sounds pretty easy right let's talk about what actually happens so we have a plan timeline all good projects should have a plan timeline and they should all be as optimistic as I was so we started this off in May 2022. um June we had it all rolled out to or we wanted to have it all rolled out security it infront execs so on and so forth basically one month after another we're like cool four weeks is long enough to get some address and ship out Keys no problem it's like basically Amazon right what could go wrong uh it went pretty well um but everybody was a long tail after that you can see it pushing off the page there so we went

into this year people who were ftes we were able to get to them pretty quickly but that long tail case is a very very long tail because you're talking International Logistics at this point and if you take away one thing this talk is that when you go through this figure out your International Logistics Way in advance so if I come into a distributed Global team is really really hard and I don't work for importers but just pay for importers for these countries that are really difficult for something I wish that we did uh coming back to we got stuck in a few cases and having somebody who knew how to do customs in a country

like Columbia or the Philippines would have saved us weeks if not months another key learning as I mentioned this earlier is test on yourself so I think why we were able to roll so quickly through everybody within the companies that we knew what would work like for the most complicated cases people who were accessing infrastructure so by testing on ourselves we heard our concerns we were able to understand what this flow would be like and we knew what the most complicated cases would interact like shipping shipping Logistics really the biggest part of this we used uh the ubico Enterprise delivery portal it works really well all we did was take from our his system export basically a

dump of everybody's addresses ship them a key or two and then send it out and you can do a bulk CSV upload using ubercon Enterprise delivery super simple highly recommended for nonefts it's harder remote work and agencies make this hard again I'm going to keep repeating this work with local importers if you're working with like a business partner outsourcer firm or BPO work with them figure out the best way to distribute it to them and figure these out far far in advance Imports the biggest delays for us were in Philippines and Colombia and that's really coming down to like I've never done anything like this before we hired somebody with procurement experience like halfway through this project which

is a godsend but by that point we already had things out the door that were just waiting in customs so again highly recommend getting importers would have saved us less than 10 grand and probably four or five months in some cases by just getting importers somebody in leadership so shout out to Discord leadership and you should all be happy enough to work at a place that has leadership willing to theoretically bring a duffel bag full of UB keys to the Philippines didn't come down to that but we had people who were willing to hop on a plane with say 600 700 UV keys so this is pretty awesome we're hiring so okay uh we touched Logistics I've

touched enough on importers um uh for enforcement we have OCTA as our wedge right so OCTA is um we're going to have a long tail of apps and users people who need to be moved into OCTA apps when you want to put behind OCTA it's okay to have a long tail you know some apps just never support SSO unfortunately so you want to focus on the things that are the most impactful that people will be using on a regular basis you want to get through the rollout and then work on exemptions you're always gonna have somebody who's fish ether you be key there's always weird cases that you will never anticipate because International Logistics because of male theft or

people move especially during remote work it's hard just have exemption group and follow up on it make sure you're working with your users and saying like Okay it's like 48 Hours 72 hours whatever reasonable like how quickly can we get this key to you in a way that's not going to make your life really hard so I want to finish the call to action this is worth doing this is a painful Learning lesson for me on the pains and joys of international Logistics but hopefully my tips can help make this easier for all of you doing this phishing attacks successful ones at that are incredibly incredibly pervasive and again I want to reinforce that anyone in

here myself everybody in here all Security Professionals anybody seeing this in the future can get fished what about then is phishing resistance it's not fish proof but it's something that is really really hard to fish you have the interactions the MFA prompt which is something that is really really strong okay what about that in you what can you do to prepare for this first advice is to know where everyone when the works in the world far in advance which sounds really obvious right and it is but I'll say that like remote work makes that really complicated and people move uh we had missed deliveries because of that so get a good wedge next to prioritize

for SSO for us that was OCTA and then zendesk so a lot of our BPO again business partner outsourcer agencies are working in zendesk day in and day out so that's something we could get an app that like half of our entire user population used on a daily basis and identifying that is the right thing to put behind SSO first made it so that we could move groups in and know that they would have it in force on a regular Cadence next up identify countries not supported by your shipping partner early you become Enterprise delivery was really helpful shout out they did a great job with the portal but they don't have full coverage because International Logistics

are hard so there are some countries where they really can't help you so figure that out far in advance and then figure out what your alternative means are and I'm happy to help with this like I wish that uh we had something like this to say like hey the Philippines you're going to want to work on that in June okay and the rest of the call to action what about that in you please support it Apache browser and device support is one of the biggest hurdles we have to adoption we have an open standard here and you can sort of see up on the screen some of the patchwork support it's a little fuzzy up there this comes from

webathon.me a lot of great compatibility about like can I actually use this with my browser with my device we have an open standard here and really to everybody out there working at different companies to help improve this support it extend it so we can make this the standard so it's not sort of an exceptional thing that we all are doing this as a baseline because the more support that we have the more multiplicative this effort becomes and becomes more worthwhile doing so with that I want to leave some time for Q a what questions people have

yeah we're here actual personal devices and save mobile access to yeah so questions about BYOD which I mentioned kind of in passing how do you handle access to uh services from personal devices with doing a rollout of Web Bot then so um I think one thing that we were flexible on early on that I haven't heard everybody be flexible on is that you can use your personal cell phone as an authenticator if you want to set up your iPhone or your Android phone not a work phone as an authenticator we're okay with that so you can access OCTA apps on your phone otherwise we would disrupt some people's workflows we use people that you know they play a game

and they would review PR's on their phone in between rounds of that game we wanted to keep enabling that because they want to do that and we don't want to just shut that off so we're flexible on that we're also flexible on the type of Yuba Cube so we really wanted to focus on the security key cnfc as the standard but then if someone asks a question like hey you know I have um we have a game Studio at Discord actually and they all use desktops and so they're like it's not going to work for us using this USBC thing I was like cool I've been getting some USB a devices so that's supporting their use case one

that I didn't even realize that we had you want to set a standard so you can kind of converge around what docs and support there are and know what the standard onboarding package is that'd be really flexible after that someone's like hey got a problem here you figure out cool what's the thing I can get you that'll make you successful does that answer the question cool over there

yeah so the question is about usability feedback from non-engineering users um honestly most of it was not getting much feedback at all which sometimes is the best feedback to get I did talk to people and kind of do sampling polls of people working in different roles across the company in general people do not miss to tpmfa for OCTA we didn't allow octopush verify or octave verify push at the time we started doing this so it was comparing that to tftp um yeah people who are working in the mode case really like touch ID something people are already used to using and then being flexible on the interactions with uh you know different other ebqs people really enjoyed that too

Yeah question right there

so the question is about comparing the efficacy at scale of security key cnfc to the USB 5c Nano is that the one you're talking about 5cnfc um I don't use the 5cmsc directly comparing it directly to the 5c Nano which is the one that we were looking at primarily because the NFC thing was like it was sort of a nice to have we didn't need to have that for bootstrapping authenticators but I was about choosing for byd purposes to allow people to use personal devices there's a few things that you need to consider when you're looking at what key to support and that's a lot of like what are the authentication protocols supported by this key security key NFC

is actually pretty restrictive and that supports like u2f mode and that's all we wanted so actually some things that you may want to turn off if you have the five scene Nano like um I think we've all seen at orgs when you touch the UB key and you get like an OTP code spanned in the chat it's fun you would sneeze but you don't necessarily want that and so the key to NFC having the limitation of not having OTP mode it's actually kind of a user benefit because you don't have to worry about turning that off and you can do customization with ubico but it requires paying extra and shutting down the assembly line so it's kind of a

headache to deal with yeah a question right next to the last question

yeah um the question is about uh different compatibility and support issues with embedded browsers one of the big ones and I think the still problem is with hris workday logging in specifically on mobile that's one of the big ones that we saw and uh the way you work around it is by allowing people in temporary exception groups for setting up those apps because the sessions are pretty long-lived it's not the best experience but again because the supports kind of Patchwork on different browsers especially like um you know mobile browsers that are embedded like it's a great call out for now that's the best thing we can come up with is like when you want to log in that way we set

the session that way and it'll have some sort of refresh I have no idea what the refresh mechanism is with workday specifically but something when we want to allow people to use that from their phone that's how we did that I think setting up Google Calendar was also a pain point if I remember correctly with that and so same sort of uh same sort of issue same sort of Remedy there exception group set it up take them out of the exception group so you just exempt them from one of them one important call out the exception group that I probably should have mentioned is uh we didn't say like okay you just have no MFA now people have backup tootp

already set up or if they don't when they go into sign in with OCTA again they set up totp MFA so they fall back to that so we never have people logging in without MFA there's always a backup authentication mechanism standard so people are still using trtp when they're logging in with a new OCTA session this is about you're trying to create a session to log into an existing app we can exception group you for now and things will be okay Yeah question right up front yeah uh it's uh two questions there I'm gonna go to the next slide too so in case I have any other questions you can find me afterwards but um two questions

there first one is we use any other features of the UB key like storing um OTP codes on the UB key itself and generating OTP codes from that answer that one um we haven't used that we've been still using OTP like uh apps like authy or Google Authenticator again we like intentionally chose a pretty restrictive um key the key cnfc is the standard that we came down to the second question is about Windows hello specifically setting a pin yeah that was a pain point that we felt fortunately we have limited users of Windows and so what we did is for those users we had it go through and disable the pin mode using YK man which is command line tool

so you can turn it off that way but it is unfortunate that we have to do some manual operations this is again one of the goals here was to not put too much burden onto it just want to give a shout out it team and Discord you're awesome thank you for helping with this and thank you everybody else in security and thank you for all of you coming and seeing my talk today I really appreciate it [Applause]