Daniel McMahon - Hey Everyone, Break Our Stuff - BSides Portland 2018

thank you for being here my name is Daniel McMahon I am also known as mcmayhem I do work at New Relic on the product security team I'm an applications security engineer this is my third besides Portland so my first time talking happy to be here just so you know thank you I am a terrible socializer so if you have any questions afterward please come up to me and throw something at me or yell at me because I'll probably be sulking in a corner somewhere I'll be one of the hundred of us that are doing that so I mentioned New Relic why does that matter it's not a bitter pitch I don't care who you use what you use but we are a pretty

large SAS company we're kind of unique too we're not we're not Microsoft's eyes but we're also not a new startup we're not one product to manage we have a pretty large surface area numerous products that all do very different things that operate on very different parts of the stack applications servers mobile deployed applications JavaScript ring and browsers server software that is sending data to us all these things collect data and sent it to our our back-end software so the threat landscape for us is is gigantic we have all of our back-end services we need to maintain we have to worry about our code running on our customers devices we have to worry about our code running on our

customers customers devices so twice to say we take security seriously it's not funny it's true we have multiple teams at the company who think about security constantly we have our operations team who's dealing with the the actual physical infrastructure the data running over the wire our servers patches we have a compliance team who's talking to our customers constantly figuring out what things they need for us to build out and then of course there's my team product security who is looking at code we're doing code reviews every day we are meeting with teams constantly basically just trying to get a handle on the 50-plus teams and growing that we have the company but it's tough there's only

a few of us on any of those teams there's not that many people not that many eyes so everything that we can do to increase the information coming in to us is helpful we try really hard internally within the company to build out a culture of security sure I get advocates from other teams teams that aren't necessarily teams with security in their name engineers salespeople our support teams anybody who we can get thinking about security is helpful to us so the point is we were stronger with more people we don't have that many people dedicated to security for the more people we have thinking about it these are our job is but even with all

the work that we do internally that's that's not always enough so that brings us to coordinator disclosure Cortney disclosure is kind of a fancy name for for bug bounties but that takes a lot of different forms who here has some way for people to to submit bugs to your company that's awesome I know you do so it could just be an email address we call that a vulnerability disclosure program some way just for people to send something to you if somebody comes across a bug on your site accidentally or intentionally they want a way to be able just tell you about it you may pay money you may not you may use a service like hacker one

doesn't matter you have a way for people to actually submit stuff to you that just is that it's a safe way for researchers to disclose stuff to you without a fear of reprisal that's the big thing so we have had a bug bounty program for about two and a half years now almost three years at first we had a completely private program that we paid money out to well we had a public program we added a private program but we started paying cash - and now we have a completely public program that we pay cash - so we've been doing this for a while and we've had a lot of reports come in we've dealt with a lot of

different researchers from all over the world so benefits wise it is a humongous force multiplier for small teams we do assessments we have security researchers from big firms that we trust come and poke at our products we'll give them our source code sometimes we'll let them we'll let them do everything they can to do pen tests a specific product but assessments can only catch so much stuff we've been around for a decade now our products are very mature so repeat assessments don't always find anything new people kind of get stuck in the same thought processes a lot even trying to rotate vendors around which we do eventually they've ran their run books they've done what they know but the

really unique thing is you get a completely different perspective from random people on the internet who are submitting stuff to you these aren't always people who are professionally involved in security these aren't always people have any kind of formal training they're gonna be doing things that you haven't even heard of before totally crazy new stuff so you can do your red teaming you could have pen testers come in you can do all the things that you do normally and those are all really really great but at the end of the day you only have so much budget you only have so much time you only have so many people so anything that you can do to add

additional eyes especially these totally unique new people is a super big benefit there are risks who here has considered something like this and didn't go through with it a couple people so that's pretty common it's kind of terrifying to just have random people come and poke your stuff legal needs to be involved there's PR risk potentially a big thing for us especially with a coordinated disclosure program is most or all of things that are being submitted to you are eventually going to be public so you have to worry about your what what does your team say in a response all of their responses are gonna be public the actual ways you're working through a problem is

gonna be public it's also a lot of work for your team every single thing that comes in has to be reviewed by somebody that's a lot of work to to triage and to decide where it applies but that's why I'm here I want to convince you that even with all the risks even with all the difficulties it is totally worth it initially when I was planning this talk I really wanted to target risk-averse companies there's a lot out there healthcare governments all these all these industries where there's a lot of red tape anyway and a lot of things to worry about that can make you not compliant or put you or your customers at risk

I actually saw some folks in the automotive industry talking last year and seeing seeing how much they they want to be able to do stuff but they're tied back by all these other all these other factors so my goal today is to show you that even with all that it's a net benefit it's worth doing and it will be a drastic improvement for your team so here's the big thing no matter what you're doing no matter who you are no matter what industry you are in you're already being hacked the big thing is without a program you're never going to hear about it if you have a program good people aren't going to avoid you they're

going to come and tell you so really you're not losing anything by opening up you're losing all the good people skipping your program being afraid that something's gonna come back and bite them if they submit something to you and then you're also hopefully gonna defer some of the bad people doing that stuff somebody might seem actually mentioned a good point if you're doing something malicious you want the shortest path to catch you can get you're not always paying money even with a program like hacker when you're not always paying money but reputations big and even small amounts of money are a whole lot easier for people to want to get submit to you and get then then trying to go through

some kind of black market way or share it with friends so what should you expect if you decide to open one of these up whether it be an actual bug bounty program or just an email or any way for people to submit to you well you're gonna get some absolutely terrible reports we have seen we have seen things all across the board it might be people who just can't communicate well whether that's a language barrier or people who are new to the industry or people who are just really bad at putting their thoughts down on paper even if they have a good finding we also have noticed swarms of reports similar reports coming in at the

same time which is a really strange thing but often you'll see the same class of report come in simultaneously so there can be a lot of noise but at the same time you can get some really really amazing stuff things that you your team may not have ever even heard of before even if you keep up with the news even if you are super security conscious you learn a lot by seeing the types of reports you get and the methodology that these people use so I want to talk about some examples that we've seen and this will be kind of the second half of my talk today we over the last two two-and-a-half years have had close to 200 reports I think

that have come in via our program things tackling every part of our stack things that some we've paid for some we haven't paid for but all really really great things that have actually changed how we've thought about our products and how our partisan put together so to start with mentioned we're a SAS company what you may not have realized is that stands for SS RF as a service so what is SS r f it's server side request forgery it's a way for somebody to make your server call up to something that you don't expect interestingly we actually do this intentionally we have multiple products that perform requests for people intentionally what's not intended is accessing internal stuff we don't want

people hitting or on endpoints and things like that but as we know it's hard so this is an example of one of our tools that reaches out and does these requests you can plug in an endpoint you can have it do a get or a post request it'll just reach out and try to hit something that you wanted to hit and this is super useful if you want to test monitoring a website things like that interestingly here though on the right side it's a little bit tough to read but that's the that's the request they're trying to make and that's actually an error message that's coming back this this enterprising individual is trying to access the Amazon Web Services

metadata endpoint thankfully we filtered that so those are three blocked error messages but have you ever heard of zip-tie Oh see me heard of this before good you learned something today so this is a neat little tool what does it mean for for our for a prior example here well this is what it does you take an IP address you prefix it in front of zip that I owe and now you have a resolvable dns name for that IP address anybody can do this with DNS but this is a really quick little way to do it so

naturally that breaks every filter that we have in place so the same person took that example typed in zipped at i/o and then of course they can access the metadata endpoint because we didn't filter for that in the end they actually went on to write a proof concept for this where they were able to completely download an excellent rate that information using this smarter the the larger scripted interface to that so yeah but who would ever think about that you know that's a totally unique situation we had blocked the IP we had blocked at endpoint never even considered the possibility that it would have something like this happen the end result though is that we improved our

filtering we have a canonical way to determine where these things are going now and it improved the company across the board all via a submission we had through a bug waiting program

so similarly we have one to seven point 0.01 it's your local machine you tend to have services running on your local machine did you know that there are other ways to write 27-point open 0.1 we have dot decimal notation with hexadecimal we have octal there's also dot las' decimal notation which as it turns out also ends up breaking all the filters yet another thing that I personally had never considered I've never even thought of but coming back to having a canonical source to reach out to improve our product improve the way we do things and kept us safe it's also really cool to read these and then go back and like use them yourself if you

want to go and do some acting on your own because these are our really cool techniques so we had a really neat one come in from albino wax who is one of the researchers at ports Berger who makes burp suite he as he does was messing around a burp suite found what her brain points had kind of a weird rewrite rule and our and our load balancer there this actually resulted in a pretty awesome conference talk called cracking the winds if you haven't read it or read this paper or seen the talk I highly recommend looking it up but what's really neat about this is these bug bounty programs aren't just a force multiplier within our company these are

actually industry-wide force multipliers this bug that he found turned out to be an Apache bug which then ended up as a patch against Apache so we fixed it on our side we submitted to Apache Apache fixed it and everybody got to benefit from something that was submitted to our program

so we are SAS company we're also an IED or as a service company we are a huge company so we have a lot of products working together it's tough to make sure that everything is doing what you expect it to we have so many products doing so many different things intermingled in so many different ways it's a lot to keep track of and with 10 years of history it's it's always difficult we also have a lot of tiny startups within our company we're kind of a kind of a unique space that we're not one big company doing one thing we're a whole bunch of groups operating pretty independently of each other and we have some really smart

engineers we have some of the best engineers in the industry and I don't say that like sarcastically or blowing smoke up anybody's ass here but we have some really bright people and even those bright people miss stuff so one of our top contributors is somebody who works at hacker one mr. John bot araignee over here he is really good at finding exceptionally weird stuff he noticed a while back that he could enumerate the full names of users within our accounts not obfuscated so by adding the right the right combination of things if we had to use your account on our already existing within our system he was able to reveal the full name which is a big deal we

don't want that to happen so we fixed it after we got that submission

or so we thought so we had a resubmission of that same bug we had multiple other resubmissions of that same product I'm I believe I believe you refer to it as the gift that keeps on giving so this was an interesting and interesting case because these things kept popping up and what does that tell us it tells us we're not actually fixing the underlying problem so once again we could have just filed a bug we could have filed one for each one of these but what ended up happening is we went and talked to the team we discussed what was actually going on with this situation and we we made an effort to do have a

real correction to the root problem that actually spanned out to the entire company something that without having these reports come in we probably wouldn't have even caught let alone had a comprehensive fix for

oops there we go

so synthetics is actually a selenium driver it really is just RC as a service like we're we're intentionally running people's code within our infrastructure interestingly this is one of the very first things that we started paying money for with our bug bunny program there's a little bit of a risk for us it actually got quite a bit of attention when we took a live so we have a sandbox in place within this people can write JavaScript it ends up in this selenium driver it runs it at those things like the one I mentioned earlier where he was able to to get the metadata stuff they write JavaScript it makes requests it navigates through websites it does a

lot of stuff just like you were using a browser at home but we have a sandbox in place to keep it from doing nasty stuff the goal - the goal being that they can't run things we don't expect but I say sandbox with quotes because javascript is really tough to sandbox if you actually want to do anything useful with it there are so many possible things you want to do and so many things we want to do with our product that trying to sandbox everything and catch everything every side case is basically impossible so we actually ended up paying out a lot of money so here's this is an interesting one here's our here's our sandbox you see do you see a problem

here in the script

so there's our sandbox deactivated we ended up actually paying out thousands of dollars on sandbox escapes like this this one was painfully obvious by the time we but it's on we realized like there's gonna be more and more and more of these there's every possible way somebody's gonna be clever and figure out how to get out of a JSA inbox we did the same thing as we did before with the with the alerts names we wouldn't talk to the team and we came up with a solution that was actually comprehensive and would cover all of these different cases not just specific cases in the end we actually ended up putting apartment profiles around these so the JavaScript

scan sandbox is more of a more of a guideline than anything but we have a much more robust solution that wouldn't have come about had we not had these submissions come through

so the final thing that's been really helpful for us is keeping our documentation accurate so our customers do a lot of things that we didn't necessarily ever intend our products to have done with them we give people a lot of power to do things like write their own code embed our api's but we do try really hard to document all the places where we're doing anything potentially weird so when we collect data we're very specific about specific about what we collect we do everything we can not to collect additional data that's a bug our most important thing though is we just don't want to surprise people we don't want to surprise people within the company we don't want to surprise our

customers we want things to be very specific we want people to know what they're getting into so what happens if our documentation is wrong so we have a concept called restrict accounts and restricted accounts are accounts that basically operate like kiosks they're just read only accounts that a lot of customers use it's not a real security feature it's more of a more of an account control to help you do some things like kiosk work but we do document it heavily and we want people to use it and a lot of people like it so same person I mentioned before from hacker one has come up with some super clever techniques for figuring stuff out with Bert and he came up with a lot of

situations where restricted users weren't quite as restricted as we expected so here we can see the account license key is pretty important that's how you send it up restricted users can't see it they can only read what's already in the account but they shouldn't be sending new data to the account well he found out that if you just go to the right page and go to install it actually just shows for you even if you're a restricted user so by navigating around by looking for the by looking for the license key that's there he was able to find this in a totally random place that the team didn't catch that we didn't catch but it did show us

that we were out of compliance with the documentation we had in place which means customers weren't getting the experience that they expected so to finish up anybody here who doesn't have some kind of vulnerability disclosure program consider one it has been such a massive benefit for us compared to all the other resources that we have to get submissions look at the disclosures that are already out there look at ours at that page look at big companies who have had a lot of disclosure Shopify is a great example see the kind of stuff that they're finding and engage with the companies that are already doing it like us come talk to me it is absolutely terrifying as I

mentioned before to have random people on the internet people you've never met people you might not ever meet that may not even have credentials may just be an anonymous username having people like that poking your stuff is scary but it's already happening if anybody wants to look at your stuff they're already doing it it's just best to know also if you're a hacker come hack our stuff and if you're a hacker looking for jobs come talk to us because we want these kind of people that are submitting stuff to us so thank you

do any questions yeah so the question was are there NDA's for submissions to our program or any program we don't have in DA's and a big part of the methodology is that if it does come into our program it's gonna be public eventually but there is there's some some guidelines to people submitting generally who like join a program like hacker one there's a there's some like kind of bug buddy etiquette I guess like don't disclose it beforehand we do have a policy in place that is kind of our rules of engagement for that the policy says give us time don't just disclose it I think a lot of people operate off of like the Google

methodology where they wait 90 days or whatever but there is no explicit NDA it's a it's a lot about reputation if you're the kind of person who's just gonna go spam it everywhere we're probably not gonna want to talk to you as a researcher and I think a lot of the researchers don't want to be in a position where they're looked at negatively like that almost universally our experiences have been really positive with people and and even the ones that weren't super positive there's a mediation process for that there's a lot of there's a lot of kind of guardrails for a lot of stuff to keep it from being too painful and it's not that

onerous for somebody to submit they follow the guidelines they pick the product or domain or whatever they want to they want to start pen testing against if they don't see it there they can message us and ask why it's not there but generally I think the idea is you put everything in scope you let people submit whatever they want and then you kind of take it from there anybody else

yeah I was actually discussing that earlier I think there are probably situations where that's gonna be a problem you know if you have an iOS jailbreak bug you're probably gonna take it to some you-know-who pin or somebody like that and make a million dollars off of it but for most of our stuff you know if you find a cross-site scripting bug or a Caesar for something you could probably make some money off that somehow but you can make money right away maybe soon as us you'll gain some reputation you might get invited to other neat stuff I the kind of things that I've seen that we have had come in are pretty intense we've we've had some really good bugs

come in and those people decided to submit to us you know these are these are on par with the assessments that we've gotten so I think there's probably situations where that's actually concerned but I think for most people it's not you know Apple Google we're you know nation-state type level things are coming in but I think for web apps stuff and most of what we're doing with it hasn't really been a problem

yeah and I think there there are some cool controls around that for being competitive so you can see what everybody else is paying out so if you're a company who's only gonna pay out say $20 for a cross-site scripting bug people might not spend their time on you but they can see that ahead of time so before they submit they can kind of see like we have we have a table right now that shows what we pay for different classes of bugs they can also see all of our prior reports so it's up to the hacker if they want to spend their time to even research it in the first place and if they do that they're probably to

submit it to us if we're not competitive we're probably gonna get fewer submissions and little control for that but the overall ecosystem I think kind of balances itself out that way

yeah we operate the same way so that was if somebody submits a bug it's about the potential not necessarily about what they submit and we do the same it's if we can if somebody submits a key and they weren't able to to do anything with it for example we we would research and find that out and payout with the Maxis and yes that's that's a good a good example people want to submit they want to engage with you and and and they want the the path of least resistance for sure if it's so if if they don't if somebody doesn't provide a proof of concept to us they'll need to have some some kind of bare minimum so we can at

least reproduce the issue they're saying if it looks out of place but they don't necessarily need know if they somebody gets access to a service or something we don't necessarily need to have them just like rifle through our whole infrastructure it's it's seeing like if they open a door what does that door do and we we try to evaluate that and if we are wrong if we think you know we probably can't do anything like this we might come back and say why don't you try something else to see if you can take it further but then it's in that kind of controlled environment

so honestly I don't care how old somebody is I don't care where they're from the question was well how do we how do we hold hackers accountable if we don't know anything about them if they're a real person or not it doesn't really matter as long as we can as long as we can understand and reproduce the report coming in it makes us happy that's that's on the that's on the platform to care about me and I think that there are a lot of really really bright folks from all over the world of all different ages doing this kind of stuff and I think the more the more of that we have that's that's my favorite

thing about all that is these completely radically different opinions that you get from people who don't have formal training necessarily don't look at things the same way that you do you know they're they're gonna see things with a completely different point of view than your researchers so it's it's great and also there is a reputation system I think through all these different different platforms if people are acting in good faith that are making good submissions they could be 15 they could be 50 it doesn't matter if they're rising up the ranks of their their kind of reputation like that says something about the kind of stuff they submit well have him have him come hack on us he can

yes

so the question was if we have a we have a bug that comes in that does have a real a real vulnerability and the company wants to accept that risk how do we respond to somebody so if there is a real risk we're the ones making that determination and we have really good buy into the company to to trust us and to accept that we we have judged that for that to be real if it comes back that we don't think it's severe enough to do anything about that's on us and we're making that choice and we will we will reflect that back to the person will tell them we we are not in the

business of like taking things on like that and unless we have a really good reason to and if if we didn't follow something if we decided that it was going to be something we just kind of ignored there would have to be some other control around it I think we're really we're really lucky that we have that buy-in from the team now we have had things where it doesn't appear that there's really any security implication it might be might be a weird thing maybe you can maybe you can break the UI locally if you do something strange it's not really a security bug it is a bug we might follow it as a bug and let them

know that but we're we're as honest and straightforward as possible as we can be with that stuff and I think that's a big reason why we get as much engagement as we do and our program is we're pretty transparent about a lot of that and I think if you talk to some of the hackers out there they would probably probably say the same you know the depending on what company you're dealing with you may or may not get that kind of that kind of response

yeah that's that's a good example like if if there's something like that that is so if you have you know domme a my company Rostelecom clearly you know that that company exists or at least somebody decided to steal that name that that is an accepted thing and if we had something like that we would say it's it's working as intended this is an intentional feature we probably have it documented somewhere as some kind of accepted risk but we don't start to be confident in saying that because if it if if we do disclose that report that that conversations gonna be there so we want to be pretty confident that that's the right answer the right way to go

anything else yes

that's a good question I wouldn't say first off I wouldn't say that the that the bug bounty program necessarily picks up where the pen testing leaves off I think there's a lot of crossover we have gotten very similar reports from like a like a pen test company versus what we had come in can you repeat the second part the yeah so I I don't think we've ever had anything like that that would have been would have been useful for like to like talk to sports or somebody like that we did have the example like I said with Apache where we've done like backported patches I don't know if there's any been anything specific well I guess we have

had a little bit so the the port server one I mentioned earlier that I wanna wax found that actually became a burr plugin which he he wrote his talk and he wrote the pay the white paper on that after he made the plugin for himself got all his bonus and then publicized that so so we have had those but but I don't know if any have resulted specifically from our program yes

I'll come work for us and you can see it so I I think knowing the knowing the methodologies a wasp optin all that to actually get in is super important and I think we could probably see a situation where people could see more of the other side if we did private type bug many things there are there are mechanisms for that we've talked about that a little bit so when we when we first started in our paid program we didn't have that public at all we invited people in and we were also pretty frank talking with those people if they had questions about how things worked we would tell them same with like the mitigations here that I talked about

today if you're ever doing a bug bounty program and there's something that's unclear try to engage with them on Twitter or slack or whatever because we want you to find stuff that you may not see and we might give you clues or at least explain how like the architecture works in terms of actually seeing it from the other side I I could see more like deeper collaborative penetration testing stuff happening where you would say it from both sides there are some like live hacking events that like hacker one puts on where you're actually sitting there with the security team at the same time and then you have like face to face where you're doing that so

I suspect there's gonna be a lot more of that but I think we're still pretty early in the the adoption phase which is why I'm here

so coming back to what you mentioned I think they're I think they're a compliment so an internal red team is gonna be doing a lot of the same stuff we would expect from a bug program I would say for red teaming probably more like Network level stuff necessarily the gap SEC stuff but we whether it's whether it's a red team whether it's our own team or an Assessor that we have come in we're expecting the same kind of reports from a bug bounty program we're letting them hit the same kind of stuff the only difference is we're not handing our source code away to random up any people so they're getting more of a black box all right we probably have

time for like one more somebody wants to throw anything out all right thank you very much

you

you