BSidesSF 2020 - Non-Political Security Learnings from the Mueller Report (Arkadiy Tetelman)

all right everybody welcome Arkady gentlemen chief or head of infrastructure and security at chime is that correct there you go hi everyone welcome to non-political security learnings from the Moller report my name is arkady Tuttleman thanks for being here when I originally wrote this talk was a little bit longer than 25 minutes so I'm gonna try and go really fast but the slides will be online that's recorded of course so you'll get to see the agenda for today I'm gonna start off with some brief background about the Moller report the timeline over the report kind of give us all some shared context to think about then we'll dive into blue team learnings talk about the timeline of the

attacks recommendations for organizations what tools the GRU installed what data they stole that sort of stuff hopefully there's time at the end for questions so when we're talking about the Moller report this is really the timeline that I'm thinking about even as early as May 2014 the IRA the internet research agency started their social media disinformation campaign this was before any candidates had even announced their intention to run for president so just general sowing discord and once certain candidates had announced their intention to run then the campaign did pivot and start to favor certain candidates or disfavor other candidates a bit later in March to June of 2016 was when you saw the GRU phishing and

hacking campaign so most the time when the media talks about you know Russia hacking the election or all this stuff going on that are really talking about that three-month window that's when the bulk of the activity happened in July of 2016 the FBI opened their investigation in May of 2017 Robert Muller was appointed after James coach Comey was dismissed and in March 2019 the FBI investigation was concluded and the redacted report was a released a month later and the redacted report actually was like 95% under redacted most of the report was there so overall it took about three years to do this investigation employed about 20 attorneys and paralegals nine support staff worked on one side for the FBI

staff so 70 people working for three years pretty expensive estimated cost of 25 million dollars but actually the report brought in an estimated gain of close to 50 million dollars paid for itself twice over the reason for that was that they uncovered a lot of tax evasion and then they were able to him and that off to the relevant authorities so the report was split into two volumes the first volume was focused on Russian interference in the 2016 election which was its blood itself split into five different sections the first one was an introduction the second section was focused on this active measures social media campaign which is what the IRA did kind of like

trying to run these political ads on Facebook and Twitter and elsewhere the third section was the hacking and dumping campaign fourth and fifth sections aren't relevant for us and then the second volume was the administration obstruction of justice so for this talk I'm really just going to only be talking about Volume one section three the hacking and dumping campaign because that's what has the most broadly applicable blue team learnings that you know we're all security practitioners here that's what we'll get value out of I would say that specifically if you work at a social media company like a Twitter or Facebook or something else like that I do recommend reading the active measures section because there's a lot

of interesting information in there for you and also the report in general if you don't have the time to read it the whole thing is quite dense but if you're interested the executive summary is only like 20 pages long at the start of each volume so that's pretty interesting and accessible so blue team learnings like I said we're really just gonna zoom in on this three-month section that's when the bulk of the activity happened starting on March 10th to 15th the GRU sent out about 90 spear phishing emails to various Hillary Clinton comm email addresses starting on March 15th the GRU started to target personal Google accounts of Clinton staffers and some C n DNC org accounts and notably on March

19th John Podesta was fished and if you can't recall a John Podesta was Clinton's campaign chairman at the time so this was a pretty significant event for the election cycle it was quite damaging I think so I'm going to show you an email here this is an email from Google someone has your password hi so-and-so someone has your password review your devices go and change your password etc I would love to throw a pole to the audience the two options are going to be this is a real email or this is a phishing email can I see a show of hands who thinks this is a real email I'm a little bit blinded by light but I think

I see like a handful like two three hands who thinks this is a phishing email a lot more ants most people so it turns out it is both this is a real email that Google will send you if they think with I certainty that someone has logged into your account and it is not you it is also the template that the juror you use to fish John Podesta so if you raised your hand for either you were correct if you raised your hand for both you were doubly correct if you raised her hand for neither you probably don't like audience participation but so John Podesta had a personal aide Sarah she had access to his personal email

account and she saw this email first she saw it before he did and she thought something is suspicious about this it doesn't look quite right so she forwarded it to the Clinton campaign IT staff and this was the email exchange that happened Charles Delavan replied Sarah this is a legitimate email John needs to change his password immediately and ensure that two-factor authentication is turned on his account he can go to this link to do both it is imperative that this is done ASAP does anyone see anything weird about this email something inconsistent I'm happy to give it away if nobody wants to speak up hmm Shane I'm not sure who Shane and someone else that they decided to see see

basically the very first sense of this email says this is a legitimate email and the rest of the email seems to scream that this is not a legitimate email so it's a little bit inconsistent and so Charles Delavan later became an interview to the New York Times where he said that his bad advice was the result of a typo he knew this was a phishing attack as the campaign was getting dozens of them he said he had meant to type that it was an illegitimate email an error that he said has plagued him ever since which is just fascinating to me because I you know I think we talked a lot in security about how we have to

do things that scale we can't depend on the single point of failures like one person needs to patch the server one person needs to like recognize the phishing email like obviously these things will never scale at the same time I think this was like a pretty pivotal moment in 2016 so it's just very interesting to think about the counterfactual of what would have happened if he didn't make that typo so in terms of overall fished accounts the GRU fished numerous email accounts of Clinton campaign employees and volunteers they fished junior volunteers assigned to the Clinton campaigns advance team they finished informal Clinton campaign advisors they fished the DNC employee overall you had about 120 GRU officers

stealing tens of thousands of emails John Podesta alone had 50 to 60 thousand emails stolen from his email account so what can you do first and foremost super basic and easy password managers and hardware 2fa tokens I know I'm preaching to the choir here this is not going to be like news to anyone here I think most of the time we want to do these things and the bigger problem is getting organizational buy-in getting resource and budget and headcount and that sort of stuff but this really is like probably the number one most effective way to prevent this because both of these tools when you register on a particular website the password manager or the hardware token

will be bound to the domain name of that website and if subsequently someone sends you a phishing link that is for a different you know off by one character letter then those tools simply won't log you in so Brian Krebs had this article posted where he looked at Google's rollout of 2fa across their entire fleet of employees at the time in 2017 it was like more than 90,000 people and their phishing rate fell to zero they were like completely eliminated this class of problem because you know that the device won't authenticate to a website that is not the legitimate website and just an alert on DNS so much valuable query data and passive DNS you

know you can look at most of the time I think when employees need to get work done they really are going to be using a small set of websites to in their day-to-day work if you have one single laptop that has made one single query to this domain name that you've never seen before you can probably alert on that maybe you can join it with some intelligence data around like this is a recently registered domain or the this domain has a high threat score or something like that that's still too noisy you could decide hey I'm really want to protect these like five to ten domains like gee sweet and octave for example I really

want to protect two domains and so if you ever see a DNS query that is like it off by one character of one of those domain names that you cared about that could be alert worthy scan incoming emails lots of open source tools to do this lots of vendors that do this Airbnb open sourced a tool called email alert that'll just scan every incoming email and run yarra rules against it so that's pretty easy ingest mail audit log events gee sweet office 365 others they all provide an audit log that you can programmatically ingest and say hey someone reset their - if a token someone went through a password reset flow someone logged in from Russia or Egypt

or Brazil they're sitting right next to me what's going on pretty valuable fishing exercises question mark the reason that I have a question mark here I think there are some pros and cons to deploying phishing exercises within your company on the pro side you will without a doubt systematically reduce the number of employees who fall for phishing like real phishing attacks you will without a doubt systematically increase the amount of employees who are reporting phishing emails to you so you will have increased visibility and also I think security has a real problem with metrics and this is very measurable you can you know definitively say we are click-through rate dropped 5% this quarter over the

past quarter and you can measure that and you can like share it up in presentations and that's great some of the cons for phishing exercises depending on your company culture you might not be able to get buy-in to roll this out some employees might feel it is adversarial and it might hurt your relationship with them and and also I think you know you're never going to drive this down to zero just like with Charles Delavan it's like there's always gonna be that one out case that one edge case and so you have to assume that phishing is going to happen anyway and then defend against it no matter what like have defense-in-depth maybe deploy some sort of endpoint monitoring and see

suspicious process activity or see other behavior and so if you're going to be doing those things anyway like no matter what you still have to deploy those solutions then is it worth the like mental burden that you're placing on employees to think hey I'm being fished by my own company right now so you know pros and cons to this approach and then there's some basic like email security technologies I'm not gonna go into depth on these but like these first three help verify the authenticity of email the latter to help verify that email is encrypted in transit and their more recent so those are things that you can deploy there was an interesting blog post or a few

interesting blog posts where they're looking at the current twenty twenty candidate campaigns and their emails and so you see about like fifty percent of them used Demark which if you're a glass half full or glass half-empty could be good or bad I think it's like decent I think we're trending up it's certainly more than twenty sixteen but we also still have a ways to go okay so we're only a month in we got John Podesta fished on March 19th the GRU gained access to the d-triple-c network on April 12th using credentials that they had spear fished from those spear phishing emails that they've been sending out and there's a quote on the report over the ensuing weeks the jury

you traverse the network identifying different computers connected to the d-triple-c network by stealing network access credentials along the way including those of IT administrators with unrestricted access to the system the juror you compromised approximately 25 different computers on the d-triple-c network so over the following weeks after April 12th compromised approximately 30 hosts some people may not realize that the d-triple-c was hacked at all or that they were hacked first before the DNC but they were hacked first so you might ask the question how was the DNC hacked was it also through spearfishing and to answer that I want to take a little sidestep here into the structure of the Democratic Party you've got three components the Democratic National

Committee the DNC focuses on the presidential campaign and the party convention the Democratic Congressional Campaign Committee d-triple-c focuses on the congressional races and the Democratic Senatorial Campaign Committee focuses on the Senate races and that last one was not hacked to my knowledge like there was no public reporting it was hacked I'm just including it for completeness so the D Triple C and DNC even though they're separate organizations they do work quite closely together they work so closely together that the d-triple-c had VPN access into the DNC's internal network and so that's how the GRU gained access they were already on the d-triple-c network they were able to use that VPN to pivot into the DNC so yeah April 18th

a week later after getting access to the d-triple-c network they were able to pivot recommendations just don't allow third-party access into your network so easy like why why didn't they think of that obviously I'm being a little bit tongue-in-cheek here we you know as security practitioners I think we all received these requests of hey I just need this thing to my do my job I don't have like the resources to build it out the right way can you just give me access you're blocking me right now and like you know come on so there was a footnote in the report here the VPN in this case had been created to give a small number of d-triple-c employees

access to certain databases housed on the DNC network which is you know exactly the request that all of us receive so segregate access practice least privilege add monitoring all the usual things I think they probably could have done more to reduce the blast radius here so that given the VPN access like they shouldn't have been able to pivot as far as they did and all of these other practices that again are basic standard information security practices over the following weeks or more than a month even they compromised an additional 30 DNC hosts and then around June 8 that is when the compromise was officially removed so install tools once they were on the network what did they do there were four

tools called out in the report the first one was called X agent it was a bespoke tool not anything that you can find on the Internet they wrote it for this it can lock key strokes take screenshots gather filesystem and operating system information etc the second bespoke tool that they installed was called X tunnel we was used for creating an encrypted tunnel for large-scale data transfers so that they could exfiltrate all of that data mamie cuts if you're not familiar with me cats it's a Windows credential harvesting tool little gather credentials from memory so that one is open sourced WinRAR this one surprised me I don't these were like the four tools called out I don't I guess if

you're gonna hack the dnc you want WinRAR to compress her files that was one of the tools installed in terms of stolen data you've got key log sessions containing passwords internal communications banking information sensitive PII various internal strategy documents fundraising data opposition research emails from work inboxes in total the exfiltrated more than 70 gigabytes and election documents I want to talk a little bit about the structure of the GRU thus far we've been saying you know there a single entity but of course they have sub teams so within the report there were two units called out one of them is called unit two six one six five they primarily focused on spearfishing so they wrote those email

templates that were used they focused on building malware so the two you know X agent and external they wrote the bespoke piece of software and they were mining Bitcoin actually they used freshly minted coins to purchase some of the infrastructure that was used in the attacks so some of the hosting infrastructure and also later for disseminating information there was a particular domain name that was purchased through Bitcoin as well the other unit involved unit seven four four five five I'll just read this quote Oh while they assisted with the release and promotion of stolen materials and the quote is officers from unit seven four four five five separately hacked computers belonging to state boards of

Elections secretaries of state and US companies that supplied software and other technology related to the administration of US elections again I think that the DNC really received the majority of media reporting but there were these like peripheral hacks going on as well in terms of exfiltration and what that looked like they were already on the DNC and d-triple-c network from there they reached out to so called middle servers that they had stood up and purchased with Bitcoin like I mentioned from there they reached out to I don't know what AMS stands for but that was the name I think it's the internal geo you name maybe but this was their command and control center so this

was where GRU agents logged in and reviewed screenshots reviewed key logging sections downloaded data you know locally onto their computers issued commands to execute on the DNC Network and from there I went to the GRU so overall recommendations alert on mini cats obviously there's a lot of things you should to alert on but I think this one is kind of a freebie like if you see mimic cats on a production host you should probably be paging engineers endpoint monitoring for sure there's a bunch of vendors like carbon black CrowdStrike choose your favorite it doesn't really matter even if you don't have the resources to tune the alerts or like manually do investigations or whatever it's fine if you're even just

collecting the audit log data then it can be so valuable after the fact to be able to go you know historical forensic analysis and say here are the exact files that were accessed here the exact commands that were executed even if you can't make full use of those capabilities and if you can't make full use of them than even better network segregation I already mentioned intrusion detection system maybe I mean every company I've ever worked at we had an IDs been mostly false positives but maybe we're not tuning it correctly like maybe it could have helped hard to say so overall conclusions if you look at the attack vectors spearfishing for the initial foothold lateral movement via

over privileged permissions and mini cats pretty standard you see many things in other corporate breaches many similar things overall recommendations practice defense in depth set up to FAO and all the things have been monitoring practice lease privilege again nothing new is well-known to all of us but I think it's interesting to look at it through the perspective and lens of this attack on the 2016 election the last thing I'll say is that it's pretty easy for me to you know sit here an armchair judge the DNC but I think that realistically a few organizations can defend themselves against the nation-state so that's our thing to do with that I will take questions if you're interested in my

Twitter that's my twitter also just like everyone else wearing a blue wristband I am hiring come talk to me outside or DM me on Twitter if you're interested in that

how much time do I have I really like to FA or like the hardware tokens or password managers I think user education as good as it is and I focus a lot on user education at every company that I've ever worked at it's it's like that thing that I mentioned where like you just can't depend on getting that down to a hundred percent no matter what no matter how much education you put out still someone's gonna click the link son still someone's gonna like you know execute the attachment alright I think that's time if you have more questions I'll just be right outside in the hallway I'd love to chat about it so thank you so much