Ransomware Reaction - Lessons Learned

is this microphone okay this will work all right so our first speaker is Jeffrey Lang he is the director of cyber defense operations at Virginia Tech um Jeff Lang has been with the Virginia Tech security office since August of 2012 and brings with him 16 years of I.T experience before joining the team Jeff was Computing technology manager for the Virginia Tech School of Architecture he has experience with network monitoring and forensics intrusion detection and configuring security appliances Jeff has a ba in Philosophy from UNC Greensboro and a masters of Information Technology from Virginia Tech he is a sand certified intrusion analysis Windows Security administrator Network forensic analysis python coder cyber threat intelligent intelligence analysts and certified Enterprise

Defender so Jeff is going to be doing a talk today on ransomware I just had to and I'm gonna let him take over now thank you guys thank you good morning uh thank you for having me today I appreciate the opportunity to uh talk a little bit um we're going to talk today a little bit about an incident that happened at Virginia Tech last year we've had some departments uh get hit by the cassaya ransomware uh and we've been actually very open about what happened uh the steps we walked through and the things that we kind of learned from it and so hopefully uh this will share some information with you and uh be a good

talk

thank you so again I'm I'm Jeff Lang I mean it had the introduction there uh the responsibilities that we have uh with our defensive operations are our our security operations network monitoring incident response uh we've got computer forensics and we deploy all of our security tools maintain them uh for our monitoring operations uh my email is there if you have any questions feel free to shoot me an email I'm always happy to talk and share the information that we've got so just as kind of a background about Virginia Tech uh our main campus is in Blacksburg Virginia which is in Southwest Virginia and the beautiful mountains um we do have a large presence in Northern Virginia though and we're

actually building a uh an innovation campus there it's under construction now and that's in Alexandria Virginia uh when I took the last statistics we had about 30 000 students uh enrollment is actually up so that's probably closer to 42 000. and we're ranked number 48 in research institutions in the United States um you know nine colleges in our graduate school 110 170 programs or fairly large research one University uh but our I.T infrastructure is very distributed we have a few services that we provide centrally things like networking email our courseware various infrastructure mostly is all provided centrally our security office provides services centrally as well but most of the Hands-On I.T stuff is done at the

distributed level every department has their own set of I.T staff some of them are very competent very large staff some departments are much smaller and have a smaller set so we have a lot of challenges that we go through working with different levels of people different technical expertise and different availability with them and so those provide a lot of interesting challenges especially uh and this you know as you go through and you talk about best practices for security and you get a lot of uh gasps when people say things like never have RDP open to the internet never have SSH open well many of the research one institutions are open we have no network firewalls

except in limited places security is at the host level and that provides a lot of challenges as well we don't have that centralized management of things to try and control and do a lot of the security layer on kind of that outside edge our security office we have 10 people total um four people on our defense team including myself we've got our architecture red team our risk management team we're working on getting a 24x7 security operations center up so we have an associate director who's kind of working with that and then we also have added a software developer to try and help with the projects that we have going on uh so you know that's 10 people total

across the board doing all the security operations uh for the University itself so what ended up happening last year uh Friday about lunchtime uh right before the Fourth of July holiday weekend everybody was ready to have Monday off and ready to go and people started coming back from lunch and opened their computers and had these weird files on their desktop they clicked on them and they were encrypted and there were text files saying hey you've been hit by ransomware you need to go to this site to uh check out how much it's going to cost to get your data back um we started getting some phone calls in at the security office from one of our

departments and they were like hey we manage this thing called caseya VSA it's a you know Security Administration software package it allows you to push patches install software remote manage and they were like you know we've got all these people who are connected to this device and they're all getting hit by this ransomware so it turned out there were some flaws in that application from cassaya and there were a number of people hit by this so we started scrambling to try and figure out what was going on uh and what was happening and it turns out uh that there was a group that used sodino kiwi and attacked all the machines we had on our

Network that were connected to the VSA server uh it turns out it wasn't just us it wasn't a targeted attack at us it was targeted at cassaya the VSA Appliance so this was a supply chain attack they didn't attack us directly but they attacked a vendor that we worked with and the primary targets were actually managed service providers so there were 60 msps that were impacted there were a handful of other uh areas like Virginia Tech that got hit that weren't really an MSP but it was over 1500 clients of those msps it was a huge attack surface and it but it only uh it only attacked their on-premise software so their software is a solution

in the cloud wasn't impacted but they did shut that down uh as well because just in case it was something they didn't want that to happen so kind of the details of what it was um it turns out there were seven flaws that were exploited in order to get this attack to happen the cves are all Lister SQL injections cross-site scripting all sorts of you know pretty uh normal and and things expect not to have in a management application like this um what they did is they uploaded an initial payload through a bypass and once they had that they used the SQL injection to execute that file and once they had that they had administrative

access on the entire VSA Appliance so once that was they had that footing they were able to push out software to every client that was managed by those VSA servers um they downloaded an agent.crt file uh and they pushed it into the cassaya directories that uh were protected and were trusted so you know this is admin access this can install software and the operating system allows it because you know this is a trusted component uh so they actually were able to run a Powershell script that disabled Windows Defender and then downloaded um a some additional files uh they had the cert.exe which is a standard Windows executable they used it uh to decrypt an agent.exe file so now they had their own

agent running on all the boxes that the VSA managed wrong way um so once they had that they actually used it to download vulnerable software and this was an old Microsoft anti-malware that had a side load vulnerability so they downloaded it uh it's Ms MP eng.exe that actually looks like a you know a valid file and for a long time it was a valid Windows File so if somebody happened to see that running you're not going to have any question about that you're like oh yeah that's that's the anti-malware and that's fine um they were able to side load a dll file and then that dll file told the anti-malware to encrypt files and to do it in such a way that

you couldn't do things like Shadow recovery or rollback so basically in place you couldn't recover those at Virginia Tech it turned out we had three of these servers running only one of them was compromised it was the one that had its administrative access open to the world so uh Port 443 pretty common pretty normal uh but you know you question maybe why was it there and we have some questions about that as well um now the deed was done and we needed to recover from it uh the department that Managed IT they also allowed six other departments to use it they had already invested in the infrastructure and were paying for it and they said hey why don't you guys use

this too so all of those departments were impacted 111 servers and 805 endpoints in those seven departments were encrypted all the files on the machine all the documents were encrypted also it followed any map drives any synchronization drives like uh Google drive or Microsoft OneDrive and we actually had multiple file servers that were encrypted so we had terabytes of data on file servers that were encrypted and from the endpoints themselves so what did we do um well first we shut down all of those VSA servers even the ones that did not get compromised at the initial response we didn't know what that Vector was and so we were like let's shut them all down we also shut down the network portals

that were connected to them just in case a VM popped up uh accidentally we knew that no network traffic could get to it at Virginia Tech we have kind of our computer incident Response Team a cross-functional group of people we have some guiding documents I have a link for that later in here uh we activated that team and so we were able to bring a number of people together and start having conversations kind of across the university with the Departments uh and with additional resources that we could bring to try and discuss what our next steps were we notified all of our senior it management which is part of that cert is to get out

notifications so that the university knows what's going on we contacted the Virginia Tech police they are a full Police Department so they opened our initial police report which we then were able to report to the FBI and that was in Richmond Virginia that's our local Hub but they didn't have a lot of information because it was being actually run out of the Austin office so they forwarded some information to us as the as their regional office received it from uh from Texas and we kind of went from there uh we also opened a case with the ic3 uh submitted that so that we had kind of our paperwork done and everything going on there uh we did open a ticket with cassaya

which was very important uh since this was a now a widespread event across their product they kind of took the lead on incident response we provided data to them that we had and other organizations provided data as well for them to test and to figure out everything that was going on we started having daily Zoom calls we had actually two on that Friday and then one Saturday and one Sunday I remember standing at the grill and then going into the house and sitting down in front of zoom and talking to Senior Management and the Departments to get status updates from that and then to go back out and try and help finish off the Fourth of July party that we had going

on um so once we kind of got our our grip on what was going on the department started to identify all those machines that were out there that they needed to basically go to and and figure out what needed to be done next with them uh we decided that basically we're going to restore from backup uh and then you know wipe every machine and replace it so they began that process of identifying things and giving a status with that um and once they did get that restored they started scanning for pii so any any confidential data any Student Records anything like that we needed to know if anything was on those and then we also had to get in touch

with the Department of Education or report to them so we had weekly meetings with that update them on our status throughout everything that was going on so the request for the ransom actually came through the r evil group they are pretty widespread ransomware is a service organization um they announced on their happy blog that yes we did this haha look at us we're great um the uh we received permission to negotiate and we got a number from them that was forty four thousand dollars per decryption key there was some confusion uh with them trying to figure out exactly what that meant and ultimately it meant that every one of those machines had its own unique

decryption key and it was going to be forty four thousand dollars to pay that um we did some negotiation for about 30 Keys we had some some thoughts that we might need to potentially pay some Ransom and so they dropped that down to 700 000 for those 30 machines a few days later and you can see the announcement uh they decided that they would rather just offer out a universal decryption key for 70 million dollars and of course nobody took them up on that offer uh so as we were going through kind of the review uh one of the Departments realized that they didn't have good backups that failed and so they needed to think about paying the ransom for it

and that's how we got to that uh number of 30 they said this would probably do us and we kind of added in some for some other departments in case we needed kind of that over overhead so we reached out to our cyber Risk insurance which at that point we realized we should have done that at the very beginning it's one of our lessons learned that we'll talk about in just a little bit but they would have been able to negotiate on our behalf they have people who that's kind of their job so we potentially could have gotten you know a much better deal on that but fortunately we found out that we didn't actually

have any systems that needed to be recovered from the ransomware using a decrypter uh so the the department that was running it they had their VSA server running on a hyper-v M on a Windows server that it was itself managed by the VSA so when the encryption started happening it started happening on those windows hosts that were hosting those VMS and they basically ate themselves so it shut down all the VMS shut down the servers and when they were able to actually get at those VMS they realized that they had encrypted the data was still on them and it hadn't even downloaded the agents uh to those VMS so they were able to go through a process and pull that data and

not have to do the ransom so fortunately we decided we were done with that we stopped all communication with them and kind of went from there uh we did have a tabletop uh in earlier this year and we had a lot of Senior Management with the university and our University president Dr Sands said hey any ransomware I have to approve no one else at this University can pay it so that was it was something good to kind of get that perspective and say oh yeah you know any decisions we might have you have to take it to the top you can't have somewhere someone even the CIO of the University make that decision it had to go all the way to the top

um as far as decryption keys go uh so uh our evil and their infrastructure disappeared on July 13th they fell off the face of the Earth no one really knows what happened well someone probably knows what happened but we don't know what happened um on the 22nd of July so this was 20 days later cassaya announced that they had a universal decryption key uh so that was great we signed an NDA we actually had a our library had a digital live of scanned images we have a big architecture program so it was a lot of architectural drawings that they had accumulated over the years it was a few terabytes worth of data they had basically decided not to recover that

data they were just going to recreate it as people needed it and they were actually able to use the universal decrypter and get access back to those devices uh in September we've all found out that the FBI had had that Universal the crypto key for about a week and a half before they made provided it to caseya so there was a lot of response time and a lot of effort that was going on that you know perhaps could have been avoided if we'd had that decrypter a little uh sooner uh on September 7th uh the our evil group came back uh and in May of this year they started updating the uh ransomware software uh in its under

current development now so they kind of went away for a little bit and then came back just as uh as annoying as before so things you know some things went really well uh as far as the response you know of course not having it would have been a better thing but it happened uh and so our cert activation those things went overall very well they allowed us to bring people together and to have communication across the Departments um one of the biggest things from the first call we had was Senior Management uh it was never a blame game it was never fingers it was okay this has happened how do we recover how do we work

together to make sure that we get our data back and we ensure the Integrity of the data that we have that made a huge difference because nobody then was blaming someone else so we were really sharing everything there was no impetus to hide things that we found or you know be disingenuous about something and so that made a huge uh benefit to us overall um the departmental response was very quick and effective you know within a few weeks they had gotten most of the systems recovered that they could uh and they were moving uh forward to get things taken care of as they could we did run into some issues uh but that you know that will generally happen

we maintained our daily updates for the first two weeks and then we had a weekly call and then we had you know once a month calls and then finally we did get it finished out so that was good everybody attended everyone was there from all sorts of different we had University legal represented our Risk Management Group we had all sorts of people involved um we also during the initial event we had a lot of response from our Central I.T although we it was in a department that this was going on they supported all the work that needed to be done uh cutting those Network portals we had some centralized backups that they immediately changed the data retention

periods so that we could roll back and we wouldn't overwrite dials on Central Storage uh they even provided physical Hardware so uh the initial group that managed the VSA server they had you know they needed a large box that was able to host uh those VMS so that they could see what kind of damage was done so our Enterprise systems had a box that would do that and so they lent it to them so that they could stage uh the way that they got things going um and then we also have some staff that supports the division of I.T internally for desktop support they also handle things for our VIP clients their services were offered and they helped

reformat reinstall and re-image machines to try and get people back up and running we also had pretty good Network forensics we immediately started looking to try and find out so hey is this an encryption only event or is there also data exfiltration response is a very different different based on which one that is we determined that that we did not believe that there was any data exfiltration going on and caseya later confirmed that that this was an encrypt only attack and it was not a data exfiltration so that worked pretty well but not everything was good and again uh that first thing you know why was that uh administrative interface exposed to the internet um there were some needs for some of the

vs8 it would be exposed to the internet but perhaps they had a little too much so you know that was kind of key piece that's kind of gotten the ball rolling uh the other thing our business contacts and our cert document had gotten stale how many of you have ever put together a big document list a whole bunch of people's names only to find out later that they've all left the company and now you don't know who to contact well that that happened with us it caused us to miss out on some important communication uh kind of the risk management and some of the other areas um the university risk management also um we didn't have any contact with the

university Emergency Management they usually deal only with you know like physical emergencies a fire on campus an event that happens um you know they have a lot of resources though and they also uh you know we're bridging uh between you know things that happen physically and in the virtual world that are all linked together and so they didn't get contacted which caused some delays in them reporting the incident to the Commonwealth Emergency Management Group um we also and kind of one of the big Keys is we didn't know where our high risk data was at we didn't know what had pii or student data um you know the Departments hadn't really kept tabs on a lot of stuff we

had terabytes of file storage that had spreadsheets that went back 20 years uh and so we had to treat every device every machine almost a thousand machines as having high risk data on them so everything had to be scanned everything had to be done and that took months to do uh imagine trying to scan with a regex across three and a half terabytes of data it it's painful to say the least so that that was a huge issue uh that we just didn't know what data we had and where it was we also uh had this little thing called uh move-in day uh come August and that 37 000 students showed up all at once

and all the faculty and staff to support them all needed updates uh so uh for that start of the semester things got pushed as far as some of those scans went and it really delayed us getting the resolution finished um the Department of Education had recently made some changes about reporting deadlines we actually had 72 hours to report the incident from the moment we had any thought that something was going on um we barely made it in on Sunday morning with the with that information to them because that group was one of them that had changed the leadership and we didn't have the contact information and so when we finally got it we were already about

48 hours into that period so there are a lot of things that just kind of just didn't happen the right way um we also so we asked them what machines are impacted by this and they're like I have no idea we're like well why not it's like well VSA Managed IT so the inventory of all the machines that were supported by VSA was VSA and now it was gone so they didn't have any reports they had nothing that they could go back to and say hey these are our devices this is what we have and this is what we need to do so now again they had to go to Every computer that they owned in their

departments and say was this a VSA managed machine or not did it get encrypted or did it not so that delayed things a lot I already kind of mentioned the size of some of the files um even though the library got the decryption Universal decryption key they didn't have three and a half more terabytes of storage to do a decryption because in fact you need another three and a half terabytes of storage because you don't ever want to run that weird decryption key across your original file because if it messes that file up that was your one chance at it so you want to make a copy of that file you want to decrypt that and if

everything goes right that's great now you have the decrypted file so they actually had to do it in small batches and that just took time for them to run through that and get that process done our physical forensics we were able to support doing uh one or two forensics images at a time and now we had almost a thousand machines looking at doing forensics for so we weren't able to do that we basically just said bring us an example of every type of machine that was uh compromised and encrypted so we had a copy of a server a copy of a desktop and like a copy of a laptop that was the best we could do

fortunately we didn't need those forensics images but it would have been great to have them so what you know what recommendations kind of came out of this we had you know a lot of things worked well but a lot of things that kind of were gotchas for us um so immediately it's you know the minimum thing is if you've got an admin server of that you know that kind of admin privileges across your organization make sure that you really do have it protected and you don't uh leave it open um for us it should have been restricted at least to on campus uh and anybody who needed uh to access it remotely uh that would have you know use the VPN come

through with that uh we also recommended now that we need to take all those sorts of systems and do security reviews for them make sure that they're doing the best practices and that they're protecting logging and monitoring uh additionally with those services so that we know that they're as protected as we can make them our business continuity I mean we we made we made it through but our business continuities in different departments they couldn't cover from some of that because they didn't have servers uh we had they had to borrow servers from the central I.T they have the storage to decrypt things so all of that should be in your business continuity how do you

get yourself back up and running and that wasn't there so that was an issue that we'll have to deal with moving forward as each department goes through and kind of builds those that incurs extra cost and extra thought in what you do with that um so one of those cross-functional groups was our Purchasing Office and they were there and they were looking at it and they were like so you know they pulled the paperwork and and we did a Security review of the of cassaya and the VSA when it was first purchased but they'd had that thing for seven eight years now had done upgrades along the way and there was had never been another

Security review done on it and so it made us start thinking that you know maybe at renewal time you need to at least get a certification back from the organizations that they're doing the things that they said they were you know they said they did it at the beginning but who knows what's changed and what's happened at that organization um we were also going through a major major incident i.t-wide major incident process review at the time they were formalizing it in servicenow which is our trouble ticket system and it basically deals with how you communicate uh to your your stakeholders the University at large when you have something like email going down or canvas the course

management that's having issues and we needed to in the security side into that because that would have automated all of those initial Communications that we needed we wouldn't have missed people because we would have already formalized it had it and had review within there and we now have done that and hopefully anything moving forward needs to or would go through that process um we also want to start conducting some tabletop exercises fortunately everybody came together and worked really well but what we need to do is make sure that when we bring all those people in they know their roles they know their responsibilities and what needs to happen and take place and so we can focus these table tops around

things that we know are likely and some things that are unlikely that could happen and just to make sure that everybody knows what they're doing and you know what's going on with them so we finally got everything kind of taken care of all the machines that were encrypted were wiped completely and reinstalled there were no root kits there was nothing that caused us to have to throw away any hardware or hard drives so we were able to wipe and rebuild all of them we rolled back all the data on the share drives to a clean State and then at where they could they did those decryptions we did have some interesting Adventures trying to roll back Google

Drive and Microsoft OneDrive we had to have an engagement with Microsoft in order to do that because otherwise we would have had to have done it one OneDrive account at a time so Microsoft partnered with us helped us out with that but it still took a long time to do like I said we had to treat everything as high risk so we were scanning all those shared drives those terabytes of data for pii and for Student Records that took until December and that was just it dragged out and dragged out and dragged out and finally uh in January of 2022 we provided the last data that was needed by the Department of Education and we basically closed out the incident

at that point at the end of it there were over 129 000 Student Records that needed to be flagged in our system and that basically was saying that there was an event that happened with their account but there was no data exfil traded out of it that's a requirement by the Department of Education and most of those were students who hadn't been with the University for 10 15 years because it was all old Excel spreadsheets now everything like that is tracked you know in systems and data is encrypted in databases but back in the day all we used was Excel so we had digital cruft that just lasted and lasted and caused us to have a lot of

these headaches and a lot of these issues going forward and now so you know say oh yeah the event's done but it isn't really done um our Board of board of visitors yes our bov uh was very concerned of course uh at the end of this incident and decided to uh engage Deloitte to come and do a across the board university-wide review of security practices uh and in different areas not just the security office but within the Departments and how they treat data and what they do and how they go through it they spent a couple months going through that putting together a lot of recommendations for us and then kind of came up with

forget at least six it related recommendations there were some management related recommendations but with the idea to try and strengthen security and make sure that we have all the communication that we need together it actually got us uh three positions at the security office that we're now hiring for um and then one position to work with endpoint management that uh is was one of the recommendations to improve what our endpoint management solution was at the time it was left up to the Departments to roll whatever they thought was appropriate now we're kind of moving into something that's more centralized and can be managed in the division level of the division of I.T and used by everyone

we're also in the middle of an internal audit of course the Auditors are taking a look at you know not just the processes that we have but they're looking at our all these results and all these recommendations and things from the cassia incident and saying did you do what you were supposed to do how did you impair to what you say you're going to do and that's a good thing because it ensures that we're following the procedures it ensures that we're looking at them of course and making sure that they're appropriate for what's going on one of the big things that kind of came was that you know we had that stale data we had a document that was supposed to

be a living document that just sat and collected dust for four years without anybody looking at it so things like that are what we're focusing on trying to get that moving forward putting in some new security standards on the high-risk data identifying and inventorying what that high risk data and those high-risk machines are within each department in each area and it's going to take some time to get there but we definitely have a lot of steps there a lot of this were things that um we wanted to do things that had been kind of you know hit on an internal audits an external audits that we had before but it always comes down to money

right how do you afford to do these things and because of the severity of this incident the board said hey okay we're tired of money as an excuse we are going to fund these projects and we want results and we want them now and so we're in this process of transformation there is a link up here it's open to the public any information that's there you can read and see kind of the things that we have going on um and that's really what I have uh for you today I don't know uh if there's if anybody has questions uh whether we have some time now uh to do that or afterwards in the hallway I'm more The

Talk and go from there okay um I saw this hand first

uh so the question is was this uh preventable from the Virginia Tech perspective uh had that VSA server not had its web admin console open to the world it would not have been compromised um the group in question they actually they have machines around the world they're an Outreach group that does a lot in other countries all over and basically the VSA was open so that they could communicate with all of those machines no matter where it is that weren't on our campus that weren't in our Network basically unfortunately um the web admin interface didn't need to be open to the world so had that been done they could have left the ports open that just needed for that communication

uh and and in that case it would not have happened because they wouldn't have been able to access it

the University

uh so the question is uh with our distributed it are they part of the decision making process on how things go or are they basically given the information afterwards the answer to that is really both there are certain decisions that are made that are made from an infrastructure level as far as like networking those things that they're are kind of brought down and said this is what we're doing we do have meetings twice a year with the departmental groups we're getting ready to have one next week where we talk about what's coming and what's going on but as part of this transformation every one of those projects that we're doing has representation from our departmental I.T and the different areas

so that they can provide feedback and input into the process I believe one question in the back there

so the question is are there data retention and maybe data labeling things that we can do moving forward to try and mitigate having the information that lasted so long there is that okay um so we do have a data retention policy there are records management policies provided through the Commonwealth of Virginia and Virginia Tech policy and they just weren't followed in many cases so that's one of the kind of education things that has to happen is to make sure that we have departments doing what they need as far as data classification uh part of one of our Point protection uh transformation projects is also to do a DLP and labeling and so we're moving forward with uh

trying to get that in place so that we do have protections on that data so even if it does stay for longer than it should and that data got you know exposed somehow that we they wouldn't be able to actually access and read it so yeah those are those are projects that are underway and going okay thank you everyone um we can talk after

you have okay thank you very much it was my pleasure to speak um enjoy the rest of the conference all right