"Don’t Lookaside or you’ll miss it: Turning a Hyper-V cache miss into 200k cash" Leo Adrian, Cbr23

uh we have our talk now this rated really well in the review process and I'm really looking forward to it as well uh don't look aside or you'll miss it turning a hyperv cash Miss into 200k of cash Leo Adrien let's welcome him to the [Applause] stage cool hi everyone thanks for coming uh so hi I'm Leo um I used to be a pent tester but for the last year or so I've been independent researcher focusing on Windows um and a shout out to the link team cuz I started there less than less than 5 years ago and now I'm standing here finding Odes so yeah great team over there um cool so this talk is a

little bit of a journey um definitely a bit more winding and uh and precarious than I thought it would be from the start it follows the tale of some cool research I did back at the very end of last year and through January um and as I mentioned I'm an independent security researcher so my goal was I wanted to find a really cool or really good hyperv buug um earn some Bounty money and that would set me up for for the rest of the year to to do some other research that was maybe less profitable um but like all good research it took a bit more of a securest path than I was expecting and so hopefully uh

by sharing that with you today um I can remind everyone you don't have to be an ultra Elite hacker who started when you were 5 years old um you can also just be someone who's very persistent Not Afraid not afraid to try something new and and make lots of mistakes along the way um and I saw this tweet the other day like 3 days ago I think and I just had to had to put it in here because this is exactly what it's like um yeah that you have this amazing goal in mind and and nothing goes quite the way you expect um so hyperv what is it um hyperv is Microsoft's virtualization solution so it's what they use to run VMS so both

in terms of the Azure Cloud if you spin up a VM there that's running on top of hyperv but also if you've just got a Windows 10 Windows 11 laptop you want to run some VMS hyperv comes uh by default you do have to in it's it's free you'd have to install it um but you don't need to pay anything extra it's just kind of there alongside the operating system um and it also is used for some security features we're not going to talk about that but like secure isolation mode for Elsa stuff like that um but what does the virtualization platform actually look like um there's a lot of components it's pretty complicated you definitely

don't have to uh remember all of this but there's sort of three main components uh to hyperv there's what they call a root partition um this is just your host operating system so technically when you turn on Windows 11 with hyperv installed that's actually like a VM that you're seeing with your desktop it's just a very special one they call the root partition partition's the windows term for VM um I'll probably just use them interchangeably but they mean the same thing um then you've got child partitions this is just this is just VMS like if you spin up a Linux VM on windows with hyperv that's a child partition um and then underneath them all is this the thing called the

hypervisor and that kind of ties them together it does does some really cool lowlevel magic that's not actually what we're going to be talking about attacking today we're looking at the root partition that's kind of like the overall thing um and I think legally if you talk about hyperv everyone has to include this diagram because because every talk I've seen so about hyperb has it um so thanks Microsoft um so now we have some idea what it is what are the security properties why would we care about it as attackers um well hyperv the the hypervisor and and sort of the root partition that manages things um they broker all the hard the interaction between VMS um and the and the hardware

so if you spin up a VM and you're like I want to give that thing two CPUs and 4 gig of RAM but my laptop's got 12 CPUs and 16 gig of RAM well hyperv has got mechanisms in there to make sure that the guest VM only sees the things you want it to see in terms of hardware and also if you've got several VMS running like you would in a cloud it also Brokers the the devices and the hardware so that the VMS can't interact with each other when they shouldn't they can't see any other any other systems running on there um but as we just saw it's a pretty large and complex looking thing

right so we're thinking about security we're also thinking there's probably there's probably going to be some bugs in there and Microsoft knows all this too right so they offer a pretty nice amount of money uh if you're able to find something that even looks like a guest to host Escape so it's $250,000 if you actually have one with an exploit even if you don't write the exploit part of it even if you've just got a bug that looks like it could be exploitable they'll pay up to $200,000 um and even before I started this research there were other researchers who' publicly said yeah they gave me the money so that's kind of why I chose this

this bounty program to to put some time into um so I'm not going to cover all the components here but I sort of broke it down into like what the attacks surfaces of hyperv the kind of functionality you can interact with from from the guest um we're going to really focus on that root partition kernel so that's the kernel driv is running in the the host OS that you're using to manage all the VMS um but there's a lot of attack surface in in the userland so in sort of the actual processes not in the kernel stuff and then the hypervisor itself is also a really cool and complex uh attack surface but not something I've

di dived into directly um so most specifically I want to about talk about this thing called vmus um vmus is a what they call a power virtualization framework um so normally when you have a virtual machine right it's it's an operating system you boot it up and it goes oh cool I'm running right I'm alive um I want to talk to some Hardware like what's my CPU how much RAM have I got are there devices available to me like hard drives graphics cards like network network cards things like that and so back in the day what they would do is they would emulate these devices right you would write some software that pretended to be

a hard drive like a spinning platter disc drive um really really complex to do really finicky if you get it wrong the operating system is going who the the hard drive stopped working like it's freaking out um and it's also really inefficient uh so over time they develop this this better idea called par virtualization where you install some special drivers into into the VM and onto the host that's running it um and these drivers are kind of aware that they're being virtualized right so you turn on the VM and it goes oh hey I'm running inside hyperv so instead of just acting like I'm talking to a real hard disk I'm going to use this special set

of drivers specifically for hyperv uh they're tune to be way more efficient um and to be able to talk to the host uh in a way where we we sort of both sides know that we're not really using Hardware we're kind of in this in this software abstraction um and vmus supports a few different device types we're going to we're going to look at storage mostly today um it does require Hardware support so if you have an incredibly old CPU that doesn't have support for the virtualization extensions you can't do this stuff but we're talking like more than a decade old at this point um so the fundamental technology that that vmus is built on um and that

makes it really efficient compared to to some of the emulation techniques is the idea of shared memory so if you've ever used like shared memory mappings in in Windows or Linux programming the idea is essentially that you've got your your physical memory so like your RAM um and we do some cool stuff with with virtual memory and Page table mappings uh to map the same bit of Ram uh into sort of the root partition into the host's memory and into the guest VM um so the addresses like the Little Numbers you the hex numbers you'll see would be completely different uh but actually when the CPU does its all its tricks to resolve the page tables they're pointing

to the same place so this means one side can just write write to a piece of memory and the other side sees it pretty much instantaneously so it's a really really efficient way of communicating and that's kind of the high level idea of how how vmus uh becomes really efficient is um any communication of of sort of mass data is just done by setting up one of these these shared buffers and what will happen is the guest will like say hey I've got this buffer at physical address two3 and it'll send it through the hypervisor to the host and the host will go to the hypervisor and say hey the guest has sent me two3 that's not a real address

can you tell me what the real address is and the hypervisor will do some magic and go sure the real address is actually 456 and so the host can map that address and now they're looking at the same thing um and yeah really efficient um and if you want to think about it in kind of like an OSI model way um yeah so vmbs kind of replaces like your physical layer and on top of that each device has its own specific protocol so who've got Vore nvsp they're really small we don't really worry too much about those and on top of that you have your more generic protocols that you might be familiar with like SCSI RN

is like a Microsoft specific networking over USB protocol that they just sort of crowbar in there um but then on top of that is tcpip and you have your applications above that so it's it's kind of how you can think about it and um the the lower level uh protocols incapsulate the higher ones right so as you as you're passing these you unwrap the lower layers and pass them off the stack um and before I go any further I I I want to give a shout out to this guy Peter havav um because his blog post here is kind of what gave me the impetus for research we're going to talk about today um I had previously looked at some

hyperv stuff but this was kind of what got me back into it and going hey like this is something I want to look at again um so this describes some very successful research he' done he found like 10 12 bugs over a couple of years in hyperv um and he goes through his methodology here um and essentially at a high level his idea was like he's going to do fuzzing but instead of just doing data fuzzing like hey I'm sending a message from A to B and I'm going to mutate the message he said it was to to fuzz the state of the hypervisor so it's not that the individual messages being messed with but like sending messages in

a weird order or across multiple threads trying to find race conditions and um like this kind of stateful fuzzing something that's been done before with things like szalla but I'd never seen it talked about in the context of hyperv it was really successful for him and so I was kind of like this is cool like I I have some maybe ideas of of where I could take this and and and do this kind of stuff myself um cool yeah so now to get into something a bit more specific um so so I after reading this blog I sort of had like a bit of inspiration a bit of a plan I was like okay I'm gonna replicate

peters's work on on this sort of stateful fuzzing um but I'm going to focus on the virtual storage layer because it wasn't something i' looked at before I was like I've got this great I'm going to implement a a multi-threaded generative fuzzer written in Rust like I didn't have ai back then but I probably crowbar some AI in now right like really cutting edge stuff it's going to be it's going to be awesome um and so I spent some time reverse engineering this storage stack trying to like okay figure out what are the different components what are the actual drivers I'm working with I learned a lot about SCSI it turns out like that's just your generic storage

protocol but um yeah there's actually like some SCSI commands that are in there but they're not documented unless you're like the member of like a secretive Industry Group it's like I guess it's like the secret to fast copying is not something they can give away for free um and I wrote like a little harness uh using the Linux kernel so the nice thing about hyperv is all the the sort of guest VM stuff for the Linux kernel is completely open source U Microsoft have upstreamed into the Linux kernel because obviously they want Linux to work really well in Azure um and so I was able to go in there and play with that Cod a bit and use it to start

sending messages directly through this vmus thing into the the virtual storage layer that I kind of wanted to fuzz and Target it's like yeah cool things are going well this this seems like a great idea and then I get into the fuzzer I'm like fuzzing is awesome like fuzzing is really cool right now it's in Rust as well I love rust so I spent quite a while like writing this this generative fuzzer and what I mean by generative is again we're not mutating some some particular like data or packet instead we're generating packets that should be valid but we're sending them in maybe a weird order or we're sending them across a couple of different threads at the

same time and hoping to trigger a race condition um and and this was going to happen like in in the Linux uh in a Linux VM and we're sending it down to the down to the the hyperv uh host and the konel um and so this is basically me at this point I'm like yeah cool like great idea I'm definitely going to find like a thousand bugs with this thing um it's crazy fast like I'm doing a lot of benchmarking on it but I'm not really doing the fuzzing while I develop it you know I'm like having to recompile the Linux kernel constantly is really slow so what I do is I sort of write some

stubs to like yeah I can definitely send messages from the colonel to the host but I'm going to do most of this stuff just by like writing some Rust and have it output to standard out or write to a file like you know that's that's kind of what I'm doing when I'm developing this thing and and it's crazy fast right like I'm getting great results it's looking awesome um so I want to just have a quick interlude and refer to uh to Mark out's 2022 offensive con keynote where he's talking about the helpfulness or unhelpfulness of writing tooling as part of the vul research process so I'm going to quote him I've seen a lot of times in

my career people start doing vulnerability research and they're like I'm just going to automate this thing and then they spend forever doing that and they forget why they were even doing it and they make this huge complex program that might not even be particularly useful in real life because they're things they think elegant and are going to work in their mind don't actually work in practice um yeah so I've spent probably a good month writing this this cool fuzzer in Rust and it's so fast uh and I eventually like pluged the pieces together like cool I connected up to my little Linux kernel harness to like start sending packets down to the host get some crashes um get

a crash but for those of you who haven't seen this particular kind of message before this is the message you get when you make the Linux kernel really upset uh and so basically um it was it was pretty uh immediately obvious that uh actually I hadn't thought about a bunch of stuff and all I was doing was crashing my own guest konel um I had this amazing idea of like yeah this is this is going to work really well it's going to be so so fast but fundamentally i' I'd failed to understand some of the some of the protocols and Technologies in use um in my sort of keenness to start this project uh and my attempt to

just crowbar a fuzzer into this Linux kernel stuff wasn't working um yeah so I I've mentioned do is create a buggy guest VM it sounds like you could probably still get a cve for it these days but um not what I'm trying to do uh and in all honesty I felt a bit silly right like I'd spent a bit of time I I got really tunnel visioned into this idea of writing this particular fuzzer but I didn't give up and I started to like figure out okay what's going wrong and it turned out that um you know there's this VM buus layer that I briefly touched on that's kind of the transport mechanism and I

hadn't really thought about that all I was like I'm going to generate all this data and just send it and I was like okay but like how is it sent and is there like a cue and there's some ring buffers and there's actually like you know there's there's a threshold Beyond which it can't send more data and you have to wait for it to catch up um and essentially I was just overflowing all of these things on the Linux side and then the kernel couldn't do any storage like any disc iio and it was getting upset and it was crashing so like I slowly started to figure out like oh okay actually just like trying to make

things as fast as possible in this case is is not going to work um but a couple of days into this mess I'm just you know constantly testing the fuzz like maybe it'll magically work this time um and actually I suddenly get a crash right this is a real one um this is a wind debug a screenshot of wind debug so this is a crash I got in the actual host Kel I'm like wow okay we're back on like we have got a real guest to host crash um but where this came out of nowhere like what I thought my fuzzer wasn't working um so this point I drop everything else around the fuzzer like yeah I'm not

working on that thing anymore um and try to triage this crash and figure it out um so this is a stack Trace sorry it's probably a little bit small but the the first thing that jumps out me so there's the crashing instruction everything above that is like the windows crash Handler we don't care about that at all um but looking at that that instruction and the function and the the driver that's involved and what's below it um is there actually no storage stuff there at all so this doesn't involve any part of the host storage deack it's not in the like SCSI passing or anything instead it's in this VM bus layer that I had completely you know ignored and and

not bothered to understand um and so I don't really have any any background for like or context for like what's going on here um so I spend you know luckily in like the first sort of 30 minutes I'm able to get a like a very pinpoint understanding of like oh okay this is a use after free bug there's there's like a link list being iterated and one of the one of the nodes in the list has been freed and I sort of get a very rough idea of what's going on but I don't really have any context for like what this vmb km CLR driver is or what it's doing um so I have a little bit

information about this crash but really if I want to report it to Microsoft and and certainly get the full Bounty I need to be able to reproduce it um unfortunately as previously discussed my highly Advanced rust based fuzzer does not help um it's basically non-deterministic because of the the really buggy stuff it's doing on the Linux side so I run out a few more times it just crashes the guest colel it's not helpful like I've basically fluked my way into this into this cool guest to host crash but if I want to reproduce it I'm actually going to have to understand what's going on uh and somehow like manually trigger the conditions for this thing to happen again um so yeah

definitely not a deterministic fuzzer which is is a problem um but one thing I find that's really helpful is to start to document assumptions with this stuff like okay well how does this actually work what's going on and so starting at a really simple level like the guest is sending some data some stuff's happening and then the host is crashing right and I can even from what I mentioned before I've done a little bit of triage immediately I can sort of see some functions and like okay well I think what's happening is that this input fill Q function that's reading stuff from the guest I'm guessing um it's calling this input batch process and in Q function

and that one seems to be doing some freeing and then input fill Q sort of gets control back again and it's iterating this link list and and one of the links is free sort of crashes right that's I can sort of start to see where where this logic might be happening so I set some break points um windy bugs awesome Dynamic analysis it's like incredibly helpful um and I run my fuzzer again and I sort of see most of the stuff get hit but for some reason the first thing that strikes me is this this free actually is not happening right there's there's no there's no free so there's no use after free um so I've

got to like figure out okay well how does this code work how can I force it to free a packet because I don't know what I accidentally did in the guest that was causing this to happen Okay so back to vmus again but this time I'm actually going to figure out what's happening and try and try and understand not just that high level diagram but you know get like a better newer diagram where I can really understand what's going on inside these host drivers um unfortunately there's almost no public documentation uh was sort of really in the inner workings of hyperb which which again on the host site is completely proprietary um there's public symbols available like

bless Microsoft for providing public symbols for for almost all of the code that's in Windows um but it is a close Source driver um and thankfully though it does interact with this open source Linux client so I can kind of get some information there in terms of what's being sent uh from from the guest to the host um so I reverse engineer as much of it as I can um there's not a lot to go on there's some there's some debug strings things like that but function names especially if kind of like where I get a lot of the context information um and I'm able to to end up with a pretty decent understanding of the packet

processing flow um the sort of driver where the crash is happening uh manages what what vmus calls Channel objects so for each of the the different virtual devices and and potentially multiple like inside the device as well there's what they call a channel and a channel is just exactly what it sounds like it's a channel for communicating between the guest and the host um there's usually one channel for like guest host communication and one for or yeah there's sort of one communication layer for guest to host and one for host to guest so they aren't sort of writing over each other um but yeah the this driver essentially creates and manages these Channel objects and then also

provides an API for for the actual storage driver the actual Network driver um to go hey if if you want to use vmus to talk to the guest um here's here's code to like set up a channel and to manage it and set some call backs and stuff like that um and yeah there's lots of passing routines so definitely like oh this is actually interesting attack surface I probably should have looked at this before um so if we look at that sort of shared memory diagram again uh sort of the the central part of vmb BU is that these shared memory buffers get mapped as what's called ring buffers and you can think of a ring buffer as just an

array right so you've just got an array of items and you can add to that array but instead of getting to the end of it and being like well it's full we need to allocate new array with a ring buffer you kind of just work incrementally so you get to the end of it and then you just start adding stuff back at index zero and and so as long as the writer isn't writing way faster than the read is consuming like perhaps a certain fuzzer was causing um yeah it's actually a really efficient mechanism for for sending data because uh you don't have to keep allocating more and then like freeing the old stuff you can just keep

reusing this same buffer so each VM buus channel allocates two ring buffers one for sort of the guest to write to and the host to read and one for the host to write to and the guest to read so this way you're only writing to one at a time um it just helps with synchronization makes it more efficient um and again thankfully the Linux kernel has has a lot of uh like code in there that's used to send these kind of packets and messages and use these ring buffers um and so I've tried to minimize amount of code on the slides because I know it's hard to read but essentially on the left we've just got

like the the header for the packets so like an actual VM bus message like a packet will always start with this VM packet descriptor that's kind of the header um there's different types of them there's some offsets lengths and stuff and then on the right there's an example of one of the types um and this particular type the BM data GPA direct contains the gpa's guest physical addresses so this is where the guest's like hey host I want to send you some information like say I want to like read 2,000 bytes of data um from from a hard disk from a virtual hard disk um into some buffer right it's going to send this packet and instead of like having

just a 2,000 byte buffer in the packet um or something like that it's going to say hey here's the virtual here's the here's the physical addresses the guest physical addresses I want you to write I want you to write to so that's like the GPA data is just a way of kind of like passing a reference and then on the host side it can use that reference to find the shared page the shared page in physical memory and right to it and then the guest can just read it out automatically it doesn't have to like do a bunch of copying and it's really efficient for for um sharing like large amounts of data for things like

networking and storage and obviously in the cloud you want these to go fast um so here's a sort of a diagram of what the input fill Q function was doing this is all reverse engineered so if there's anyone from Microsoft um this is probably wrong like I'm sorry um but yeah so first it reads it reads the packet header into into a register and I just mentioned this because it's kind of like a cool optimization um you you might be thinking like wow this shared memory is kind of unsafe right because what if like the host reads something and then like the guest changes it and then the host reads it again and it's different um well they they avoid that

with the packet header they just read the whole thing into a register and they never read that header again right so you there's no like time of check time of use bugs that I could find maybe there are um but kind of cool um and so it does a check is is the header valid and if it's not just freezes everything um and just it stops um but otherwise it allocates a buffer from from a look aside list get into that a little bit later um it writes the packet header and the packet data into this buffer um and then it adds it to a link list so already as I'm understanding more about this driver I'm like okay cool this kind

of lines up with what I what I found when I that initial crash triage right is there's there's some kind of Link list structure of presumably guest data and something was being freed so like I think I'm on the right track um and next there was this input batch processing in Q function so this was the one where I thought like the actual free was happening um and sure enough there is some logic in there to free essentially the purpose of this packet this function is to go through that initial list of packets and for any packets that have these gpas these guest physical addresses um it just does some validation like is this kind of like a

reasonable address or like a reasonable description of of guest's physical memory and if not uh we're going to free the packet um and then we're going to return an error and everything's going to everything's going to stop so that's kind of like okay cool that's that's like the error condition um if the everything's good though it just adds it to a different link list they really love link lists and passes that back um but this was like okay now I understand maybe why that free wasn't happening when I when I just ran my fuzzer again um is we've deliberately got to be sending uh malform data and that's I wasn't trying to do that it clearly it

happened luckily at some point um but that's kind of what I need to do to trigger this this crash right so now I'm like cool I have some understanding of how to reproduce this can write some code in the Linux kernel to send a slightly more malformed packet so it'll get to that point it'll do the free um then we'll have this iteration and we'll have our user after free and the host will crash um no no it still doesn't work unfortunately I yeah got all done all this work and and something's still not happening right so um the one thing I haven't really looked at much yet is is how the freeing happened there's just this function

called input free packet and I'm like cool it frees the packet right well not quite so I go into input free packet and actually it does this check right and it's it's checking was the the packet buffer allocated from something called a look aside list and if it was it freeze it using this special API and if not it freezes using this other API XF free pull tag is like the default one right if you if you've used like Malik and free andc in the windows kernel you just expect everyone to use like X allocate pull with tag and x free pull with tag that's that's what I was expecting um but it seems for some reason like

instead we're not hitting that code path and again I set a break point like can I actually hit this piece of code that I think is doing the freeing right X fre pull tag and I I just never see it I spent quite a bit of time going down this Rabbit Hole I'm just never able to get that particular piece of code to execute and eventually I kind of just submit to the fact that I'm going to have to understand even more and dive even deeper into what is a an endpage look aside list and and like is there a way to to cause this to cause a free um the lookaside list API is documented and

you don't have to read all that essentially a looks side list is like a caching allocator so what that means me is you go to allocate some memory from the operating system you use it for a bit and then you free it right um but if you're allocating a lot of stuff and then freeing a lot of stuff this can get pretty uh inefficient and and like expensive in terms of constantly having to go to the OS allocator and be like give me more memory here's it back give me more memory and so what the look asite this does is you allocate a bunch of memory and it will just uh when you free it it'll just hold on to it it

won't actually go back to the West say I'm just going to hold on for to it hold on to it for a bit because I know that you're probably going to come back to me soon asking for more memory and I'm just going to give you the stuff you'd already allocated but you thought you'd freed um so it's that's kind of why I'm not seeing this this crashing condition again is because um what's actually happening is we're just freeing it back to this looker side then it's never really being freed like nothing else is able to use that memory um and so that's kind of like blocking us from having the like we know the crash can occur we've

got this crash dump we we stumbled upon but we can't get it to trigger again because this look aide list is kind of blocking us um and it's unfortunately not very well documented like in terms of how it works internally like there's nowhere in the documentation is like oh yeah if you just want to get one one packet to go back to the OS just do this thing um and so I was like all right well I have to understand how the look side list structure really works and and dive a bit deeper into that um to get a better understanding of of how to force like one specific packet to be freed so I can actually cause a user after free

reclaim the memory with something else if I want to um and so yeah again little bit of code this is the the rough structure of the looks side list that I was able to to reverse engineer a bit um it's not documented but a bunch of people um like react OS have got some definitions I found the one in the version of the colel I was working on to be a little bit different um but they they were pretty close and definitely good starting point and the three the three main things I realized you sort of have to um learn about the a look aside list uh is is the the list header so it's

essentially a singly linked list so just like a list of allocations as you allocate memory from the OS it just puts them in this in this big list and then these two other fields the depth and the maximum depth um and this definitely took me a while to kind of get my head around it it wasn't explained well on the doc the developer documentation but the depth is essentially like the number of freed packets that we're holding on to right so we allocate like 10 packets uh and then we and then we free 10 um there the max depth says oh we can only we can only hold on to four of them but we'll hold on to those four so if you

need to allocate again we've got up to four available before we have to start going back to the OS um my partner is an amazing designer and she's very kindly made some animations that hopefully will make this like a little bit um clear about what's going on uh cool yeah so allocate from the OS they go into this list there like the control structure and then we free them and they don't go back to the OS right we hold on to them and then we can just use them again right so pretty simple pretty cool uh trying to get to the next one cool um okay cool but what happens if we allocate like more than more than we

need and More Than This max depth figure right well so we allocate a bunch but as soon as we we try to free more than four right because four is a max depth then that one goes back to the OS so I can just run that one through again like we allocate in this case eight eight blocks of memory with the look aide list they go into this list and then if we free them if we free more than four which is our max depth then we can get it back and so I'm like okay cool I finally sort of understand how we can get the look aside L list to free uh piece of memory

back to the OS and then we could reclaim it and have like a use after free condition where um the the driver is using memory um that it thinks is of one type but we've actually like it's been freed and we've replaced it with a different type and you know maybe we can we can get some exploitability um like an exploitable bug out of that so um sorry just trying cool so like this is kind of where we're at um and I've got some idea of of how to get this to happen but I'm not really sure how to like make make this happen in terms of like using this look aside this like in theory I Now

understand like okay so I need the max depth in this case four was the real number right it would hold up to four of these uh these packets um in the list before it started sending them back and go okay we're not going to hold on to like 100 packets um but I need to perform like an extra step and so I was like all right how how could I make this happen right so I'm thinking about the packet passing flow um I read I read some data from that the guests written to me like as the host um and assuming the packet head is valid I then go to this looks side list and I

allocate a buffer and I'm like okay so this this buffer um like is contains a valid packet I'm going to add it to a link list um and then once that first link list is full I'm going to pass it to this other Pro um other function and this function is going to check that like if there's any guest addresses in there they're also valid and that's all good um and if it finds any that are bad it's going to free them um or it's going to fre sorry it's going to free the first bad one it finds and then and then stop so we we want that first bad one to actually be the one that looks like side

list goes yeah no I don't want that one like send like I can't hold anymore send it back to the OS and it's just kind of like a fundamental issue with the way the look side list Works in terms of thinking it as like a state machine which is that every time we allocate a buffer a packet from this buffer we reduce the depth right depth is how many unused things do we have and so if we use one of them well the depth goes down and so just kind of like the act of allocating this packet like if I send a bad packet uh that's dropping the depth and then it goes to free the bad packet

and the depth just goes back up to four like even if it was a 4 before so I was like like I'm so close but just something isn't isn't quite working and it um it took me a little while of thinking about this to be like how can I get how can I get this packet to be freedback that West right I know it can happen I've got this crash um and if you if You' sort of done a lot of VR before you might have some idea of like there's still one thing I haven't really talked about or looked at um and that's that's like how does a good packet get freed like that's kind of the the missing

piece of the puzzle that that I ended up on um so I set What's called the hard Ware breakpoint on just like a regular Packer I was like okay a hardware breakpoint is um like a regular breakpoint you may have used on in your code you're like hey when this bit of code runs I want you to stop and like tell me we've hit this but a hardware breakpoint you can actually say like when this bit of memory is accessed I want you to like stop and tell me so I set a hardware breakpoint on uh on one of these regular packets I hadn't messed with it like all right what what's like the lifetime of a packet where does it

go where does it end up um and I got this this huge call stack again you don't have to look at the whole thing um but it ends up in this uh vmb CMR like uh it ends up in like a packet completion function that's like hey we've finished using this packet we don't need it anymore um and that's where it gets it gets freed and the interesting thing about this call stack that I found though was there's no there's none of the other vmb buus code in here that we've been looking at right so it's actually a completely different call stack and it's happening in a completely different thread and this was kind of like the key to this puzzle was

like actually we can have packets being freed in one thread while they're still being allocated in the in the other like like finally things things are starting to make sense um and so I I had like a plan right like this was was like finally coming together if we spray some number of good packets so the number ended up being about 50 we wait just a bit of time for the packets to start to get processed and then completed right and so when they're completed they get freed and so that look aside list it's going to start having a bunch of free slots right we finished using the memory well we're not going to give it back yet

because we're a looks side list um but if we have enough of them free then actually they will start going to the OS and so I could uh line up the timing so that a bad packet gets allocated kind of while the other packet's in a different thread of being freed uh the bad packet will drop the depth right because it's like hey I'll use one of those slots I need I need some memory um then I can Pat it with some larger packets so the the other thing is um these channels had quite I think they had like 10 lookaside lists of different sizes and so if you sent like a radically different Siz

packet it would go from a different lookaside list so it's not messing with like our little like bad packet look look aside list State machine um and and then hopefully like while the packets are still being processed uh in our sort of uh input thread off in the other worker thread um the packets are still being freed and and try and just get the the mathematics to line up um so that the pack gets allocated and then another one gets freed somewhere else and so we can bump that depth up to full um and eventually uh we we can actually have our packet get freed for real back to the OS and we can have our us off to

free condition um and so kind of like this was the bit where I was like okay I finally understand enough about this whole process uh to to get it to actually crash and again hopefully like a nice little animation to kind of like explain a bit more about what's what's happening um so like fill up a bunch of good packets send it down to a network device uh like the network device thread which does the like main processing um and then here so the the yellow packets are just the different size right so they don't they don't interact with the lookaside list that the green and red do they're just kind of padding um but here

you can see like while this thread is is sort of processing in freeing these these regular packets um we send out one bad packet the depth goes down to three um but then they keep processing right and they they free a few more and so finally uh we're able to get this bad packet to be freed actually back to the OS um and cause a use after to free this was kind of like the yeah like after all this work finally understanding the like four or five different components I needed to bring together to get this to happen um again trying to exit this video is interesting Okay cool so yeah the final steps to to write the code in the Linux

kernel to um to just spray a bunch of regular packets um so I used the Ed the networking driver actually for this to just send a bunch of packets like hey I want to set the MAC address and and they just went off and did this um then send one malformed packet which which required a bit of tinkering because there wasn't really code to send malformed packets in the kernel for some reason I don't know why they don't provide that as a feature um play with a bit of timing um to try and like have a little bit of a delay because you you need to all this happens asynchronously and and you sort of need that worker

thread to be going in the background ground by the time the bad packet comes um and then finally put it all together and I got a crash like I got it working I was able to reproduce it not just once but you know like repeatedly um and yeah write up the write up the report thankfully we'd done all this work to like understand how this worked so it was a really good report and I could explain the root CES really well um and send it off to msrc and get some sleep basically everything from like finding the crash to this point was like 10 days uh of of a lot of work like I was kind

of obsessed um so yeah that was a really good feeling um and yeah after a few months uh got probably the best email I've ever got in my life and like serious props to mslc because um they really put their money uh the money where their mouth is um I know a lot of bounty vendors I don't I haven't worked with them but I hear some some rough things about other places um so yeah like incredibly cool especially after putting it all that work can only imagine if they'd come back and being like yeah we're not really interested so very cool and and I'm sure some of you like well is it exploitable um um we'll

get to that in a second not as much but but basically mslc was was happy that like there was enough potential that that they were willing to pay out the full amount um so sort of looking back at some of the lessons learned so far um so we've covered a lot of stuff so we've learned about hyperv um sort of like choosing one target you know having this great idea of like where I think the research is going to go and like how it's all going to work and my awesome fuzzer um getting a bit a bit sidetracked U but then yeah like getting this crash right and just just sort of being persistent working through it and

and really understanding what was going on like the the real key to it was was when I initially started this research right I just I just skimmed over some some things um and it turned out like actually learning about it and not just trying to automate everything uh was really pivotal to actually being able to reproduce it and turn it into something cool um and yeah modeling the look aside list date machine was was something I was definitely not expecting to do at the start of the project and hey we ended up with it with the cool proof of concept that we could send to Microsoft exploitability look in all honesty I I think this would definitely not be the

bug I was going to choose if I was trying to write an exploit very small window to to like reuse the site after it been freed because it's essentially just in like iterating a link list and we can pad the linked list with those other siiz packets um but we don't have a lot of time um the proof concept worked about 30% of the time if it didn't work all you had to do is restart the guest VM right so if you think about Azure like you can do that as many times as you want like um it's not really an impediment um but once you've caused the use after free if you like fail to to

put something else in there it may end up getting captured by a different free and and like that could cause a crash um and the use of free was just a read so I spent a bit of time trying to see if maybe there was somewhere else in the code where maybe like you know it try and there was some function pointers in this object like does it call a function pointer or like maybe do some kind of right based on it from what I could tell it was just a read um but I'd previously reported stuff like this just as a dos and Microsoft had like upped it um and like given it a higher rating so I was

like you know I'm going to say it's RC and they they agreed um so that was kind of cool they added me to a like an old CV retroactively for some reason um but the the old CV was found by Peter havav so I guess I did a good job like replicating his work right because I essentially just found a variant of that um so in terms of the actual Lessons Learned I think the big one for me was like research is fundamentally about understanding and I I got a little bit sidetracked at one point with like hey like rust is fun I like writing it it was really cool um but also like just

coming back to that understanding and like you know I I did I did get lucky and like yeah like just trying stuff and getting lucky isn't something you can really replicate but what what I can replicate is like if I get a bit sidetracked just remembering like am actually understanding this or am I just like writing code for the sake of it uh multi-threading increases complexity so much I think I think it's really easy to underestimate that um just like that look aside list State meant that we we sort of in like a single threaded model we wouldn't have been able to to get this crash or to make it exploitable like technically like an academic level

the use after free was always there right the look aside list was always going to mark that buffer as freed but because of the way look aside lists work we couldn't really put like a different type in there like it wasn't useful at all until multi-threading came in the lookaside list object was being shared across threads without any synchronization um and so suddenly like we're able to to get some um behavior that perhaps wasn't intended um again like a yeah it's a non- default allocator right so it's it's going to also hide things like special pool that are supposed to find these these kind of user free bugs and transport layers matter right so you know it's easy to be

like Oh I'm working on this cool thing it's over TCP like there's no bugs in TCP right well no people are still finding bugs in TCP and in this case yeah I was like oh VM bus is just like the transport layer right it's not like a thing I have to care about all the cool stuffs up in the like device drivers um but actually it turned out to be a really interesting attack surface and even if I it wasn't right like I actually needed to learn about it just if I wanted to use it for sending those packets um and yeah variant analysis is awesome um there's been some really good talks Matty Stone actually gave a talk a

few years ago about varant analysis where she gave a bunch of really cool examples of like people don't patch things properly um but it doesn't just have to be like Oh I'm going to recreate this bug that someone else found it could also be like oh this person had a really cool idea and I'm going to like reimplement that idea in my own way um and the advantage of that is you're probably you're almost certainly going to do some things differently and so hopefully you'll you'll trigger some code paths uh that they haven't and sort of and find some cool bugs that way um yeah good tooling is inv valuable but don't overcommit to tooling before

you've kind of confirmed that it's going to work the way you think um and when it comes down to it you know we' got to be persistent um so some shout outs to these guys who helped me with like the reporting process and also reviewing the cfp and everyone who who gives this kind of research right like um I learned so much from reading and watching presentations that people have given of their research so hopefully you got something out of this one and a big shout out to everyone who does sort of present publicly um yeah any questions [Applause] thanks do we have any questions in the audience it's fine it's lunch time I'm not

offended uh there's one just in the middle over on the

right hello hey thanks for sharing great insights um it's fascinating how we came from open source is a concert to um how windows and Linux are now interlocked inter integrated so the question was was um in the process of understanding um of both uh you mention you rever engineered some window stuff you spend time in Linux kernel um what did you use for Rivers Engineering in Windows space and um to what degree you had to have de understanding of source code and Linux kernel for research similar like yours thanks yeah yeah so um on the on the window side so I I like binary ninja but honestly like I I think whatever tool works best for you so I started when I

started doing this stuff um it was just honestly it was when gidra came out cuz it became accessible and that was really cool um it's a little bit an eyesore I think but really whatever works for you I think biner ninja is awesome if you are wanting to pay a bit of money but not like item money like I don't have item money um so biner ninja is great and that's that's what I use for all my reverse engineering now just because once you commit to one tool you can write some scripts and stuff on the Linux kernel side um I did have to spend like when I was triaging like why is my fuzzer crashing its own kernel um I

spent quite a bit of time reading that um and that was that was certainly helpful in terms of when I then went to write the proof of concept I did actually have some understanding of like okay so vmb buus has got these ring buffers and it's got some cues and like here are the functions that already exist in the Linux kernel source that I can leverage to to write that stuff so um yeah that was that was mostly useful in terms of reproducing the proof of concept but as I did show having the um like some of the type definitions but data that was being sent from the guest to host was also really useful when on

the host side there's no documentation and I'm in this like huge function in binary ninja I'm like where do I even start well I can see this is coming from The Ring buffer and I can open windy buug and like oh yeah this is a packet typ scent and I can kind of map that structure into the um into biner ninja and start to understand what the like structure definitions were and work outwards from that any other questions hello thank you for the talk um before when you were mentioning how um the freed packet goes to the looker side mhm um the second array and then that gets freed and goes to the kernel um you said that it gets freed in the

first place from the first list and then it gets repurposed to the second and then freed there and that's when it goes back to the kernel so the question is why does this end up being a use after free and not a double free vulnerability um so it was being uh so it goes to the worker thread the worker thread uh yeah sorry the the the graphic might have been a little a little bit confusing so essentially like what was shared between those two was the look aside list that was kind of like backing the memory so the work thread finished using a packet and it freed it and so but what really happened is the

lookaside list just marked that as free and then so the uh when the bad packet was then allocated a bit later um it was allocated from that same luide list and so the luide list just reused um memory that was never really free right it never really went back to the OS it just like used that same slot um and then when the bad pack was detected that's when like the real free happened so I guess it wasn't double free because we're not um like I don't know I I I definitely think it's is more of like just a use after I think like maybe uh depending on how like the semantics of the look side list you could classify as

a double free um but from my perspective it was more of like the packet was was like the packet slot was supposed to be reused because that's how the looker slide list works right it's it's caching these buffers and then it's just kind of like providing them to you know whatever driver is using them for reuse um but I don't I always just thought of it as a use after free maybe there's an argument that it's actually a double free certainly in terms of like how it's being used um is that it's like there is a memory read it's actually like a memory compare of a value inside this this location that has been freed so

like when I did play with like trying to replace um that buffer with other allocations um it was just like a like a use after free read that was essentially occurring is's a question right up the front just over there uh you mentioned that uh this sort of Architecture is used in is have you tested it in is does it work no no so in fact basically as far as I'm wear identical architecture like hyperv is what's used underneath Azure there's probably like a slightly different version of these drivers that's used there um they do not like you testing the stuff because again like even if you Doss Azure like that's a big deal they

don't like that um but from my like if you look at the Bounty page um and you can actually get some like vers like Azure like versions like you can get Azure on Prem now um and I've looked at those binaries and and they're very very similar so I think the behavior of some of the particulars might be a little different for like obviously scaling purposes but all the mechanisms to my knowledge are basically identical um that's one of the reasons I think hyperv is kind of the only Cloud hypervisor you can really Target in this way for bug Bounty like they offer a gcp and AWS Bounty but like I I don't know which

custom version of KVM they're using like I could look at the KVM Source but I'm sure they would have forked it and changed it and and so it be way harder to find a bug in KVM and go hey give me some Bounty money cuz they probably come back and go well we don't even use this B of the code or like we've already fixed that Upstream on on our version so yeah there any other questions in the audience have a look around we have a speaker gift for you that's out backstage let's thank Leo one more time for an amazing presentation