Solar Flare: Pulling Apart SolarWinds ORION

to stop talking to be great go ahead and sit down you're not staying for my talk that's jacked up dude see ya oh ouch hi we're I'm tall thank you can you hear me okay now yay alright so I submitted this talk awhile ago and I couldn't think of a name for it so that's what I picked it was a pretty horrible name and then later on I had a better name and I forgot to ask them to change it that's my better name can you write all right so that's me what is Orion so Ryan is this unified IT monitoring with the SolarWinds Orion platform with network performance monitor and server application monitor if you say monitor one more time I'm

gonna slightly right so what does this thing actually do it has all these things and these these pictures and stuff is strolling the street from the SolarWinds website but the thing is as a pen tester when I see something like this I cut my eyes kind of glaze over right there like more infographics yay but I did notice a few things that it does network configuration manager cool virtualization manager right hey network configuration manager virtualization manager it does these really cool things I want to manage that kind of stuff and then this thing at the bottom centralized settings and access control fun so I got really interested in this thing so the reason I got really

interested in this thing was because I was on a test I was already been test and one of the things that normally I do on tests is looking around for you know sequel or any kind of databases because that's where the data is it's in the name so I found this database and I did what I normally do is look for credentials or passwords or user fields or whatever in there and I found this data and it looked like it was encrypted because it had a little encrypted checkbox next to it because that's what crypto does right that's how it works you check the box and your encrypted and then I I tried to figure it out like I

tried to figure out if it's hashed if it's salted if it does all these things and I spent probably two full days trying to figure out what what the crypto algorithm for this thing was and then a buddy of mine said you're being stupid why don't you just download the trial and check it out yourself so I did and this is what came of that yeah evaluation time it's awesome when people allow you to download their products for free and try them out so that's what I did but one of the things that really irks me is like every single technical talk that I've ever seen it's all about hey if you find this thing

then you can do these awesome things and they never talk about how you find it it's all about hey once you get there here you can do all these things but they never tell you about how to get there it's really frustrating so this is how you find it and so this is the TCP dump of it but I'll get more into it so that's a scan of it and map scan so we're gonna take out the actual Microsoft stuff it's installed on Microsoft Windows Server anyone notice anything important in here any interesting things what would you key in on if you were and this is totally interactive because I'm not continuing until I hear something

message queue yeah RabbitMQ this is fun what else HTV 87 87 in particular that's the interesting one SSDP UPnP weird stuff that's interesting well if you go it on on the orion's website that's where the web server for this thing is so if you look for it on the network and map by default does not scan for 87 87 but if you scan for it or port scan specifically you can find this thing pretty easily it's also where all of the agents are talking back to so if you're on one of the servers or workstations that have the SolarWinds agent then you can just see it in a net stat there's a lot of other interesting things in there

too and when I was pulling this information I noticed that thing 2 5 6 7 - anyone know what Erlang is it's a programming thing right something like that anyone know what this C C is this C C is a distributed C compiler I absolutely love this binary because what it does is it takes in C code compiles it and spits out whatever the result was and one of the cool things is if you ever find this on a pen test you can tell it hey here are the commands to run the compiler with and then you get code execution well Ruby thought this was a great idea so now they sure Ruby has this

distributed compiler as well and if you run it you can do it there's all of these for almost every language I've ever come across there are these distributed compilers or distributed node type things or you can just say here here run this for me and it does it now I have not looked into this part specifically because I just found it in the back while I was waiting but there was one thing that I found when I googled for what this thing does I think I'm onto something so I'll look into that more later I haven't looked into it yet but that port was definitely open on my test windows system that I had the evaluation on so

more to come later so the first thing you got to do getting back to the actual talk is because you know you're probably like hey why don't you stay on that track now is authenticate to the web app so 87 87 what do you think the default account is admin admin you are wrong you'd fail login admin blank guys tried too hard so the cool thing is that it is admin blank but they do force you to change it in later versions if you're still we're doing old versions then you are not as lucky because I did find admin blank on the assessment so getting back to this where are all those credentials stored because

this thing what it does is asset management it has to have credentials to do that guess what it manages vmware ESX credentials snmpv3 credentials windows credentials Orion credentials so the first thing I did was hey hey if I'm logged in to this thing it's admin blank I should be able to look at the credentials right I should be able to just just pull it out and you know look at the the box and it shows me all the credentials because that's how things work normally right unfortunately they do it pretty decent it is not reflectively low the the actual credentials into the password change box or the current password angel that's great on them but then I started looking

around on the trials version and seeing what binaries there were and inside the Program Files and so I can't find the sequel manager of her this thing like it's not installed by default but it's managing the database somehow and they're in there in the directory was database management management exe awesome let's load it up anything notice anything like you guys obviously didn't just click this but this is the first window I saw when I double-click that binary I didn't have to authenticate it's amazing plus they have this little button add default server boom and it just works so even if you're not a syndicated so how does that work well I'm not going to worry about that

for right now I'm looking for the credentials and then I found this directory or this database it is a credential and I'm not as observant as you guys probably are I said where are the stupid creds they're not in here can't find them until a buddy of mine who was also on the test said hey look one more down there's credential property like haha so I looked in oh I'm sorry it looks awesome on my screen so I looked in and there's there's over user my awesome user has encrypted has the password in there and that's exactly what I saw on the test so I said awesome I'm on the right track so how did they

encrypt these things I got to figure out how they do things so I looked in the directory again and they had this wonderfully titled dll called security DLL awesome I was gonna look for encrypted DLL but it wasn't there either so security should should do it right anyone know what this tool is it's called DN Spy a lot like I'll spy basically they are D compilers for c-sharp code so I dragged and dropped the DLL into AI DN Spy and when that happens I became a reverse engineer

yeah I love you guys all right that's how that coming right so when you look in there when you look in D in spy or any of the are aisle spy it'll decompile us stuff unless it's obvious gated there are D n spied does a pretty good job at D obfuscating code but there are ways to make C sharp a lot harder to see obviously SolarWinds did not do this which is a recommendation for later but anyone notice anything interesting no it's hard to see sorry it looks great on my screen I'll make it bigger for you so there is a decrypt function awesome great I can G crypt things looks like a really small function right so I had to

go deeper and find decrypt XML and D kept short and then I found that it was certificate-based Wow so now I have to figure out what kind of certificate this thing has I mean it is at 128 256 cipher or CBC but it uses a certificate to do things that really sucks so let's find the cert we're aware is a certain oh there it is it's in the system store for the service but it couldn't be exportable right they don't make things exportable like that like that would be a security risk oh yeah you can get the private key you can put it on your own system and you have it forever you only have to be

admin to do it the secret sir Tiffa kit doesn't ever change even with revisions so the same certificate was installed in revision I think I don't remember the version numbers anymore I think it was like 5 version 5 all the way up to version 14 or whatever the versions are so on the client themselves during the test they had tons of different versions that they had gone through and there was the same sort as installed based on the certificate generation date from the first install of the application so it is generated per install though so you got a little bit there it's not like everyone has the exact same cert across to every single install which would have been really bad

so let's decrypt it and here's my awesome coding skills copy paste generate code done and there's my password cool except for you don't have to be an admin to do it you don't only have to be system to do it you don't have to be anything to do it you can use the cert as any user on that box awesome so if you get code execution or console access that the web that allows or sequel access that the sequel server allows you can then just parse the actual certificate and decrypt it but they couldn't use these for everything else right they couldn't use the same certificate for every single set of different creds on on the box right that

would be bad yep they encrypted the same way so just copy and paste in and there you go but wait did you catch that back here there's this password field that doesn't look like a password does it what do you guys think that is it's not base64 I know what do you guys think that is it's a bunch of numbers hmm maybe hex could be but all those look about the same and they're different length passwords so that's not hex so I looked in and tried to find where this thing happened and I saw this really cool comment in the code can you guys read it here I'll make a bigger do not use this decrypt algorithm

it is not fips-compliant use hash password instead so it's commented in the code - not to use it so why is it even in the database why is it being used at all well because you don't have Phipps compliance turned on and you would never turn on Phipps compliance on a Windows box it basically breaks everything and they warn you saying hey this could break things it does I'm exaggerating but like calc wouldn't even work like there's tons of stuff that just didn't work after that so I looked and it said don't use this but if you looked at the code it used it right after the actual hospital right before the hash password thing that it told its

use so the coder said to himself hey this is a bad way of doing things and then another coder said hey I'm just gonna use this anyways like they're right next to each other's there's no if if v compliant there's no like don't do this if the other ones there like it just doesn't make any sense to me so I copied and pasted the code and I got a really weird response my password was all uppercase like wait I've seen this before I've seen this before and knew no they wouldn't allow you to login that way right they don't allow you to log in to SolarWinds Orion this awesome enterprise-level product that the us that a lot of people use

with an uppercase password right oh yeah they do so yeah you're easily frightening number two easily reversible encrypted password quote/unquote encrypted it doesn't use any of the system data certificate it is disabled when Phipps compliance is disabled but Phipps basically breaks everything and essentially all it is is bit pooping so it takes the password uppercase is at first for some weird reason and then does a bunch of bit flipping that you can just copy and paste and now if you have access to that database you get all the passwords for Orion then you can log in and then you can use the cert to decrypt all the other passwords and then you're done I wasn't done I wanted all

the passwords and one of those passwords was the database password I remembered that I had just double clicked that stupid little binary in it logged me in automatically so where does that anyone ever used bullets pass view it's a nurse off tool don't trust nurse off but I did on this occasion there in a VM so I don't care it allows you to look at the bulleted stuff so there was the password inside of the configuration wizard for the database and I had the password but it has to get there somehow it has to get it from somewhere right so I used a bunch of other tools that had something to do with the database including the

configuration wizard and it showed me all these past the password multiple times multiple times I kept trying to figure out where it came from like the configuration wizard wasn't telling me the database manager wasn't showing me where it was getting this stupid password from and it wasn't in the database obviously so I went on process monitor and tried to view where it was coming from and I finally dug deep enough to see where it was loading the password database or the password for the database from the sequel database MS sequel so what kind of database do you guys think s/w net perfmon DB password database thingamajig is in Access database flat text sequel light fox DB what do you

guys think not even that cool you guys are right flat text file it literally had connections string equals in the text file but a hold on before you guys laughs its looks encrypted right nope so it's encrypted with the the certificate just like it was the Orion guys say it wasn't encrypted with the certificate but in my instance for some reason it was they say its DP API but it can't be DP API because I'm using it as a standard user to get the database and it wouldn't work that way because Windows doesn't work that way so they say it's DP API I say it's the crypto certificate we'll see who's the right answer I guess when they start suing me

so no screenshot of proof because the old configurations were as a client site but what happens when you install a new version of SolarWinds Orion is it just moves in the in the database file it moves the connection string instead of replacing it down and adds the new one and then down and adds a new one and do you think they were are always encrypting all of the the database connection strings so on this specific client they had 1520 connection strings to different databases they had used including a dev one over the years that's passwords probably still works and it stayed in this in this file so they don't remove anything all right so here's the results so the fix is is to

make the RSA key that is there in there in the Orion database non exportable the the storage of creds is there easy reversible format just enable fips-compliant s-- because that's the only answer I have right now unfortunately that's not an easy thing to do what I would recommend is separating the the actual SolarWinds website and an application from the database so if it's not on the same box then you can't do sequel base stuff to get the key out or the let's figure it out and once you do change it to v compliance or do that make sure that the passwords are at least changed once so that the password field that is now disabled is no longer used makes sense

just like LM and ntlm if you don't change the password even though you disable LM the password for the LM hat or the hash for the LM password is still there and still valid and then clear out York SW net perfmon database with old connection strings now one thing I will talk about that I briefly talked about at Kiwi con when I gave this talk was there is a pretty vexing and horrible DDR denial-of-service phone ability here can anyone think of what it might be what could you do to this box to make sure it never worked again turn on Phipps compliance that's funny I should have used that no delete the cert they can't regenerate the same cert if that

cert goes away that box and the encrypted passwords are gone so and the system itself won't work anymore because it can't connect to the database because it can't decrypt password so you're out of commission for at least a little while so be careful with your SolarWinds boxes generic solutions here are ensure the Orion server is protected as much as possible this thing holds your credentials for access controls and all of the passwords for Windows boxes and ESX boxes and all the fun stuff make sure it doesn't have access to anything that it's not supposed to generic generic generic security security security right this makes sense yes and make sure that you firewall off the 2 5 7 62 port because

I'm going to be playing with that shortly now when I when I started working down this talk I submitted to both besides filly and kiwi Khan and the Kiwi con people put up the abstract way before I was ready to and they and the SolarWinds representative for their security people reached out and said hey can we talk to you and they were great about it they were awesome I was doing a lot of traveling at that time they were really considerate of all of the all of the delays that it took me to get them information I finally sent them all the information they were like yeah we understand some of these these are part

of our security best practices yeah we're going to where we have the and the other things ready to fix so if you hear or see this talk either recorded or now I urge you to go talk to your SolarWinds representative and ask them about the security best practices for these specific issues see if they have anything better than in naval Phipps compliance because that would be awesome one request that I haven't asked specifically about to them is the and I realize later is that their interface does not enable two-factor authentication in any way shape or form and this is one of the biggest qualms I have with all almost all security products are security products these

days Ross Laborde 90% of them at least have zero ability to add security to them like two-factor authentication we have these great instant response tools like Splunk is one example that does have it right there are a ton of these different features and and great products out there and there's sort of great products ok products out there for security that don't have two factor authentication or any kind of multi-factor authentication please ask your vendors to add it or if you are one please have that ability because if you're not I'm sorry I'm staying on my soapbox a bit but if you're not enabling security with your security product then you're more of a problem and you are the target so

without completing my talk right now I would if I completed my talk right now it would be a failed talk because I don't have a meme I don't have a son Zoo quote and our cat picture thank you very much [Applause] I went really fast didn't I nice questions comments concerns hate mail yeah

so it wasn't land man it was a it was a it was an encoding of sorts so they were doing bit flipping on the password they were up casing first doing some bit flipping which was really complex it took me a while to figure out what it was doing and then I gave up and just copied it but like he wasn't exactly land man it wasn't that it was turning it into these numbers and it was just ones and zeros and nines and fives and whatever with dashes so it wasn't exactly and man it wasn't there for backwards compatibility for Windows it was there for backwards compatibility for the old SolarWinds products and one of the one of the scariest things are we

stopping recording we