Log4Shell If You Don't Know Any Java

hopefully hi one i i know we're the last session before lunch we won't run too late don't worry so let's see um who remembers look for shell hopefully everybody um and it was a bit like the thing where in december mid-december look for shell hit and we were all kind of ready i think for christmas more or less and then somebody decided to join the party and it changed how things were working out a bit um i normally run the poll like what logger are you using if you're a java audience but i'll skip that for today i still have another poll if you go to slido.com and then sierra which is my twitter handle and

everything or you just scan the qr code you can fill out the poll i'm just curious to ask anonymously who was affected by law for shell because nobody wants to raise their hand and say that publicly that we were affected so i'll let you fill that out anonymously also there if you have any questions at any point in time you can just type them in and i'll come back to those later so if you don't want to shout we can also do that you can just raise your hand and ask in person of course but if you type in any questions i will come back to those and we'll see how that's going so let's see

um so i see that we have a 50 50 tie um and well i guess it keeps switching

cool so i i see that we had a whatever share of distribution we had um i'll i'll put that up one more time so if you haven't joined yet you can still fill it out we can take a look at it at the end like who was affected and who wasn't affected um but let's um jump right in and see what's going on so when i was asking at a recent event like who is using log4j2 that was the one that was affected um for the java audience like that was the most commonly used logger um there are some alternatives some still use println and others are what what is logging um and in that case actually what is logging

was a good thing because you didn't have that security issue then but overall probably not what you want so i have broken this down into like four parts what even is log for shell how can you exploit it how can it affect your products and how can you protect against that and well let's dive right in why am i talking about that i work for elastic the company behind elasticsearch we have used log4j2 and have because of that dealt with log for shell a lot um so we have that all of in our product so i spent a large amount of my december on lock for shell issues so what even is lock for shell um so those

who have kids who play minecraft they might have come up with that question and you can then give them a very long explanation of why there was a lock for j update and what it means and we can take the entire day or night um to explain like where this is coming from from strings in c um down to jar files but we'll do the the compressed version of that i think um so what what happened was there there was this update coming out i think it was thursday night our time when somebody said like oh there's this security thing in lock4j and everybody was like oh well interesting i'll i'll look at it i don't know tomorrow or

after christmas or whatever and then on friday pretty much the internet was on fire and everybody like yeah we need to fix that so what happened um or before let's looking at what happened like taking a step back what is even enabling this security issue so at blackhead back in 2016 um somebody said like there's this interesting concept in java called jndi the java naming and directory interface or sometimes the java people also call it gindy which basically allows you in your application to look up some other values from some source and that source could be ldap rmi or corva most of the attack vectors are now doing ldap and it's basically a way to dynamically look up

some value from a remote source um to fetch that which back in 2016 people are already like oh this looks like an interesting concept to exploit and then it was very quiet until somebody rediscovered that in log for shell and basically what you could do is you would give log4j the logger a string like this which basically says use the gindy plugin and use this ldap address to fetch whatever code you find here and then run it in the context of the application and that's really what all was to it and it was a or is a straight 10 out of 10 rc um so that's pretty much how that worked um to take a look at how this works and i hope

this is large enough to see but how loggers in in java generally work like you have data coming in and you have that logger that should write out the data and then there is a layout that defines like how how should the log be defined like this is the time stamp and this is the log level and this is the message and whatever so that's the layout and then you have the so-called appender that writes it out to a destination that would normally be a file and in the case of log4j it has a lot of so-called features and one of the features is that it has lookup plugins so you could use um that

dollar curly brace syntax and then you have a plugin name and a value for that and what it could do is like you could say java colon version to actually look up the current java version which might be helpful to say like oh this is a java version on which this is crashing or the java version we are running on um and that's potentially a good thing um what then people figured out what with jindy which is another plugin you can then load this remote code and that's really all that happened so it's a combination of a remote code execution that you load this remote code and that you also then just execute that so it's not just loading a string which

might have been fine but it's really loading that remote code and then executing it so what what happens is an attacker gives you a bad string and redirects you to their ldap server under their control gives you whatever that a thing is doing and you execute that and that's really the attack vector so somebody makes your application log that bad string it will fetch the code execute it in the context of your application and that's all there is to it you can also phrase it a bit more positively because it's not just arbitrary code execution but it's a surprise extension api which it really is to some degree but well it's not what you wanted

and then there is the big discussion is this like more of a remote code execution or more of a serialization issue um because to make this a bit more complex depending on your java version and there are a lot of java versions out there um loading that remote code has in more recent versions been changed to false so it wouldn't just load remote code and execute it but it would just fetch it and then not execute it but there are still other ways around that or you could still use that to trigger local code that already exists so a new java version is not protecting you completely against it though it is closing some of the attack vectors in

that so it's in part that remote loading that is a problem the other part is like do you even execute that remotely loaded code so that is kind of like the thing that is coming at play here um and then obviously how to get rid of that was you upgrade um log4j and log4j still supports jdk 7 which is very old by now or modern versions and so you need to update to different versions of log4j to get to the latest version which looked all good and people even adopted their everybody knows little bobby tables somebody adopted that to call it little bobby jindy so this is the the remote code execution that you could get with that and as we

know from little bobby tables you should sanitize your inputs and that still applies for this gender vulnerability the other thing that we learned personally was that most security scanners for your applications are pretty bad who's using a security scanner for their application very few okay so the problem with security scanners for us mostly are that most of them just look for the version of a library and then they say like you're vulnerable you or you're not vulnerable but lock for shell is not that simple so what log for gel does is you have some options for example you had could say format message no lookups to true which basically says like don't load remote code because most

loggers don't really need that and you could set that but most security scanners don't know that or don't take that flag into account also because of the jdk version it might not be exploitable the same way on all versions and security scanners also don't find that also the actual feature to to load the remote code in the log4j library is there is a java class called jindi lookup if you remove that from the jar you would basically remove the the attack vector or the door to it and no security scanner finds that and we have done that but then you have a constant fight against false positives funds from security scanners so we have been

answering hundreds if not thousands of emails about automated scanners that were detecting the wrong things which caused us a fair amount of pain over the time so we had the first lock for shell vulnerability but why not a second one because you can have always not just second breakfast but second block for shell and what happened was that the first patch that they brought out was actually incomplete at first they only thought that was a limited denial of service attack but it turned out it was limited remote code execution again because what you had to do is you had to change the attack vector from this to this because the patch they implemented at first was looking at like is it only

doing something local then it's fine but it turned out the way that this is using code or whatever is that it will ignore that and if it says sees attacker.com afterwards it will still load the code from attacker.com so this was a not so complete patch and the thing that makes it a limited remote code execution is that in that pattern how to output the logs um you needed to have something called uh or you needed to use this context um with whatever name you give it um in your log output and then if you ever log something into this distinctive variable um only then could you trigger it so it needed a non-default output pattern for your logs but you

could still load remote code through that not as bad as the first issue but still um not pretty to get to that um and again you could upgrade or again this one could also be solved by removing that jindy lookup class um which we have done and for us it's kind of like it's a stupid hack but it's also kind of like effective or you could say it's honest work by removing that because it really closed that attack vector because very few people actually need that remote lookup of code to execute something it's just that i don't want to be too harsh and say like it's an unnecessary feature but it's something that almost nobody in the

java ecosystem ever uses and it still opens this entire class of attack vectors so you can do a stupid hack to actually kind of like get around that and obviously there was another lock for shell security issue because this one kept giving and more people started looking at log4j and all the features it had and then they found another one again you needed a non-default pattern this was only a denial of service attack the reason was they had recursive lookups in the patterns so you could just kill your java application by having a weird pattern that did recursive lookups here the solution was this feature they just removed because again nobody uses recursive lookups or almost nobody uses

recursive lookups in their java application it's just a full class of attack vectors that nobody really needs or uses um and it wasn't the last one there was one more so there was one more security issue here um this is a remote code execution but only if you control the login pattern because you basically needed to have a specific configuration to log to jdbc like a jdbc is like the database interface in java so you would need to control the pattern of how to log to a database and only could you then exploit that so that the cvs score is a bit lower on that one but they brought out another patch for that from that one there was a big discussion

then how bad of an attack is that yes it's kind of a remote code execution but you also need to be able to actually provide that custom pattern to exploit it which then led to this discussion um if the attacker can already change your configuration files on the server is it really still a security issue or not um and then some people are insisting well it's still a remote code execution um even though you need write access on the server and so to actually exploit it you would probably have just to wait until you find somebody who can edit on your servers um so that one i think is more like a not such a big deal because it couldn't

be really exploited in practice that easily okay the final point here is how many features do your loggers need um and that is even in the java ecosystem um which tends to throw all the features on things a bit of a debate because they're the two main loggers that are used are log4j and logback and logback doesn't have a lot of these features and to make things more interesting that the two groups of people who do these loggers don't get along with each other and they constantly fight on twitter and say the log back people will always say you shouldn't have all these features and the log4j people are like well we have all these features and we're the better

logger so this is kind of like leading to these discussions and if you should really have them i'm not sure but for attackers it's a great field um because it has so many features you could potentially exploit um and by the way the the people who found the initial security issue they created that initial logo and then every time they found a new security issue they made the logo worse and this is the final result i mean if we find another one it would get even worse so this is the the final look for shell logo by now and that's where we ended up any questions for that part so far did anybody type any

questions in here no any other questions otherwise i'll move on so how can you exploit it um there is plenty of code on on github to actually see that so here i'm taking this one somebody wrote a vulnerable application just to show you how easy it is or how little you need in terms of exploitation what you need here so this is using spring boot which is the most widely used framework in the java ecosystem spring boot itself by default uses lock back and not lock for j so it's not vulnerable but if you they have the option to switch out the locker so if you have done that then you can actually exploit it

so for the purpose of this application um we have added that and all you need in your application is um we're having this class and on the under the the root url basically what you need to do is you need to log something that is vulnerable and here i'm just taking the the header if you provide the header x api version we're logging that and that would be the necessary vector to actually get that bad string into your application and then load remote code so any application that takes any user input and doesn't sanitize it and locks it anywhere is potentially vulnerable and attacker will just need to find some header or some other place where

they can inject their strings somewhere that the application will pick it up so that's all that is needed with a vulnerable log4j version um and then to exploit that that the repository has a has a an ldap server that you can run so from there you could fetch the payload and then you would just need to run against your vulnerability application the the curl request where you say here we have we provide that header with x api version and then from the the gingy server up there we just load the code and that then it will be executed and that would then write something to the local file system to show that it has been exploited

so that's really all there is you need to run your gender server you find need to find the vulnerable application provide the string that is somewhere being locked and then it will fetch your payload and you're done so that's that part um something that led to some discussions initially was like why was there no virus or should there be a virus or worm around that and there wasn't in part because every application is different and every application is locking some different things so there is not not one general attack vector but it is somewhat application specific what you're logging that's why it's kind of hard to exploit that automatically what you could see in the days after the exploits

for this came out that people were throwing um their payload strings into all kinds of headers in your applications and you would see a firework of logs if you were logging that where people were trying to exploit anything they could think of like any field you could enter on the website any header they could think of people would um try to throw in their dollar curly brace strings to somehow attack your application um but because it is application specific it was not really easy to to write a worm or automate that any questions on the attack cool then let's continue um how can it affect products um this is a bit where we at elastic come in because

elasticsearch um has been using a vulnerable log4j version basically forever um so 5.0 came out like five and a half years ago or so so for the last five and a half years if you have used elasticsearch or any of our products basically um we had a vulnerable log4j version in that but the vulnerability is actually not that simple to exploit and i have made some tables to actually show show you so what you need is you need to figure out what is the elastic search version that you're using and i'm this is just a slide for for seven um then based on that we have a specific log4j version shipped with that so for

anything older it was a vulnerable version then it depends like which jdk you're using which attack vectors are open um so what we have done initially the the very old versions on an old jdk could leak some information but you couldn't do a remote code execution because of the java security manager and i'll show you the java security manager in a moment with newer jdks you couldn't actually exploit it and then we removed the gindy lookup class and said that don't look up any remote values in any case so this closed it this version was then still flagged by all the the security scanners um so we had to update log4j pretty quickly after that to get around those security

scanners we are still removing the the gindy lookup class because nobody needs that feature it is still just a precaution of what we are doing here why is this an information leak so what you could do is you could do dns lookups from your application and through the subdomain for example you could exfiltrate some secret information from the server but you couldn't execute code and i'll show you in a moment why i've done the same table for six point x where it was very similar where it also was not a remote code execution but only on old jdks and old versions and information leak and then on five point x which is not supported anymore luckily um there it

was a full remote code execution because we didn't have strict enough java security manager rules in place so there you could have both the remote code execution and information leak issue and on more recent versions you could set that format message no lookups to true and which basically doesn't do remote lookups on older versions the lock4j version is so old that it doesn't know that flag anymore there you would really need to go into the the jar file and remove that that class file from the the system um and if you're on an even older version which is definitely not recommended but then log4j was so old that it wasn't vulnerable yet so that kind of saved you

in that area the thing is with all the things that you're running in your network who knows what is the jdk version of everything what is the logger that your job applications are using and is it vulnerable or not and especially if it's somewhere in a docker container where nobody really dares to look into it it's very hard to figure out what do you even have running on your network somewhere um yeah and for example for us you could on the api we have you could do an http request um to just check the jdk version and we're always shipping for example the latest jdk version but it really is hard especially in docker to know for all your

dependencies what is running in them and where um yeah so the java security manager it's a piece of software that's interesting but that also nobody in the java ecosystem is using basically because it's a bit hard to use but it's really saved our bacon here so what the java security manager is doing and you could basically say in this area of code you can use specific features or you cannot use them and the common question from java developers is always like is it a big pain to use that and yes it definitely is and it took us a long time and effort to get it right and structure our code correctly but then it was totally worth it because

what we basically did is anywhere in our application you can do [Music] a resolve which is a dns lookup in the end and you can set some other options but in our code by default you cannot fetch any remote code so in the package that is logging you couldn't load remote code um it couldn't open a network connection because your locker doesn't need to do that we only have so that the logging or that the networking library that we are using is called neti and only in a very few places and we give our application socket permissions to actually fetch remote code and otherwise it couldn't do a network call because the java security manager

wouldn't allow you so even if you managed to get the attack vector in you couldn't actually get any remote code that's why it wasn't in any recent version of elasticsearch a remote code execution because those network calls were not allowed and that's what kind of saved us here and it's only in very few places where we do allow that and because of this one here it was the information leak because you could still do anywhere in our code you could do a dns lookup and through subdomains you could exfiltrate some information but you couldn't fetch any remote code that's why you couldn't exploit it um but almost no java application is using that because it's such a pain and also

that java security manager is going away and we are currently working on actually replacing that with more modularization which was almost ironic is we had we wanted to upgrade lock4j for a long time but we couldn't because something changed it wouldn't play with our java security manager rules anymore so it's kind of like the java security manager um saved us but it also made upgrades for some older versions harder but we pushed past that and we did manage to upgrade in the end but it is creating a complex ecosystem though the java security manager in general is complex but it's quite an amazing piece of software if you have a widely used application because you can

limit where in your code can you do write operations to the disk where can you do network calls you can really limit the attack surface in your application if you manage to get it up and running because that's a bit of a pain and yeah we're currently trying to replace it which is its own problem because this java security manager because nobody's using it um it is being removed from the jdk so you need to find replacements you can also set this this header or that configuration option to actually not allow lookups and we're doing that in any recent version as well just to be extra sure to mitigate that what we're doing now in all our builds

so we're using gradle as a build tool in java for example we're taking the the log for jar and we're just kicking out that jindi lookup class in the build so it's not there so that attack vector is pretty much closed uh for any ongoing version that we have um if you need to remove that from an existing installation you could do that any jar file is basically a zip file so with zip you could just kick out that the class file uh from the library where we are including it and then did this command would for example check do you have that jindy look up anywhere in your jars um and you could potentially remove that and that is the

most effective way to to actually close the attack vector in the first place um there was also an interesting thing from amazon is bringing out its own jdk distribution which is called coretto and they have a hot patch so you basically with java reflection you close the hole you always need to apply that at run time but it looks like amazon has so many java processes running that they wanted to have a hot patch option so that they didn't have to patch and replace all the running jars so they could just at run time fix that this is not officially tested or recommended for elasticsearch and it was harder to get up and running with the java security manager because

they blocked again some calls there but i i work with them and they they get got that to work with elasticsearch i tested it it works it's still not recommended um what is kind of funny is that this patch um had its own security issue so in april they found that that patch created its own security issue but they fixed that pretty promptly but it looks like everything around log4j is full of security issues um to some degree uh yeah also don't try to be too smart i've seen various things that people suggested why how you could avoid that um so i've seen some who say like just don't log anything right and you're good um

and you can actually do that in elasticsearch so for example through the api you could say on a cluster-wide setting and just set the root logger to off and it will not log anything anymore but that's not what we recommend if you ever restart the server and it doesn't come up properly you won't see anything in the logs so you probably want to have those logs in the first place that's not really going to be a good fix the other thing that we have seen is a bit of a yolo approach that people just replace those jar files with the log4j version which at least the case of elasticsearch is really just not working so we

depending if you just update the jar the process might not start at all it might throw an exception from the security manager depending on the version or it might look like it's working at least for some time it could still break at any point in time at runtime so we really don't recommend doing that just replacing the jars it it doesn't work it's not recommended to a proper upgrade um otherwise it will just be a pain and what we have seen was that our support we had a couple of dedicated support engineers um just answering lock4j um support questions for weeks and we had a our own support rotation for that which was not a lot of fun and

it just sucked up a lot of time unfortunately any questions on that let me check did anybody add any questions so far no good let's move on then so how can you protect against it some things that make this a bit trickier is that you don't really need to have the vulnerable log4j version in your application if you for example use elasticsearch as a search engine in the background or as a logging system it would need to or it could be exploitable just by having the dependency in there even though it's not directly in your application as long as your application is handing over that vulnerable string to the search engine and it gets executed there it would be enough so you

also don't need to be able to access a vulnerable system directly even if your elasticsearch for example is behind a firewall and no one can access it from the outside if your application is handing over the vulnerable string to the application you would still be vulnerable and as always you can sanitize your inputs uh and an outgoing firewall would have been very helpful here again that your application cannot just do random calls to the internet because it probably doesn't need to uh but obviously nobody is doing that so yeah so how you how can you protect against that um you could use a web application firewall or anything else that blocks anything that has dollar curly brace

gindy because that's probably not something anybody is ever throwing at your application which would be kind of simple um you could patch log4j or disable logging but that's probably not such a good idea you could disable those jindi lookups by either removing the chindi class or setting that setting that it shouldn't fetch remote code or for example you could disable some features like the java security manager you could disable some features in your code base and all of those would help you to protect against that security issue what kind of like made this tricky is that what everybody does is to monitor their applications they have logs right um it's kind of unfortunate if your logger then has that security

issue and sucks up all of those uh vulnerable strings and might be potentially exploitable by that but that's another issue and somebody made this great map of am i vulnerable to lock for shell and then it's like am i running any code in the jvm um if no do i use any third-party tools like a data store or whatever that uses java if no then you're actually safe otherwise you can then go these various paths like i'm using code in the jvm do i have log4j anywhere which major version is it is it one or two one has its own security issues which are not so severe uh but if you have two then you

need to check like what is the specific uh minor version and then you need to check and on java 8 it's different version than on java 7 etc and then you can work your entire way through that to actually figure out am i vulnerable or not which is really as complex as it looks um and the effect that this had was that a lot of companies were kind of like a freaking out and b also trying to outsource all the work to their vendors um and what happened then um daniel stenberg he's the the guy who does curl what programming language is curl written in c and he was getting requests by companies that said like you

have 24 hours to tell us if you are vulnerable to log for shell which is a a bit ridiculous because it's written in c and b also like oh you're just a free user i'm downloading the software and then you'll expect somebody to answer in 24 hours but he was complaining on twitter quite a bit that he even he was overwhelmed and he wasn't even using java so anybody using java was having a lot more trouble just keeping up with all the support requests people were sending out so it was a lot of fun for everybody and then there was also this drama around lock for j1 because it still had its own security issues and i said like there are these

two camps in the java logging world which is almost surprising it's like it's really only a handful of people doing all the logging in the java ecosystem but they don't get along and then one group didn't want to patch that and then the other group forked it into another package and they just this week there was another this big discussion if this should even have happened and whatever um so it's always also a people factor to some degree um how the ecosystem evolves what is quite interesting that if you use tracing in your application to see like what is slow or what is failing you can actually watch these exploits happen in real time as well where you can see

oh my application is running and suddenly my application is doing a get request to this random ip address here which is probably shouldn't do um that is a probably a good indicator that something is off here and you might want to step in um another thing that's also from our products like we have full security suit but now um where you can see like what application is calling what are the other applications so for example if you see like you have your job application running and that's calling a shell that's calling another shell and that's calling w get um that's potentially also not a great pattern to have also if you have these weird child

processes we have actually alerts on those to tell you like java is starting a child process that it really shouldn't do or that i have never seen before and then you can fix that i don't really want to make that a product demo i if anybody wants to see that in action i can show you afterwards um also i think my colleagues are actually in town very soon um to show you stuff like that um that's the only picture i'm adding here um any questions on that part no cool then um let's wrap it up um log for shell is definitely a major mess all over the place and it's probably still somewhere in your infrastructure and you don't know

it um it might be um something like this where log4j on your toaster and then you're like no that's not a thing but really are you sure because i think the oracle is always promoting that java is running on one billion devices or four or whatever and by now this might be more of a threat than actually advertising and so you might find that in random places where you didn't expect it to what to keep in mind is that the first security issue that really was a big deal and and that should be fixed the other three are annoying and should also be fixed but are on a much lower level so i also feel like people were a

bit like capitalizing on that big first wave of security issues and then just to get their name out like oh there's another one and they were a bit overblowing those even though they were by far not as severe as the first one but obviously if you clean up the first one clean up all of them in one go um and for most java developers um december was a rather stressful month and uh nobody aged very well then it's easy to exploit in theory as i've shown you in practice it's sometimes a bit more complex as you've seen with java version and java security manager and there are lots of pieces that play like which make it a bit harder to

actually say like you're actually vulnerable and you can exploit this it's still a bad thing in your code that you want to find um is it still a problem uh yeah just very recently in end of june um i think that was some us governmental organization that was uh giving out these warnings that ex people attacking systems are very actively looking for log4j that it's probably somewhere in the infrastructure and it's probably something if somebody managed to get a hold in your network anywhere they will be looking around for any vulnerable leftovers in lock for shell because it's just something that is potentially still waiting to be found in your network so it is still a good time

to try to clean those up and with that any final question i think we have like three minutes or so left anybody to ask added any questions here no um yes please i'll i'll try to repeat or yeah shout out and i'll repeat

so the question was how could you counter the hysteria um yeah i think if you know that you could make a lot of money or you could also make your life a lot easier i i don't want to say we we didn't manage to do that but it was a lot of work i mean the hysteria is just sweeping over you and i think we had 100 people working over the weekend on lock for shell on the first weekend in our case and then we had like i said the support rotation where we had a couple of engineers do nothing else for weeks then reply to lock for shell things so we yeah i don't think there is an easy fix

to that it's like i think it's it's like that it's unfortunate that a lot of security issues are not getting the necessary attention and then there are a few that get all the attention and it's kind of like the other way and everybody freaks out about them and it's like there is no reasonable middle ground to to work on that um and i no i have no idea how to fix that but i think that's a hard people problem yeah any other questions before lunch otherwise a thanks a lot b i have a lot of stickers here if you want to have any stickers take stickers if you have any other questions i'm here for the rest of the day find me

over lunch or at any time thanks a lot [Applause] you