Mitigating Java Deserialization Attacks

Thank you very much. Good morning everyone. Thank you for coming. Thank you for the invitation. So my talk is a bit more technical. Closer. Okay. Thank you. Like that. Okay. Okay. So I will discuss about the technical topic and I'd like to ask you, are you familiar with Java? How many of you are familiar with Java, the JVM? Okay, so bear with me. I'll try to make it simple. So it's a complex topic. It's a complex topic and I'm not going into details. I will explain the problem at a high level and I will also explain the solutions, the existing solutions to the problem. So you know about me. David introduced me. Key takeaways before we start. I

will explain the existing solutions and why they're not adequate to solve the problem. And then I will introduce a new approach, how I see the problem. Why you don't care? Because it's technology, the serialization and serialization, it's technology that's being used everywhere. Like, if you use any of these protocols, you use serialization. If you use products from any of these vendors, you use serialization. Or even your own application, either directly or indirectly, you use serialization. And it's very popular. Tomorrow, my understanding is that the OWASP Top 10 2017 Release Candidate 2 is released. Finally, after so much time. It contains deserialization in number 8. It proves how popular it is. And it is very... reliable. So an attacker, if

you have this vulnerability, they can exploit you really easily and reliably. It always works. We had some incidents like the SF Munich San Francisco municipality agency. Last year, I think, someone exploited this vulnerability and installed the ransomware and they managed to take their business off for a couple of days and they had several thousands of dollars By the way, Oracle, you know Oracle, right? Yeah, the software giant. Releases every quarter a critical patch update, we call it CPU. How many of you know that Oracle released the CPU two days ago? Great. Did you install the CPU? Okay. Yeah, but you do use Java, I guess. The thing with the latest CPU is it contains four new vulnerabilities in serialization. And the previous CPU contained

four and the January CPU contained one. Very critical vulnerability. So it exists not only in the application layer, it also exists at the Java platform itself. It also exists in application servers like WebLogic, JBoss, Tomcat, etc. And any other framework. So, what is serialization? Really quick, at the high level, it's a mechanism to transform an object that lives in memory into a byte stream. And why do you want to do that? Because you might want to transfer it to another computer. It's a way to exchange information between a client and a server. So the process of creating this byte stream from memory is called serialization. The opposite, creating the object again in memory from the byte stream is called deserialization. And the problem here is the

deserialization part. We're not going to deal with the serialization part. During deserialization, we get the byte stream and the byte stream contains only data, it doesn't contain code. And the problem with that approach is using data, the attackers have the ability to create specific data structures that when it gets deserialized, it will manipulate the code in such a way to perform malicious or unintended behavior just by manipulating code. And this is fascinating, this is something new. So, I understand that not many of you are familiar with the Java code, but this is a piece of code, these three lines of code create a problem. And this is something that we call a deserialization endpoint. This performs deserialization in Java. And this is vulnerable.

Any of you know where is the vulnerability?

Okay, so it's simple and I'll explain. The first line of code receives an input stream from a request. In this case, let's say it's an HTTP request from the user, from the browser. The browser sends a request to the server. The server receives that request, gets the input stream, and then passes it to an object which is called objectInputStream and invokes that special method, the read object, which is the method that performs the deserialization. The problem with these three lines of code is that it performs deserialization using an input stream coming from HTTP and HTTP is untrusted. Anyone can send a request to the server and the attacker can create these special crafted objects and

send them to the server and the server will happily deserialize it. And the next problem is that any deserialization endpoint, that's how Java works, that's how the deserialization works. Whatever object you sent it, as long as it knows that type, that class, it will deserialize it. No matter what object you sent it, it will deserialize it. Any available class can be deserialized. And that's a problem. It makes no sense. And this can cause any of these malicious behavior. Arbitrary code execution, denial of service... To be honest, the four vulnerabilities in the latest CPU that was released two days ago contained four serializations in the Java platform. All four of them denial of service. How to solve the problem? Well, it's so fundamental in the protocol that you

have to stop using serialization. Just refactor your code, re-architecture your application. That's the solution. But is it feasible? Well, no, because it takes time, it requires effort, it may not be even feasible because if you use a legacy system you cannot touch the code. And what about the other software layers that you do not control? For example, Tomcat. What if the vulnerability exists in Tomcat? You cannot fix Tomcat. You have to depend on when the Apache community, the Apache Foundation releases a new version or JBoss or Log4j. So let's patch our software. When there is a vulnerability, let's patch our software, like the Java platform itself. Well, it's not that easy. Sometimes when you patch, you know the problems, right? What's the problem with

patching? It says on the slide. It can break your application. The latest CPU that was released two days ago is not backwards compatible. And that's scary. If you blindly try to apply the CPU, the Critical Patch Update, to fix your vulnerabilities, suddenly your application might break. Have you ever heard of Apache Struts? Yeah, a very popular framework. Again, a few months ago, a new deserialization vulnerability was found in Apache Struts, in the Apache Struts REST plugin. Okay, let's patch it. But if you patch it, here's the advice from the strats community. It is possible that some rest actions might break. So you need to be careful with patching. So again, patching is not the proper solution and you depend on the vendor when the

vendor will release a new patch. So here are some, not some, My understanding is that these are the only ways to solve the problem from the platform itself, from inside the JVM. So what are the solutions? Using the Java Security Manager, using filtering to filter blacklist and blacklist filter known trusted and untrusted classes, and something new that I will introduce, the runtime virtualization and the privilege de-escalation. So, the Java Security Manager, I don't want to go into details. My understanding and everyone that I have talked to, no one is using the Java Security Manager in enterprise. No one, even though it's there. It can benefit you greatly. No one is using it. It has many disadvantages.

I will come back to that point. Just really briefly, the latest CPU that was released two days ago contains 18 vulnerabilities that can bypass the Java Security Manager. The previous one in July contained 26 vulnerabilities that can bypass the sandbox. So fundamentally it's broken and I will prove to you why the Java Security Manager is broken later. And also even if it wasn't able to bypass the Security Manager, it doesn't protect you against deserialization denial of service attacks and it doesn't protect you against another type of attack which is called deferred deserialization attacks. are not important right now. And of course it's extremely difficult to configure it. That's why no one's using it. So the next option, the next alternative is filtering classes, class names. So we

need to define which types, which available classes in the system are allowed to be deserialized because as we said, deserialization, when you feed it a class, it will deserialize it blindly. Filtering, is it good or bad? What do you feel about filtering? How do you feel if someone asks you to do filtering? Why? The names. Would you use a blacklist approach or a whitelist approach? Why whitelist? Great, great. Yeah, so whitelisting, definitely. It's the best practice, right? You have to do whitelist. Blacklist is never complete. It's never secure. You have to depend on known exploits and then blacklist the dangerous classes. "Oh, that's dangerous, let's blacklist it." So, you don't have zero-day defense, right? If

a security researcher finds a new exploit, you're screwed with a blacklist. Also, there have been some researchers in the community, they found ways to bypass blacklisting. Whitelisting is extremely difficult to do. it requires profiling. You don't know the classes that you need. So, what you need is to profile your application, identify all the classes that you need and then whitelist them. It's not agile. Every time you need to push a new release into production, you have to profile your application again. Identify again the class names and do the filtering, the whitelisting again. And if you mess with the whitelist, you have false positives. You broke your application. And what happens if, let's say, You found a

class that is vulnerable. So you don't want to add it in your whitelist. But your application depends on this class for actual functionality. So what do you do then? If you don't add it to the whitelist, you will break your application. If you add it to the whitelist, you add a backdoor. It doesn't make sense. So we need another approach. And also, human intervention requires human intervention. Humans need to create these lists. Whenever a human is involved, they're prone. And also, as a concept, people tend to mistreat the whitelists. So here's an example of what I consider to be a mistreating whitelist. This is ActiveMQ, the Apache ActiveMQ message list. very very popular framework in the enterprise

world. So, they have their own whitelisting approach. They don't depend on the JVM. And they have this system property which is called serializable packages. And they say, look, for us, for the Apache ActiveMQ to work, we need these packages. The Java lang, of course, the Java X security, Java util, and some Apache packages, the X stream, etc. So, these are whitelisted by default. What's the problem with that? Do you see any problem with automatically whitelisting these packages? How do you mean? Dependencies? Well, yeah, pretty much. So, when you whitelist a package, you whitelist every class in the package. Even the classes that you don't need. And if one of these classes is vulnerable, then you're screwed. Let's say, for example,

the hashMap. Do you know what a hashMap is? Hashtable. and not a list. These are collections and these live in a package which is called java.util. Now the hasmob is vulnerable to denial of service attacks through this realization. The moment you whitelist the whole java.util package, you're vulnerable to denial of service. Guess what class was vulnerable in the CPU that was released two days ago? The hasmob. the collections. 5 or 6 collections in the Java Util. So, ActiveMQ automatically whitelists the Java Util package. You are vulnerable to denial of service until you apply the CPU. So, yeah, this is a backdoor and this is by design. They're forcing you to whitelist these packages. And of course, for

convenience, they also do this. White list everything. Of course, it makes sense. And it's a city job. No one wants to maintain lists. When you ask the security engineer and security team to, "Oh, you know what? You have to pay attention to the community. Whenever you find a new exploit, you need to add it to your blacklist." It's a burden. They have to do it again and again on every release and profile their applications. It doesn't scale. It's a city job. No one wants to do it. The JVM engineers created a mechanism inside the JVM that's generating. Now it has been, they did it for Java 9, but it has been backported to Java 8, 7 and 6 as

well, the latest releases, that allows you to do blacklisting and whitelisting inside the JVM. But again, it's whitelisting and blacklisting. It has inherent disadvantages. Regardless of if you do the whitelisting or blacklisting at the WAF, or inside the JVM, or an IDS endpoint protection. It doesn't matter where you do the whitelisting and blacklisting, it has these disadvantages. So anyway, now this is inside the JVM, you can use it. But one of the problems is that it requires, one of the things that it does, first of all, it has three types of filters. A global filter, I will explain later, it applies the whitelisting for all deserialization endpoints, from all software layers. Custom filters for if you can create new code. This is for

developers and built-in filters fine. And then it has some limits. Graph and stream limits. The problem with that is that you need to understand what a graph is. How many of you understand the details of a graph? The details of a byte stream? This is meant for JVM engineers, not security teams. This kind of configuration is not for security people. It's for JVM engineers. But they expose it as a configuration for people to be protected. And you know what the problem is when you expose such a configuration to the user? This is the problem. So this person in Stack Overflow says, "Oh, I'm trying to address the security vulnerability in the Java RMI, and I'm trying to use the serialization filtering mechanism in the

JVM. So I created this configuration." So he set these graph and stream limits to address the RMI vulnerability. What's the problem? First of all, no one understands and this is clear because he misused the stream limits, the graph limits. He maxed out the values. These are the maximum values that he could use. And he maxed out the values, which means in his attempt to address the vulnerability, he's exposed again because this makes your application work. It doesn't break his application. Smaller values could break his application. So he maxed out the values. And let's say you understand how to use that. It breaks your applications. Several Stack Overflow questions, I applied serialization filtering and now my application broke.

Yeah, of course, because no one understands how to do it correctly. Or if you upgrade to a version of the JVM that has serialization filtering and your application is not aware of that, it will break, of course. Most probably. Yeah, so the built-in filters are enabled by default. The global filter, you have to configure it yourself. But this is how they configure it. The users, this is how they configure the global filter. *. Enable everything, or at least everything, which is not a protection. Okay, so it breaks the applications in many cases, but it doesn't only break the applications. It also breaks the JVM. They shot their food. So they introduced the serialization filtering inside the JVM, and suddenly

the JVM broke. Why? So the building filters have these values, these graph limits, and they arbitrarily selected a graph depth of 5 as a default value. That 5 broke the RMI, I think, the registry. It broke the registry filter, the RMI registry. Fine. What's the solution? How do they solve the problem? They increased 5 to 20 from one arbitrary number to another arbitrary number. And this is not even documented. This is in the JIRA, but there is no documentation for this change from 5 to 20. You just need to pray that it works. Okay, so instrumentation agents. Serialization filtering is good, but it's not perfect. It has several problems. Let's see another approach: instrumentation agents. The instrumentation agents are a very popular approach to

solve the problem. Again, they use blacklisting and whitelisting. Again, they have a global filter and custom filters. Known open source agents are notes of serial, contrast through, and a few more available in GitHub. People use these agents. What's the problem with instrumentation agents? Any of you know any instrumentation agents like the one that I mentioned? Okay. Are you familiar with instrumentation? What is instrumentation? App dynamics? Hmm? Okay, so instrumentation is a technique inside the JVM, in Java, that allows you to manipulate bytecode or allows you to change the behavior of classes at runtime. And the problem with instrumentation is this: it wasn't designed for security. From the specification, from the Java doc, it says: "The purpose that we created the instrumentation API is

to gather data and to create benign tools that do not change the application state or behavior. That's by design. So, if a vendor creates a security tool based on the Instrumentation API, they are abusing the Instrumentation API. Why is this bad? Because the Instrumentation API lives inside the same address space as the application. This creates a single fault domain. If the application is compromised, the agent is compromised. This is significant, this is big. Your agent, in other words, the module, the component that provides security for you is compromised the moment your application is compromised. But you need your security to be unaffected by any application vulnerability. But that's not the case in the agent space. Think of the browser plugin paradigm. You have a

browser like Chrome and you install plugins. If one of the plugins like Adobe Reader, Acrobat Reader is compromised, you don't want the whole browser to be compromised or you don't want one plugin to affect another plugin. This is exactly what happens in Java. The security solution, the security control lives in the same address space as the rest of the application. And if the application is compromised, everything is compromised. And also there is no protection against insider attacks. Let's say again for the browser paradigm. If someone creates deliberately in your application a rogue plugin, they can control your browser inside your organization. The same applies in Java. If one of your employees, your colleagues creates a Java plugin class that

deliberately wants to mess up with the agent, they can do that. There's nothing to stop them. And the problem is the instrumentation agents can turn into double agents. So if you can manipulate the agent, you can make it do whatever you want. You can disable the protection, you can tamper with the log entries, install backdoors, do whatever you want. And as a matter of truth, I'll do a demo. but I'm aware of time and I'm happy to do the demo at the discussion part later. So I will continue with the presentation. So I'd like you to do, I'd like to examine the attack in a bit more detail. Not the vulnerability, which is a bit complex. I'll show you the effect of the attack so you can understand it.

So this is a stacked race. Don't be afraid of the stacked race. This is simple. the methods that are invoked at the point of the attack. This is the effect. What we see here, I'll explain it to you, it's very simple. At the beginning, we have the deserialization process that has initiated. And during deserialization, it tries to create a map. And it gets the elements of the map. During the operation of getting the elements from the map, it gets... It invokes GET. So far so good. Inside GET, somehow it tries to fork a new process. Why? What's the logic behind it? Why the JVM should allow this kind of behavior? So when you create the map or

when you want to get the elements from a map, that's it. That's a specific API behavior. Like, I want to do something very specific. You're instructing the JVM to do something specific. Or the thing about the deserialization, again, what's the concept of deserialization? Get a byte stream and create an object in memory. Why do we need to fork a new process? That violates the API. And this is not just one example, deliberately, to prove my point. Here's another example. During this realization, it tries to add an element to a link asset. And during that operation, it creates a new class. When you create a new class, this is called injection. This is the definition of privilege escalation. And of course, when you create your own class,

what you want to do with it is fork new processes, remote command execution. So again we have a violation of the API. When you add an element to an asset, you don't want to create a new class. But the JVM allows you to do that. It's irrational. It's fundamentally irrational. The JVM in terms of security is not your friend. Whatever you tell it to do, it will do it. Happily. So we need a way to constrain it. There is a way to constrain it. The security manager. But it has so many problems. We cannot depend on the security manager. We need a different approach. We need a security solution for Java that doesn't live in the same address space as the application. And

we need a way, a security solution, that mitigates API abuse. So when someone tries to do something benign, like creating a hash map or calling toString or something benign, it shouldn't be allowed to do privileged operations. So what do the standards suggest how to solve the problem? So, the standards say during deserialization you need to minimize your privileges. And deserialization methods should not perform potentially dangerous operations. Also, you need to compartmentalize your system into safe areas and create boundaries. And then drop these privileges within these boundaries. No one has done it, even though it's in the standards. So, me and my team at the company I work for, WorkAttack, we created a solution based on these standards,

based on these recommendations. And the reason it works is because of runtime virtualization. So, think about... So, what we have done is we virtualized the Java, the JVM. The JVM, we separated the JRE, the runtime execution, environment from the virtual machine itself. Now they're separate. And we added a hypervisor in the middle. And we added the security controls in the hypervisor. So now, if the application space, the address space of the application is compromised, it will not affect the security controls anymore. The security controls are inaccessible, completely safe. They cannot be compromised. So that's significant. The second thing that we did is we created boundaries based on the standards. So this realization is a boundary. And after the

boundary inside that operation, we dropped the privileged operations. So during this realization, you're not allowed to do something that violates the API. So that's privileged de-escalation. And also it safeguards the JVM itself. So to conclude, maintaining lists is not an enterprise solution. It doesn't scale. No one wants to do it. Instrumentation agents can become double secret agents. They can be manipulated. They can be tampered with. They're not reliable. They're not suitable for cloud. And my approach, my suggestion to you, my suggestion to the community is we need a solution that is secured by default. We need a platform that is secured by default, that requires no configuration, that understands basic security concepts like API abuse and privilege escalation and it is not affected

by application vulnerabilities because we need to secure our controls, not expose them as in the case of the security manager. So that's the end of my discussion, my talk. Now I'm happy to answer questions and if you want we can have a discussion about any of the topics I mentioned. Any questions? Okay, I have one. So I actually probably gonna have a project coming up with web application firewalls. And when we were just discussing what kind of tests should we do, I threw in like, yeah, and by the way, we don't have on the list the serialization things. And I was thinking like, that's gonna be interesting to see how web are performing against a Java serialization attack. What is your experience? How well webs are protected?

Like, I mean, how well you can configure them if you spend time on it and out of the box, because let's admit, web application firewalls, usually people just drop it off with the default set and they're happy if they block something. You don't really invest in time, even though you should. So what is with the defaults? And if you spend time on configuring a web application firewall, how well they perform against such an attack? Very good question. I'm happy you asked that question, actually. Okay, so... Okay, great. So what's the problem with web application firewalls? Anyone uses a web application firewall? Open source or commercial? Okay. Are you satisfied with your web application firewalls? What kind? Mod security, what do you

use? Okay, cool. Okay, so have you ever seen a rule in mod security? Have you tried to create a rule in mod security or configure a rule? They're this big. They do exist. They're not human readable, in my opinion. There are some... geeks that they have devoted their lives to write these rules, they're not meant for the average security engineer that wants to protect the enterprise. What they do is they use the OWASP core security mod security ruleset, take it as is and they use it. You cannot configure it based on your own needs, first of all. Second, a web application firewall by definition lives outside of the application. doesn't know what the application is doing. All

it does is it filters input and output. So when you filter input and output, what are your options? What can you do? How can you protect an application against any attack just by seeing what is incoming and what is outgoing? You have to apply some kind of heuristics and you have to apply whitelist and blacklist and say, "Oh, I saw this in the stream, so this is bad. I will block it." But this creates false positives. This will break your application. You need to tune it. You need to configure it again and again. And some of the web application firewalls like ModSecurity, you cannot configure them easily. You need specialized personnel. Log the attempt. If you

log the deserialization attempt... Yeah. Yeah. Yeah.

Quick question. I don't have experience with WAFs, using them in production. I have experimented as a researcher, but yeah, they're not effective. They depend primarily on known exploits to blacklist them and they do have false positives. Let me give you an example how to test your WAF against a false positive. I have an example here. So create this code, open your Eclipse, open your IDE and create this piece of code. Now this creates a serialized stream and what it does is it serializes a hashMob and it adds a single element into the hashMob. which is a hasmap of two strings. So the key is this and the value is the calc. A WAF, when it sees

a serialized stream that contains this value, it will believe this is the invoker transformer by the way is one of the most known exploits in deserialization. When a WAF sees this string, it will believe it's an exploit and it will block it. But it doesn't do any harm. It's benign. It doesn't do anything. it will successfully deserialize a hash map. Fine. But it didn't do anything. So they have false positives. They don't understand the context. They see that string and they do pattern matching. And they say, oh, I know that. Let's blacklist it. Let's stop it. Or let's log it. And regarding logging, is logging security? Yeah, but you're compromised. You know what the problem is with deserialization? There are critical vulnerabilities. Remote code

execution. If a successful attack is managed to compromise your system, it's game over for your enterprise. It's game over. You don't get it. Excuse me. You don't get it. So a single exploit took their business out for two days. You're happy with false positives. And you're happy with letting... Okay, I don't want to talk everything. So you use your WAF in a reporting mode. So you let the attacks come in. I don't consider it useful. If you want to log your attempts or your attacks, fine, do that. But bear in mind that your business might go out of business for a few days or even completely. It's a game-over scenario. When you allow a critical vulnerability that can

perform remote code execution on your server, on your infrastructure, it's game-over. So all that stuff. I wouldn't want to be in that situation. So what I'm proposing here is an alternative. There is a way to block any privilege escalation during this realization. And I see that any alternative, it's pointless. They have false positives. They cannot be used in blocking mode. Not only they perform poorly, they perform even more poorly compared to serialization filtering. Because serialization filtering, for example, does the same thing inside the JVM and it knows the context. It knows that in the example that I showed you before, the invoker transformer is not a class, just a string. So they will not block that. The WAF will block it. It's even less effective.

For me it's satisfying to hear this, especially that I'm going to test the VEF. Thanks. I was wondering if you have any statistics on whether deserialization, it's a hard word, decks are often authenticated or more unauthenticated. Can you say anything about it? Yeah, very good question. So the question was, I think everyone heard the question, was if the deserialization attacks are completely exploded without authentication. I don't have statistics. My understanding is that yes. Log4j or JBoss or Weblogic, sorry, Weblogic I think it was, or JBoss version 7, very popular exploit, they expose the deserialization endpoint without authentication you could take over the server. As simple as that. It's very common not to add authentication to the deserialization endpoint. One of the most common

scenarios. I've seen people saying that the majority of deserialization endpoints usually are not in a public-facing server. That's a good thing. It's not always that case, but it can be. One of the things that you can do as an immediate defense is identify all your deserialization endpoints using some kind of tool. And then for all your deserialization endpoints that are deserialized and untrusted stream, put them behind an authentication mechanism. Don't let anyone send arbitrary data.