Xavier Mertens - All Your Logs Are Belong To You!

the the first talks today where it talks about was the thoughts were more offensive security and now I will switch to the defensive part so we will see how via looks we can extract useful information from your systems and uh so I will start with a more technical representation and then I will switch to more research that I perform how to increase value of their logs so once you're once one thing is your audio secretive events have belonged to you because my name is some of you are already following me I'm working as a security consultant for a bedroom company uh I had several certifications I'm also an active security blogger so I try to

to give a lot of info I speak about what's interesting for me and I try to share as much as possible and I'm also a volunteer from some security projects mainly based in Belgium Europe so the well-known neurotrastic treaty podcast uh I maintain the website so the every time you visit you download the mp3s it's on my systems and I'm also active an active volunteer in Brooklyn so the Belgium security conference where I'm responsible of all the technical stuff so the networks the infrastructure during the conference and all the websites and all the information um some disclaimer so it's also a classical slide so I try always to uh to split my business and my personal stuff

so this presentation has nothing related to my job in my job I'm also involved in a lot of log management scene projects but let's split everything I'm here for personal purpose organization is this so you have a lot of people which are responsible of checking what's happening on the system and it's really boring so it's really a nice picture why because if we try to to check what's your look through inside your organization your companies first in most of them the bad reaction will be locks which Lots what do I have to take care of my loss it's really boring because uh nobody is happy to process your logs and often you are forced to process your Lots because

you are you comment there are some compliance requirements so you are forced to put money to put time and it's really boring because it's so huge to process them and of course most organization has they do not take care of their loss they are really not prepared to face secret incidents if you don't have flops you don't you will not be able to investigate what what happened on your platform so you are in a bad situation and of course Murphy's her best friend is already there if you never face a secret decision today it will come in the future Indonesia for sure be careful and finally um do you have enough internal resources inside the organization because it costs

a lot of resources we will see that later in terms of people in terms of money but you need to to assign some resources uh to this project first regarding the looks what's very important is you need visibility so everybody can read blocks but if you cannot extract the right information on the right moment it's uh senseless hopefully we have no decay use computers and we can also use computers to manage yours because a computer is small definition that I could be based on the internet so via computers so computers generate Lots but the same computers can also be used to process them and to extract the right information it's very very good um to increase visibility we will see

later that we can also integrate them with multiple source of information coming from other systems for third-party systems and we have we will have more chance to detect suspicious in our suspicious students and finally the goal of the road management is certainly to detect activity below the radar because as we saw in the previous presentation you will maybe have program running on the system disabling some firewall saving some some security stuff but such event will be hidden in the global flow of your platform so the goal is to really put the right Bond or the right on the right device this one's suspicious I need to take some action but we have a lot of issues to manage to

do there are three types of issue the first one on technical issue is easy to understand all your networks are more and more complex today because you integrate a lot of Appliance operating system applications so you have maybe Mainframe Windows unique system you have routers firewall whatever Vision firewalls a lot of nice Supply Appliance and all of them generate blocks and in different uh in different ways so they can use some different applications you can they can use API they can lose some protocols some Protocols are open like system SMP other Protocols are completely closed like uh let's say offset for checkpoint firewalls and they generate a lot of events pretty Millions thousands millions maybe billions of

even for very useful transition and also some logs can be also outsourced because you delay you you they get some management to a partner you you maybe have some application in the club but also those application generated blocks you need and you you have to reclaim your access to the lot because you need to add them in the in your law Management Solutions and also all this the networks components the whole Dom is the home that console and Tool management so when you need to investigate something will it's impossible to login on the console from a checkpoint firewall then you have to switch to your 24 you have switched to your system your your Linux system

and so on so it's uh it's time consuming and of course you increase the risk to do not see the right events just to give you a practical example the classic name for children find the difference between these two plots the first one is coming from Mac OS 10 So based on IP a package heater so either it's working fireworks coming from an old text device as you can see they give a lot of information you have timestamps you have Society addresses Sports protocol blah blah blah but it's not displaying the right in the in the same format so if you need to read this information it's impossible so the computer will be helpful because you

will we will decode all this information and we will be able to extract the right information the next type of issue are economic issue everybody agrees even more managers time this morning so it's impossible to do to put 10 20 people in front of screens from yours a day to just watch London because it's uh it costs a lot of money and also it's very boring we need to perform real-time operations because if you have a secret incident you cannot say okay let's stop the device for 12 hours to do to have time to investigate because the business most of the show was wrong the business program and also such downtime has a very huge financial impact if you

maintain an e-commerce platform your website is known for 12 hours for the manager will became press uh okay let's see maybe uh better though but it's still crisis so all companies try to reduce the budget stuff and stuffing and after shareholders always are always better for a company because the goal of companies to make business and finally uh you have to see lot management as an insurance so it's basically a low risk management and some manager they don't agree to invest into log management because like an insurance with maybe pay you pay your your country your whole Insurance every month but you will never use it you you you may see okay it's money lost but if in two weeks

you have a big incident in your house fire water I don't know it will be very helpful it's the same for your loss you will pay but you don't the management we then we will not see immediately the original investment until you have something to investigate and finally boring issues legal issue uh you have legit deploy log management for different uh reasons first compliance requirements compliance may be a quick name like stocks PCI DSS and so on for all kinds of nicer but it can also be in seated by the group of the business if you maintain your whole small company based in London and your company is overbought by a big International Company maybe they will come to you and

say okay from now we have to manage your notes you will have to send me reports we have to send me such information every month every year every week every day so we have to prove something so you you will have to deploy a loan application loan allows so every country has its Home Loans you have to follow them you are in Belgium you follow the benjaminers you are in UK you from UK and uh well known for csb and all the other uh security uh certification you have to apply due diligence and you care regarding uh the legal requirements a very important dislikes May easily related to the Belgian load because I spoke to the the bedroom guys the fccu

federal computer premise regarding the law because my question was to the to this guy okay I maintained my loss but but in my notes I have a lot of personal information I have five stamps of users IP address login so how can I manage from a little point of view my dose we have two big uh too big size first one the internal side you cannot play as big brother so if you maintain your loan management solution it's so inside your organization you cannot use it to track the users so it means okay I know that this guy is not doing his job every day I will check my lock every morning to see to try to

detect if it visit Facebook if you went to Twitter I know you cannot do that but your team members must be aware but the procedure exists and that they are tracking and recorded because if you find something if you if you are suspicious and you off you will be fine uh just by reviewing the logs you find oh why this guy went to playboy.com I know it's outside your CPU policy you can take an action so it's primarily it's some Brazil but it's important the two members will just be aware of this from an external point of view if you have a website and you allow users to create a login because they need to buy

some product they need to write some posts or to to generate some content you have to notify them users and restore which information is locked hope and for which purposes it's a legal appointing very important no another example in Belgium but it's really it explains really what uh what's happening for the for much companies in bedroom we have the cdfa which is which mean in France in French polish so it's a financial banking and insurance commission and in April 2009 they released document which say any institution that connects to the international numbers as security policy which shaped into at all the creation archiving eventlocks which permit the analyze product and Reporting and the big terms are in red we don't

see exactly differences time right because they really Express The Challenge and what we have to do to maintain our loss so the challenges the first one is the creation and accounting across on lot files we have analyzed and the previous challenging is the normalization a few slides ago I you saw two different events coming from a Mac OS and a piece the goal of normalization is to analyze the events and to extract all the information timestamps ID addresses and all information that can be reused later for the for the processing of products the follow-up so follow up that mean that we have information in database we have to use it to generate more detail more

information reporting and correlation is the latest step so the state of the art of load management release calculator so that's really the the business challenge and we can put those challenge in some layer approach so from the the bottom to the top we start with the work collection so it means that we have we have to protect via different methods protocols to the device via Push Pull methodology and creates the the events we have to normalize normalize them store them and then we can perform three types of operations to search it's like a Google so you are looking for something an IP address or username a Time window to to try to find some relevant information the reporting which

can be automated when demand so you produce more information valuable for the technical teams for the management and correlation uh and above correction we have the final step which can be The Incident Management so if you have all this process in place you can restart your Incident Management procedure and really investigate what happened on your platform about the raw material uh the very good news your rocks already belonging so all the logs that you have on your system there are Euros so you can process them you can use them if they are not stock internet platforms if you also use cloud infrastructure claim access to them because if you Outsource a website on yeah a cloud provider this cloud

provider generates also locks to claim access to the logs via automatic system in real time once a day once a week but you have a universe have access to them all application devices generate events all of them every time sometimes it's in a good way in a bad way but all of them generate events and finally if you have developers inside your organization they must generate events and in a good way when I say in a good way it means that usually developers write too much information or no not relevant information with strange hexadecimal products and so on so please developers generate good events because we must be able to extract once again the right information

once you you have access to all your logs you can also use third-party sources here it became very very funny you can use reliability database there are a lot of database available on the internet for free just grab a copy of them include this database inside your local management system so you can increase the value of your events detect a juvenile RBT regarding a nakachi for example on the Linux server just give more relevance to even generated by your web server because you can find immediately something relevant a lot of blacklists also availability of Internet lots of companies or organization maintains like this with IP addresses autonomous systems domain names and so on so you can grab a copy

of them and check them in your system and also the physical data like geocalization and much reader so it's about this one the next slide I would like to focus a little on secret conversions which is in fact a mix of logical controls a logical controls is a password an access list so we're switching online PRS Automotive Systems and domains and a physical control like bad readers or the organization I will give you two examples let's imagine that I'm swiping my batch at the door of building a in the morning and a few minutes later I'm my login is used to open a session on a workstation in Building B for me to security incident maybe I lost

my badge so somebody finds it on the street and use it or maybe I share my credential with another people inside the company so it's something irrelevant regarding your organization if you read IP addresses and host names of the fully clarified domain names it's quite difficult to immediately see if the request is coming from when I graduates in India China Romania UK version France we don't know but if we match with geological geologization if I'm I maintain a website based on bedroom market and I'm selling bedroom Goods why do I have people trying to to buy things from deutscheon from the deutscheon for example or from from Poland or from France it's irrelevant so it can also increase the

the value of your audience so it starts the security convergence uh uh so no a few resides so what can we start to collect so some love so the the first technical part um uh uh the first pass away about the theory is done so let's review how to connect and how to process different steps so the collection we can use push or pull methods there are a lot of protocols available some are open other populatory protocols what's very important in the collection process is also to ensure Integrity because uh example the system protocols rely on UDP so when you have a Cisco switching a system message to a system concentrator it just send a UDD packet

it will never take care if the packet reached the destination if the destination correctly process the packet so we have to take care about integrity Once A lot has been generated as soon as possible we have to connect it and send it in a safe way so using other protocols to the central system very very important that's why you also have to collect the events as close as the source so a load management solution is often based on distributed systems we have a lot of Agents collectors connectors whatever the name the different Implement you should use but we collect as close as possible from the shows and once it has been collected we are sure that it will really it will

reach the internal system we can use some ready-made to avoid uh bandwidth overload we can also Implement some some tracking system we can Implement some earthbeat systems so if the smart the the connector detects that the central stand is done it will kill all the events to kill them later so each Integrity is really really important the next step is the normalization so we have to once again to pass the events and the goal is to fill in common fields so by common fields that that's a field that we can reuse later in in search or in reports like timestamp Society address destination IP address the username the device type a board an arrow HTTP error quite interesting a 404 38302

uh two zero zero and so on and so on so the passing is really important and what's really important with the normalization later if you are looking for a user for example by giving the user will be able to get all the events coming from fibers FTP server SSH server all kinds of devices and you don't have to the the people performing The Incident Management don't have to to have knowledge about the source devices whatever it's a firewall it's a switch it's a server that just grab events to to perform the job really important about the storage you know so we collect it we normalize we need to store somewhere in the platform so we will

create a big database with also bit index because for performance we need to invest as much as possible the the most important fields we need a non-carving system because we can sorry oops oh

all right

it's back yes so we need to store all the events on Central base for immediate processing but we need also to store them in an archiving system for long term retention here long-term retention what does it mean if you fall into some compliance requirement like PCI or you have PCI something like 90 days so maybe I'm not sure I'm not an expertise but you have to take your all your events for specific amount of fat or archiving is important and once again Integrity is really important if a system admin has accessories are clarified it should be able to modify some event so it's completely even so you need to to maintain Integrity by performing uh

ashes on using ssg1 and MP5 on your archive or the system between your true integrity we have know the system so we can perform the first uh action a search one let's search uh action a lot of people are still using common line tools like the well-known red out Softail so it's it really it remains today the the the most used tools but there are a lot of other Solutions available everybody knows Google so you can deploy systems which act like boogers we just search regular expression look for specific information and you'll receive everything you can find by clicking on turn so it's really really easy uh the search feature is very interesting for investigation and

forensics operations and what's really important here is we are looking for also for small seniors so we don't know exactly what happened but we have some suspicious window that suspicious activities happen last night so we start with the time window between 2 am and forehand then we we are really looking for smoke signals the next one is reporting can be automated on demand very important we either only if the first step is currently processing so when you deploy the solution reporting is the next step first be sure that all the events operating processed and are in your system otherwise you will lose information and the re I need to reserve uh like in this thing the report the

report must address the audience so it means that you will have technical reports top 10 top 10 denied IP addresses top 10 username and we will change your password top 10 I know top 20 whatever but you can also have business reports addressed to the managers

is also very very business oriented and finally the correlation a small definition also graph from the internet so operation is the rich the generation of new events based on the way other events occur based on their logic time and requirements uh example if you have an access denied user administrator access denied on server a it can open always type Wireless I don't know so it development but if you have an accident from administrator on server a a few minutes later on server B A fueler administrator or server C for me to secret incident somebody's trying all the admit password on different servers so it's a correlation rule again like the reporting correlation will be effective only if the other

steps are working properly and correlation is mainly used to in Incident Management because base operation instead of receiving a lot of alerts you just receive some errors which indicates you need to investigate those ones immediately no let's see into the different tools I don't know if there are vendors of C so what what we call Sim Solutions here but dear vendors please don't shoot me but for me uh it's not because you're a big player but you provide the best solution to log management for sure it's like for me it's like a Formula One it's a very nice car but it's very touchy to drive it should we ask questions as you want Christian I didn't have a question

so uh why do I have to pay a huge amount of money if I use less than 10 percent of all the features in the product for me it's like the Microsoft Office effect everybody use Microsoft can use Microsoft Word Microsoft uh uh uh PowerPoint for example but I really use less than 10 percent of all the features very very important even if you decide to go for a free software solution it will have huge costs because we have to implement it you will have to decorate to support it to train the people to maybe find Birds to write patches to to for the to match the solution exactly to your needs so your hand you will have

always a cost and finally for me big players they give a full sense of security because they will send you a box you do deploy the Box a few days of consultancy next finish check their customer you have the logs in the box but it will say nothing so create a full sense of security um just regarding the the commercial products there is there are two uh two lines of the rich through the first one is a lot of management uh line of product which address the lowest layer from the production to the reporting if you remember my uh layer approach and finally a c security information event management it's always the correlation layer plus the other

tools which permits to manage secret incidents like a ticketing system integration with a third-party ticketing system and so for its rate two two different kind of of products so when you need to decide okay let's go for all my dream solution you will have to to show the solution or to deploy the solution depending on those checklists do you need compliance for those uh do you need to detect suspicious activity or just a repository of all the logs to be sure that we have something in place do you have to perform web application monitoring do this perform correction what are your supported devices because if you go for a free resolution maybe you will have to write

yourselves some parcel because if you have a very exotic commercial device it won't be supported off of the box and finally if you decide to buy your sim it's a very very specific project because uh it's a regular project first to deploy the scene often customers don't have any ID of what happened on the system so they don't know how what which amount of logs they have per day which type of devices so you refers to make some some assessment to be able to size and to design the platform and then don't forget that you also have to deploy procedures in up front uh of the scene because your platform is always changing so if you do not attack

adapt your romance resolution when your infrastructure to change because you have a new server coming you have a new internet connectivity you have a new ntls network you need to adapt your configurations to take care of all the logs but on the other side at the output if your system generates alerts but nobody is able to manage them return investment business so yeah you you have you will have to deploy huge procedures on both sides no let's forget the commercial products and let's switch to the free tool because it's free it's so so funny um the system of demand for system is a well-known protocol and still today well implemented in all a lot of appliances

and tools there are a lot of implementations so everybody knows system D running on all the uh the classic Unix environment but even for example Ubuntu switch to all system e4y your system NG which are Forks of the system and they had uh some nice heaters like multiple sources so you can drop even from uh from the kernel from uh from this e from UDP flow dashboard TLS dashboard system property CB which can be interesting and they also support plugins so you can for example with our system can immediately store your events into a mySQL database backend so you have nice of plugins and also uh there was several tools on for the windows environment which can

also export even Windows even fewer or even even even your feed to a system feed for example snare which is a commercial product but it has also a free implementation is it's a head to parse because you have no money regarding the RSC you must respect a specific number of fields but a lot of manufacturers of developers they just send free text into the slot so it's a helping us it's really really boring a nice tool to start up is SPC simple event for lateral correlation um it's it's a big Pro screen so it performs correlation of logs based on Broad regular expressions so to use it you must be a pearl regular expression to check this one the pattern I don't

know if it's really good for everybody but it's it's not a very complicated one but you need to have some background in a regress question and this tool is very interesting because it can produce new events it can trigger scripts and write to file if you detect something suspicious so these are just this example for documentation when we just have a single event it's a regular expression even if we match this pattern if you have Cisco device you know what means this commands this Five Below means of device Reloaded and you can have the description so warning is we don't regress it for device blah blah blah and you can action via buy you can mail the everything to

my Gmail address so it's quite simple like the name says but it's not very easy to implement because you need to play a lot with very Expressions but it's better than nothing the next one my favorite I really like is this project oh let's see is a whole Space IDs it has a long long story so the name was different it was uh it's now a project uh inside Trend Micro and so on but you it's very very nice Tool uh what it does so it's platform low connection and parsing so out of the box it can pass a lot of different uh logs all the classic open source implementation products are recognized all the same buzzfeeds Apache

product FTP servers The Cisco switches some firewall is really really easy it provides an active response mechanism so it can also treat your scripts when it detects an event you can generate something else send an alert but you can also for example inject a temporary or firewalls if I detect SSH Brute Force attack inject The Source IP address in my drop list on my firewall it has a rookie detection very important to find Integrity checking these features are made via agents running on all the Unix levels and Windows systems and it has also a nice roller carrying system so basically if you have nothing you need to start your load management process have a look at

this one really really nice what about the protocol so so when I I say a few minutes ago that when you collecting events you send them to a Central site but how do you send them and there are a lot different implementation there is no standard at the moment some of them are free some of them are based on the proprietary protocols those ones are really common really well known so oxide develop the problem even format so outside is a big commercial product but it's free so you can you are free to deploy your safe product self-compatible products is also a nice one and you have also help really even looking protocols Cisco has its own one so there is no standard

protocol to call at the moment osake use also another proprietary fund is home protocol to send information to the to the server so it's not very easy uh we can also find a lot of interesting information by generating events uh uh between osc and all the tools like my Square uh we can use ipt World Europe D the Google Maps API a lot of color codes only block norms and so on a nice project some people started to deploy to develop a library to be ejected as a modeling system server to automatically pass some even receive so it's very dark it looks that there are a lot of activity on on domaining lists and this library is

actually alive and you can also use group Services cloud services uh Don't Be Afraid uh to pass the event to to make powerful search so though uh I will show you some research that I made how to increase the value of some logs using osc and the first example is USB detection uh osc comes with a Windows agent so you can deploy a small engine on all your windows devices and the purpose of this of this exercise was to to detect when someone insert an USB stick into the system why protection against datability or security policies enforcement so you don't you don't permit to install investment or you must track the USB stick what we what I used to always think

Windows agents and the Windows registry and in OS X you can Define rules based on the registry so if you check uh in the registry in HTML system the new USB store every time you connect a new mistake it creates new and try with this Defender also the revision the name of the Republic as well uh it's just example so if I need to go deeper how to write to a sequence it's really not the purpose of the of my code here but you create new rules and you just say you will save business registrative for any if I detect a real sort of a change in this rate uh rig entry IV pop up another 80

million receiving money mistake an event which we say on this device this IP address I edited a change with the name of my new USB flash so it's very very efficient the next one um um I use a lot of MySQL databases and my opinion is that MySQL lacks of Integrity auditing system so it means that when you have a table it's impossible to know when this table has been changed if you have a database users with a table login for example if somebody adds an administrative looking change your password you cannot be automatically notified but you can actually just leave it for a sec how to achieve this MySQL has triggers so it's a classic SQL

feature so by with a trigger you can say okay if I need a SQL sentence this sentence will trigger another one for example writing another database myself also a nice feature called UDF user defined function when we can inject some Dynamic code into the MySQL to add new functionalities and the one that exists is right into a log file you create triggers I will show you it's under the next ring ufu High School Credit clients and um insert update and delete I I have my database level and I create three triggers and inserted it's on insert update then delete every time this trigger is generated it pulls the user-defined function which writes a Network login on the file

system and using my osc I'm able to access this flat file because osak has no way yes very very quick question and if if you do that and somebody has access to the database that really in practice can be defeated because they don't disable the triggers first then manipulate the table then enable them back to back triggers uh are different yes if you are that's why when you define your database you need to define the user with the right access on your database of course of course if you are if you have a root access on the database you can do everything you want of course but in fact the goal is if I have an

application here if you are my application somebody adds a new user a new admin user you are new account you can detect it so ossec has no way to directly attack a MySQL server so we have to use the different information right which writes a flat file I can manage everything temporary tables are very very useful in scene environment the purpose is to detect suspicious for example users and addresses it's a common feature on potential products it's not available on Osa that's why I wrote a patch to have this functionality basically you have a table when you put IP addresses and you on YouTube usernames uh these two Stables can be populated via external sources so you determine information

coming from the internet of the Acron and so on and if you if you receive the difference and the the the IP address we generate this event is also present in a suspicious table you can increase the value of your event so if you are for example a table of users accessing a web base in your temporary table you could use admin if you see admin accessing this application you will you can increase the value of your events how's it working so I have perfect sources on the internet and VR figure I create in mySQL database suspicious ID and suspicious users I my even sources my switch application servers and from and the osc code

collectors send everything to the other to the analyzer this analyzer reads the bicycle database and if it takes a suspicious IP address for user it increase the the level of the address

to populate the the tables we can use whatever we want because we don't have uh to restart the process we don't have to to to work at osc level we just populate mySQL database Google Maps why do I use Google Maps just to to match on the map where coming from my tax if I check those two IP addresses what's different between that between them you have 45 46 but if you read this it's impossible for the for your brain to immediately detect what's the difference the first one is coming from Netherlands the seminar is coming from Spain so the ingredients the Google Maps API available free for everybody some first cutting and I use geocyclit jio iprs

zero organization API and the result is this one so I received a lot of IP addresses under generated by my OST it sends everything via the active response script to my uh my backend my Google Maps backend and I have a live map on what suggested they know that I see if I click on on Twitter I presume I can see up to the city where I'm coming from in my attacks

and so on let's take dashboard because one picture is doesn't work so you have a lot of attacks a lot of information coming in Lots but it's also impossible to process that manually uh so MySQL always say has I have a nice person so you can write everything into mySQL database and using an Apache my spiritual alarm server I just deploy developed my own State dashboard and you can add some nice drafts some reporting live on your web server with one page so I have a trend level So based on the on on the right you can select which interval of time you need and based on Instagram you have to train it compared to the previous interval you have the

top 10 alerts top 10 super issues so top 10 60 shoes is which occur or more only one time during the last interval to be sure that we can detective and agents so what are my machine is generate the most number of events some timelines top 10 locations also we are geocalization and top 10 attackers with IP address I've always access to the database so I can see a lot of nice information about the visibility um you can so always think it's very nice to grab an event to generate alerts but we say right by default alerts into a flat file sent to an email it's not very convenient so you can use why not

develop I'm using globally you can use spring save this but here is an example of Lobby so locally cloud is an lrs so it's login as a services so you can send all your alerts into the cloud and locally will create you nice graph and you can also really search your alerts using a nice curling Giant Sprite is nice but it's just between commercial free producer I don't like exactly log Nissan is very very interesting my conclusions the raw material is already yours so you have all the events on your system so feel free to use it the amount of data it's impossible to review it manually because it's so huge today suspicious activity occurs below the

runner always so that those uh events are really the most important ones your requirements because if you start to deploy Your solution without exactly knowing what I need to catch and you will be lost because you have so much information so stick to requirements it cost a lot of money and time even if you choose a free of a commercial solution and make your looks more valuable the actual sources so you map them on the map you add extra information coming from external sources that's all uh just important remarks all the examples are on my blog with patches what is the scripts are available and all the materials available on my group so feel free to to visit it download it

send me comments good one