Fe-fi-fo-FIM, I Smell The Monitoring Of An Elastic Stack!

hello hello hello everybody's wide eyed bushy tailed I'm great you're not me go it off no one's ever ready welcome so this all come very happy so we're very excited to be here and a lots of light touches and I've described the stock and I've named it but it is touching on elastics topic is searching on the thin searching on heads it's all about seeing data and what people do that for I want to take you through the story of Finn and heads and getting to where you have dates are going somewhere what you're doing with there but we can kind of go with that the general idea set behind that and it's fun so I start

with me stop me for this Brett I'm a security analyst engineer now we'll probably have manage people people told me people also call me the chief executive brew maker some obviously secretary and I've been doing and post that for just under four years now it's four years the preparations around that well I'm enjoying so excited and the reason I've been in this talk is it's nearly where I started fin hits and also have a point monitoring that's pretty much where it began beyond some of the GRC and compliance stuff I did years ago but twice first start enjoying in terms of technical stuff so I thought why not go back to my roots and talk about that for

45 minutes so I'm gonna start from the beginning of my story and that's going to be about FEM and what Finn is a famous file integrity monitoring it's about looking after what's important to you on a very selective basis it's a very reactive skew to control it's not something it's not preventative in any way it's all about an internal kind of events in-depth solution it's completely stateful it's not particularly real-time people can sell it as real-time but it's a technical impossibility to really be real-time well it's also really adaptable because it's in that defense in that solution you're able to put it anywhere effectively a monitor any specific thing you want and how does this work I suppose is the first

question and it's mostly to do with the specific file attributes so that's hashes us the content of configuration files that's any kind of changes like the capabilities on the file like the security in writes or file and monitoring those different changes there's also ways for it to my turn api's and things like that an AWS or is your well that's a bit more advanced or talk about that a bit later but i've told you a little bit why Finn I suppose why I've chosen vendors this time quiet for me so why do people actually use FEM and it's all about compliance more than them it's particularly when you're like an SME kind of size not like a full

Enterprise your need to comply with PCI DSS or an American company and I survived with Sox necks a few other shaunb acronyms but that's always gonna be with the core of why people use it it's kind of a typical hit solution for any kind of SME as far as I Stanny way and he gives them that those eyes that the kind of movers and shakers on their infrastructure based on their baseline or the baseline they're generating from this information on this data and he does offer that Neal real-time I mentioned before of course technical impossibility but it gives them that initial visibility and that's why it's an important kind of fundamental first step towards having

security controls and actually investigating a bit further there are of course ways around it as I saying is the foundational control there are ways of dealing with non real-time functions as the deal with ways of dealing with our dynamic functions and points moves around more advanced controls or other controls or in terms of process and policies more than anything so the goal I'm trying to aim for with Finn right now he's all about that kind of the film and hits solution combined with somewhere to actually aggregate and analyze your data and that's why I've gone for the power couple of was 'when l cuz they're pretty much built for each other and I'm really quite excited to talk

about each part it's really cool it's really cool promise nobody's home good I'm very excited about everything so let's talk about that film solution I talked about the head solution but that was ooh head solution obviously had to come from somewhere and that's evolved from a open source solution called the OSF which is open source it's set purity rolls off the top without provide your fears your registry monitoring level one a train and he provides a rudimentary rootkit detection based on file names and hashes nothing too crazy it's really small file actually it does extinguishers it's definitely not an ActiveX placement at all but it provides those regulatory needs like I said for the compliance once were as well as you

don't start it's your start fen your fundamentals are all there in this piece of camp but why do a lot of people typically use on a sec it's because a lot of people already use or a sec it's open source so people but typically looking out the source of my eye it's free and not giving back so here well that's a completely different argument but it is fundamentally really really easy to use truely's develop or documentation averse it's a bit confusing we first look at it takes about half an hour to understand the installation process initially after that the documentation is fine it's got to have everything but you know it's not very my need to be so what has that

changed into it's like resume the home nation of all that's a fork of was that that's the next evolution hasn't pull up there but it's taken that base open source software and they'll talk with it and added extra features things like open sky things like the tie-ins without an updates relegation point an analysis point but it's a combination of all those features into a bigger wider platform and the while it does provide those features over set those fan those hands that system monitoring it also provides the features of open scout and elk so open Skype is I was inspired in the formulas management and disco as well as the more compliance focus compliance tained configuration

management and access and discovery but by providing that next level of features and the actual easier ease that's coming naturally from overset you're creating really immediately almost powerful tool for people to start working with to start using to start changing to configure into environment and again you can go back to that baseline and you're able to baseline further and further and understand more a warrior environment particularly when you're an SME versus an enterprise and it makes it so so much easier more than anything else as your own SME and you're growing bigger and larger Europe if you're not able to do this kind of constant discovering you're going to end up a bit behind with all

this is gonna be loads of one walked after infrastructure and information lying about that you will have time or manpower or investment really so looking to and that's may become a bit of stock well they think you've got lots of sticky stuff you can't get off your hands you're blind to what you actually really need to do and this is providing those the first step that initial investigation point or your infrastructure for providing that monitoring so how does it do is you know it's a basic kind of agent server communication three different processes sis check rootkit D forgive me forgetting the other process demon but it basically exists as stole agent and server calls back to manager that's

really manager so he conferred goes as well you switch off and I'm from stuff to Impala key on the edge but now that you've actually got that set up that initial part you actually need somewhere to put all this logs all these alerting all this telemetry day at the old gathering it's there in the middle you want the top as well that's that's that's the part it ties really really tightly and without so what it's this diagram itself is just giving you two different options on the single mode for masters or distributed nodes that's its own thing that's just giving you choices and design let's give you choice in how you want to set up a lesson but further

its hope right saying it's being able to tie in with this day our group aggregation platform and ask this platform is immediately really really useful and really really interesting because it gets it gets you further and further what would sure towards where you may want or need to be by providing the place there's a few different ways suppose you can look at architecture if you want to talk about it you can talk about some four hours maybe of a few tequilas few all the dreams but that's it's up to you you can if you've got elastic already can stand with that if not sticking on single server look after that so for new elastic histor and

stopping it went day or whatever type again into that point there's a load of different choices you've got that or let's take deeper dive into what is and it's a combination of three main projects or last search long staff and cabana elf okay and that's about a search analysis engine which is our search you've got long stuff which is a suicide pipeline project for digesting that data sending its receiving it all the good stuff and then you've got cabana which is all about visualization actually interacting were there on a GUI web interface level and that combination is an immediate start to the whole dinner aggregation platform makes it pretty pretty cool pretty interesting and oh it

does this is by taking that log stash so the pipeline the data the server-side pipeline creation white takes an end does whatever you need staring up at one takes into our search then you can try to think about it's not it's a it's beautiful it's really easy you may have noticed this is another one where they do vertical beats as well that's an age of these kind of setup so if you can't send the logs necessarily but you still want to receive them or gap with them you can pull hey agent on the endpoint pulls them off or sends them wherever so all the options that human get really specific with speeds is working at

metric beats for metrics get all it beats for order box I can get generally far beat or any kind of file log if you want they're all available so I will get an idea but that will be going straight to my search rather than viral and typically you can still send it by and long staff or its new you know that that's your choice and the pipeline's on log stash are really really I don't want to say complex and I want to say lengthy they're available to configure you've got lots of different choices of how you want to take that data uses a language called crop to be able to take the data in editor change it

label it however you want it to be labeled plus into a search and then put it that your index of many endeavors you've got answer is available so identifiable you know what it is you know where it is hidden know what it is well why help why do people choose out well I chose now I've chosen helpful of some obvious reasons so ties into was they obviously but it is overall just a powerful powerful powerful search engine it's got lots of choices in terms of architecture design in terms of how you want to set it up and destinies there you can make lots of different choices in however you want set it up you can

centralized or decentralized there's options for you wanna look at identity or user behavior its options for how you want to store that data you want to get rid of it up to six months all the options in that all the difficult kind of options but there's also so many ways of analyzing that data as well especially when you're gonna end you've got pro bono that and talk about statistical analysis you can talk about Israel just looking through the day you can talk about machine learning talk about what's the name leaves its timeline is a narrow platform it's analyzing data over longer time varies don't remember natural name for it unfortunately but there's so many ways

of using those there if you look from it can monitor it however you like as well completely little choice but again it's Razoo and elk they are those are those two pieces stuck together it's got Wu Zhu on those end points on the manager or separately or with out like Oh showing to the potential choices and you've actually reached that goal a combination of the two you've got a platform for security analytics or starting to independently view you can actually start moving forward with we've got and you've got the whole kind of three main points can do the data so can rich it ionizes navigator bar pair so we would have say an aggregator you've got

your data in that place you've got your day - in the centralized or decentralized architecture it's available to you now you can combine it with all the other data like oh say you can change and annotate and labelee how you need it - it's all your own choices I apologize if this isn't too technical of a deep dive but I'm touching on a lot of points here and what e makes available to people is if you don't want to learn sequel or you don't want to do a full scene on yourself you still want to start something like this don't wanna try something out this is what this is on what I'm telling you is you can move

from this to a more powerful platform which is something that we're talking about so I'm quite excited about that but you can from that aggregation from putting your data in this platform this analytics platform you can actually enrich it using those pipelines as I mentioned before you're able to buy bees you're able to push in all this together you also thoroughly import threat intelligence and things like that right into your data and he really lets you take a lot of work from your analyst my assumption at this point is that if you're getting this far you're the least high in the person that's looking at it but that's not always the case deficit is the case but there's plenty

of options for taking a lot of the work off a lot of the enrichment so when I'm saying about identifying and assigning all our dates with labels and been able to look at the easier that's what the pipeline comes in that's the initial part of the initial touch point and and not only does it you can do it manually you can do it in the animal file that's how you live your line or you can do it in a visualization if they offer this now and it's fantastic and it makes it very very easy and thoroughly recommend good weekend sit in get approved then hop on the Obama dashboard and throw together a few highlights

it's been fun right from there saying a lot you caramelize it it's a fantastic platform we've got lots of different options you've got lots of different options in machine learning is difficult analysis as I said before but it's also got lots of built in features like monitoring like a PM like the choices you can make in the data you want so the Daisy don't want you can drop anything archive it it's a great platform for ECI because it gives you the money - I could hear all that data but it also lets you make the choices of great got the PC idea to the spot that aside still compliant don't worry about that my absolutely is

if something this is just compliance and the features that with it allows it to be very controlled if you need it to me it allows you to do things like some single silo it allows you to do things like a buck a buck they also like to because of that this to be a nature so I won't fall he allows you to easily orchestrate it and thrown over on another note throughout another knows where you need it to be doesn't take a lot it's really really straightforward but from this platform we've then got some like took it over now we've got something really really cool and interesting I a big fan of Cole's apti Metron this is a much much

larger platform this part of it is ELQ which is why I've got here but it's CAF gates Hadoop it's a lot of advanced stuff suddenly in your face but this is more towards I full Enterprise people dedicated to this work this is where this is takes what's up its welcome understand just want to get running but of course it's a much larger platform but it builds on top of that puffle has been built with the zoo and elk and give you a much larger choice of how you want to aggregate it are going to rip here there's lots of law built in enrichments as well things like threat intelligence things like how a metrics reggae and once you got my initials and

set up all the way sold they're available in front of you it's easily manageable I say easily it's easy for me might be a hard few subjective right but it's that next level you come from Bob head to pop in X brain you've got four months instead of two it's fancy it's real it but he'd provide it's a massive framework in comparison to who was who and health and that platform its befall from Cisco open sock it's become this platform when it was taken on by Halton works I want to say don't quote me on that moment believe it is and they took it to a completely new level completely new idea set a new framework

but they've really built on some incredible stuff and it provides four key ideas it's all about in Richmond it's all about right it's all about rich men it's all about getting logs in their Slovakian network with stuff in there it's all about threat intelligence for Maine it is what drives its framework but it takes in Knuffle it takes and things like pickups things like logs things like sticks and taxi it's all available there in front of you right there and immediately available and so so interesting I'd recommend giving it a go if you get a chance it's a fantastic framework but what it's done is it's taken that core elk it's taking that core aggregation platform and gone ok so

how do we make this easier and this be easier and this be easier isn't very careful not to over engineer obviously but you're taking all these functional parts plan together into this beautiful got these layers really good stuff so you takin it from a car okay normal beef burgers like that triple decker stacker delicious cheese feelings on top Oh beautiful especially when younger but you're taking it to that new level is what I'm saying cause you've got so many people behind and working on it got lots before working out don't get me wrong but this is dedicated for what scoot analytics there's lots of different use cases and ideas driving it that aren't necessarily driving elastic so it's important way of

where you want to be almost time you wanna invest in that and decide what you need that but the fact that it's this dedicate is huge a massive framework means it's much more extensible it's built for the ideas that is built for friends that cannot see analysts still for cyclist is built for people doing the work rather than somebody who needs an understanding of data analysis and how to actually configure that and you mix it takes a lot of the work or guesswork out of the work for the analyst and it's really really fantastic but how does it work it's got it's pretty simple as well it's got the key kind of pilot serves the data ingestion

so it's giving you examples of miffy and generic telemetry but of course you have things like the farm these things like the optometric these things that logstash still there and available to you but when it's aggregating it it's pulling in every single thing you can't and put in that place or distributed and those are brain but then when it's going from there is going into an enrichment in Richmond isn't optional like elastic is going to be able to ideate it with all these kind of Trent Arjun stops it stays taxi a user import understanding the data is what's driving that driving that in richmond is your analyst your threat intelligence people all these all this work is actually

going into versus it just being advanced on a day like then from there you can take similar so that's it can take the alerting the monitoring obviously they've got a bit more mature in terms of how you might alert on monitor as in its naturally love for a right stick in jail and stick it in emails texture if you want that's you what form needs your input by less and your understanding or work going into it so it's important to know understand your data it's important to map your infrastructure it's important to baseline understand their i know these can be really really difficult you've typically Carmack's their workload but it's about trying it's about finding out it's about knowing

what you do would know understand and think in those risk decisions how you want to approach these things well les amis I've seen don't have security strategy so filled have that we need to comply how do we get there and you know Razoo and elf is a perfect time place for PCI DSS it's easy this reform is simple and he gives compliance and then he just lock it up put it away still do this thing still compliant no really well you do need to actually analyze and look at the data but there's no definition of how regularly beyond daily its long-standing what you're buying and selling as well what your services are a business what do you get

in terms of data or what do you give into there and knowing so if you're not so you know use it for and against yourself whispers in terms of securing infrastructure and the standing yet but it's important that you user rights available to you and you've got these platforms available to its once you got past that understanding stage once you've got past our baseline stage you've then got something to work with because something available to you and you're able to do something with it it's important we do because it's a rel is ready for you nine months no no this particular interest within this s log out over random house observer except to you or anybody's gonna break in your

structure so it's probably best that you use it first you're followed here but it's knowing what you're after how to get out there and how you gonna do it and this is just one of the choices of analysis and analytics and aggregation I mean more than happy to feel that little questions but it's up to you it feels euro British questions to understand what you've got what you need and what you're going to do with it all but Frank you