A Day In The Elastic by Oliver Creed

learn more about elastic um i'll talk why i wanted to do that and place it through that a lot more so here's my general pre-made what i did and why i did it [Applause] [Music] [Applause] um

[Music] experience set up my own cluster um so there's that single note it's still a plus plus um i want to use different features with elastics use log slash files um i want to use a python library for interacting with elastic so i didn't want [Applause]

i wanted

[Applause] [Music] um so bringing security i think timely security news is really important and being having them real time obviously there's so many different things to monitor it's difficult um to like go to all these different websites so it's a solution for that i'll see our attention whether um see what we do with that um [Applause] really nice way to visualize different types of data um fancy gas sports graphics utilization metrics etc um [Applause]

um first one was the rss plugin and what this does it calls an rss feed every so a couple minutes or time you specify i better pull down all the events transformers and log stations then finally um it's great it seems really simple but i found out probably wasn't the best solution um

[Music]

[Applause] credentials um if you give some keywords um and it will pull that data basically as well so that's the main point is um with the twitter youtube information you get retweets the likes people tweet to retweet i'm not retweeting so you can actually kind of look at trends so say you've got a new cd pops um and a lot of hype tv once you see a spike if you're doing this correct search and it's a really interesting way of actually keeping tabs on what's popped at the moment or if there's a cd you want to know more about you can search for that cd and i asked it with the data there and it brings up if you

feed all this information for that cpu so this is an example rssc [Music]

um [Music] [Applause] visualizing better um

[Applause] [Music]

[Applause] i was getting all these events over and over and over again into elastic i was like well i've got that twice it's useless um so the way i resolve that is uh the fingerprint filter would be to take the hatch so i take a half of the title of that article and i use that as the document id and elastic and so again instead of creating a new document you just update the current box so we talked about los angeles pipelines uh registered new electric cars ingest pipelines these are similar quite hard but they're elastic and if you don't think two packs of them i sort of wanted [Applause]

[Applause] to standardize um across all the feeds uh platinums and visuals so also i think it's really cool um so on the hunt for more feeds you can never have enough beads i was like it's really unscalable to add a mod slash pipeline for each individual url there must be a better way so there's like news api there's like how do these news apis get all this feed are they scraping themselves quite a lot of them just use google news [Music] [Applause] um

[Music]

there's one major downside um so that brings us into like some problems um so not everyone's planned so i didn't talk quite yet but basically just cd database [Applause]

um

[Applause]

uh first issue i ran into was with the duplication documents the second issue is it was very strict about the attack so it wouldn't like take all the feeds it has to be very thought well formatted in any sort of issues you just would not just in any events um was really annoying so obviously perfect so whoever run that or um

really simple just take just go to the euro take all the events and then just convert suggestions of work placements um

[Applause] [Music]

so instead of like saving into a file indexing it directly to elastic similar with creating a my own id for the rss i use the cd id so when that cd is updated and then updated and i asked it um so we're going to go on to the results so hopefully there's a couple of quick dashboards to make some use cases um some a bit more fun some are like a bit useful um

[Applause] [Music]

um

[Music] [Applause] [Music] um

[Music] [Applause]

[Applause] [Music] [Applause]

um [Music] is [Music] um

[Music]

[Applause]

[Applause] um

um

[Music] [Applause]

so i want to display this value and make it a clickable link make it really easy like view this they can go to the actual page so this is

um [Music]

[Music] [Applause]

[Music] [Applause] but you can um quite a complex one so you look at these keywords what these keywords on top of the filter that's already done [Music] um [Music]

[Applause]

uh

um

[Music] [Applause] [Music]

um

um

specific to your organization um

um [Music]

um [Applause]

yeah yeah it's definitely not the best way of doing it the other way to do it uh maybe more reliable ways but i just wanted to do the last one yeah huge opportunity for social media like yeah

thank you