How Microsoft Made Me Love SIEMs Again

[Music]

hi everyone welcome to b-sides i'm really excited to kick off this conference we've been working really hard as a team all the b-sides founders and presenters as well as all the people that are able to volunteer we know that this year with covid and not being able to be presenting in person has been a real bummer but with this conference we hope to bring your day and on top of it bring some education and community back so with this presentation what i want to do here is really go over our azure segment i had a really difficult time over the years trying to build out sims especially in bigger companies and moving to a startup has been even

extra difficult with staffing needs and as well as being able to hire security luckily microsoft came up with a really great solution they put together this this product called azure sentinel which is a sim and i'm going to explain why i'm falling in love with microsoft sort of their platform has been really easy to build upon as well scale so let's get into the presentation so my name is chris mainer i already introduced myself i'm going to be around the conference all day i'm going to be not only presenting but hosting the conference so i'm one of the the b-sides individuals to being able to monitor as well as put together a great conference for you guys you can find me

on linkedin you can find me on github twitter on discord so if you want to communicate or ask any kind of questions during the conference or post conference please feel free to hit me up i'm a native philadelphia individual it will be easy for me to find me around in the city now why am i talking about azure sentinel uh sims uh security incident event monitoring tools have been the bane of my existence um but not in a real negative way just it's been a pain to manage them over time i had to deal with vendors had to deal with data problems had to deal with staffing problems had to deal with scaling problems but overall

they've been challenging now why i really want to highlight azure signal here especially as a one-person security engineer at the time when i was first building out the the startup security program i really needed to figure out a really quick method to not only help the business grow but at the same time figure out a method to better protect as well as offload my time i had to figure out some kind of method to not only help grow the business but you know take a method so that i could be able to build out infrastructure build out a team build out staff but at the same time make it flexible and easy for myself so as a challenging

dilemma the limited security staffing needs were very hard by the time i was able to get some staffing that actually knew how to do proper security devops sre work uh they were lacking in regards to building out sim so what they really knew were good security tooling knew how to handle instant events how to handle incident response so this is where azure center was a good fit now the other challenging problem that we've had um from a startup is we're cloud first we're cloud native so how do you actually scale and protect uh a global community uh we had to figure out you know not only the the challenges of business but how to take uh a

10 a an engineering staff of 10 at the time and in about a year grow to 150. with that not only are we growing the engineering staff but also all of the business development all of infrastructure operations executives i needed to figure out a method because we only had a staff of two to three security engineers and they were basically trying to secure a lot of our applications in our platform how are we going to secure our environment as we grew so cloud native that's another big problem that we ran into and then again as a startup we're lucky during covid we were able to figure out a really good business case and make it grow so with exponential

growth means securities looked you know negatively upon it's not be that they don't want security first they just think of security second or maybe even third it's mostly business it's products so how do we tackle this problem so my objective when building out a sim was not for maintenance didn't have third-party vendors i can't have vendor support we don't have the time or the or the the money so we have to be creative so my objective going into our security program and figuring out how to build a system is autonomous serverless and mutable infrastructure i know this is very you know popular to words that are getting thrown around a lot but azure sentinel affords this not only azure sentinel but

azure as a whole as the second leading cloud vendor in in the world this was a really great method to be able to build out a security program not only minimum staff but we're more of a developer focused startup so i'm lacking some of those engineers that can think very low level lower on the osi stack so with immutable infrastructure it's you want to look at a server servers and containers that are never modified after deployment and that's really important for me so that i can just focus on code i can focus on rules and i could deploy things very quick and efficient bring in new staff and be able to just figure out more of a business problem figure out

more of business risk and figure out how to secure all of that all in one shot so the methods that go into creating a strong security incident event monitoring platform is you need to figure out what are your strengths what are your weaknesses and most importantly what are the business problems that brings in azure signal so i've been highlighting and talking about azure signal so we're going to go very high level into what it does i have some videos later on at the end of this presentation that you can look up they do a much better job of explaining the technical i want to try to really focus on the use cases here as well as why i want

to use serverless based infrastructure throughout our security program now uh azure segmental again is a cloud native security incident and event monitoring system and you want to look at it as as four primary objectives here it's collect detect investigate and respond it's pretty much how the sentinel platform works the sim platform is nothing new in regards to the the methodologies so you can come in and get your team scout up really quick not worry about the infrastructure and focus on the data and the quality of how you actually action on alerts i like to break down sentinel in an order of operations so there's three main terminologies that you need to know about with with azure sentinel it's the hunt look

at it as like an investigation query the syntax the how you perform your queries so in this case you're using custom language uh you can search you can take some video tutorials it's it's a very flexible language set that has a very low learning curve so you don't have to know a lot about programming you just have to know a basic sql and it adds some really nice scalar based functionality so you can break apart json objects or arrays or any other syntax that you're looking for with basic um you know data structures second part you want to look at is the workbook so the workbook look at it is like the the trigger of how you would want to build out your

hunt this is where you do dashboards this is where you would take those queries and implement them so that your analyst or your product teams can look at what's occurring against their product or their platform and then the playbook is really what the action is taking and what's cool about the playbook methodology is that you can really make this very reactive or proactive in regards to remediation i kind of blend the two with our team i do this on purpose so that i can kind of bring in new security engineers or non-security engineers that really understand it operations or even like product and development and tell me a problem that they're they're facing then they will be able to articulate to

me what i'm trying to solve i'm able to basically create the query and then on the remediation piece of it figure out how to apply azure's logic apps or any other serverless infrastructure to proactively try to remediate the low-hanging fruit or alert on any complex problems so that we can then action on as an incident response team so we got a little crazy as we wanted to learn azure sentinel my team at the time was like you know like i said like two to three we had a really aggressive schedule we had a growing business we also had covid so how do we solve this quickly my boss kind of like threw out ideas like hey

there's a hackathon going down by azure and i know that you're getting ready to deploy as you said no why don't you take a stab at the hackathon while you're at it it's something fun you can build out a team organization build out a trust as well as win some money so azure signal had three main objectives quality the value and the technical implementation so we didn't want to you know tackle this in any obscure way we wanted to do something that was quick and easy because we only had five weeks when we found out about this so we had to move quick luckily the team that i had they were very skilled they're they're they're

very high they're very they're good engineers that can think very objectively as well as learn quickly even though they didn't know the tools i could kind of direct them to say here are the problem use cases that the business is facing here are how here's how you build out your queries and then you implement how we want to remediate now week one again planning week two build and deploy at this point we're able to build out a strong ci cd uh the infrastructure was a blend of containers they call this azure container instance um we were using aks azure kubernetes service very lightly at that time because we were still a little bit newer to kubernetes on azure

as well as their platform as a service using all paths deployment so that that's where the serverless piece comes in and what's good about that is that azure does all the security and bakes in all of the identity management and integrations with other bindings of their services week three and four was really our development and testing phases so we can verify that all the use cases worked they operated the way we wanted at that point we were able to document and submit it was a really fun project but what i want to really show you here is three methods of what we focused on in data collection data collection has always been the real big issue with sim

sims wants you to deploy agents on a server vm they do all the compilation they're able to do the the extraction the transformation the loading of data i always was very irritated with this method with from vendors because i hate being forced into using something i like to be very creative in regards to tools that i like to use something that makes me be able to integrate with database systems or servers or frameworks from a developer stack of choice so what was great about this is that it was very flexible with the data collection they have some standards that they like to use but in this case i wanted to focus on three major objectives because

one we had our facilities we had all of our network equipment two we had all of our our stack in regards to how we deployed all of our code as well as how we presented all of our applications like we said we're cloud first and then three we had multiple multi-cloud so i had to figure out an easy method so the first part was syslog syslog was a really easy method here because why how we were able to deploy this was using elastic beats and elastic beats has really really great integration with kafka and uh i put security in place so that we could allow syslog in just from a subnet perspective but i needed it open on the internet and i

had to figure out a method so that i could easily deploy and also protect our environment so using beats uh deploying this on a kubernetes stack we had a load bounce up front we had a firewall only allowing our facilities traffic to come in and then from there on the back end it was able to integrate um into it integrate natively with azure event hub which is the kafka piece so it even if we get exploited or someone is able to figure out our assist log in just from a facility level it would never take down our system because kubernetes was able to scale as well as be able to collect information on the event hub and on the

event hub piece is where we would build out the filters to take in data that we needed and we didn't have to worry about having any kind of system failure on the back end with a traditional syslog server second part of is our web hooks because our use cases are are are using hp methods the the web hook method here we used azure functions that is look at it as a serverless hp trigger that binds to other services and what's really nice about azure using paths their platform as a service is that these azure functions uh if you come from aws world azure functions are a lambda so just so you can know the comparison between the two

uh we used a nodejs stack here are our startup um uses node.js a lot of javascript type script as well as a little bit of golang but mostly we're in node shop so when we were deploying these node applications it was really easy to do very simple um methods to connect into other cloud first services and then with azure functions what's nice about these things called bindings that all you have to do is integrate with any of services any of the type of services that are available on azure and it will be able to write to it so with those function apps i was able to collect data send it to the event hub and that event

hub would basically be able to send the information into the azure log analytics which is the platform that powers sentinel the third part is the connectors if you are not a developer you're not an infrastructure person you don't need to really go into level i'm talking about i'm i'm building things in a very unique way for unique methods so i can scale and not worry about having staff but what's nice about azure signal you don't even need to worry about the webhooks or syslog you can just use their direct or grand list of cloud service connectors what that means direct is any service that you run within azure you can consume that information and be able to alert on it so if you're

running azure active directory it will collect all the off logs if you're using the pads it will collect all those those application logs then with multicloud it has native integration with aws so you can collect all of your aws logs from cloudfront and be able to action on things in one single pane of glass this makes it really flexible for instant response teams especially where they're not used to using different multi-clouds if they're in like a lot of our analysts they're very used to using azure tools so this was a really easy method for us to collect data and action on things so the use cases we wanted to make it simple um we had two major problems one

we had slack and we had to use slack that was not really configured in the most secure manner uh we we eventually were able to catch up by getting the slack enterprise capital it's very expensive slack enterprise runs roughly 25 to 30 000 a month but what's nice about it is that you get native api integration and with that uh we had this we had this use case where we had to figure out how to do dlp um to look at what people were using to download information that was dormant over time so instead of integrating a dlp solution which was an extra ten thousand dollars an extra twenty thousand dollars an extra fifty thousand dollars

a month uh we instead just bought out these use cases with the api because we can build out those web hooks we can collect information at will and then we're able to focus use cases on what we wanted to collect in this case we wanted to know high number of downloads of sensitive files slack file names so if people were you know crawling slack for any kind of information were able to detect on this that was the not that was abnormal to their user behavior and then a maximum daily download and basically what that means is like not many people download a ton of information from socks so if we saw an uptick of that we would alert

an action on their account and depending on their role we would automatically deactivate their account um we we had that native integration with slack to our i.t operations team so it was like easy push button so they didn't even have to go in the senate it was all just one easy flow the second use case was the cisco meraki network i chose cisco meraki because again we didn't have staff so i needed a programmatic way to control our environment so we did a lot of orchestration and with that that allowed us to control our sd-wan um at the facility level as we tested sd-wan um it was it's a nice platform but because we're cloud first we can do a

lot more controls in the cloud but with that we also have to worry about lte backup our facilities run really lean and the code base that we control in a lot of the the tools that we build we can uh make those calls and we try to keep it under like 50 milliseconds for every api call so we don't need a lot of bandwidth but we did have a problem because our managed services as we were growing our facilities we had to take shortcuts on what equipment we had to get because of cost uh we didn't get to control the routers because of that we could never tell if an ip changed if say for example

uh isp failed over to the lte so we had to have these native web hooks that just listened on the meraki network for ip changes and if they did we would then change a configuration on the fly so that we would be able to keep the business going and limit bandwidth usage by our employees and this allowed us to grow over uh hours of the night because our business is mostly you know more more evening hours and we didn't have staff or instant response at the time when we were building this use cases out so with that this allowed i like to call sentinel autonomous mitigation i was able to sleep a little bit easier

at night because i knew sednal was doing the hard work that my staff couldn't afford to do because they were more focused on building out products and building that business and building out organizations so cental was a nice complement to our security team by focusing on specific use cases implement our security posture and then at the same time allow our employees to rest easy and focus on business he needs instead of worrying about rules and policies so here's just a 24 hour collection of data so you can kind of see over under 10 million requests on app services uh we over that 24 hour period we have 150 alerts and this was this was removing a lot of

the false positives so these were actual events we found that it wasn't that they were being malicious our user base is very creative they're they're they're a young bunch that are trying to grow a business so they're they're doing a lot of different things and by setting up these alerts we mitigated a lot of abuse of infrastructure of of the integrity of our systems so by doing this kind of work we were able to go out and focus on training and then this allowed our alerts to start going down over time so a couple results that really came out of this the hackathon we came in second place uh we couldn't believe that we actually

qualified uh we we created uh a serverless infrastructure and later on i'll have um links so that if you want to go check out the code or just check out what we did um you could look at it after the after our presentation um that that was a really big booster to our security group because not only did we get the morale up but that showed our leadership like wow you need some budget and you need staff let's get you some so now we were able to increase our budget we increase our staff and at that time we had around like three security engineers i was able to grow the organization from i'll say those three

to a staff of 20 split the organization then have it ops with a new director so that they could focus on enterprise security and then splinter our security team into product to focus on the future of our cloud and now we're able to focus on more secure devops build on an sre group as well as have a strong enterprise security posture another big thing that came out of that is discovery so that we can prioritize how we handle our slo or slis now why is that kind of important it just helps us with priority and this was able to set expectations to our facilities to know when we should have our availability higher up on the network

versus our having our cloud infrastructure be more prioritized with development so following up uh azure sender is a really great platform it's not the best when it comes to you know just plug and play and making things work you're going to need to know how to use a sim so it's going to be a little bit of work and a little bit of training luckily azure does a really great job the azure sentinel level 400 ninja training they like to call it um this will give you like 13 modules that will teach you everything about azure sentinel as well as azure and it's free with videos with code with credits so that you can help build out an azure

signal tool if you have experience with building sims i have been using sims for a good you know 10 years if you are newer to sims this might be a little bit of a little overwhelming but i would say still go after it it's really helpful because a lot of clouds are going to be offering this not only with with um azure but also with aws and google cloud and then eventually something beyond that and then i also want to highlight is the the sensory platform that we built so if you go to that link you'll see the work that we did you'll see the code base as well as individuals that help and that's a

special thanks so the two security engineers that really helped me was joseph rivera and damian walter meyer and damian uh you might be seeing him throughout discord he's volunteering for besides philly um without their help and we could have never got this done so it was really fun project that the three of us were able to put together so thank you very much for attending this conference again thank you for attending this presentation as well if you want to reach me you can reach me on twitter at the finga i'm also running our b-sides twitter so if you just go to b-sides philly or you go to our website you'll be able to interact with me or the team

and my github is cmaner i don't have a lot of my company code on there it's a lot of just like little personal projects i'll be expanding my portfolio but uh thanks again for attending besides philly and i'm looking forward to communicating with everyone thank you so much

you