Why security initiatives are doomed to fail & what you can do about it

all right thanks everyone um I was saying before I'm after Peter this time I was here in 2019 I was before drinks so I'll definitely take this trade I think it's much better time to talk so uh as the slide very much says white skating ships are doomed to fail and what you can do about it now over the next half hour what I'm all going to talk about is not how to do security but more how to get security done um it's a bit of a nothing statement but hopefully it makes a little bit more sense as we get into it so there's three key takeaways that hopefully um you'll be able to take away from the

next half hour and if they don't make sense hopefully they will by the end the first is that understanding how workflows is fundamental and this is how work flows through a business through teams all that kind of stuff understanding how work gets done and where it goes is fundamentally important the second is agile doesn't scale across teams uh those who know me know I brag on so much of agile especially skills agile all the time but I'm going to talk about something that actually does work across teams and kind of look at why agile really when you look at security teams agile really doesn't seem to do much for you it seems just make everything a little

bit harder um than it was before and you're really getting any advantages I'll talk about why one of the reasons why that is and the third one is you need to be constraint aware and I will Define what I mean by constraint and what I mean by constraint aware as we go along um the least important slides um is the one about me I'm a senior engagement partner at cognizant Serbian I wrote a I wrote the cloud native security with O'Reilly that came out last year to lukewarmaclaus um I um I'm also the uh one of the Perth organizers of birops which is Australia's biggest networking event and we have the first one Happening Here in

wa this year in mid-november um tickets free tickets are available now um You can find me on LinkedIn if you want to know more about that so uh today I had way too much fun with mid-journey doing these slides hence there's some ridiculous breakfasts all the way through it we're gonna talk about fictitious company uh nice base wa quarker Incorporated uh I use them far too much they're trying to digitally transform um if anyone has a good definition for that can you let me know um the second thing is they're doing agile very much in inverted comp and inverted comments and always will be the security team is working harder than ever I want to stress as I go through

kind of what what some of the problems are it's not because people aren't trying hard enough if anything they're trying harder than ever but Security's struggling to keep Pace from the fastest teams and you're struggling to get movement from the slowest teams this is really common where across a larger company you've got the teams that are moving at the speed of light and you've got the teams that are barely moving at all and the this is just because this is what I run into on a daily basis and now for a few years now do employ the new cspn Cloud security posture management six months ago and they thought that was going to solve all the problems

um smaller alert it didn't okay so looking this is pretty standard this is what I see a lot of the time in terms of controls that the amount of controls they have in the cspm keeps going up that sounds good right they can see more of what's Happening that's good the problem is they've got violations are increasing at a faster rate than they're adding new controls and the average age of a violation is increasing now I can't remember the last time I went to a client and I didn't see this maybe that's because I work with particular kinds of businesses but this is so common it's a trope when we look at the security posture

really it's eroding over time and I know we picked on cloud security here because that's what I generally do but you know you can apply this to any kind of security really defining problems faster we can fix them it's forcing them into this reactive of a proactive approach it's not about it's about all something bad happened to me he's going to fix up right now you can see from the numbers that the the derivative of the security posture is going in the wrong direction I'm gonna talk far too much about value strings for those who haven't come across them before um a value stream pretty much is the mapping of the process from police to

thank you so from idea to and ideally in the hands of a customer when you look at them they can either be owned by one team or they can be split across them and that's a key point I will hop back on at length when you look at Value stream metrics and I've gotten the simplest possible I remember this from like looking at devops back in or early 2010s and they said this is how everyone delivers software but this is kind of a very very very simplified value stream for a kind of any kind of development team they do development they do build they do testing and they deploy it when you look at a value stream there's

many metrics you can look at I'm only going to look at two today the first is throughput which is the unit delivered per unit time and the second is the lead time which is the average time for a unit to be delivered that said there's many more metrics but those two are probably the most important to in the two I kind of need to make my points um you can think of an organization that's been made up of value streams that's effectively the way they deliver value to customers whether that's internal external or have here but there's kind of important ways to look at things now when you look at the value stream of a product team or kind of development

team ideally and yeah there are caveats on this but they generally own their value stream end to end this is generally what the devops movement was trying to do despite what's happened to it in the intervening years the point with this is everything's in their sphere of control um you know the development team owns everything they have to do they don't have to rely on anyone else they don't get blocked on other teams and not waiting on other teams admittedly your mileage may vary but this is kind of where things are going where things hopefully are this is to date the most effective way of delivering customer value we've found is to bring together cross-functional teams

to go and do the thing so a slight sideways movement into the theory of constraints from Eli Gold rats now people may have come across this before I'm going to do a very very light treatment on it but in terms of relations to Value streams they're exactly one constraint on the value stream there is one thing that is defining what the throughput of value stream can possibly be and anytime you invest effort into improving something that isn't the constraint you are wasting your time you're just kind of making more work that just sits at the constraint not really getting worked on the classic example I see in development teams is testing you have developers they go hit features and there's a

manual testing team that they have to wait two weeks them to be around and then they're on testing that it takes two weeks so although you can make the devs code faster it doesn't actually matter because it's the testing bit that defines how quickly you can go so just following on from that and I mean I don't have enough time to go into this level of detail I ideally like to but we want to keep the constraint optimally fed ideally wherever that constraint is in your process you want to make sure it's always working but it never has too much work so anytime your constraint doesn't have work to do you're actually just losing value you're wasting your time because

it should be working and at the same time you don't overload the constraint because then there's too much to do they have to multitasking I'll go into that in a little bit in terms of all of a sudden having to prioritize 10 different things nothing really that gets worked on everything starts to fall apart so for a security team your the value stream you're trying to actually on spans teams it's very rare security thing can execute stop in isolation normally and I was saying this to some guys outside that developers are the source and solution to all security problems they create them but they're also the way to fix them the thing is that your work generally

enter of influence here you're no longer in control of the work you're just influencing how the work is getting done the constraint is almost certainly outside of the security team so you can optimize what you're doing doesn't actually matter instead I'm going to prove this with some simple modeling in a bit um so being this is the case where your security team you have to get product teams to do things eventually work kind of comes back to you for trust and verify you need to understand understand how to optimize when this is the way you're going to work when you look at agile they go back to that kind of key Minds everything we're just going to optimize

on that they randomly end up approximating theory of constraints-based view on that but it's kind of by chance Not By Design so just to quickly walk through like what in the search configuration change might look like we've got a security team there are three product teams that are independent and then comes back to the security team at the end and I put indicative lead times in all of them so remembering lead time was the average time for something to get actioned and completed by a certain team now I'm going to use a variety of colors to show um and why the colors are there will make more sense in a second with the show colors in terms of all these

changes they impact every single team so the security team is going to pick something up and for every piece of work they're going to need each product team to do something and then ideally you'll get security at the end we'll do stuff and we'll just kind of model what happens after like five weeks of work so after the first week the security team's done what they need to do they've now passed after the three product teams after two weeks again the excluding double to jump through work feed into the other teams you can only see there's a bit of a backlog happening on two of the other teams I bet if we just step through

you know it takes us actually five weeks to get any work completed any work actually signed off because it had to go all three private seams to get there we have a variety of backlogs sitting in front of teams of work to be done and you'll notice that for product team three that backlog is actually measured the top one is slightly in progress so it's actually eight weeks worth of work they have to do so that means climate team three is your constraint that's the simplest way to find the constraint there's a lot more you can do on this and there are other not going to go into it don't have time anyway so protein 3

is your constrain so at the moment they are defining how much work Azure comes out of the security team because they're the slowest moving part so yeah but finally constraint for a given backlog of work the team is the longest queue in time is your constraint not necessarily the most they have to do but when you take the amount they have to do times by the time it's going to take them to do it that biggest number that is the team that's your constraint multitasking is a myth um I argue with my wife on this all the time if you reckon she can do a million thing up things at once against annoyed me for

only doing one but um the psychology and all the research backs me up on this one when you get to constraints you know cash switching debating priorities those are both inherently wasteful things if you're you know there's context um residue that happens when your attention residue that happens when you strip between different things you know when you have if any of you worked in like an environment that scale that job or anything else like Pi planning where you spend two days arguing what you're doing for the next two months or even on a scrum level where you're spending you know generally maybe a day doing backlogger I mean argue on these priorities all the time

ideally you don't have to do that right that's all wasteful time not actually adding value or trying to find Value in there when you have these constraints and you have a team oh I've got 10 these 10 different security holes I need you to fix and like which one's the most important there's all waste right what you want to do is release work as a constraint becomes available you kind of want oh you've got time to work on this here's the next piece time to work on this here's the next piece you're not going to try and introduce multitasking into a system where you've got where the constraint is because you're just gonna waste time

sorry waste time the performance of the constraint actually goes down the more work you give it um a little bit on queuing Theory because I can't help myself but Little's law the length of a queue is arrival time by the processing time so yeah effective the bigger the backlog the longer the lead time so as you when you go back to that model before we have product team three that had eight weeks of work ahead of them the next thing you add this still is going to add another three weeks to that right so you're starting to lose I said the idea of agility because you've already given them other stuff to work on they're just getting

more and it's like well now that's two months away guys that's three months that's six months this is really get to the point where all of a sudden you know I when I work in Enterprise like I need this simple change cool that's three months away but it takes five minutes yeah yeah I know but it's going to take three months because we can't get to it till then so let's look at a slightly different model where we're trying to think about what our constraint is yeah so you will notice in the Middle with the product teams the colors have changed slightly so product team three only works on white and red the product Team 2 works on all colors

and product team three works on what ended up being the French flag not by any means or design just worked out that way but effectively what we're trying to do is look at well actually if we try and pick work that fits around our constraint can we actually just get more done so if again we model after one week after two weeks after three weeks we've actually got work done faster than we did before because it's actually flowed through the system but in the five weeks we've got two pieces of work now that doesn't sound superly impressive um but bear in mind on the early model we only had one if we extrapolate this out to 12 weeks the difference starts to

become quite Stark that in the constraint naive mode where we're just throwing work at the system that everyone had to do we only got three things done and the average lead time of any piece of work was four weeks if we go to this constraint aware we're actually stylizing the work we're doing basically we know where the constraint is we've got the same three items done plus an extra four and our lead time for actual making improvements has gone down to 1.7 weeks on average this is this is just like a step change in difference and no one worked any harder this wasn't hard this was just working smarter in by just modeling this way by

looking and I I'll get into I understand the model was simple right and it's not as if it's quite that simple like everyone has a number there and I'll get into that in a bit but just you can just see by thinking this way you're actually changing the way you're approaching the work relative to the system you're working within not just kind of throwing stuff on like this is the most priority well actually is it when you actually understand things a little bit better so hello by agile planning or professional fortune telling as I like to think of it so the most common one you'll see is called wsgf or where the weight is weighted shortest job first

which comes down to theoretically simple formula cost of delay over job size which you can kind of Go Value over effort a little bit more Nuance to it but not not for now this works okay in single team scenarios I'll come back to that product team that was kind of owning everything end-to-end this kind of works um you know uh planning poker aside which is horrific but you know it generally works when you're doing this in the multi-team scenario job size uh this isn't quite so simple anymore because if you look at the constraint aware here it's Retreat on the way here and the constraint away here it's drop size really necessarily changed that much

um from this from the team from the security perspective they have two weeks of work to do on each of these job items the what really means is what this really means is job size is impact to the constraint it's not actually amount of time it takes to get the work done in its entirety is purely the impact of that piece of work at the constraint of the system all the other work doesn't actually mean anything because the constraint is what's actually focusing the throughput in here again I don't have time to kind of model this to the level where you're gonna have to tell me on face value I apologize I apologize all right

the constraint is outside the team so as much as within security we go all the work harder will work harder we'll work harder it's not actually going to make any difference because really the thing that's going to make the difference is by you know focusing on how are you going to fit the work the fit your system or try and change the system and I don't have time to get into how you change the system here but what we did the change that I made in the constraint aware point is I actually recycling working that was outside away from the constraint had no impact on the constraint at all it was free effectively in the system

so that's why we got all the same work done but also extra bits because we actually moved stuff away from the constraint here um bear in mind the constraint can move the kind of default thing is you'll see product in three three week lead time or they've got to be the constraint right well if you actually have the backlog on the left actually the one weekly time the fastest team actually becomes a constraint because they have the most work to do um there is five weeks of work for them to do there's three product team three and two for product team one so the constraint can move and again I don't have time to get into kind of all the

modeling you need to do I'm trying to sell the point um as opposed to selling the solution um facing predictability so this is probably what I'm going to go through is what I think is probably the easiest and best bang for buck change or debate you can have from security into development kind of in the next week so when you're looking for predictability you want a stable system what do I mean by stable the system I showed you were stable and those those lead times they don't change I kind of went there it's going to be three weeks three weeks three weeks every time simple as we all know Security rates get constantly deprioritized because

everyone gets featured factored into Oblivion right so when this happens you start to see like this so product team one is two to five weeks product thing two is one to three weeks product team three it's three to nine weeks this system is inherently unstable now so when you're trying to actually plan out how you're going to plan your workload yeah good luck with that you've got no idea what's actually going to happen on these because you may give stuff you know do you take the averages of these is that representative do you take worst case should take best case how do you actually do it on this you actually can't um because it just varies and you'll see

you'll plan your workout you'll have it oh this is going to be amazing these start to go a little bit volatile and all of a sudden everything falls apart in the Heap and you've lost time the best thing you can do is feel uh from the project Potter book by Dr MC Kirsten who talks about what framework in general but I'm just going to talk about one piece which is flow distribution and again the people know me I'm sorry I'm bringing this up again um I have a bad habit but effectively this is a Consulting so Consulting friendly for hey we've got four buckets and all work goes into it it's amazing right so you can look at

all work a team will do um in terms of a product team buy four four different things you've got features risks defects and debt one of the most important thing here is each of those buckets is under the single control of a single stakeholder so feature your product owner is going to say hey these are the features you need to do feedback's going to cover me customers complaining about things or customer support risk is going to be security potentially governance but you know kind of one part of the business trying to manage risk I'm going to Define what goes in there and debt comes from the team and this is where you get the verbals on board we're going to

build Tech that time resolution and um resolution into how you work and they're all going to jump for joy because that's all they ever wanted in life but effectively it's this important thing about the risk bucket being singly prioritized by security because now you don't have to argue with everyone potentially for how your stuff gets prioritized um and I've got a Tim Ferriss paraphrasing quote here then what's the one decision I can make that prevents the next hundred if you can get this kind of modeling I'm going to go one more level on it in a second you get that in place you stop having to debate with every product team how your work fits into their prioritization scheme

because what you've done is you've defined a bottle up front and what you end up doing is you go okay and it doesn't have to be 20 you just need numbers here so we're going to go up every iteration whether that's a Sprint whether that's kanban measured over a period of time whether that's Pi like product increment however you do your iteration planning you know actually we're going to spend 20 of our time on risk security work so what we're going to do that's negotiated up front it's the product team I don't decide what goes in that bucket so risk decide what goes in that bucket and the relative priorities of those so I will just pick

off the top and work on that as it goes so all of a sudden I you don't have to argue why making I don't know configuration change for the necessary bucket is important I don't really care I don't have to know what you've said is that's the most important thing for me to work on and we'll pull that in and do that and you know you do the same for debt as well the idea is that the risk and debt buckets get filled up first and then you put features and defects on Top This is how you take like a product-centric mindset to it this is how you look after things over the Long Haul so stuff

doesn't start to fall apart over time and the allocations you use it or lose it right if security Can't Get Enough work to pull the 20 too for that iteration fantastic that spare capacity goes into features so it makes this Dynamic where Security is incentivized to fill those buckets but now we don't have to argument about it we just have to put it there and then you'll work on it that's so much better right this is where I kind of come back all the way to the start where I was talking about there's a focus on controls on detective controls over actually fixing things things the security posture was getting worse but we're able to

understand it was getting worse better that's not really a value in isolation right the problem and this is just kind of basic psychology is you naturally gravitate towards the easy work over the hard work well deploying new detective controls is something you earn you deliver you do by yourself trying to get devs to do things very frustratingly hard and difficult you're naturally going to tend away from this right you want to focus on value over points and yeah I I just have to bash story pointing every every slide deck I can't help myself um you want to focus on what the value of this is more detective controls it's not going to give value in and of themselves

because you need the corrective controls as well you need to go and fix things or prevent things so you kind of you can make the harder easier so you can do kind of what I've been suggesting through all of this where you actually go through and look at the corrective controls and look at how do we get the correction work happening to best fit the system that you can go along with that's that's ideal bit um is making it harder easier but the I I come back to this analogy of um sort of a American Professor was talking it had like a John it's filling up with rocks and then Pebbles and then sand and then

water right and then you ask everyone at the end like why do we do that it's like oh well then you were able to fill it up why do they do it in that order if I filled it with sand first I can't get the big rocks in anymore without focusing on what are the big things you need to do to provide the impact and you're feeling around that you don't just fill with the easy stuff and go hey we're doing amazing because we're getting so much done yeah but is it really driving the value you're trying to get to you know like really these are the bits they're important the top one that the

Cs screen controls that's more vanity metric than anything it doesn't actually really mean anything in isolation the problem or your North Star is more around the violations and the age of the violations you've got more problems than they're living for continually longer those are bad things that's what you need to go after if your work isn't helping attack of those you really need to be debating about whether that's the work you should be doing so after all of that and don't talk well some level of detail on all of that just to kind of look at is can we do something else that sounds like a lot of work right of modeling and trying to

figure out everything's going on now not everyone should be Google we're gonna have you up with that Google have a what they call large scale changes and I'm going to link the book that talks about this in about that at the end there's three books at the end um because again I can't help myself so what they've tried to do with lots of that removed multi multi-themed value streams here they've got it where the accountability for the service and the service Health that still belongs to the product team SRE that doesn't belong to security however security teams can actually make changes autonomously so you know we had um the guys giving a great talk about um

S1 earlier and that kind of stuff or the way Google work and they have a whole bunch of stuff around this that wraps it up that makes it possible but effectively if they have a vulnerable dependency security can go and change that and it will automatically proliferate out to all their all their services and systems they don't have to go and get the product team to actually do it but if something breaks in production that's not the security person's fault for breaking it it's the product team's testing quality engineering wasn't actually good enough to allow it they it should have been caught on the way it's not screws problem it's a product team's problem they need to go and fix it but

it actually enables security go and do things you can get away from this old multi-team problem because actually you're actually operating within yourselves again the problem is that you need kind of world leading development operations to make this happen and quality engineering all this kind of stuff right I just I like to put this in as kind of like a well what the what's the actual Zenith or where you could be versus what's actually realistic to do now the stuff I talked about today is stuff you can start looking at now especially flow distribution I have kind of a long list of clients I've introduced out over the years because I I do it every single time because I get

in and there's um like there's a giant backlog that's measured in years I remember one I joined in like the Monte Carlo simulation the backlog was five years okay okay okay we're gonna kill everything here and we're going to start again and we're going to look at it with much more kind of rigor around it right getting on the backlog doesn't mean anything absolutely means nothing um only it's an option that you've yet to exercise you exercise the option as soon as you pull it into work so you know the easiest thing in the world's got oh yeah it's on the backlog when someone asked can you just yes on the backlog doesn't doesn't actually mean

anything at all right because it doesn't mean it'll ever get worked on it just means I've made a note of it fantastic right um so I'm going to circle back around I'm actually running a little bit ahead of time so there might be time for a couple of questions if people have them um but just come right come right back around understanding how workflows is fundamental this is where I'm talking about this multi-team scenario it's understanding how security is done is arguably more important than how to do security because all you can do is optimize kind of how do you do security but if you understand how work actually gets done you can optimize the wrong

thing you're going to local Optimizer for Global optimizing you're going to end up in a worse problem than if you did nothing at all agile doesn't scale across teams the notion of planning and everything else and trying to look at things in isolation the way agile really does doesn't really go to scale to multi-team scenarios I was thinking about it earlier that if people have been worked in safe and you have Agile Release trains I'm pretty convinced that someone trying to introduce the trolley problem into agile where you're just like how can we actually take out as many teams at once by bringing them all together and the third one is you need to constraint aware so you need to start

thinking about how your work is done is it how it impacts constraints where your constraint is based on the work you're trying to do and then there's a whole lot of stuff you can do after that um which I do not have time I can't I haven't got time to help you fix the problems just maybe give you a little bit of a heuristic to where they actually are if you're more interested in the theory of constraints stuff standing on bits you can get online Club it's fantastic if the value streaming stuff was interesting project to product is a great place to start if the how good will do it with large-scale changes interesting software engineering at

Google is a great resource as well um but yeah for more kind of reading or if you've read those and you want something else to go into more detail uh believe me I have more um but those are those are the best place to get started but that's um all I have