DevSecOps: The Good, the Bad and the Ugly

hi everyone my name is nevada murphy and i'm here to talk about tessa cops the good bad and the ugly um about me first um i'm a senior security consultant at synopsis based on boston massachusetts i started in security after i got inspired by movie senses from sitcoms and i started with security operations where i worked on network incident and application security moved on to sas pentas scr one bitty triaging risk assessments and finally into decicops i've been working in this field for around like three years index course actually um and then while i'm not working i like to travel read books and take photos and then when i can so what am i going

to talk about today all about their cops i would like to first introduce you to the whole world of dessicops and then explain the need go on to move on to the good bad and the ugly parts of it as in as you can see by the name it's an amalgamation of development activities operation support and security checks it is a cultural mindset altogether um it requires coordination and sync between teams involved in the sdlc process this synergy has is obviously helped due to automation desk ops is not a quick fix or a temporary solution it is the long-term implementation you play to gain long-term goals in ensuring uh an organization is able to achieve

and maintain secure stlc practices it requires development teams to follow a standard stlc process um such as waterfall or agile and ensure that they don't break the process um well one is easily able to detect this as well and while not going too much into details developers ensure they maintain versions of their code and follow a pure review process um before the code is moved to various environments segregation of duties is a must uh to avoid one person from one team or one team to have full control on how things are updated in the you know code or environment this means a separate team is responsible for development a different team is for testing and a

different team for deployment etc operations team is also is responsible in supporting the operating environment um the entire development process where it runs which includes maintaining updating the operating environment defining and implementing a deployment process by logging every detail of the same security team is responsible in identifying any vulnerabilities being introduced through this process and ensuring it doesn't reach production if it does read production there's a clear trailer why it could have moved over there tessa cops or even devops insist on traceability so that it's easier to identify what change resulted in a problem or a security incident or like or an incident itself um while changes in the code is traditionally what starts the process

there's a cough starts made before that and it requires other activities to complement this process design reviews and post-production monitoring for example um tesla cops is also a lot of moving parts like the different wheels that run behind a block to ensure it reflect uh to ensure that it reflects accurate time deficit cops as mentioned earlier is development security and operations each stage of sdlc requires some kind of security review but these reviews don't end at the end of the sdlc security checks are done uh before and even after that because devsecops is not a process but a cultural mindset that needs to be adopted and encouraged uh hence every activity related to development even if it's not

actual development must work hand in hand with security and operations you start off with training developers in writing secure code you surround development teams with um you know secure development standards and frameworks to work on and have security champions supporting teams in their secure development efforts um champions are chosen because they are part of the development teams and have a keen interest in security this way they can evangelize about security within their teams what follows are several security reviews um along with the path starting with risk analysis uh architectural reviews threat modeling um which come into picture and then during design and planning stage um this is followed by source code reviews and then either manually or using a

various id plug-ins that can help you run a quick scan and then sas sca das i asked scans that run during coding and testing phase these can be repeated at every stage of development such as a bills test and release date as applicable then we have penetration testing done on post production but not on the production environment to identify any vulnerabilities found using any vulnerabilities that were not found using the previous methods you also continuously monitor your production uh environment for threats using tools such as rash and inviting external researchers to test your critical applications by offering a bug bounty uh program all of these offers with various methods of testing your application for resilience again security

decicos insist the integration of all of these along with functional testing methodologies to test both your surface resilience against bugs as well as security we talked a lot about development activities but deficiox is not just limited to that the third part of this whole thing that we often ignore is ops or operations all operational activities need to be reviewed one can now build up infrastructure as food this makes it easier to stand up or stand down um environments easily and consistently we now have various options available to run our software um app our application the best way to observe and watch um an application work and identify security issues in its truest form is while it's

it is running in its environment which means you will also need to run checks and identify for security issues within the environment itself some of these issues would directly impact your application hence it is important to secure your environment in which the application is hosted in all right so does this mean that this can be done with just one tool absolutely not while there are several tools out there that offer mixed services there's no one-stop-shop tool um that does everything for you you have to tools that help you run the whole process and tools dedicated to identifying vulnerabilities using different methodologies um one thing that i observed is that now a lot of companies that used to

exclusively offer offer one tool now for you know something else within this like uh with a different methodology um like an assassin with sc or sda with tests uh and then another thing that i have also observed now is that a lot of uh scm 2 companies that offer stm tools um have also included sas and sda capabilities in their environment uh you have tools from open source as well as uh property know that no one cool fits all the environments and in some cases no no one tool fits the same in the company itself in my opinion we it's not how it should be uh the second the latter part but it is very rare um

but possible i think in such cases organizations should uh run a bunch of pocs and see what fits the environment best i think there will be a difference of opinion among this among different uh you know uh different implementers and security folks in general um which i would definitely like to talk about um desiccops tools are not just uh restricted to doing just scans but also to do with others such as reporting um defect tracking management uh environment building tools etc also please note that it is not restricted to just these two tools that you're seeing on your screen right now even a simple thing such as a script uh written either in um shell or

powershell or even python or for various capabilities there's a large world out there plus a world out there and choose wisely so why does everyone you know talk about this um i don't know about you guys uh you're all out here but i when i see the whole desk implementation happening come alive and watch as the changes trigger jobs which in turn triggers scans which in turn triggers generation metrics and you know reports issues as applicable and make or break the deployment it is really magical for me um watching a lot of work being done in minutes or even sometimes even seconds is mind-blowing uh there's a course definitely helps team up team scales up big time we all know in

general security teams have always dealt with a resource crunch but that um doesn't stop bad actors from uh you know taking advantage of a vulnerability that's available out there with more and more applications moving away from a monolithic structure to a more you know um multifaceted multi-pronged agile method security teams need tools to catch up and back up and make sure they have control over what's being released in production plus there are only 24 hours in a day minus the eight hours uh you know you know sleeping which is important um desi cubs lets you crunch a lot of work in a really short time um speeding things up cutting down on delays and scalability

um are the biggest advantages of desktops with global teams across different time zones organizations are increasingly seeing the need to for processes and frameworks that helps get teams together and uh help and reduce dependence and helping you know while achieving the goals through the same process what i found in several organizations is that they have different teams working as the independent verticals or businesses having their own independent silos how does one work in such a fragmented environment how does one managed to get some consistency while still allowing for leeway into each team or silence which they require because it process demands how do you get teens working on a goal they all agree upon but unable to change

the entire development process can we do a gradual change instead of a full-blown change um deficit cops is able to accommodate all of these requirements and maybe more no matter at what stage the organization is in terms of security uh or how fragmented um or centralized the organization is uh dessicops lets you initiate and implement security activities and lets you adapt um to different functional teams you may have like often heard this phrase security is not one person's responsibility it is everyone's responsibility um jessica's makes everyone participate in the process um and practices this very same rule developers app managers ops team security teams reviewers testers are all involved in this process one of the biggest wins of desktops is

traceability since processes are interconnected and automated it is extremely easy to determine what change resulted in what issue our problem in terms of idea tickets this would be which change change ticket resulted in a problem ticket or an incident ticket uh this level of traceability also makes it easier for people to uh make it makes it easier to make people accountable and responsible for their um changes with additional monitoring it induces developers to be more careful while writing code and writing secure code actually which prevents the pipeline from breaking in the current project that i'm working an email gets sent out you know once the entire uh development uh to the entire development team once the job completes

if the it notifies the team whether you know if the the application has met the security requirements or not and in both cases it lists down the the name of the developer who checked in the changes at what time and what were the changes out there now you don't want to know that you don't want the entire world to know how bad you're like right and you're writing the code right yeah you saw this right time and again i have seen client expectations out there that i first thought what were you thinking when you said this um but then the next moment in my mind you know it starts turning on how can i make this possible

um yes tessa cox can make a lot of those father expectations into reality but it requires a lot of time and effort um to and a lot of other factors that will uh work it out before it actually happens unlike other methods a full desk of implementation that's a implementation takes time and a lot of patience is required so what are the good things about devsecops the first and obvious answer is automation automatically triggered bills scans deployments evaluations approvals automation works for all now since these tasks are automated security teams can focus on their important activities rather than just the operations part of it all as an example with 700 apps in scope and counting uh

it would have been very difficult for a security team of uh let's say four uh to be able to monitor regular releases if these were all manually done the time taken to do this would have been really long however with automation the team can now balance out the work a bit uh recently the team received a request uh to onboard and scan 30 micro services two days right just two days before uh production where the automation um automation piece we were able to complete this in two hours not just that right like 30 news microservices onboarded on two tools we ran the scans um evaluated it and triaged the scan results all done in two hours that is the power

of automation earlier i mentioned global teams uh being the norm and with work from home um we also have other challenges as well as distance team members um less accounts helps to create a framework which reduces the dependencies to do tasks and activities uh a lot of organizations have been spread across different time zones such as asia uk emea uh in my client's case it is like the while the security team is in u.s majority of the developers are in india and china the security team does not have the budget nor the bandwidth to support all time zones lesser cops helps them to support developers in india by monitoring port being checked in late esp or early morning est

um we are we are also able to create portals through which uh developers could run on-demand scans in some cases for on-prem tools uh another big advantage that i really liked about the standardization is that um about this is standardization sorry instead of it being a chaotic world out there it is more standardized which makes it way more easier to pay through the steps in this process with standard standardization it is easier to scale up the process and make updates and additions as applicable anything more of course there is acceleration speeding things up agility um another pillar of the seconds fast-paced world needs fast-paced solutions does the cops offers that with manual steps and dependency cut out the whole

process start to finish gets done faster and earlier in the earlier example all the all of those 38 micro services uh got done in 20 uh two hours that is the speed that our teams are looking for with less than a week available for the whole sdlc cycle uh security gets very less time to run its activities with a lot of security team tools today have improved in terms of on time taken to scan and provide enhanced capabilities to customize the scan and select checks to run so as to cut down on the scan time this way optimizing it speed speed speed is extremely important in desktops um traceability with all things connected one can actually connect a change

request to a problem ticket or an incident ticket or even a security incident um an example would be where an organization had to deal with a bridge studio vulnerability found um and one of the turns and turns out there was this was actually interviewed as part of the process and uh uh we it went through your normal dell circles sas tools were able to find it and it was reported and then an exception was granted the respect of the application t application team had taken responsibility for it because they didn't consider it as critical or they would be fixing it in the next release um so uh this way they were able to trace it down that this is something

that the organization didn't our rather the application team did not realize the impact of this um the risk that they accepted um now uh so next is scalability one of the reasons why desktops is so popular is that it lets security teams scale up really high with limited bandwidth um just again like the same example four members can you know uh in a team are now still able to provide support to 700 plus applications that would have not been possible without the current automation piece in place another client that i work for uh they were able to drive remediation efforts for around 600 plus applications and change their blocker uh you know requirement uh from critical to

high to mediums as well flexibility let's say the organization wants to transition from one tool and to another i think of that think of that to be done for thousand plus applications the whole scene looks like a nightmare but with their circles it is actually possible um to do this in a much more faced manner and easier less stressful way uh than what it would have been without that so if it's so good is there anything bad about it uh well we haven't reached the top here here in devsecops because there are so many pieces involved it is not necessary it will work as one happy big happy group out there every organization has to deal with

prioritizing their activities and desiccops may not be on everyone's priority list even though they do understand the advantages of it uh in some cases it could be that the devops team itself doesn't want to introduce security into the process because it does require a lot of uh coordination a lot of dependencies as such um the other reason would be that not everyone is the same you have apps and then you have apps that use legacy tools or framework time and again you will find teams that uh that are supporting legacy because they have no there have been no plan to transition them yet example while majority of your application would be using visual studio 2019 there would be a few apps out there

that would be using vs 2012 or even 2010 for that matter or for that matter sharepoint apps you know using the old framework even now we do find um some security tools that may not integrate that easily with the other tools um you know with the whole process or out here you have to create a layer of abstraction and so as to make it use of it in cyclops process for example burp until recently did not have a ci plugin um hence it wasn't easy to integrate a bob scan into the automated process then we have those legacy applications apps that organizations can do continue to use it uh continue to use it's critical to functionality

but it was written ages ago and as there can be no changes done on it or no one actually wants the changes in it um assigning resources to uh like automate such apps uh does not make sense or in some cases it is partially part of their circumstances but like you may get the app uh being evaluated using the standard process but the since there are no changes happening it does not make sense to you know add some web hooks or monitoring for not for any changes because that will never happen but the same the same features will utilize the scm or ci server resources as well while evaluating you need your an automated process you need to check

what is the area of getting it done you need to take baby steps with desktops it is not a quick fix at all it requires patience and tenacity to implement it all any lesser cost implementation takes a minimum of a year for sure anything less than that is incomplete delta cops involves a lot of planning um design and designing before you actually start with the solution um you need to first identify the gaps in your current process followed by then to see what tools would be required and how do they support the process you are implementing then comes coordinating with different teams to get their buy-in and then followed by all the teams implementing the required changes

this does not happen overnight if you were to make changes on to your process it affects all involved in the process and all applications following the process and what someone say we will go and make this big bank change with this one if all of your applications are being scanned using a common set of libraries unless you put in specific conditions any changes in these libraries will impact all of your apps getting a new application um added to the process may take long depending on um certain factors but uh for example i find that dotnet applications usually take a lot more to a time to be onboarded because you need to ensure that the bill

they like the bill correctly i would say 30 of my time actually spent in resolving issues coming out of doctrinals um because visual studio tends to hide a lot of build errors and provides dependencies on runtime which is not done by ms built countless of times i've encountered scenarios where the app team has built the application using visual studio and checked in but since our automated process uses ms build command line it breaks due to a variety of reasons incorrect direct directory structure missing dependencies in direct dependencies etc and then the ci tool itself is not up to market times a lot of organizations start their ci process using jenkins and uh because well it's free and open

source and pretty popular as well um but you have other ci tools in the market uh for example git lab team city etc um but if you're using gen games you are probably aware of the fact it is buggy like a lot uh i've lost a lot of time and you know a lot have been stressed out with the number of issues that keep on coming up with jenkins um in fact one time i lost an entire day or actually two of that because of her i had to reward and upgrade it's pretty recent um but that doesn't mean it's not useful yes it still has its um pros and cons of it i haven't used team today and gitlab as

much so i cannot comment on the bugs or the you know cons of those tools as of right finally the worst part developers themselves can be deviant little tricksters though it is a very tiny fraction of it there have been times when i have had to escalate to it to the manager um the other thing is that tools do not have the level of maturity to do everything that is needed every tool has its limitation and what on what it can and cannot do especially in an automated process though that area is declining to give you an example jenkins can not um probably allow conditional parameterization maybe there are some plugins which offer some workarounds but not that actual

requirement and finally the biggest headache any security team that has to deal with is false positives i have this one app that keeps on throwing up 360 plus critical every time you scan it but i have no idea why and we have to determine that um but i don't um if you have any good ideas out there um on how to get rid of false positives i would definitely like to hear that out yes and yes uh we still need difficulties it really makes work the most simple and streamlined um i hope uh this whole presentation actually helps you understand the nitty grittiness of the seconds uh there's still a lot more to talk about in terms of opsec

which is part of the dev circums but then that will be a whole new topic itself um now you can connect with me on linkedin and twitter trying to use it a bit more but let me know if you have any questions thank you okay