Salman, Khwaja: Story of Implementation of SecDevOps in Fin Tech Organization and beyond

[Music]

[Music] yeah it's it's it's been a pleasure to be speaking at uh b sites cal uh calgary uh i have just one uh set of monitor with me so i might be switching the applications and that those switchings would be displayed right in front of you to see what's coming in chat so let's start um my name is salman khawaja i will be sharing with you the true story of application security scaling efforts in a fintech or how we developed devs how we how we implemented devsecops in that fintech um who am i i am a manager application security in dps pakistan in my day job and in my free time or let's say after the day job i am a certified

ethical hacker i am asana certified pro and i am running the ovas karachi chapter obasp is basically um an open source web application security project everyone i think everyone knows that in this community and i am running the pakistan karachi chapter of that community my profile links my social media links are right in front of you to contact me further so what is gps the company that i'm working for it's a fintech company which is providing solutions to banks of pakistan payment space and we have different proprietary products which enable banks to provide different services over different channels one could take a big um spectrum this is the the spectrum of our payments uh offering that we provide to

to banks of pakistan and to many different banks of um abroad i won't be going into details but you could see there are digital banking there are um mobile commerce there are digital wallets then they are only channel banking that means uh there is one uh state bank of of a country and and through that state bank many other different banks of the same same country are connected then we are doing something on robotic process automations as well then we are we are providing open api platform we are providing payment gateways as well which payment gateways are generally implemented in in one country so wherever you see you see electronic payments we are providing some solutions

of it and our our solutions are more dynamically uh implemented in an organization so that's the full spectrum of our payment products you could see uh different logos of our companies that that we are offering right now different clients that that we are doing we are we are giving you giving solutions to one link one link is is basically the central you could say central uh controlling bank in pakistan then we are giving solutions to central bank of uae that is united arab emirates to name a few and in in other words we are providing solutions to microfinance banks we are providing solutions to telecoms and fintech then we are providing solutions to exchange houses then we are providing

solutions to central banks and as a national payment processor three payment processors we are doing that and then we are doing commercial banks as well so the reason to tell you the uh what is my company is offering is the wide spectrum of the products that is being offered and what if a hack is is done on the same same platform what would happen and recently uh a hack was done in pakistan and if people are in security they would be um they would know what what is the hack that i'm talking about so today's talk agenda would be um the story of hack in major bank of pakistan and the problems that came that came

along with it um we saw the opportunity we started fixing it and then we implemented um seg devops or ssf in our organization um it's a story so sit back relax and enjoy so the hack was done in 2018 uh a few years back uh six million usd were lost in 23 minutes of cyber attack the hack was done in bank islami um we you you could see the url the date and everything else and the url itself says a lot about about the hack um this this bank was not not uh our client but yes it was running on the same technology stack so after that that hack was done um all the financial institutes in

pakistan they started thinking about how do we protect these things um even our regulator with that is state bank it has it it gave me a mandate to all the banks that you have to do variability assessments every three months regularly and the banks didn't use to do those um assessments so and on top of it regulator speed was slow with comparison to the zero day attacks coming from all sorts of attack vectors so that was the the challenge that we had to face uh basically when we what we realized is um there were three types of things that we need we need to address one is people courses and platforms and this is based

a general um triad that that has to be looked into for for people uh in my organization we have we have the capacity issue we have hundred developers to for those hundred developers who are churning out the payment application new features new new code they we had 30 keyway engineers now those two engineers are um functional keywords and to top it all off we have only four application security engineers so that's the ratio that that that we are we are going going towards and with that ratio we are just left with nothing but to do automation and manual testing accordingly or at the same time when we come to process uh as a fintech we have to deal with car payment

industry and as a card payment industry we are dealt with pci um pci councils council's pa dss guidelines uh pci council says that whenever car details are asked in a in a payment environment if they are stored if they are processed if they are transmitted you have to do certain um you have to do certain things in your application you have to encrypt the data you have to encrypt the data when it is in transit when it is at rest and when it is being being input in in the browser so those things are there um i i won't be going into details about pci pci standards but we are mandated to certify our products with pa dss that was uh

payment application data security standards and now the pl tss has been retired and new guideline is coming forward which is software security framework and that talks uh not only on card payments that generally talks about the whole software the whole payment um the whole security um around payment so in in terms of platform or technology we have to stay obsessed with the latest technology being being used and applied basic uh level of updation which has to be much faster than the slowest speed of financial institutions or let's say our regulators and we should be able to match the speed of zero-day threats coming towards us so that was the basic challenge that we that we were faced

so what we did we um we hired the the best people who were who were the top of the line uh secured security engineers we went to the community we i actually went to the community and hired bug bounty hunters which um who were actually doing bug bounties on on different applications uh i i discovered the internet bug crowd uh hacker one um every every place i could i could find those those those people who were doing bug bounties and then i i asked them if if they are willing to come and join my team on process aspect we went in and we studied all the process of pci all the price of apps all the process or vast

and then we started implementing them and on the platform thing we started doing automations which i would be telling you so first things first of asked we grabbed the asbs wsdg application security verification standard web security testing guide and then and then on mobile mobile and we grabbed a mobile application security verification start standard mobile security testing guide we grabbed obas cheat sheets and we actually went through all of them and started implementing the bits and pieces of these these standards into our manual testing and bits and pieces of these standards into our automated testing so if we talk about security pipeline um which is a secure dev pipeline it basically revolves around business requirements design development testing

implementation maintenance and if you implement different things in between different review cycles different security review cycles i should say that would be requirement review and doing the threat modeling followed by doing static analysis security testing and then dynamic analysis security testing some aspect of interactive analysis security testing iast comes over here as well then we started doing support documentation and then sdl so basically these are the things that in community people people do say we have to implement it but we took a different approach we took the reverse approach so what we did in reverse approach uh all our applications were being um um vulnerability assessment vulnerability assessment was being done to our to all

our applications so i was getting all the reports from many sorts of banks and everywhere else so what i did was i went in and put all those applications in one excel and i started categorizing them so you could see um the real issues that are being reported what i have done is i have changed the bank names and the product names of our of my company and i have basically redacted them but the issues are real the categorizations are real and uh these categorizations categorizations are real so what i did was i went and took the past uh six months last six months penetration testing reports i started writing the type the main issues that that was being

reported the categorization that that was being done and what we have to do in terms of fixing that that issue um now if i um foresee it in terms of uh fixes they there are only three types of fixes that that could be done in an in an application one could lead to server hardening the other could lead to port phase and the third one is the hybrid approach so i just wrote that this is server hardening this is code fixed and then i did a small pivot to see what's the the ratio behind it so if i if if i look at one one one product that is product six j we could say uh we could see that six

of them are of server hardening and only one is of cour code fix once again 80 20 ratio 85 14 so if i take any random um all the products and if i take any random uh bank so it is almost uh clear picture of uh 80 20 um the the result was was once again 80 20 um towards us that 80 were server configuration issues the 20 off of those issues were were related to code fixes so we started doing um we started documenting all the the stuff and uh whenever the the banks uh personnels were coming in asking me that hey do do i do have to you know harden the ias or i have to harden the apache what

should i be doing so basically i documented everything um in one nice document and these documents are basically what what ovas has done much in a much nicer way but i made made them available in in a user-friendly format to for my audience and this was this is still being being maintained with all that like when ova changes some stuff i comment change over here so that was done for maintenance space for system harding phase we took the uh cis hardening guidelines and we implemented it them in our web server and right now we are doing automated is hardening and apache hardening through powershell scripts and through batch bash scripts uh those scripts are also provided by

cis but we we have taken the cia's benchmarks um freely available from the uh from the ci's website and we made our own proprietary hardening guides for operating systems for database and for networks um the way in which we are operating our client usually uh does the the hardening for them so that was taken care from system hardening's perspective and then we went towards secure deployment we made our own uh deployment engine which deploys our proprietary software along with doing the hardening of these these things as well so from the aspect of next comes the aspect of das that is dynamic application security testing we started using ovaz zap we started using birth suite we did complete uh avast zap um

automation through ovas app apis and uh yes this tool has helped us a lot in doing uh an automated testing um so basically whenever an engagement comes we first do the automated testing through overstrap for like first uh first two days and then we have a basic level of assurance that that we have covered around 50 of the app of the application and then we do the manual uh security testing um so one more important thing that i would like to highlight is tps is uh developing software as a product like one product is deployed on several different banks so if i do the security testing of one product and if i save that that session the same

section could be replayed on different projects of the same product so that's one of the good things that i'm enjoying in my security arena the future plans would be to implement kali tools inside dust we have implemented test ssl we have implemented an app scan as well and we might be implementing automated reporting things as well so minimize the the reporting aspect of of security engineers on on sas uh static application security testing we implemented solar cube so narco is basically a software which uh which works on plethora of different programming languages ranging from c sharp from java from java service servlets from flutter from dart to php to many other languages and it provides code

reliability management technical debt and then it provides security hotspots and vulnerabilities as well so we implemented sonar cube in our source control and um yes it does provide a very good documentation about implementing in source control and whenever a developer checks in um a code it goes through this goes through sonar cube and it scans and then it provides a gateway rating about whether or not the code is good enough to go into the repo and if it if it doesn't go into the repo it goes back to review for the same developer or the senior senior developer who would take take a look at what what where did things go wrong then we implemented a vast dependency

check this is also a very very very good tool um it basically goes through the feeds of nist um nitrate and i'm i might be forgetting one thing but um wonderful database 1db one open vulnerable database it goes through these things and then it checks with your code whether or not the old components of your code or the third party components you have implemented in your code are retired or not or if they are having some security vulnerabilities it lists that as well and yes we have implemented in our security pipeline feature plans would be to implement a vast dependency track sonar type and to check credentials inside our code design for designing we have implemented

threat modeling through microsoft threat threat modeling in design time we might be introducing to you know to play um evil games or brainstorming sessions to to to really think about the new features that we had we are developing what could what could go wrong and how it could go go wrong in design time so and lastly we we did security you know security trainings as well we we uh provided overalls top 10 to developers we provided generalized peer lisa trainings to all users then we provide specialized trainings to all developers then we provided paradises in plain english to project managers or business developers business um bd personnel who were directly facing their customers my future plan is to run a private ctf

in our organization just like a ctf is being being run in in in this conference lastly we did uh trainings across the board which which is related to to general browsing habits and and all those things uh and then we implemented in lms as well so basically um all the trainings that i've been been providing it really is it doesn't make sense to you know do those trainings every three months and i'm a big fan of doing recordings so i could plan implement and execute a training but i would like to do it once and for good until the content gets old i'd like to that content to be replayed and it all the newcomers all the existing

people who who might not attend that training they could go through those those recordings as well so the lms was was a very good thing um that was done um not for only security things but all the trainings aspect of the whole organization um i would like to thank all the people um we i am really standing on this on the shoulders of giants um security community cis community over zap community burp suite committee uh community microsoft sdl network twitter friends of infosec appsec stack overflow github open source repositories and of course my my team uh my future plans again would be to implement ctf's dependency track oversam is a very good project and i'd like to

implement it in in from from management from top to to towards bottom sneak is is another very good very nice tool which which provides a developer you know to to to the extent of uh saying look you have you have implemented this third party tool and it is having this vulnerability and uh this version of that third party tool fixes that vulnerability it goes to one step ahead um with this i'd like to close the floor and i'd like to open open the floor for any further questions thank you

thank you

if there are any further questions i'd love to take it

okay the question did they have to report the incident to pci council um sharifah um i think the banks where our applications are running if they do go through a breach yes they are bound to report those incident to pci council um i do not know about the hack that i have mentioned the bank islami it might have been been reported that that's why it was you know circulated in the whole security industry but yes the banks and the the payment industry um they are bound to you know report these these things to to to private networks but reporting is generally done in terms of um not publicly it is being reported it is it is reported in a very

you could say subtle manner if you go through ovas top 10 the reason why ovas top 10 is called top 10 is is because ovas actually takes takes all the the beaches that are being done every two three years whatever the this the cycle is and then they publish the top 10 um top 10 of those breaches and the reason they say top 10 is because they say that look we have gone through the the beaches of whatever world the whole beaches that are being done in in over the world and then these are the top 10 threats that are that have been used in those features and if you protect your application on those top 10

you have basically applied 80 20 rule so so i guess there is a system called very sis if i'm not wrong that system is being used to to analyze all the threats and in terms of reporting um [Music] basic uh generalization it goes like there's a there's always a difference between between incidents being reported and incidents being happening and they are not being reported so that gap would always be there whether that that gap is is on on actual murder crimes crimes or cyber attacks i hope i have answered your question

most welcome

uh if you do have any further questions you could also leave it to my linkedin and twitter i am sharing it right now uh lots of great information here thanks salman if a small business needs to be pci compliant what would be your advice to those well malik peace going for pci compliance is is a very uh very tedious task in terms of experience in in pakistan but yes for those um uh the the process that that an organization goes through is basically for the betterment of um of that that organization and yes my advice would be to go for pci uh compliancy but you have to see um what you are dealing with if you are a

small organization and you are dealing with part payments that is you're accepting payments through cards through payment cards or you're doing some tokenization or what netflix or amazon prime does then yes um pci you are in scope of pci so if you are in scope of pci it would always be good to to gets pci certified because pci is certified basically you know tests you or audits you on all those three aspects of people platform and process so um and generally if you if you look at any any breach that is being done uh it is it is done on on each of these three three areas i hope i have answered you malik there

uh could you briefly compare pci dss and pci ssf definitely yes okay so pci dss is applied on all those organizations that are actually in production environment that are processing the card data um where the card data is being either transmitted either stored or is being processed so you could be a payment gateway where you're just processing your processing the card card payment details of merchants and and acquirers and custom customers you could be a merchant as well where you are just you know dealing with transmission of the the the car data or you could be an actual um bank where you are storing the card data so by by card data i mean um the card

basic the card um the details that that you're offering um the card details comes as a pan the the card number the expiry date and the cvv so cbv is is also called sensitive authentication data through which the card the internet um card shopping is being done and those things has to be have to be encrypted in in transit and they should not be stored cvv should not be stored so this is one one big big thing about pci dss so pci dss is is implemented on those organizations which are dealing with car data in a real production environment pci ssf that is applied on those organizations which are developing the software for for payment industry um you

could say my com company which is actually developing the software but they're developing the software and those software is not being used in production environment of my uh my production environment my company could be and could become a company as well which is dealing with the production environment but then pci dss would be applied on my company and if my company is developing though that that software pci ssf is applied on them i hope uh simon i have uh given you the the comparison of pci dss pci ssf

thank you that was my pleasure

uh i have given my linkedin and my twitter um the the username is the same so you could contact me over there if you would like like to once again i would like to thank besides calgary yep that's right uh cherryford it has all the details there are four levels of complex requirements for merchants depending on the number of transactions per year yeah um you're so right uh sharifah if i hope i'm pronouncing your name correctly yeah there are four four levels of compliance requirements and then there are multiple level of um what what what yeah you uh you ate thank you so you're you're absolutely right there are four levels of compliance requirements along with

with the level of the the car payments that you're dealing with once again if you are a small company dealing with cards uh if you want my recommendation go for pci compliance you would be safe um at least you would be safe you would become another name in in history and your your name won't be you know shared in a presentation like like mine or or any other person so it's it's good to be to be sharing your your knowledge on security community but if you become a history um then it it really gives you your company a bad reputation so i would definitely say yes you should um you should go for for for pci compliance

right um any other question

thank you darcy thanks a lot thank you everyone thanks a lot

um [Music] dorian should i leave right now i if this is my first time in calgary i do not know the customs and everything

thanks a lot thank you durian thanks a lot uh would this um presentation be shared publicly or am i at liberty to to share this presentation to my social media if needed thank you dorian thanks a lot thank you everyone thank you so much um for the audience who have been burying me who have asked me questions thanks a lot it was a pleasure to be at b-sides calgary