My container application has 100 vulnerabilities, now what?

hello everyone i am rajna vishwanathala and i'm a developer advocate at ibm and the title for my talk today is my container application has under vulnerabilities now what in this talk will understand what is what are containers what is application modernization and jump into what is container security and how to ensure container security in our docker applications so as part of enterprises id digital transformation applications have expanded from traditional bare metal servers to vms and now to containers this is driving containerized app deployment and development with that enterprises are faced with key challenge how do you ensure that the containerized applications are secure containers are standalone lightweight portable and they provide accelerated time to market and infrastructure

efficiency they have broad appeal because they allow users to easily package an application and its dependencies into a single image which can be promoted from development to test and to protection without change containers make it easy to ensure consistency across environments and multiple deployment targets like physical servers to vms to public or private clouds and this helps teams more easily develop and manage the applications that deliver the business value and because of this application modernization and migration to cloud is significantly increasing the container option and in the next two years 89 percentage of increase in container usage is expected and despite the high adoption rate uh container security continues to be a concern to id professionals 50

percentage of id professionals cited security as primary issue with their container strategy so what is cloud native so cloud native applications are a combination of automated devops containers and microservices cloud native applications are built using multiple independent microservices that are deployed in cloud environments this avoids the monolith scenario of maybe tangled code scope creep and over collaboration that ultimately releases impacts the release schedules and using automated devops and continuous delivery model software development teams can quickly and iteratively add new features to an application so if you look at the diagram it it pretty much explains how cloud native works and as a result we get a lightweight modular and highly automated approach container security so

devops isn't just about development and operation teams if you want to take full advantage of agility and responsiveness of the devops approach ie security also plays an integrated role in the entire cycle and with this approach there is a way for us to integrate devops and security for best dev ops practices times by putting a certain controls in place right starting with secure by design culture and practice this is a apsec program that starts with the people culture that lead to secure and best coding practices shift left design and development what this simply means is instead of thinking of security as an afterthought once the application is created maybe then i'll write a firewall and patch something

instead of thinking of it like that we have to shift left and incorporate security in every cycle of development enable core security controls which will implement core controls across data identity network and points and containers and continuous validation and offensive testing uh we'll have to make sure that the applications are continuously tested then finally ci cd delivery automation uh you could use something like red hat ansible to establish a robust devsecops tool chain while helping while helping automate deployment and development of security controls which are embedded into the deployment of new workloads so there are multiple reasons why security can be challenging for container environment and the following considerations are critical for securing a container environment speed of

container deployment can be a double-edged sword right the benefit of container technology is that they accelerate application development employment processes making security updates upgrades and vulnerability patching fast and easy yet the speed of the deployment can be a challenge too often there is not enough time for a quality assurance or security testing which means that companies need to go through manual processes of consistently checking that the latest versions of containers are the ones that are being used that all the code is patched and fully up to date and micro services adds complexity the fact that containers typically requires applications to be broken into smaller micro services resulting in increased data traffic and complex access controls

and more roads you have a lot lots and lots of services often too many ports so that means there are more doors to secure plus each door has less information about what's going on so it's harder to identify if someone's the bad guy and the news of public container repositories the traditional software development process build test deploy quickly becomes irrelevant in the age of containers in fact developers often grab ready to use and scan images from public repositories and throw them into the cloud and by the time someone notices that the containers are there they've done their job and are gone so the life cycle might be over by the time security team can actually

go in uh lack of isolation one of the nice things about virtual machines is that if an attacker takes over your vf the damage you can do is limited to that particular vm uh this is not so with the containers an attacker who compromises one container could potentially gain access to the others that are running on the same post and the like and the lack of strict isolation between containers is another uh inherent container security challenge lack of visibility right the portability makes containers more susceptible to in motion compromises many security proficience professionals don't have the right tools to monitor transient containers and micro services as they appear and disappear additionally most of the recent attacks started from within the

data center and more laterally due to the lack of visibility and control for east west traffic hackers exploit this oversight not addressing this traffic flows leaves the organizations vulnerable to potential security threats cloud native environments cloud native technologies such as container serverless and other microservices based technologies present real challenges to security teams especially when coupled with fast-paced devops processes cloud native environments require new approach to security and lack of skills and expertise can create that challenges so whether using whether you're using containers to build your application from ground up or porting your existing monolith apps to a containerized environment you need to realize that container environments introduce you to a unique set of security risks that

you should be ready to address from day one as you begin to pull your base images to build your containers until they're deployed and running in production environments and there are five major container risk areas starting with images right containers are built using either a parent or a base image images are useful for building containers because you can reuse the various components of image instead of building a container image from scratch however like any other piece of code images or dependencies could contain vulnerabilities registry for developers the quickest and easiest way to incorporate standard open source components into an application is to pull the appropriate container images from a public repository when you do this you are by necessity

trusting that the registry to maintain the least at the basic level of security regarding the contents of image orchestration the orchestration layer of container security focuses on the environment responsible for the day-to-day operation of containers there are multiple moving parts and orchestrators such as kubernetes this makes it more challenging to secure not only because of the number of components involved but the way in which those components interact right some communicator via api some others via host file system all potential points of entry into the orchestration environment must be addressed and then comes containers uh besides the entities such as apps and images they hold containers themselves are often considered vulnerable to security risks through container runtimes that

manage the containers a vulnerable runtime can expose all the containers it runs and also the host os to potential security risks and finally the host os itself the os that hosts your container environment is perhaps the most important layer of the stack because an attack uh this is an attack that compromises the host could provide intruders with access to everything else in the stack um container security risks are majorly categorized as compromise of a container image or compromise of container as a whole and then the other type is if someone's misusing a container attack or containers that the host os or other hosts among others so if you look at the threat vectors that the container environment

introduces us to starting from images right you have image vulnerabilities configuration defects embedded malware embedded clear text secrets and the use of untrusted images right uh when you are reporting your applications to a cloud strategy these these are certain things that you must address as far as container images are concerned then comes registry you might come across insecure connections to registries stale images in the registry insufficient authentication and authorization restrictions to the registry orchestration unbounded administrative act access unauthorized access poorly separated inter container network traffic mixing of workload sensitivity levels and orchestrator no trust right so these are certain set vectors that orchestration layer uh presents us with then comes container itself the vulnerabilities within the runtime

software unbounded network access from containers insecure container runtime configurations application vulnerabilities and raw containers right and finally comes the host os where this is the largest attack surface it has um starting from shared kernel host os components vulnerabilities improper user access rights and posters file file system tampering these are some of the things that could happen uh with host os right so container environment introduces us with uh whole different types of thread vectors so our containers are i mean our container services should cover all the risk areas right it should address major risk areas that we've discussed about starting with images application images are among key vulnerable areas for security risks in container environment these images can be outdated once

insecure versions of software applications carrying bugs those containing hidden malware and those are those which are loosely configured moreover these images often carry along authentication keys or certificates to address this risk we will have to design validate and then detect the configuration defects and define the policies to block the instantiation of unsecured images and then comes registry so when we are accessing from a poorly configured registry the access should happen through an encrypted and authentic authenticated connections also the registry should undergo continuous monitoring to ensure all stale images that offer scope to vulnerabilities are cleared so we should ensure that the audit processes implementation of secure connections and then develop policies for effective authentication then comes securing the orchestration

it recommends tight access control to cluster wide administrative accounts through effective authentication methods something like a multi-factor authentication just about uh password authentication segmenting the containers by purpose sensitivity thread posture provides higher defenses and defense in depth and overall container orchestration platforms should be configured to features that make them facilitate a secure environment for all apps that they run therefore it is recommended to implement rbac policies for improper api controls design and implementation of workload security zones right securing the container so the most common security issue with containers occurs when container run times that manage themselves contain vulnerabilities a vulnerable runtime can expose all containers it runs and also gives the host to us to

potential security risks given the flexibility that containers enjoy through dynamic ips over the network there is a need to identify network anomalies in such an environment and apply relevant filtering tools to address any possible vulnerabilities we should ensure the vulnerability management monitor and help control unbounded network access detect and fix the insecure container runtime configurations and finally the host os host os is a key to successful container environment given the fact that it lies at the lowest level of container architecture it is the more critical target to security threats a compromise of host os can lead to the compromise of all containers that are running on it hosters should undergo continuous scans fundamental abilities and any required

updates should be immediately applied so this is not just uh the level of container runtime it should also be done to the lower level components such as kernel which is which are key to secure container operation proper configuration is also important to security of host os it should be run as immutable infrastructure with no data and application level dependencies this makes hostwars highly reliable and effective in functioning so these are some of the security controls uh to address the container risks that we've discussed about so here are some of the best practices right when we are writing our code what these are some of the practices that we could follow right starting with white listing the dependencies

so it's a trade-off between convenience and security right so solution is that let that developers be developers create a dedicated team to white list approved set of goal and base images for developers and if you if you this is a typical example that you're seeing on the screen uh remove the bloat that's found in your typical vm instance right you need to whitelist the dependencies that you need for your process and then you remove the system libraries or tools that hyphen hackers sometimes love right and we have to use managed security services uh ship the security consistently across the environment so don't use the root user in the container right so it it prevents hackers from doing

anything that requires root with your container right it all if you if you don't use the root user in the container it also prevents hackers to from using apt-get to install additional tools that they might need to exploit further on in your container base so this is a typical example just put user x in your docker file and after running any commands that uh requires a root and then limit the container resources so when you are when you are running a container or a pod on an orchestration platform you can write a machine config and limit the container resources limit the resources by memory and cpu right when you are putting a certain limit it it prevents

any um dos attacks right so one way to prevent dos attacks is to place memory and cpu limits on the container and these are some of the uh security options that kubernetes environment offer so use a non-root user and give only read-only file system when you're if you're writing a pod or if you're creating a container for a certain image you can enable read-only access to the file system uh when you're writing the security context and finally disa disallow the privileged escalation right so in the specs that you're writing for your for instantiating a container image as a pod on orchestration platform all you can do is write a security context run as a non-root user

enable read-only access and allow the privileged escalation disallow the privileged creation so here are some of the other best practices when it comes to using kubernetes as orchestration engine you can use iam to configure role-based access policies you can manage the access uh to the kubernetes resources you can use our back to define part security policies so with part security policies what i mean is you can configure policies to authorize who can create and update parts in a kubernetes service then comes encrypting the data in kubernetes in versions point in certain versions of kubernetes the data in etcd is stored on local disk of a master node hence it would require us to encrypt the data

that's being stored right so we will have to make sure that the data is encrypted and then enable tls on your ingress so that you can encrypt the data that is going into your apps and then implement network policies using calco and you can protect the app config with kubernetes secrets instead of writing your application configurations uh directly in your code hard coded you can write a secret uh for kubernetes or any orchestration platform through which you can protect your app config and finally sdo issue is a service mesh that provides a uniform way to connect manage and secure your micro services you so what you can do is you can use this to establish a mutual tls among the

microservices that are running running in your environment securing container workloads so this is how enterprises most of the times work you have a typical pipeline which mostly consists of build dev test app scan pre-prod prod etc a pipeline that typical that typically is a setup and this is this is this is what uh this is the kind of devops pipelines that the enterprises are using every time a new change is made instead of actually performing each of these steps manually we can automate it by setting up a delivery pipeline so in delivery pipeline a build stage is where you're just fetching the code from fetching the code changes from the source repository and you're building

the code and then you once the build stage is passed it automatically starts at dev stage dev stage has again certain jobs where you're trying to deploy the application i also perform some functional tests once the development stage is done automatically the test stage is triggered uh once the test stage is done apps can dynamic prod so these are some of the things that happen uh typically right so some of the benefits that it of it offers is delivery pipelines are very easy to set up and they enable continue continuous integration continuous integra testing continuous deployment in your uh application and one one one more advantages you can deliver your application to multiple cloud platforms or multiple

deployment targets anything so this is where we have to understand uh so when transferring data among network systems trust is a central concern in particular when communicating over untrusted medium such as internet it is critical to ensure the integrity of the publisher of the all data a system operates on so you you use docker engine to push and pull images to a public or private registry so what content trust is a concept which gives you the ability to verify both the integrity and integrity of the publisher of all the data that is received from a registry over any channel docker content trust provides the ability to use digital signatures for data that is sent to and received from

remote docker registers these signatures allow client-side or runtime verification of the integrity and the publisher of specific image tags through dct image publishers can sign their images and image consumers can ensure that the images that they pull are signed so publishers could be individuals or organizations or automated software supply chains signing the content as part of their release process so when a publisher who is using docker content trust pushes an image to a remote registry docker engine locally signs the image image with publisher's private key and when the user pulls this image docker engine uses publisher's public key to verify that the image is exactly what the publisher has created so it ensures that the image wasn't

tampered with and that it is up to date a dockercon interest provides a stock strong cryptographic guarantees over what the code is and what versions of software are running in your infrastructure uh it integrates the update framework into docker using notary notary is an open source tool uh that provides trust over any content so here are some of the use cases that uh that it provides starting with image forgery right so let's say uh publisher has published uh his his image a certain private key is associated with the image and if the if a hacker is trying to tamper this doc docker image the key the key has will be the key since the key is tagged once the

publisher has once a hacker has tampered with the image when the user is trying to pull the image he will check for the digital signature which is available on the image and since the image has been tampered with he will not get up i mean any anybody who's trying to pull the image will not get a validation on the digital signature which the image has so you there is no way um to forge the image using doc if the if if docker contrast is in place image 4g can be detected replay attacks a replay attack occurs when the cyber criminal tries to eavesdrop on a secure network communication um intercepts it and then fraudulently delays or resends it to misdirect the

receiver into doing what the hacker wants the add-in danger of free play attacks is that the hacker doesn't even need advanced skill to decrypt a message after capturing it from the network the attack could be successful simply by resending the whole thing right uh the way it happen the way dct avoids this is every like i said every image has a tagging key and if someone is trying to perform a replay attack on it you once once a user is trying to pull this particular image you'll see that the digital signature is expired over the docker image so if a hacker tries to delay it or tries to resend it after certain time after each

dropping on a certain network by looking at the expired digital signature over the docker image the user who is trying to pull the image knows that it's no longer valid and then finally compromised case in in the cases of compromised keys the hacker simply tries to mis hacker simply tries to replace the keys with something else in this case if a publisher is publishing an image with a private key and let's say the hacker is trying to tag it with a compromised key what happens is when the user is trying to pull it with a public key the mismatch is easily recognized and the use and the and the keys and the fact that the keys are

compromised is easily noticed so that's docker content trust one of the practices that can be followed when you're using docker and then comes vulnerability scanning right so we face attack vectors like os vulnerabilities uh which could be vulnerable kernel versions or in patch libraries and then application weaknesses something like xss buffer overloading or sql injection so with containerization every layer in the container is just one link in the chain the containers that you produce can be used as base for another container and so on with the link in the chain susceptible to the attack vectors and threads so that's when we use vulnerability scanning when containers are loaded into the container registry service or are registering any registry service

they should automatically be scanned for vulnerable vulnerabilities and once this vulnerability scanning is done it should generate a it will generate something called assessment report and this assessment report is something that consists of cves the common vulnerabilities and exposures that are identified for the image right so uh whether it's exempt the cva id the security notes information and the summary of the vulnerability that has been detected so this is a typical process that happens and these are some of the issues that are being that that will be captured by using va scanning so this is how it actually looks like uh so if you see once the image is scanned for one or a scan with

vulnerability scanners something like openscape vulnerability ids cvs are captured and once the cv is captured you also have to understand which package has been affected uh once you identify which package is like identified typical va scanners usually advise us on how to resolve it so it's not limited to cvs it va scanners usually identify configuration issues too uh using these configuration issues you can identify and accordingly address the patch i write a patch to it and then comes docker image signing right so in the same pipeline that you are setting up one way to do is uh automate the entire process using tool chains with the with the same for this for an application you can write it to to

separate tool chain one which focuses mostly on image signing so if you see there are two this is example that it could look like there are two stages that are set up init dockercon interest stage starts by the city there it will create the dct keys right once the dct keys are created uh this stage also allocates the keys and provisions them to be the delegation keys that are used and to sign the docker images in future and the second step second stage that you're seeing here is to set up iks runtime enforcement in the deployment cluster so what this stage does it does is it creates image enforcement policy that is applied on the kubernetes cluster and

the stage also specifies that an image can't be deployed until the signatures are found and validated so you can control where images are deployed from enforce va policies vulnerability advisor policies and then ensure content trust is properly applied on the images if the image does not meet your policy requirements the pod is not deployed to your cluster or updated or maintained or anything so cisc container image enforcement policy retrieves the information about about image trust and vulnerabilities from the registry you can choose to block or allow the deployment of images that are stored in other registries so these are these are some of the steps instead of having your pipeline mostly focusing on building and containerizing and

deploying these are some of the stages that could ensure the security policies are in place you have build containerize validate and deploy stages in your devops two chains writing unique tests in the build phase would make it easy for us to test uh that that the application build the code is properly being built and in the containerized phase you also build uh signed docker images instead of building just docker images so that you ensure that the images are tagged and signed for for us to validate before instantiating them and the next stage comes which is validate stage this is where you're checking vulnerabilities this the check vulnerability stage is a stage where you're checking for

if the image has any cvs or configuration issues if they are not there if the stage passes move on to the next job where you sign the image for validation uh once the once this is done you go to a deploy phase where you do a pre-check of signatures and once the signatures are confirmed you instantiate the container image which would mean that you're instantiating a pod on your orchestrator and apart from this your tool change must contain vulnerability advisor software docker image signing source control issue tracking online editing and then finally deployment or kubernetes service or orchestration service that you're using so if you look at the stage where we're talking about vulnerability scanning it

is a simple assessment that should give our overall status of the image based on the vulnerabilities or any configuration issues that are that are present in the image if no nothing is present that's when you go for that's when you approve the image and instantiate the image so if the image has uh if the image has indeed been found with vulnerabilities or configuration issues just make sure that this make sure that your uh tool chain will exempt it out and accordingly stop stop the instantiate instantiation of the container as a part so that's my talk thank you