Up next in 10
Container Security with CI / CD Pipelines
14K views
Oct 30, 2023
"Here we talk about deploying large scale container & container orchestration toolsets, such as Docker & Kubernetes, and their respective artifacts into cloud environments using CI / CD Pipelines. We will also cover how to secure these containerized workloads when deploying into cloud environments We will use Google Cloud, Cloud Build, ArgoCD, and Aquasec to design our architecture here" Software Architecture Conference 2023 (https://softwarearchitecture.live/) #SoftwareArchitectureConf23 #CSharpTV #csharpcorner
View Video Transcript
0:00
This presentation will be on container security and CICD Pipelines
0:06
So we'll go into a little bit about what container security is, what CICD Pipelines is
0:12
how can we incorporate CICD pipelines within container security, and how this will work with Azure Pipelines
0:21
And I'll give a small demo that will explain how we can use Azure Pipelines and Azure Dependines
0:30
DevOps to better, you know, enhance our container security. So just before we get into any further technical information or technical knowledge, a little
0:43
bit about myself. So I'm the CEO of Viral Software Solutions Inc
0:49
This is a consulting agency specialized in DevOps engineering. We have done cloud migrations and data engineering
0:56
I myself, I'm a Google Cloud architect. So I have come from a Bloomberg background and I've come from a Cloudera background
1:05
So I've worked in many different verticals. I've touched a little bit of pure software engineering, data engineering, DevOps engineering
1:15
And in terms of my hobbies, I learned to play music for several years
1:20
If anyone is familiar in the audience with what a tabla is, I have learned to play tabla for eight
1:26
years. So that's that's my hobby. So let's get a little bit into what is container security
1:34
So container security containers are just like any other cloud asset or any asset in your organization
1:41
whether that be a virtual machine or an on-prem machine, right? The idea is that you will need
1:48
to maintain the libraries on that machine and ensure that they are patched and that the libraries are up to
1:55
date and that there are no security vulnerabilities across any of your assets, right
2:03
So this also applies to containers. And your global security office, I'm sure, agrees
2:10
And it's important that your containers are also compliant with the policies that your
2:15
global security office holds. So in order to do so, we need certain tooling that we need to put in place with regards
2:23
to containers. image scanning tools, the ability to scan live containers, the ability to scan Kubernetes
2:33
pods in the case that containers are deployed or used within a Kubernetes environment
2:37
and so on and so forth, right? So there's a plethora of tools that will need to be used
2:44
Today we will specifically be discussing how we can scan images. So in this particular case
2:53
I understand that there's many different ways OCI compliant images, we call them, that a container can use
3:00
In this case, we're talking purely about a Docker image, standard Docker image that can be used
3:08
and that you are likely using in your environment. So that Docker image can contain many vulnerabilities
3:15
For example, a Docker image can contain OS libraries, can contain Python packages, NPM packages
3:21
can contain many packages that have vulnerabilities baked in within them. Those packages can also have dependencies
3:30
and those dependencies can also have further vulnerabilities. So depending on how your global security office has the underlying written rules established
3:43
these vulnerabilities may be an issue. So, for example, if your global security office says a low or a medium
3:51
vulnerability that's specified by the NVD, the national vulnerabilities database. If a low or a medium vulnerability is, they might consider that to be an okay vulnerability
4:04
if a library were to have that vulnerability type. However, a high or a critical may be vulnerability types that they will not allow into the
4:14
environment, right? And when it comes to container security, what we should really focus on is that they will not allow
4:21
on is the idea of container registries. So we do not want any container images to be stored in a container registry unless the image
4:34
has been scanned for any vulnerabilities. And the idea is that we need some form of a CICD pipeline or an automated pipeline
4:42
which will scan the images and ensure that there's proper security checks before the image
4:49
then goes into the container registry that you can host on Azure
4:53
Azure Container Registry, you can host on Google Container Registry. There's many different locations you can host your image
5:03
So let's move on to the next slide. So for today's talk, we will be describing Sneak
5:11
Sneak is a specific type of container security tool. Many of you may know Sneak as the established, integrated
5:19
integrated security tool that comes directly with Docker Hub. So if you've ever used Docker Hub
5:24
you will see Sneak integrated into it by default However Sneak does many other things besides just scanning images on Docker Hub
5:35
It will also scan your code. So if you were to write Java code, C code, so on and so forth
5:41
Sneak also performs scanning of that code as well. And the idea is that we want to be able to establish a pipeline where Sneak does the scanning of our images
5:51
And those images are then vetted before we push them to our registry
5:56
And just to be fair, if you don't prefer to use sneak, you can use SISTIG, you can use AquaSec, you can use Verac
6:02
You know, I've used Aquasek in the past as well. And there are all tools, different tools that you can use in order to perform similar procedures
6:15
There's differences between one or the other, but that's not what this talk is about. So before we move on to the next slide, let's talk a little bit about how this would be useful for your organization
6:28
So for some of our clients, what we have seen is they have large GitHub repositories with many Docker images and developers are constantly building images
6:40
They're pushing images to put out a pull request. Pull request gets approved
6:46
The code looks fine. However, the Docker image itself and the libraries are not vetted, right
6:51
So now what happens is you have these large repositories of images that are Docker files that you will need that you haven't vetted
7:00
and your global security office will then come and contact you and say, hey, you know
7:06
we actually need to start implementing a policy where we need to scan these images
7:11
And that's where, of course, your team now has to start incorporating better security
7:16
practices in place. One common place that we see or one common theme that we see is folks like to
7:25
build out GitHub actions CICD pipelines or they like to build a code build AWS code build CICD pipeline
7:32
or an Azure Azure DevOps CICD pipeline which is triggered every single time you were to write
7:39
code and push the code to the GitHub repository. Once that code is pushed to the repository, the
7:46
pipeline will then kick off. And from there, the scan will then take place, which will then check
7:53
to see if there's certain critical or high vulnerabilities, which will then either allow
7:58
you to push the code to the registry. And then from then on, there's much, we can, we can
8:05
then create an even more extensive pipeline that integrates into Kubernetes and whatnot. But that's the
8:11
idea. So going back to Sneak, so really there's two, two main commands that we have used for our pipeline
8:20
The first is with a Sneak container monitor. So the idea is simple
8:25
The idea is just that Sneak is going to scan your image and in the case that there are any vulnerabilities
8:32
found, or even if there are no vulnerabilities found, Sneak will allow you to display those
8:38
vulnerabilities on a dashboard. And that dashboard can then be presented to leadership, presented to other folks within
8:44
your organization so that they understand, you know, what the current lay of the land is in terms of security
8:51
And the same applies for the sneak container test command. The idea with the sneak container test command is you simply have the same functionality of
9:03
sneak container monitor, except instead of publishing the vulnerabilities to, you know, a dashboard
9:09
you would instead just generate a report locally on your laptop or wherever the machine is
9:14
and the idea is that this sneak container test command will return a non-zero exit code in the case that there are any errors
9:23
And you can use this non-zero exit code inside of your CICD pipeline to determine whether there are vulnerabilities in your image or not
9:32
In the case that there are not vulnerabilities, then the command will return a zero exit code
9:38
In the case that there are vulnerabilities, you will return a non-zero exit code
9:42
So that's the idea. So let's talk about the actual image pipeline or I guess the CICD pipeline that will get triggered on commits and the CICD pipeline that will get triggered on merges
10:01
So for example, let's say you were to put out a pull request and you were to add a commit to that existing pull request, right
10:09
So you put out a pull request with one commit. Once you put out that pull request with one commit, once you put out that pull request with
10:14
one commit, the image scan is performed, the sneak container test command is ran. Then from there
10:21
there's checks to see if there's any critical vulnerabilities in the image. If there are any
10:26
critical vulnerabilities in the image, then you essentially, well, these should be switched
10:33
But if there are critical vulnerabilities in the image, then there should be a block where the
10:38
image is not, well, the pull request is not merged to the main bridge, right? That's the idea
10:44
So this happens on a per commit basis, right? And this will give you a better idea as to whether your pull request will pass or not by checking the status of your commits On the other hand in the case that you are actually performing a merge what will happen is the same CICD pipeline will be triggered
11:07
However, in the case that there are no vulnerabilities, then the push will be made to the Azure Container Registry
11:16
So this is a common pipeline that we see in the industry
11:22
And this can be extended out. So from this point onward, once the image is in the registry
11:28
then your Kubernetes pods can then reference the images in the registries to start spinning up containers
11:34
And then we can start talking about scanning live containers or live pods
11:39
in an environment where containers are already running. But these two CICD pipelines are the CICD pipeline is something that we see that's very common
11:52
So let's move on to the demo and we can show a little bit about and sorry, that was someone buzzing my apartment, maybe an Amazon delivery man
12:06
Don't think that's for me. Okay, so moving on to our demo
12:12
So what we have here is really an Azure pipeline or Azure DevOps environment where we have a agent running locally on my MacBook
12:22
So this Azure agent is not running on a Azure VM or hosted by Azure in any way
12:29
This is running purely in my MacBook environment just for peer demo purposes
12:35
And then from there, the idea is I can show you the Azure pipeline
12:41
Let me see. I'll show you this pipeline. So I'll go through the YAMO file of the Azure Pipeline
12:52
And then from there, we can talk about the results of one of the image scans that were performed
13:01
So the idea is that pretty standard a Git diff is performed to figure out all the files
13:07
that have changed from the current commit to the previous commit. And then from there, once we figure out all the files that have changed, we then iterate over
13:16
all of the files. And then for every file, we simply just check to see if the file that has changed has the word
13:25
Docker file in it. So usually in the industry we see Docker files with the naming of dockerfile
13:31
X, Y, C. So Dockerfile. EngineX, Dockerfile, that your image name, right
13:38
So that's the usual typical standard for file name. So we just check to see if the file has the word Docker file in it
13:47
In the case that it does have the word Docker file in it, then the idea is we will first need to build the image
13:53
So this can be a little bit cumbersome. Docker builds can take a little bit of time
13:58
They will get stored in your BarLib Docker directory and all the layers of that image can then be stored in that directory, which can take up some space
14:08
So you want to make sure that the Azure DevOps agents that you're running on have enough space, have enough disk space to be able to withstand larger, larger scale
14:17
images. But the idea is that we simply run a Docker build for demo purposes, even if the image
14:23
is large and you don't have to consider the disk base size. Hopefully you have enough. And then from
14:29
there, you will run a sneak container task command, which will generate a report. So that report
14:37
is just simply going to be displayed in the council. Once you view the report in the council
14:43
then you can determine, well, the idea is that the pull request will fail
14:48
the check that's on the poll request will fail saying that, hey, there's a vulnerability
14:52
because this sneak container test command will return a non-zero exit code
14:58
And the idea is that the severity threshold simply just says anything from
15:03
that is high or higher than high, which is a critical, so anything that's a critical or a high vulnerability
15:10
will cause this command to fail. And the report will be generated. So yeah. In the case that the exit code is not zero, in the case that it has failed
15:21
then you simply just write the results out to sneak UI. So I should have sneak UI here
15:33
So for example, you will see, you know, there's some issues, here's some criticals, 47 criticals
15:42
89 highs, 85 mediums, 129 lows. And you can get more information on that
15:49
This is the base image that's being used by Docker, that specific Docker image
15:54
Here are the number of vulnerabilities. And here are all the libraries that have the issues in it
16:02
And there's all sorts of tools just kind of going off a quick tangent that will provide you information as to whether how to upgrade
16:12
the library in order to remediate the vulnerability. So for example, if you were to use WIS
16:17
WIS is a great security tool that I've used in the past, which allows you to essentially provides you the exact path to remediate the vulnerability It say hey upgrade GL G Lib C from this specific version to another version
16:33
And you can see even Sneak provides the same thing. It says simply, this is fixed in version 2.24
16:41
So this then provides your team a little bit more information. Okay
16:47
So now that's the idea with the Azure pipeline, you know, sneaky Y
16:52
Then from there, the, let's see, is there anything else? Oh, yeah
16:57
So let me try to show you some of the. So here's a sample Docker file that I put together
17:06
Very basic. Alpine is the, I believe, the slimmest Docker image that you can use in terms of size
17:13
It barely has any packages, libraries, directories on the Docker image, if you were to spawn a Docker container
17:22
And this is quite useful in the case that you're trying to find zero vulnerability
17:25
So this is a good use case for testing to see if there's zero vulnerabilities
17:29
critical or high vulnerabilities rather. And the same applies to, well, the same does not apply to Engine X
17:37
Engine X is a very large image and it contains all sorts of highs and critical vulnerabilities
17:44
And then you have Redis, which also has the same vulnerabilities. So for demo, as standard demo day goes, I was having issues with my Azure DevOps agent on my Mac
18:00
However, I am glad to show you the CICD pipelines that ran about two hours ago successfully
18:06
but now are no longer running successfully. So you can see here the update for, let me see if I can provide a good one
18:14
So let's say DockerFile. EngineX. Dockerfile. EngineX, if you go to jobs and we check our output of this job, you will see that this is a sample Docker file and we have built the Docker file
18:37
And then from there, it simply gives you the report of how many vulnerabilities there are
18:41
So there is no vulnerable paths found in this specific EngineX image, which I believe is at the time of running this, I had set the severity threshold to be critical
18:56
So that would make sense. But if I were to show you an example of a pipeline that fails, let me see if I can show you a better one
19:08
Oh, no, this won't be good. Let's see, which one
19:25
Yeah, so this one contained vulnerable pads, and this is a good example
19:33
This is an example of a sneak report that will be generated
19:37
So in this case, I'm trying to see what the exact image was that was used here
19:44
But the idea is that critical vulnerabilities were found for this specific image
19:51
And then from there, the output of this job had failed. So this is a non-zero exit code
19:59
Now, if you were to try to set up your CICD pipeline
20:03
to cause the pull request to not be able to be merged in the case that this is a non-zero exit code
20:09
you can simply go to your GitHub or Azure pipeline, your Azure repository, Azure DevOps repository settings, and you can set up a check
20:19
which will see if there's a non-zero exit code. In the case that there's a non-zero exit code, you can set it up so that a protected branch
20:27
like a main branch, will not allow merges into it if there is any failed checks on your
20:35
pull request. So that's an example of a successful and a failed pipeline
20:41
Simon, how are you doing on time? I think I might be a little bit 225, right
20:48
I have seven more minutes. Simon, you're there? Yeah, you're good. Okay, cool
20:57
So those are the two pipelines that I wanted to demonstrate. I would have liked to have showed you the demo live
21:06
However, I did have some issues as typical demo fashion goes. In terms of more information about our organization, here's our website, HGTPS-C-S-colland-slash-slash
21:21
I always like to include that. Viral Software Solutions.com. You can find us also on LinkedIn, www.com slash in-slash-V-I-R-A-L, V-I-Hifin-R-A-L
21:33
Feel free to connect with me. We can talk more about DCP, Azure, anything cloud-related
21:40
happy to talk. And yeah, Simon, I think that's all I have for today. So thank you so much
21:47
everyone