Up next in 10
In this session, we will explore the crucial role of automation in ensuring software quality throughout the Software Development Life Cycle (SDLC).
We will delve into the key components of Static Code Analysis (SCA), Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), and more. By leveraging these powerful tools and integrating them into CI/CD workflows and your origanization's policies, developers and organizations can automate critical quality checks, identify vulnerabilities, and enhance the security and reliability of their software.
Software Architecture Conference 2023 (https://softwarearchitecture.live/)
#SoftwareArchitectureConf23 #CSharpTV #csharpcorner
Show More Show Less View Video Transcript
0:00
Hello everybody, good morning, good afternoon, good evening, depending on where you are
0:04
So this talk is on software supply chain sanitization, and there's an alphabet soup
0:09
of tools and technologies that we're going to cover. So a little bit about myself, I have been in the industry as the gray hairs show for
0:23
over 20 years. I spent 12 of them at Microsoft in consulting services, traveling around the country
0:29
and internationally. I've been a startup founder in 2018. I found a startup that was accepted to
0:39
play labs at MIT, and then COVID happened and brought a quick end to any discretionary funding
0:48
we may have gotten. And so here I am now as a partner technologist involved in product security
0:54
architecture at Aviva. Aviva is focused on industrial IoT. And my hobbies, aside from
1:03
software development, of course, include filmmaking. I do have a profile on IMDB
1:08
I am also a home brewer and enjoy photography. All right, so the goals of this presentation. We'd like to establish a common vocabulary and
1:20
know which tools we're talking about here. We also need standards. For example, we need to determine
1:29
how vulnerable a security vulnerability is. What one person may regard as a high vulnerability may
1:36
not be a high to somebody else. We'll cover regulatory and industry requirements and how we
1:43
got here. Specifically, there's an executive order that has a direct impact on this
1:48
and we'll define patterns and practices for sanitizing your software supply chain
1:53
which go beyond just the tooling. Having tools involved in your pipeline are great
1:59
but without guidance and governance to manage that, you're going to get spotty implementations and adherence
2:11
So let's start with the software supply chain itself. What is it
2:16
It's composed of components, libraries, tools, and processes used to develop, build, and publish a software artifact
2:24
So it's a short sentence, but it captures quite a bit. And that software supply chain is vulnerable
2:35
We've had a number of breaches in recent years, most notably from SolarWinds
2:41
The malicious hackers were able to infiltrate the SolarWinds network, check in malicious code directly into SolarWinds repositories that affected the server-side software that pushed updates out to customer systems and infected customer systems with malware through that update process
3:06
18,000 customers were affected. it was it made national news and it got the attention of our government which resulted in
3:18
some changes we've got a few scary stats there have been as of two years ago 18,000 to 20,000
3:32
new vulnerabilities reported through the Common Vulnerability Enumeration Database. We'll cover that a little bit later
3:42
Database breaches were quite expensive at an average of four and a half million, and
3:50
it's projected to get worse. Notably, there are numerous misconfigurations in software repositories that expose secrets
4:02
artifacts, source code. If you look at GitHub, they're alone. It's got machine keys
4:12
certificates, API keys that people have checked in. Hopefully, they've recycled them and retired
4:18
them, but not everybody has. One of the common vulnerabilities that was discussed at Black Hat
4:24
for example, with ASP web forms were reusing machine keys that people found in sample applications
4:32
were on GitHub without changing them. There's even hackers keep a list of these exposed machine keys
4:41
and you can bet they use them for exploits. So we've got a few things to worry about
4:48
Our external threats. We've got our IT departments that monitor for intrusion detection
4:56
We've got phishing attacks coming into our emails. but the ones that I'd like to focus on in this talk are malicious developer tools, open source components, container registries
5:06
generally the public repositories that we use for our open source development
5:13
Those repositories can be published to by anyone. There's no betting process
5:18
For example, anyone can publish a GitHub Action, and anyone has. There are GitHub actions that will exploit GitHub itself for crypto mining or exfiltrate secrets from your repository
5:36
So you've got to be careful. And it's not restricted to that. We've got Visual Studio Code extensions that are malicious
5:44
Microsoft does not vet what's published there. NPM is no outlier. It's got over 12,000 suspicious packages. And recently, even for packages that we've come to trust, like a lot of people on this call probably use mock for unit and integration testing
6:07
And Mock recently had a privacy violation in which they sent developers' private email to a third party for soliciting donations
6:23
And there's Docker. And we can say that we'll trust our trusted publishers like Microsoft, like Amazon
6:32
but some of these actors will rename the container or the module to be something close to something
6:40
we trust, so it's easy to slip up. So how do we prevent ourselves from falling into that trap
6:47
And you're probably thinking, well, how do I put this into action? What concrete steps can I take
6:52
to enforce security on my pipelines. So we've got a number of types of tools
7:04
There's a bit of an alphabet soup there. So what you should have is a multilayer defense
7:12
No one single set of tools is going to be your silver bullet
7:18
You need to be checking for secrets secrets and make sure that you not pushing passwords or certificates into your pipeline When you do it should notify you and that needs to be rotated out Your secrets ought to be stored in something more secure like Azure Key Vaults or encrypted pipeline variables
7:40
GitHub has secret scanning capabilities. There are secret scanning tools available in ADL
7:47
So software composition ysis is going to be a big topic here
7:54
This yzes and identifies third-party components in your software packages. It identifies the packages that your packages use
8:01
and ties them into known vulnerabilities with those packages. Static application security testing is something that we've kind of been doing
8:14
If you've been using Pitchell Studio and you've gotten code ysis warnings and violations, that's been a form of static application security testing
8:25
It's part of the solution, but it doesn't track the issues and make them available to the rest of the team or for auditing purposes
8:33
Dynamic application security testing, as it states in the name, implies some active component, and that typically involves an API and an automated test that attempts to reproduce some of the common OS top 10 vulnerabilities
8:57
And then there's IAST and RASP, Interactive Application Security Testing Runtime Application Security Protection that runs while the application itself is running
9:09
That's beyond the scope of this presentation. We certainly don't have the time to cover that, but you should be aware of it
9:15
And lastly, there are policies and procedures for governing these tools. these tools can generate quite a bit of noise and you need some governance to drive
9:27
filtering that noise and determining what should have priority. So at this point you're probably dealing with this alphabet soup of technologies and let's
9:40
break down a little bit further. So what is software composition ysis used for
9:51
So it scans our redistributables, our MSIs, EXEs, anything that deploys to the cloud
9:58
anything that we deploy to a production environment like containers. The risks that it should identify are operational
10:05
For example, if you're using a component that's 10 years old and hasn't been updated in 10 years
10:10
that's a problem. You should consider replacing that. It certainly should identify security risks, the CVE enumerations, common vulnerabilities that have been identified, and lastly, licenses
10:25
So if you're producing commercial software, you should be aware of copyleft licenses like GPL, which state that if you utilize this given component, then the license obligates you to releasing your software with that same GPL license
10:43
It's commonly referred to as copyleft. And that's great for open source projects
10:49
But if you're working for a company that wants to make money, you might want to reconsider using these specific components
10:57
And there are some that are meant explicitly for non-commercial licenses, like JRL
11:05
So I've been referring to these CVE vulnerabilities. So what are they and how do they get registered
11:14
So there are organizations that will do cybersecurity research, like Aquasec and CrowdStrike
11:26
Companies themselves will kind of eat their own dog food and also self-report
11:31
they report to the CVE program, the Common Vulnerability Enumeration Program, which has an aggregate of all these publicly known vulnerabilities
11:42
And typically, this researcher, this cybersecurity researcher, if they find a zero-day vulnerability, the ethical researcher will report it to the company when they find it and notify the company that, well, you have 30 days to remediate this or 60 days to remediate this before we go public
12:07
And then the report process begins. unethical researchers are called hackers, and they'll exploit that zero-day vulnerability
12:19
So the CVE identifier gets issued from an authority, so we have a well-known
12:26
common number to refer to this vulnerability with, and ultimately it'll get published to NIST
12:35
Now, once again, what you might consider a high vulnerability item may not be high for somebody else
12:43
So NIST has established the Common Vulnerability Scoring System, CVSS, which is not necessarily a measure of risk
12:54
It's a measure of vulnerability. So consider this for a moment. Let's say you have a component that you're using, a third-party component that's got a vulnerability that says that if a given method is invoked with a given set of parameters, that you'll experience a denial-of-service attack
13:12
The package will fault. But if your application doesn't invoke that method, your application is not at risk
13:27
So customers have been running their own binary scans and ysis and have been coming back to organizations with expectation for getting remediation statements
13:39
for being reassured that, A, whether they are vulnerable or not, or B, if they are, that a hotfix is forthcoming
13:49
So let's take a look at what one of these vulnerabilities looks like when it's reported
13:55
So here is the NIST vulnerability database. Here is one that I ended up having to investigate
14:04
Zlib allows memory corruption when deflating, if the input has many distinct matches
14:11
So to boil this down, this was a vulnerability that was exposed
14:18
when calling the deflate method, specifically deflate2 on Zlib. ZLive, by the way, is used by .NET Framework and .NET
14:27
Even if you're not explicitly calling it, it's used by the System.io Compression Library
14:33
Oh, I see a question in there. What's an alternative for mock? There's X units
14:40
I think facets have been used in testing. But I wouldn't abandon mock
14:48
I would just give them some time to work out that particular privacy issue
14:54
There been a bit of uproar in the community about that and they probably going to roll that back So don give up on that So with respect to zlib
15:06
you're using it implicitly if you use system I compression or if you are using gzip to send HTTP requests
15:15
This is a denial, effectively, this is a denial of service vulnerability
15:20
There's this vector score which breaks down the nature of the severity of the vulnerability
15:29
this CVSS score is determined using this Common Vulnerability Scoring System Calculator
15:38
And this score shows us that, well, you need to be on network
15:45
The attack complexity is pretty low. You just need to call a given method. No significant privileges are required
15:52
but the availability for it, the availability of the component, if it is exploited, is high, total loss in events
15:59
So that results in a high vulnerability score. That's how we get there
16:05
If it were low, it would be a medium severity. So this results in a vector string
16:15
And there it is. If you are tasked with investigating one of these vulnerabilities and coming up with a mitigation statement, this set of hyperlinks is incredibly helpful
16:29
But something is conspicuously missing here. And typically, affected components that require a hotfix from a commercial organization that supports that component, like Microsoft, typically has a patch
16:46
And a link to that patch is usually found in the set of NIST hyperlinks
16:52
There is no Microsoft patch for this. Ultimately, the Zlib was being used through .NET in a safe fashion
17:03
It was not exposing this deflate method by calling these set of parameters that were edge cases
17:13
It was using it in a safe manner. so microsoft had no need to patch it yet z live shows up as one of the components
17:21
that ends up getting used in a binary ysis because it's packaged with dot net
17:28
so some of the useful links here include things like the issue in github that addressed the
17:36
particular vulnerability so this is your friend if you need to drill into the vulnerability
17:44
investigate it and create one of these mitigation statements. Let's return to the presentation
17:57
So this is telling, right? There are, at this point, there are hundreds of thousands of CVEs
18:06
So how do you know which ones to pay attention to? Well, the best majority do not get exploited
18:13
They don't get used by a malicious hacker. Usually the complexity for exploiting these is so high that, well, they can't quite easily do it
18:23
But this Venn diagram tells us what we should be prioritizing on
18:28
CVSS scores that are high that overlap with positives that have been used for an exploit
18:37
Right. And so to close the gap between what is considered a high vulnerability versus what is a risk to your application, there's this exploit predictions scoring system, which is still evolving
18:55
It hasn't been widely adopted yet, but you should be aware of it, that risk or rather severity does not always equate to risk
19:07
We've had a tool at our disposal for free that we can use right now
19:17
Nothing else needs to be installed to find if you're using any NuGet packages that are vulnerable
19:25
I'm getting over a cold. You can use .NET List package, dash dash vulnerable
19:31
execute it from your project directory, and you'll get that list. along with helpful URLs
19:43
So, one of the clauses in the executive order 14.028 was to include a software bill of materials
19:55
which is a list of all the third-party open source and commercial components that your application uses
20:03
So as stated in the executive order, it's a formal record containing the details and supply chain relationships of various components used in building software
20:14
Now, that's rather vague, right? That's not a JSON schema. That's not an XML schema
20:19
That's a rather open-ended definition. We need something a little more concrete if we're going to take action against this
20:27
And that's the SBOM format itself. And we've got three definitions. They've been evolving over the last few years
20:36
SPDX, Cyclone DX, and software identification tags. We can really disregard the software tags
20:46
They might meet the legal requirement of the SBOM definition, but they don't have enough
20:54
Both SPDX and Cyclone DX include copyright statements, licenses, a hash value to uniquely identify the third-party components
21:07
SPDX is a bit older. It's been around and evolved a bit
21:11
It tends to be preferred by cybersecurity professionals, whereas Cyclone DX is more machine-readable and preferred by developers
21:18
And here's a small comparison between the two and how two different tools identified
21:29
Well, two different tools used. One used SPDX, the other one used Cyclone DX, and how it identified the Microsoft Extensions Dependency Injection Abstractions package
21:43
And as you can see, Cyclone DX is a bit more verbose on the right
21:48
The SPDX does include a copyright text, but the tool did not pick it up in this instance
21:56
And here's a set of, a minimum set of fields that are expected in an SBOM, irrespective of whether you're using Cyclone DX or SPDX
22:09
This, and there's a reference, Simon, I'll send you this PowerPoint and please feel free to share it out with the audience
22:16
There are a number of links in here. So this is what you can expect to find and be required to produce
22:24
And these are recommended but not required However customers are expecting licenses and copyright statements So even if it not meeting a legal requirement even if you can release it without that customers are going to come back and ask for it
22:41
So there's not just a software bill of materials. There's a bill of materials for hardware, for formulation bombs, tools and applications used to produce the software
22:52
Like if you're using, if you're building a UI, a single page application, you're using Angular, you're very likely using Webpack
23:04
That's not something that's shipped, but that's something that you would expect to see in a formulation bomb
23:10
Why they're not calling in an FBOM is, that's a lost opportunity there, as far as I'm concerned
23:17
There's an operation bill materials for what operating systems were used, your runtime environments, your configurations, and there's others
23:26
This is just a short example. There's a comprehensive link at the bottom
23:32
But one you should be aware of is this Vulnerability Exploitability Exchange, VEX
23:38
So the SBOM does not include any vulnerabilities. It does not include CVSS scores
23:44
VEX fills in that gap and includes the components, its associated CVSS vulnerabilities, more specifically, it's CVE vulnerabilities
23:57
I want to make sure I'm using the right acronym. And mitigations
24:04
So once again, this is another evolving standard. You can expect to find some open source and free tooling in this space
24:19
If you've used GitHub and have a repo there and have been notified that you've got a third-party component that is due for an upgrade, you may have used Dependabot
24:31
We saw the screenshot from the .NET scan tool. Microsoft has an open source SBOM tool that Microsoft is maintaining
24:43
Cyclone DX has their own .NET tool. The screenshots that I showed earlier were from those tools
24:49
from this Microsoft SBOM tool and from Cyclone DX. And OASP has a pretty comprehensive list of open source tools for SC
25:02
well, in this whole space, SEA, SAS, and DAS tools. However, if you're trying to assess operational risk
25:12
security risk, and license risk, you're not going to find a tool that does all three
25:21
You'll end up having to cobble that together. And if you're doing this at scale
25:25
quite honestly, you're better off with a commercial tool like MEND or Qualys
25:32
So let's say you've got this in place, you're rolling this out to your organization
25:41
but you need to give them some guidance. These tools can create quite a bit of noise
25:50
So I'd like to reduce that a bit by focusing only on the components that ship
25:57
at least initially. You can use things like package config in your NPM projects that have a package dev tools section to determine what's internal and what's external
26:13
Customers are expecting these mitigation statements. But you're going to get a lot of noise
26:21
You're going to get a lot of low and medium vulnerabilities. Prioritize on the critical highs
26:26
define what your CVSS maximum criticality is for release. Can you accept an organizational risk to release with a medium CVSS score
26:45
Possibly. You should have a process to block components with licenses that put you in legal jeopardy
26:52
like those copyleft licenses. Replace your components that have high operational risks
27:00
You know, like that one that I cited that might be 10 years old
27:04
that hasn't gotten any updates. The open source developer has given up on it
27:08
Maybe it's time to rotate that out. And developers are often under the gun to deliver on functional requirements
27:16
that mean, you know, features that we can sell to customers that will make us money
27:23
Alleviating this is technical debt. its overhead but your organization ought to allow for some time to cover that overhead
27:34
typically like in companies i work with i've seen teams get you know 20 of their
27:41
sprint time allocated to tackling technical debt and that this falls under
27:48
and make an s-bomb available to your customers they're going to want it sooner or later
27:53
and you should be familiar with this so at this point i i have managed that mess and that this
28:01
is how i learned to stop worrying and love the s bomb and get it under control
28:08
you're not expected to solve everything but solve the high and critical items prioritize
28:15
So moving on to SAST, software application security testing. So it's helpful in finding common security and code quality vulnerabilities
28:28
like writing a plain text password to a log file, or missing a null check somewhere, or missing a using statement
28:36
It finds code issues early in the process. You may have heard the term shift left
28:43
some of these tooling some of these tools are starting to run in the IDE as opposed to like in
28:53
a server that scans your repo separately asynchronously and produces a report that's
29:00
preferable catching it before it even gets into the repo in the first place is preferable the
29:05
advantages of a lot of these tools are that they they run passively passively they don't break your
29:11
application. They don't break your development process. They're easy to launch. They have high
29:16
code coverage. But once again, you're likely to see some false positives. It's bad at yzing
29:24
highly specialized libraries. Like if you've got a very detailed business process, it's not going to
29:32
catch business logic errors. You still need a code review for that. And it's not going to see
29:39
your code in a runtime context. It may catch some cases where you missed a thread lock
29:46
but it's not going to find all race conditions. You still need a dynamic scan strategy
29:57
It looks for Common weakness enumerations. These are defined by the MITRE Corporation that works with the CVE program to identify
30:09
based on the vulnerabilities that are getting reported in, what are the most common
30:14
And here's a top 25 set of common application vulnerabilities, like null pointers
30:25
And this does have some overlap with the OS top 10. So we've got a few free open source tools
30:33
Like I mentioned code ysis earlier, GitHub includes code QL. There's ESLint, and it's relatively easy to stand these up in your pipeline
30:44
These screenshots show you how to get code QL stood up, and this is just a simple build process to get it on your build agent
30:54
do the build, and then perform the ysis. and we've got a number of commercial tools and so once again if you need to track these long term
31:04
or create work items off of these then you're better off with a commercial tool rather than
31:09
using something that's open source if you're working on a small team that might be fine but
31:13
if you're in a large organization you need something more comprehensive so let's take a
31:19
look at github and what code ql tells us so let's see simon how am i doing on time
31:25
So five to six minutes, I would say. Okay. So I'll be quick on these because I still like to cover DAST
31:39
So this is a personal repo I've stood up and I've enabled CodeQL on it
31:49
and it reports a number of security vulnerabilities and quality vulnerabilities. like nested if statement can be combined
32:02
So identifies the code. Now, one can argue that one might be more readable than the other
32:11
But more interestingly, it also detected an issue with my Azure Key Vault firewall
32:21
that I haven't protected it well enough. So it's not just code that's scanning
32:27
It's also scanning BICEP and ARM templates. Now, this is just my open source project
32:32
This is where a commercial project, I would block on this. This is something that would be, that should not make it into production
32:41
So you need to figure out, okay, what are your blockers here
32:44
And I'd recommend blocking on high severity security vulnerabilities. these And I also like to since we only have a few minutes left I going to skip ahead and give you an example of a DAST scan
33:05
So dynamic application security testing involves setting up a process that scans your web API
33:14
And these will typically look for the OS top 10, like cross-site scripting vulnerabilities
33:20
that will exploit your cookies, SQL injection attacks
33:30
second order identifier attacks. So this tool is a freeware tool available from OSZAP
33:38
If you are not doing any dynamic scanning today, this is better than doing nothing
33:47
This is relatively free to set up. Well, it is free to set up. It's relatively easy to use
33:54
And this scan was done against the Microsoft To-Do API tutorial in ASP.NET documentation
34:06
So all I did was took the To-Do API, stood it up, and ran that scan against that
34:12
And it found a couple of medium priority items like missing content security policy headers
34:28
This one is interesting. It tells you that you're informing the client what type of server you're running
34:40
and that could be used by a malicious actor to exploit the server
34:45
They might know that, oh, if you're using, say, ASP.NET Web Forms
34:50
that there are a suite of vulnerabilities for that technology. So, you know, remove that header from your responses
35:00
Now, of course, this is just a to-do application, so simple stand-up
35:04
But this shows you what a website might look like prior to hardening
35:10
Now, there is a lot of noise here. You can see that this one vulnerability came up in 1,140 tries. So in this case, removing that header will alleviate all of those vulnerabilities
35:26
So once again, prioritize on the high vulnerabilities first, determine what your minimum release bar is, set a bug bar, and set up a DESC policy test
35:41
This can also be run at the command line but it does take a little more work to set this up because you need to deploy the application you need a host and so you need a live site to throw this against Let return to the slides and cover what we still have time for
36:11
So, these DAST tools also do fuzz testing, some degree of fuzz testing, like sending in long strings
36:21
sending in values that may fool regular expressions. There was a vulnerability that was exploited recently on an OPC IoT server
36:36
that sent a string that was badly null terminated, and it abended the server
36:41
The string had a null terminator and then a few characters after it, and the server didn't know what to do with it
36:49
The code wasn't expecting it. Now, DAST tools typically have limited dynamic content
36:59
And what I mean by that is they're focused on known vulnerabilities, like the OWASP Top 10
37:05
It's no substitute for a human. So it's not a substitute for a penetration test, where you've got people using things like Burp Suite and are professional white hat hackers
37:19
This is good for day-to-day, but if you've got a public-facing API, you know, consider doing a penetration test at least once a year with an external organization
37:32
We've got some commercial tools for DAST, Rapid7, Detectify, Qualys. Once again, if you're doing nothing today, take a look at OASP Z-Attack Proxy
37:44
That's worth looking into. and these tools typically focus on the os top 10. you can see what some of the changes look like
37:55
between 2017 and 2021. a couple of the new items surface like server side request forgery where the
38:07
client will try and send a header that should only be attached by a server behind the firewall like a
38:14
load balancer like an x forwarded for header that a load balancer will attach you could use that to
38:21
possibly reroute uh requests if you know something about the web farm environment that the
38:27
um that you're targeting to attack and and you'll see a common theme here um
38:36
One that's worth pointing out is to reduce your attack surface. If you have retired a route
38:46
if you've retired parameters from a request, get rid of them. Give that desk scanner less to look at That is one way of securing your APIs reducing simply the number of routes you have
39:07
And you're probably wondering, well... Excuse me. Still a little bit of a cold I'm getting over
39:24
Pardon me. So if you don't have these tools in your organization yet, start with a pilot
39:39
Pick a team, pick a development team that is willing to try something new
39:45
Use them to get some early feedback before rolling it out to the rest of the organization
39:49
And when you do, roll out incrementally. You know, introduce one tool at a time, give it a quarter, you know, give it three months to get some feedback, back it up with documentation and training, give teams that are managing legacy software some time to catch up
40:08
Because if you haven't been doing these scans in the first place, and you've got software that teams have been supporting for, you know, several years, no doubt they're going to get overwhelmed with findings
40:19
So don't expect them to solve everything that it reports on day one
40:28
Give them time to catch up and iterate both on the tooling that you use
40:34
and the processes applied to the tooling. And as a best practice, sign everything you release
40:45
Also use two different certificates when you're signing, One for internal releases only
40:51
That'll tell you if something accidentally slipped out that should not have slipped out. And for external releases, of course
40:59
Container signing is an emerging space. Vendors have started to support this, but it's very spotty and uneven
41:12
I'd give that a little bit of time to work out. but we should be in a place in about six months from now to expect to have services in Azure
41:21
in the Azure Container Registry that limits the permitted containers to those that are signed
41:29
And after you do sign, validate the signatures using something like Binskim
41:35
The Microsoft Security DevOps tool includes a number of subcomponents, and Binskim can be used to validate
41:42
that you've signed your binaries and you've signed your executable releases
#Business Services
#Computer Security
#Hacking & Cracking
#Software
#Public Safety
#Security Products & Services
#Computer Science
#Antivirus & Malware
#Network Security