Code Quality and Infrastructure as Code by Bill Penberthy

Hey, everybody. As David mentioned, my name's Bill Penberthi, and we're here to kind of talk about code quality and infrastructure as code, or IAC

So I'm currently a consultant at Unify Consultant, which is headquartered here in Seattle

My specialties in systems and software architecture with an emphasis towards operations in the cloud

I've written a couple of dot-net books, and I'm two chapters in three weeks away from completing my third, which is called Pro

dot net on AWS for A Press Publishing. I've spent, oh, 20 years in consulting plus an additional

10 plus in industry. And I've worked in literally every aspect of the software developer

lifecycle. You know, I like to claim, and I don't know why I like is the right word for it

But my claim to fame is that I've probably made every single possible software developer

mistake out there. And I have the gray hairs to prove it. So hopefully the things that we'll go over

today may help you avoid some of those problems yourself. So this is our agenda. A real quick

description of infrastructure is code so that we all get on the same page as to what I mean here

We'll then talk about code quality and how infrastructure go hand in hand. We'll then go into some of

the differences around code quality and an application versus code quality and infrastructure as code

And then we'll talk a little bit more about what you should do or what you should do about those

differences. So what is infrastructure as code? Well, you know, I've got Microsoft's definition up

on the slide, so I'm not going to read that. So what I'll simply say is that IAC is basically defining

your infrastructure, such as your compute resources, whether it's a virtual machine or a container

or a serverless function, your networking configuration, your firewall rules, system rights

anything and everything that you use to provide the infrastructure on which you're your applications run. All of that, in IAC, you define in code. Infrastructure is code. The code is then

checked into source code, and whenever you need an environment, you rerun that code. So this solves

real-world problems. Environmental drift is one. IAC offers item potence, where a deployment always starts

the infrastructure in the same configuration every time. And since this is checked into source code

and item potent, you can then roll your environment back to a previous version

You can check out previous versions and do some work against that. These are all solving real-world problems

IAC also offers real benefits beyond how it helps solve those problems

You know, one of the coolest things about the cloud is how you can spin up new instances

up and down really easily whenever you need it. It's something going on and all of a sudden you need a new testing instance of an application

maybe to test an integration or to give a partner a sandbox to play on something like that

Well, now it's really easy to do that because you have your infrastructure to find in code

You know, in the old days, way back wins, typically you would have a, like a, what I call it, a playbook

a written set of steps that you would take to create and configure your environment

And a new environment took time. And since nobody ever really had time, it would be a big

deal and it would get a lot of pushback to create in a new environment. Well, in the cloud with

IAC, it's now not that big of a deal. And lastly, and it kind of doesn't fit in here, but I think

it's important to start bringing this up now, is that IAC should be used with a declarative definition

This means that the code that you create and manage describes what the environment is rather than

how it's built. IAC is, you know, typically you look at it as it's kind of based on a product like

Palumi or Terraform or AWS CloudForm. And those products are kind of responsible for filling the

how based upon the declarations that you provide in code. In other words, your job isn't to do it

Your job is to define what it is. And we just briefly mentioned environmental drift. Let's go to this

a little bit more because this is, in many ways, this may have been my biggest waking nightmare

for the last few decades. So, as you may know, environmental drift is when there are discrepancies between environments

This happens when you have a VM or a system on which you deploy without redoing the environment

Very, very typical last few years, especially when deploying into Windows VMs, you know

so I could guarantee that like two years ago when you first created those

environments, they were exactly the same. Now they aren't. Test will be

different because it's had things tested on it. Go figure. Whether it's an OS

patch or an application update that ended up getting rolled back, at some point

those environments are going to evolve differently. Which means there'll potentially be

some different behaviors that are on those environments. They could be performance-based

say you bumped up production resources to 16 gig while your dev or tester at 8 gig

it could also be something more significant like different versions of installed frameworks

especially when you don't take those frameworks with you when you deploy. And that's what IAC prevents this drift

It means that you'll be using the same environment, just like you use the same bits, hopefully

when you test the deployment and then when you push to prod. So what you're evaluating and test and what you're running in production are the same

The advent of containers has done a lot around helping management. manage this, and that's great. However, those of us, as I mentioned earlier in the dot net world

and especially at enterprise companies, are still have a lot of applications running on dotnet

framework 4.x, which means most likely running on a Windows VM and more than likely that

system hasn't been updated for a long time. And you will have drift. So why IAC matters

You know, we've kind of talked about all these points, but I wanted to reiterate them. IAC matters

Using IAC makes your environments more predictable, and that has a lot of value in and of itself

So here's an example of IAC using Pallumi and C Sharp to define an Ubuntu instance on a on a T3 micro instance type

So what this is is we've defined the operating system and some other ancillary kind of software that we want installed on it

and we've determined with the instance type what size of processor and memory availability

and network connections do we want to have available. So we're basing this on a well-known machine image here, the Ubuntu slash images

You can see that in the first part of the Get Amy Filter Args. But what this means is if you choose the proper AMI here or Amazon machine image for this one

that you can get Ubuntu with dotnet core already installed on it

which means that before you use this EC2 instance, before you do anything, it's already defined and it's ready for use

So by using this, all of a sudden now, any work that you have to do after starting your environment is already done for you

And so that eliminates a lot of those problems that we used to have with the playbooks

around having to manually manage some of those workflows. So here's a different approach where the instance is defined in JSON

This is an example for the template definition for creating an AWS EC2 instance

So you can also create these definitions in YAML but personally YAML drives me crazy so I didn include those Filling out these values and running in AWS Cloud Formation will create the instance Now if you look at these you see what you doing is you describing what the configuration

is going to be, what the hardware item that you're going to be deploying is. You're not telling

the system how to do it. You're instead just defining this is what I need to run my infrastructure

Now, an interesting thing about this is that the code that we looked at on the last slide

Typically what you'll see with that or with like the AWS Cloud Developer Kit is that you're able to create code in the language that you're familiar with

And what it ends up doing is it outputs these definitions. So you're able to use those kinds of C-sharp constructs that you're used to

But you're going to end up building these kinds of JSON templates that the cloud providers themselves are going to understand how to work with

So that's a huge value of that. So think about that. We'll talk about this a little bit more when we get into it

But think about it as your code quality and approaching how you're going to be building this infrastructure, how they are starting to interrelate

And that's kind of what we'll go over now with, you know, IAC matters

And the rest of this conference kind of talked about code quality and how that matters

So let's kind of mash them together. So first of all, let's go over what code quality looks like because

these are the points that I'm going to kind of come back to as I go deeper into IAC and how that works

You know, we won't worry about a definition or anything like that. Let's just talk about the characteristics of quality code

And when we touch on these, think about them in the context of your infrastructure and not just your application

Does what it should. Okay. That's something I really want out of my infrastructure

I don't know about you. But I really want it to just do what it should. And next is easy to understand

That gets a little bit more complex when you're starting to think about infrastructure as code

because let's be real. Not everybody is used to IAC. And there's a lot of developers, especially in the older generation

Dot net world that don't have much experience in systems and infrastructure at all

So everything we could do to make the next guy that comes along be able to perform something meaningful with the infrastructure code becomes important

and that's where quality comes out. So the code should be well documented

This seems to go hand in hand with easy to understand, but consider what that means for IAC

First of all, the rate of change in code in IAC tends to be much lower

than it is in an active application. It's pretty unusual that you'd be changing your environment every release, for example

You could. I mean, definitely you could, but generally you'd be running the same environmental code

for a longer period than the applicable application code. I mean, you'll still include it in every release

You just won't have had put any change sets against it. So obviously, once an application goes into maintenance mode

it's completely possible that the only changes that you'll do is maybe like a quarterly OS security update

But that's generally not the case. But because it's updated the least frequent amount out of all of those parts of your system

documentation becomes even more important because, you know, I don't know about you, but if I go back and look at code that I wrote three months ago, I still have to spend that minute or two scratching my head going, what was I thinking

And the last point here is that quality IAC code could be tested. This is huge because a primary differentiator between IAC authored by a systems guy and an IAC authored by a developer

We'll go into this in more detail later, but think about your infrastructure in a way that means you have common coding concepts like separation of concerns and code reuse and all of the other coding techniques that you use to ensure that you're writing quality code in your application that you apply those to your infrastructure as well

So what does quality code affect? Well, first thing is reliability, the probability that a system will run without failure over a specified period of operation

operation. Okay, that sounds pretty good. I think we'd want that as our infrastructure as well

Maintainability, you know, how easily the software can be maintained, size, consistency

structure, and complexity of the code base are all part of that. Useful documentation plays into

that. Testability measures how well the software supports testing efforts, how well you can

control, observe, isolate, and automate testing. Portability measures, you know, how usable

the same software is in different environments. And reusability, whether existing assets such as

code, can be used again. So when we go into the different solution points, these are how we're

going to compare them back to, these five points right here. So we've talked about IAC. We've talked

about code quality and the importance of quality code on successful IAC, let's go point out some more

of the differences between normal code or code in an application and code used in infrastructure

as code and how could affect things differently as you think about it in code quality

So there's four major points that we're going to talk about. The first is the footprint of effect

The second is of the failure vectors or the different ways in which a failure can occur. The third

is the complexity of testing. And the fourth is the unusualness of the problem set

I put that in quotes because I wasn't sure it's a real word, but I really like it, so I'm keeping it anyway

The footprint of effect, how much stuff can be impacted by poor quality code

The first point is that an environmental failure or misconfiguration or simply poor quality can cause an outage in the application

and that may be the only cause of the outage. But of course, since it's the application, everyone will blame the application first

unless, of course, you know, you're one of the app developers, in which case you know it's the environment, but nobody will believe you

But there's a lot of problems that a poorly configured environment can cause

Think about it. If the database driver that you may need isn't installed or a route isn't open in the firewall

or access rights aren't given correctly, the application won't work, but it will take some research to figure out why

Is the code wrong or is it something somewhere else that's making the problems

And even beyond the application, a lot of things that a modern system relies on, such as tracing or alerting, logging, all could be broken because something went wrong with the environment

My favorite was once where the environment was so broken that the alerts we used to tell us if something was wrong, well, they wouldn't fire

it was so broken that our systems couldn't report that it was broken

And so we didn't expect it to be broken, and we just kind of skipped that whole part of the troubleshooting process

And all of these shows how these dependencies trickle down. You know, if logging doesn't work, for example

you may not notice it until you have to go look for a specific problem. So now you have two problems

One, the app is hiccuping and you're not logging it. And why

Because your infrastructure code is of low quality. So let's visualize what I was just talking about

The blue box is your application. Everything else around it is what you manage an impact when using an IAC to manage your infrastructure

Thus, an IAC deployment can affect everything inside and including the white box, which is your network boundary

So not only is the problem or potential problem the application it everything that the application communicates with So all of these different points become your failure vectors or the most common direction in which errors can occur

Poorly designed IAC could affect networking and connectivity. Imagine something like a firewall rule misconfigured or a routing table entry missing

or anything like that that talks about network and communications within the network

Monitoring, as we mentioned above. But, you know, that's another direction. of failure if you have to ensure that your code quality extends to that code that's managing the

monitoring of configuration and setup. And lastly, the actual processing of the infrastructure

creation. Much of this is out of your hands, but there are definitely some quality decisions

that you can control that have a significant impact on how everything is put together and

processed by your IAC tool of choice. And here's another visualization that kind of shows the extent

of IAC slightly differently because it can also impact your ability to connect with other systems

or even managing the rights that you assign your own resources. And because of the failure vectors, managing the testing is very difficult

Think back to when we went over the characteristics of quality code and how one of those main characteristics

of testability. Well, the complexity of this testing effort makes it even more critical that

that the testability of the code is maximized. Sure, more than likely the release schedule for your infrastructure changes will be different

than your application changes, but you need to make sure that you have enough time to manage

these testing needs. And we'll go into these in a little bit more detail a bit

And lastly, is the unusualness of the code in IAC. Well, why is it unusual

Well, first of all, it describes infrastructure. Many developers do not have the most complete grasp on the intricacies of the entire cloud system

A VP, for example, or virtual private cloud is generally pretty well understood

But knowing that you need to add an internet gateway to that VPC before you can access the internet

requires a deeper understanding of the peculiarities of whichever cloud provider are using than most developers really wish to have

Another unusual... I'm sorry, I went too fast. There we go

Another unusual difference is that the IAC describes what the output is rather than how to build it

This is a strength of IAC because that makes the outcome more predictable

But it's the opposite of what the normal developer behavior is, is developers tend to focus on how it gets done because they're generally the ones that are responsible for doing that

And lastly, is that it should be treated at the same level as on application. What do I mean by that? Well, many times you kind of see structure considered to be

part of an application, generally because of how tightly bound they are. But just like those

visualizations that we went over a little bit ago, it puts the infrastructure in an inferior

position. And what that means is that people then start looking at the infrastructure as being subordinate to the application, when really you should

be thinking of them as being at the same level or partners in it rather than one of them owning

the other one. Even though, to be honest, the sole purpose of the infrastructure is to run the

application, we need to break that mental model about the infrastructure being subordinate

to the application. and arguments against it being different. So I've heard the argument that the infrastructure should be considered to be similar to a framework or an SDK

They're similar that there will be changes and updates to the framework, but there are very likely to be a lot fewer of those changes

And also any quality issues in the framework could result in incorrect behavior in the application, just like it would when looking at the infrastructure

So that argument says there's no real difference between being a framework module and being IAC

And there are definitely some similarities. But we talked about those vectors for failure earlier

And that's where I think IAC needs to be treated differently than a framework or other application inclusive items

But once again, you know, if you're going to look at it as subordinate, then it would make sense to kind of consider them even more

but the fact that the infrastructure can have many other things that can cause problems other than the application makes me think that it should be separately

Another one that talks about complexity and how there are other systems that are complex as well, you know, this too is true, but I think it kind of misses the point

The examples that were given speak specifically about systems. So a group of applications working together to fulfill a set of requirements

perhaps being complicated is still really application bound. Whether an application has, whether

you know, an application is one of them with a million lines of codes or a hundred separate

applications with 10,000 lines of code, the failure vectors are different between the applications

and between the infrastructure. And lastly, code is code is code, so that in regards there

should be no differences between IAC and application code. And in many ways, I actually feel that

this is the most compelling of these arguments. However, there are some things that you'll need to do differently when you look at IAC as opposed

to regular code. And lastly, there are some additional considerations that are mostly burdensome on the

infrastructure team. The first of these are pre-established company's standards around generally IT

governance or security or things like that. You know, the standards talk about when and how and why IT infrastructure changes can

happen and how that change needs to happen, who has to approve it and so on. You know, you can't

just throw those standards out when you go to IAC or exempt your infrastructure from them

Imagine what would happen if, say, some big audit came through and you have systems that aren't

following your company's published IT standard. Well, you'd probably fail that audit

and that could be a very expensive proposition. The next consideration is industry compliance

There are additional controls put on the infrastructure if you're in a PCI compliant environment or GDPR, any of the other kind of industry-led standards and rules

Both of these are considerations that put additional burden on using IAC, and it helps, you know, I think explain why code quality needs to be important and how those decisions need to be made

So what do you have to do to ensure quality in IAC

Well, we talked a little bit about audits and governance, but these are definitely factors that you have to take into account

when building infrastructure. While not a quality code construct in and of itself

audits and governance, they provide the structure and methodologies that are going to be enforced through infrastructure as code

You know, these structures need to be managed, managed in a more rigid way than just assuming they're one of the set of requirements

that you're going to use to evaluate your testing success. Instead, reviews and audits should really be set up before the infrastructure work can be

performed, such as deployed to a new environment. These reviews are all around compliance and whether or not the infrastructure meets

the various needs and requirements that they've got. And that the key to this is whenever the infrastructure has changed no matter how small of a change you must confirm that you still in compliance And that more than just a QA test You need to be much more formal about it so a governance audit

These help support the reliability of your infrastructure. And next is a golden construct, because once you have governance, this is the best way to ensure compliance and audit management

This is your definition of each part of the infrastructure. If you're using virtual machines

then you should have the golden construct for VMs that you want to use. You should keep these number to the smallest possible

For example, don't have a golden construct for Ubuntu and a different construct for Debian

unless you really, really, really have to support both operating systems. Instead, standardize whenever possible and use variables to manage those real differences

If you think back to the IAC code that I showed earlier, this one

think about what makes sense to support different variable-driven settings. For example, I think that we'd want to easily configure the Amy value because that could be different from one another

And in many ways, that could easily impact the application. And we certainly know that we want to do that with managing the instance type or where we define the size of the machine that's to be started

That's a logical place to get started because it's easy to see what you may want to use a different size of, you know, hard drive and memory based upon the workload that you're looking to work. with

So your golden construct is really a series of constructs. Each one of your infrastructure components needs a golden construct

Thus, your infrastructure is really a bunch of gold kind of stitched together as a unified

hole, kind of like how a chain necklace is made up of golden links put together

Since your golden construct is literally defined as the definition of that construct that

could be used anywhere, then the reusability expectations for code quality are satisfied

These golden constructs also obviously help drive reliability as well. So automated testing, as I hinted about earlier, is massively important and massively complicated

You can obviously use automated application testing to get some assurance, but when you have to ensure that you have a way to manage the testing of all the rest of the different aspects

Some of this you could do through post-deployment integration testing. You know, I saw one approach where they ran an automated UI-based testing after every deployment is a smoke test

And that test series was a set of test pass that were really carefully calculated to hit every service database and external data feed that the system relies on

thus making sure that any potential infrastructure or coding changes didn't impact the ability of the application to function

Generally added 15 to 20 minutes to every release. So at a release cycle of every two weeks, the cadence and the extra time was pretty minimal

The other area of concerns, the alarms, performance, observability, those kinds of things

There's no real standard way to do this because it really depends heavily on how you manage your operations

but there are some ways that I've seen put this together quite successfully

I've seen, for example, reports set up in Splunk or whatever log monitoring platform that you may use

that evaluates all the incoming data post release and yzes it against previous releases

and then calls out any differences. Some of these differences are expected because of code changes, new features, and things like that

But others come as a real surprise, and those are the ones that you look for. Imagine back to my example I gave earlier where our system was so broken that we're

we couldn't even tell that it was broken. This would have helped us figure that out much sooner

These don't replace your traditional anomaly scanning, but instead are special dashboards that look for expected input that's missing

And lastly, is the code quality tools. We've just seen several talks about those

You know, these tools use various approaches to scan your source code

and look for code smells or, or areas where things like maintainability based on coding standards such as test methods

test method names that need to follow a standard convention or naming conventions that are

followed for constants, methods, and property, those kinds of rules. Boy, I'm drawing some blanks on the rest of them now

My favorite line about this is Sonor Cubes got the coding standards rules promote civility

I really enjoy that. And if you saw Abyshex talk earlier, you'll realize that you've seen this capability in more detail

I think you went over Sonor and Terraform. And we just saw some things with the editor config

But these are all powerful tools to help you statically yze your code

for potential issues around quality. Typically, you most commonly see these being configured or run during a build

say when your unit tests are ran. But I've also seen it set up as a gating event where you can

not complete your code check-in and because of concerns raised by the tool

Generally, these tools are best suited for application code, but they are getting smarter and smarter around code smells and IAC code

So this will be interesting to watch over the next few years as a fully matures

I expect that, you know, probably in the next year to things, you know

open source tools like AWS's cloud developer kit will be pretty powerful

about helping us understand and diagnose the same amount of problems that you're seeing with things like SonorCube or checkmarks

But unfortunately, we're not able to fulfill every aspect of code quality when you consider IAC

One of the examples that I mentioned earlier was that portability is a good goal of code quality

Unfortunately, at this time, that's not possible. even when using a third-party tool such as Pulumi

they're unable to build out an abstract model that can run across multiple providers easily and safely

However, they're getting better and better at this every year. So there is still the opportunity that at some point

you'll be able to write infrastructure code that can be deployed against many different targets. Fortunately, I don't think we're there yet

Now, in summary, there are four different approaches that help assure quality code

when working with IAC. Audits and governance, you're most likely going to have to perform these anyway

So rather than looking at them as a chore, take advantage of the structures that they offer

and use them to support quality code. The next is the golden construct

that one representation of your infrastructure. You should only make updates to your golden construct

when you're 105% confident that they can do the job. Because this is your definition of your infrastructure

structure. Remember, I call it a golden construct, but think of it as a combination of little gold

things all kind of combined together with each infrastructure component adding to the strength of

the overall construct. Next is automated tasting and evaluation. This includes regular

application testing as well as integration testing that stresses every part of the system

This should help you determine things like system rights or installed software problems. As part of

this, you also need to come up with ways to ensure that other areas relying on your system is validated

like alert, slogging, all those other things that you rely on to be able to ensure that your

operational systems are up and working correctly, well, you need to make sure that those still

work after an evaluation or after a change as well. And lastly, take advantage of code quality

tools. They're getting better and better at accessing IAC code