SOC2 Compliance Best Practices with Chef Habitat and ShuttleOps: Compliance Agents

Agentless and Agent-based Scanning Solutions

A Compliance Agent is software provided by a Security Vendor in order to supplement traditional network scans and authenticated scans conducted by a centralized software installation dedicated for such scans. These agents run on a Linux host as a daemon, or on a Windows host as a service, and report to the control plane supplied by the Security Vendor. Some examples of such agents are:

  • Nessus Agent by Tenable

  • Insight Agent by Rapid7

  • Cloud Agent by Qualys

The former scanning solution, done with a centralized scanning software installation, is often referred to as an agentless scanning solution.

Agentless Scanning Solution Architecture

Because an agentless scanning solution needs to perform outbound network connections to all of its targets, including SSH connections for authenticated scans, the solution works best in a centralized-hub-and-spoke Virtual Private Cloud (VPC) architecture. In this architecture, one centralized ‘hub’ VPC is peered with multiple ‘spoke’ VPCs, with each of these spoke VPCs typically representing an environment for a user-facing service. Often, the centralized hub VPC contains services used by the organization for internal uses. For example: scanning software, a VPN server, an artifact repository, a CI/CD system, etc.

However, not all organizations make use of a centralized-hub-and-spoke architecture. For example, picture the following organization’s overall infrastructure:

  • Four unpeered VPCs, each representing an environment for an AWS Elastic Kubernetes Service (EKS) cluster with four self-managed EKS worker nodes, which will run the organization’s user-facing services.

  • CircleCI as the CI/CD system to build Docker images to run on the EKS clusters.

  • AWS Elastic Container Registry as the Docker Registry for the aforementioned images.

  • AWS Systems Manager (SSM) Agent running on the EKS worker nodes, for remote access directly from the AWS console (no need for a VPN server).

  • GitHub as the VCS provider for the organization.

In this case, a centralized VPC would have nothing to host! However, this does not change the fact that the organization may need to undergo a SOC2 audit. In this case, a couple of their audit deliverables may be to create organizational procedures for periodic scans of their assets (the self-managed EKS worker nodes), and also a report of their assets run against a CIS level 2 benchmark and the resulting list of security objectives that need to be rectified.

In order to create these deliverables, the organization needs a scanning software, especially one featuring predefined controls, such as the CIS level 2 benchmark for Amazon Linux 2 – which corresponds to the base Linux distribution of the EKS worker nodes. Security Vendors such as Tenable, Rapid7, and Qualys supply products for this exact use case.

Thus, the organization described above is able to leverage the Compliance Agents provided by one of these three vendors in an architecture without a centralized VPC, and without VPC peering:

Here, rather than having a centralized, self-managed scanning software installation, each agent on the organization’s self-managed EKS worker nodes has an outbound connection to a Security Vendor’s managed control plane. The organization can perform scans, generate reports, and pursue outstanding security objectives as needed, without needing to change their mostly vendor-managed, hub-free architecture.

Enter ShuttleOps

ShuttleOps believes that minimizing operational overhead for an organization will maximize its ability to continuously deliver software and hence value for its customers. ShuttleOps acts on this belief by providing organizations with a fully-managed CI/CD system for building and deploying artifacts using existing codebases, and also providing the service orchestration control plane to monitor and manage application environments.

ShuttleOps supports Chef Habitat Packages as artifacts for its Build and Deploy Pipelines. We’ve created a Chef Habitat Effortless Package which deploys the Qualys Cloud Agent on an instance in order to promote the use of vendor-supplied Compliance Agents, which we consider to be a security best practice.

Using a Chef Effortless Package for the Compliance Agent

We can leverage a pattern of Chef Habitat Packages called Chef Effortless. This pattern builds Chef Infra Cookbooks and packages them inside the Chef Habitat package. The Chef Infra Cookbooks are pieces of Configuration Management code, similar to Ansible Playbooks or SaltStack States. However, the benefit of the Chef Effortless model is that we can leverage the Chef Infra Cookbooks without the need for a Chef Server: Using ShuttleOps, we can create a Build Pipeline and a Deployment Pipeline for a Chef Effortless package, link the two pipelines together, and thus every time we push to the codebase’s master branch, ShuttleOps will rebuild the package and redeploy it.

Using a Configuration Management solution such as Chef Infra Cookbooks, we can perform a vendor-supported installation of the Compliance Agent:

rpm_package 'qualys-cloud-agent' do
source "#{node['qualys']['assets_dir']}/#{node['qualys']['pkg_name']}"

(link to code)

execute 'link-agent' do
command "/usr/local/qualys/cloud-agent/bin/ ActivationId=#{node['qualys']['activation_id']} CustomerId=#{node['qualys']['customer_id']}"

(link to code)

Deploying the Qualys Cloud Agent using ShuttleOps

We start by forking the following GitHub repository: We can then visit and immediately log in using GitHub.

Once that is done, we’ll see in the Connections tab that our VCS provider, GitHub, is automatically connected:

We are going to be building the Chef Habitat Package from the repository we’ve just forked on GitHub, and then deploying the package on AWS EC2 instances. This means that we need an artifact store for the Chef Habitat package and AWS credentials, respectively. Therefore, we need the following connections enabled:

Creating the Build Pipeline

Once the Chef Habitat Builder and AWS connections are created, we can create our first Build Pipeline. When adding the codebase to our build pipeline, we need to make sure to set the file path to be that of the effortless-compliance-agent package. That is, effortless-compliance-agent/habitat/

We also need to set the Chef Habitat origin to upload the package to. We’ve chosen shuttleops-tutorials – an origin which belongs to us – but any accessible origin can be used:

We can then kick off the Build Pipeline and watch it run to completion:

Creating the Deployment Pipeline

Once the Build Pipeline is complete, we can create a Deployment Pipeline. We need to make sure that we set the following Chef Habitat configuration variables for our Chef Effortless package, in particular:

  • attributes.qualys.is_remote_pkg = true

  • attributes.qualys.pkg_name = “qualys-cloud-agent.x86_64.deb”

  • attributes.qualys.pkg_remote_source = “https://[your artifact store]/qualys-cloud-agent.x86_64.deb”

  • attributes.qualys.activation_id = “[retrieved from the Qualys dashboard]“

  • attributes.qualys.customer_id = “[retrieved from the Qualys dashboard]“

Then, we need to select the deployment targets for our Deployment Pipeline. For this demo, we’ve selected 3 T3.micro EC2 instances.

Finally, we can kick off the Deployment Pipeline and watch it run to completion, provisioning our Deployment Targets and loading our configured service:

Seeing the Compliance Agent in Action

A few minutes after deployment, the Chef Effortless package will have fully completed its Chef Infra run, and the Qualys Cloud Agent will have connected to the Qualys Cloud Platform and configured itself.

Now we can even begin generating reports, for example, we can generate the following CIS level 2 benchmark report for Ubuntu 18.x (the Linux distribution which ShuttleOps’ Deployment Target EC2 Instances run on):

With ShuttleOps building and deploying the Chef Effortless package which performs the vendor-supported installation of the our Compliance Agent of choice, we can deploy an application package built and deployed by ShuttleOps side-by-side with our Compliance Agent Effortless package:

We hope that this article gave some more insight into Compliance Agents, why organizations use them, and how to leverage them in the context of Chef Habitat and ShuttleOps. Check out ShuttleOps’ free tier if you’re looking to get started on deploying a Chef Effortless package today!

Related Posts

What Making Coffee Taught Me About The Art...

Back in the day at Starbucks… Over two decades ago during my sophomore year in college, I got a...
Continue reading

ShuttleOps Introduces Enhanced Visibility to...

It’s frustrating when critical information is not available at your fingertips. For DevOps teams, having the right information available...
Continue reading
ShuttleOps No-Code blog

DevSecOps and the power of efficiency!

As the Head of Sales at ShuttleOps, my job isn’t to hard sell you anything. As cliché as it...
Continue reading