Agentless and Agent-based Scanning Solutions

A Compliance Agent is software provided by a Security Vendor in order to supplement traditional network scans and authenticated scans conducted by a centralized software installation dedicated for such scans. These agents run on a Linux host as a daemon, or on a Windows host as a service, and report to the control plane supplied by the Security Vendor. Some examples of such agents are:

  • Nessus Agent by Tenable
  • Insight Agent by Rapid7
  • Cloud Agent by Qualys

The former scanning solution, done with a centralized scanning software installation, is often referred to as an agentless scanning solution.

Agentless Scanning Solution Architecture

Because an agentless scanning solution needs to perform outbound network connections to all of its targets, including SSH connections for authenticated scans, the solution works best in a centralized-hub-and-spoke Virtual Private Cloud (VPC) architecture. In this architecture, one centralized ‘hub’ VPC is peered with multiple ‘spoke’ VPCs, with each of these spoke VPCs typically representing an environment for a user-facing service. Often, the centralized hub VPC contains services used by the organization for internal uses. For example: scanning software, a VPN server, an artifact repository, a CI/CD system, etc.

However, not all organizations make use of a centralized-hub-and-spoke architecture. For example, picture the following organization’s overall infrastructure:

  • Four unpeered VPCs, each representing an environment for an AWS Elastic Kubernetes Service (EKS) cluster with four self-managed EKS worker nodes, which will run the organization’s user-facing services.
  • CircleCI as the CI/CD system to build Docker images to run on the EKS clusters.
  • AWS Elastic Container Registry as the Docker Registry for the aforementioned images.
  • AWS Systems Manager (SSM) Agent running on the EKS worker nodes, for remote access directly from the AWS console (no need for a VPN server).
  • GitHub as the VCS provider for the organization.
Like what you’re reading?

Subscribe to ShuttleOps and be the first receive notifications on new blogs, releases, features and technology integrations.

In this case, a centralized VPC would have nothing to host! However, this does not change the fact that the organization may need to undergo a SOC2 audit. In this case, a couple of their audit deliverables may be to create organizational procedures for periodic scans of their assets (the self-managed EKS worker nodes), and also a report of their assets run against a CIS level 2 benchmark and the resulting list of security objectives that need to be rectified.

In order to create these deliverables, the organization needs a scanning software, especially one featuring predefined controls, such as the CIS level 2 benchmark for Amazon Linux 2 – which corresponds to the base Linux distribution of the EKS worker nodes. Security Vendors such as Tenable, Rapid7, and Qualys supply products for this exact use case.

Thus, the organization described above is able to leverage the Compliance Agents provided by one of these three vendors in an architecture without a centralized VPC, and without VPC peering:

Here, rather than having a centralized, self-managed scanning software installation, each agent on the organization’s self-managed EKS worker nodes has an outbound connection to a Security Vendor’s managed control plane. The organization can perform scans, generate reports, and pursue outstanding security objectives as needed, without needing to change their mostly vendor-managed, hub-free architecture.

 

Enter ShuttleOps

ShuttleOps believes that minimizing operational overhead for an organization will maximize its ability to continuously deliver software and hence value for its customers. ShuttleOps acts on this belief by providing organizations with a fully-managed CI/CD system for building and deploying artifacts using existing codebases, and also providing the service orchestration control plane to monitor and manage application environments.

ShuttleOps supports Chef Habitat Packages as artifacts for its Build and Deploy Pipelines. We’ve created a Chef Habitat Effortless Package which deploys the Qualys Cloud Agent on an instance in order to promote the use of vendor-supplied Compliance Agents, which we consider to be a security best practice.

Using a Chef Effortless Package for the Compliance Agent

We can leverage a pattern of Chef Habitat Packages called Chef Effortless. This pattern builds Chef Infra Cookbooks and packages them inside the Chef Habitat package. The Chef Infra Cookbooks are pieces of Configuration Management code, similar to Ansible Playbooks or SaltStack States. However, the benefit of the Chef Effortless model is that we can leverage the Chef Infra Cookbooks without the need for a Chef Server: Using ShuttleOps, we can create a Build Pipeline and a Deployment Pipeline for a Chef Effortless package, link the two pipelines together, and thus every time we push to the codebase’s master branch, ShuttleOps will rebuild the package and redeploy it.

Using a Configuration Management solution such as Chef Infra Cookbooks, we can perform a vendor-supported installation of the Compliance Agent:

rpm_package 'qualys-cloud-agent' do
source "#{node['qualys']['assets_dir']}/#{node['qualys']['pkg_name']}"
end

(link to code)

execute 'link-agent' do
command "/usr/local/qualys/cloud-agent/bin/qualys-cloud-agent.sh ActivationId=#{node['qualys']['activation_id']} CustomerId=#{node['qualys']['customer_id']}"
end

(link to code)

Deploying the Qualys Cloud Agent using ShuttleOps

We start by forking the following GitHub repository: https://github.com/ShuttleOps/soc2-habitat-best-practices. We can then visit shuttleops.io and immediately log in using GitHub.

Once that is done, we’ll see in the Connections tab that our VCS provider, GitHub, is automatically connected:

We are going to be building the Chef Habitat Package from the repository we’ve just forked on GitHub, and then deploying the package on AWS EC2 instances. This means that we need an artifact store for the Chef Habitat package and AWS credentials, respectively. Therefore, we need the following connections enabled:

Creating the Build Pipeline

Once the Chef Habitat Builder and AWS connections are created, we can create our first Build Pipeline. When adding the codebase to our build pipeline, we need to make sure to set the plan.sh file path to be that of the effortless-compliance-agent package. That is, effortless-compliance-agent/habitat/plan.sh

We also need to set the Chef Habitat origin to upload the package to. We’ve chosen shuttleops-tutorials – an origin which belongs to us – but any accessible origin can be used:

We can then kick off the Build Pipeline and watch it run to completion:

Creating the Deployment Pipeline

Once the Build Pipeline is complete, we can create a Deployment Pipeline. We need to make sure that we set the following Chef Habitat configuration variables for our Chef Effortless package, in particular:

  • attributes.qualys.is_remote_pkg = true

  • attributes.qualys.pkg_name = “qualys-cloud-agent.x86_64.deb”

  • attributes.qualys.pkg_remote_source = “https://[your artifact store]/qualys-cloud-agent.x86_64.deb”

  • attributes.qualys.activation_id = “[retrieved from the Qualys dashboard]“

  • attributes.qualys.customer_id = “[retrieved from the Qualys dashboard]“

Then, we need to select the deployment targets for our Deployment Pipeline. For this demo, we’ve selected 3 T3.micro EC2 instances.

Finally, we can kick off the Deployment Pipeline and watch it run to completion, provisioning our Deployment Targets and loading our configured service:

Seeing the Compliance Agent in Action

A few minutes after deployment, the Chef Effortless package will have fully completed its Chef Infra run, and the Qualys Cloud Agent will have connected to the Qualys Cloud Platform and configured itself.

Now we can even begin generating reports, for example, we can generate the following CIS level 2 benchmark report for Ubuntu 18.x (the Linux distribution which ShuttleOps’ Deployment Target EC2 Instances run on):

With ShuttleOps building and deploying the Chef Effortless package which performs the vendor-supported installation of the our Compliance Agent of choice, we can deploy an application package built and deployed by ShuttleOps side-by-side with our Compliance Agent Effortless package:

We hope that this article gave some more insight into Compliance Agents, why organizations use them, and how to leverage them in the context of Chef Habitat and ShuttleOps.


Yonathan Koren

Yonathan Koren

Yon is a Solutions Engineer who helps our customers achieve their business goals by implementing technology that promotes productivity, autonomy, and collaboration between developers and operations. You can catch him on webinars and conferences talking about the importance of consistent, composable packaging of applications for deployment across VMs, bare metal, HashiCorp Nomad, or Kubernetes.

  • LinkedIn

Related Posts

The ShuttleOps Factor

DevOps February 2, 2021

In the DevOps space, we all too often focus on the technical tools and practices that you should adopt...

Continue reading

How To Dockerize an Angular Application with...

Docker January 29, 2021

Introduction In this blog post, we will go through a step-by-step guide on how to write a multi-stage dockerfile...

Continue reading

What’s the Difference Between...

Blog January 28, 2021

In the DevOps world, the terms continuous integration, continuous delivery, and continuous deployment are quite common. What’s also quite...

Continue reading