Day 5 - Least Privilege using strace

By: Shaun Mouton (@sdmouton)
Edited by: Jennifer Davis (@sigje)

Security in software development has been a hot-button issue for years. Increasing awareness of the threat posed by supply chain breaches have only increased the pressure on teams to improve security in all aspects of the software delivery and operation. A key premise is least privilege: granting the minimum privileges necessary to accomplish a task, in order to prevent folks from accessing or altering things they shouldn't have rights to. Here's my thinking, we should help users to apply the principles of least privilege when designing tools. When we find that we have not designed security tooling which does not enable least privilege use, we can still address the problem using tracing tools which can be found in most Linux distribution package repositories. I would like to share my adventure of looking at an InSpec profile (using CINC Auditor) and a container I found on Docker Hub to demonstrate how to apply least privilege using strace for process access auditing.

At my prior job working at Chef, I fielded a request asking how to run an InSpec profile as a user other than root. InSpec allows you to write policies in code (called InSpec Profiles) to audit the state of a system. Most of the documentation and practice at the time had users inspecting the system as root or a root-equivalent user. At first glance, this makes a certain amount of sense: many tools in the "let's configure the entire system" and "let's audit the security of the entire system" spaces need access to whatever the user decides they want to check against. Users can write arbitrary profile code for InSpec (and the open source CINC Auditor), ship those profiles around, and scan their systems to determine whether or not they're in compliance.

I've experienced this pain of excessive privileges with utilities myself. I can't count the number of times we'd get a request to install some vendor tool nobody had ever heard of with root privileges. Nobody who asked could tell us what it'd be accessing, whether it would be able to make changes to the system, or how much network/cpu/disk it'd consume. The vendor and the security department or DBAs or whoever would file a request with the expectation that we should just trust their assertion that nothing would go wrong. So, being responsible system administrators, we'd say "no, absolutely not, tell us what it's going to be doing first" or "yes, we'll get that work scheduled" and then never schedule the work. This put us in the position of being gatekeepers rather than enablers of responsible behavior. While justified, it never sat right with me.

(Note: It is deeply strange that vendors often can't tell customers what their tools do when asked in good faith, as is the idea that there should be an assumption of trustworthiness in that lack of information.)

I've found some tools over the years which might be able to give a user output which can be used to help craft something like a set of required privileges to run an arbitrary program with non-root privileges. Not too long ago I discussed "securing the supply chain" on how to design an ingestion pipeline to enable folks to run containers in a secure environment where they could be somewhat assured that a container using code they didn't write wasn't going to try to access things that they weren't comfortable with. I thought about this old desire of limiting privileges when running an arbitrary command, and figured that I should do a little digging to see if something already existed. If not maybe I could work towards a solution.

Now, I don't consider myself an expert developer but I have been writing or debugging code in one form or another since the '90s. I hope you consider this demo code with the expectation that someone wanting to do this in a production environment will re-implement what I've done far more elegantly. I hope that seeing my thinking and the work will help folks to understand a bit more about what's going on behind the scenes when you run arbitrary code, and to help you design better methods of securing your environment using that knowledge.

What I'll be showing here is the use of strace to build a picture of what is going on when you run code and how to approach crafting a baseline of expected system behavior using the information you can gather. I'll show two examples:

• executing a relatively simple InSpec profile using the open source distribution's CINC Auditor
• running a randomly selected container off Docker Hub (jjasghar/container_cobol)

Hopefully, seeing this work will help you solve a problem in your environment or avoid some compliance pain.

Parsing strace Output for an CINC Auditor (Chef InSpec) profile

There are other write-ups of strace functionality which go into broader and deeper detail on what's possible using it, I'll point to Julia Evans' work to get you started if you want to know more.

Strace is the venerable Linux debugger, and a good tool to use when coming up against a "what's going on when this program runs" problem. However, its output can be decidedly unfriendly. Take a look in the strace-output directory in this repo for the files matching the pattern linux-baseline.* to see the output of the following command:

        root@trace1:~# strace --follow-forks --output-separately --trace=%file -o
    /root/linux-baseline cinc-auditor exec linux-baseline

You can parse the output, however, if all you want to know is what files might need to be accessed (for an explanation of the command go here) you can do something similar to the following (maybe don't randomly sort the output and only show 10 lines):

awk -F '"' '{print $2}' linux-baseline/linux-baseline.108579 | sort -uR | head
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/minitest-5.14.4/lib/nokogiri.so
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/train-winrm-0.2.12/lib/psych/visitors.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/i18n-1.8.10/lib/rubygems/resolver/index_set.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/aws-sdk-cognitoidentityprovider-1.53.0/lib/inspec/resources/command.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/jwt-2.3.0/lib/rubygems/package/tar_writer.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/aws-sdk-codecommit-1.46.0/lib/pp.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/ffi-1.15.4/http/2.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/bcrypt_pbkdf-1.1.0/rubygems/package.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/aws-sdk-databasemigrationservice-1.53.0/lib/inspec/resources/be_directory.rb
/opt/cinc-auditor/embedded/lib/ruby/gems/2.7.0/gems/aws-sdk-ram-1.26.0/lib/rubygems/resolver/current_set.rb

You can start to build a picture of what all the user would need to be able to access in order to run a profile based on that output, but in order to go further I'll use a much more simple check:

        cinc-auditor exec linux-vsp/
    

Full results of that command are located in the strace-output directory with files matching the pattern linux-vsp.*, but to summarize what cinc-auditor/inspec is doing:

• linux-vsp.109613 - this file shows all the omnibussed ruby files the cinc-auditor command tries to access in order to run its parent process
• linux-vsp.109614 - why auditor is trying to run cmd.exe on a Linux system I don't yet know, you'll get used to seeing $PATH traversal very quickly
• linux-vsp.109615 - I see a Get-WmiObject Win32_OperatingSys in there so we're checking to see if this is Windows
• linux-vsp.109616 - more looking on the $PATH for Get-WmiObject so more Windows checking
• linux-vsp.109617 - I am guessing that checking the $PATH for the Select command is more of the same
• linux-vsp.109618 - Looking for and not finding ConvertTo-Json, this is a PowerShell cmdlet, right?
• linux-vsp.109619 - Now we're getting somewhere on Linux, this running uname -s (with $PATH traversal info in there, see how used to this you are by now?)
• linux-vsp.109620 - Now running uname -m
• linux-vsp.109621 - Now running test -f /etc/debian_version
• linux-vsp.109622 - Doing something with /etc/lsb-release but I didn't use the -v or -s strsize flags with strace so the command is truncated.
• linux-vsp.109623 - Now we're just doing cat /etc/lsb-release using locale settings
• linux-vsp.109624 - Checking for the inetd package
• linux-vsp.109625 - Checking for the auditd package, its config directory /etc/dpkg/dpkg.cfg.d, and the config files /etc/dpkg/dpkg.cfg, and /root/.dpkg.cfg

Moving from that to getting an idea of what all a non-root user would need to be able to access, you can do something like this in the strace-output directory (explainshell here):

    find . -name "linux-vsp.10*" -exec awk -F '"' '{print $2}' {} \; | sort -u >
    linux-vsp_files-accessed.txt

You can see the output of this command here, but you'll need to interpret some of the output from the perspective of the program being executed. For example, I see "Gemfile" in there without a preceding path. I expect that's Auditor looking in the ./linux-vsp directory where the profile being called exists, and the other entries without a preceding path are probably also relative to the command being executed.

Parsing strace output of a container execution

I said Docker earlier, but I've got podman installed on this machine so that's what the output will reflect. You can find the output of the following command in the strace-output directory in files matching the pattern container_cobol.*, and wow. Turns out running a full CentOS container produces a lot of output. When scanning through the files, you see what looks like podman doing podman things, and what looks like the COBOL Hello World application executing in the container. As I go through these files I will call out anything particularly interesting I see along the way:

        root@trace1:~# strace -ff --trace=%file -o /root/container_cobol podman run -it container_cobol
        Hello world!
        root@trace1:~# ls -1 container_cobol.* | wc -l
        146

I'm not going to go through 146 files individually as I did previously, but this is an interesting data point:

        root@trace1:strace-output# find . -name "container_cobol.1*" -exec awk -F '"' '{print $2}' {} \; | sort -u > container_cobol_files-accessed.txt

        root@trace1:strace-output# wc -l container_cobol_files-accessed.txt
        637 container_cobol_files-accessed.txt
        
        root@trace1:strace-output# wc -l linux-vsp_files-accessed.txt
        104754 linux-vsp_files-accessed.txt

So the full CentOS container running a little COBOL Hello World application needs access to six hundred thirty seven files, and CINC Auditor running a 22-line profile directly on the OS needs to access over one hundred four thousand files. That doesn't directly mean that one is more or less of a security risk than the other, particularly given that a Hello World application can't report on the compliance state of your machines, containers, or applications for example, but it is fun to think about. One of the neatest things about debugging using tools which expose the underlying operations of a container exec is that you can reason about what containerization is actually doing. In this case, since we're showing what files are accessed during the container exec, sorting the list, and removing duplicate entries it's a cursory view but still useful.

Let's say we're consuming a vendor application as a container. We can trace an execution (or sample a running instance of the container for a day, strace can attach to running processes), load the list of files into the pipeline we use to promote new versions of that vendor app to prod, and when we see a change in the files that the application is opening we can make a determination whether the behavior of the new version is appropriate for our production environment with all the PII and user financial data. Now, instead of trusting the vendor at their word that they've done their due diligence, we're actually observing the behavior of the application and using our own knowledge of our environment to say whether that application is suitable for use.

But wait! Strace isn't just for files!

I used strace's file syscall filter as an example because it fit the example use case, but strace can snoop on other syscalls too! Do you need to know what IP addresses your process knows about? This example is using a container exec again, but you could snoop on an existing pid if you want then run a similar search against the output (IPs have been modified in this output):

        strace -ff --trace=%network -o /root/yourcontainer-network -s 10241 podman run -it yourcontainer
        for file in $(ls -1 yourcontainer-network.*); do grep -oP 'inet_addr\("\K[^"]+' $file ; done
        127.0.0.1
        127.0.0.1
        693.18.119.36
        693.18.119.36
        693.18.131.255
        75.5117.0.5
        75.5117.0.5
        75.5117.255.255
        161.888.0.2
        161.888.0.2
        161.888.15.255
        832.71.40.1
        832.71.40.1
        832.71.255.255

Have I answered my original question?

With all that knowledge, can we address the original question: Can one use the list of files output by tracing a cinc-auditor run to provide a restricted set of permissions which will allow one to audit the system using CINC Auditor and the profile with a standard user?

Yes, with one caveat: My Very Simple Profile was too simple, and didn't require any additional privileges. I tried with a few other public profiles, but every one I tried ran successfully using a standard user created with useradd -m cincauditor. I looked through bug reports related to running profiles as a non-root user but couldn't replicate their issues - which is good, I suppose. It could be that the issue my customer was facing at the time was a bug in the program's behavior when run as a non-root user which has been fixed, or I just don't remember the use case they presented well enough to replicate it. So here's a manufactured case:

root@trace1:~# mkdir /tmp/foo
root@trace1:~# touch /tmp/foo/sixhundred
root@trace1:~# touch /tmp/foo/sevenhundred
root@trace1:~# chmod 700 /tmp/foo
root@trace1:~# chmod 600 /tmp/foo/sixhundred
root@trace1:~# chmod 700 /tmp/foo/sevenhundred
cincauditor@trace1:~$ cat << EOF > linux-vsp/controls/filetest.rb
> control "filetester" do
>   impact 1.0
>   title "Testing files"
>   desc "Ensure they're owned by root"
>   describe file('/tmp/foo/sixhundred') do
>     its('owner') { should eq 'root' }
>   end
>   describe file('/tmp/foo/sevenhundred') do
>     its('group') { should eq 'root'}
>   end
> end
> EOF
cincauditor@trace1:~$ cinc-auditor exec linux-vsp/

Profile: Very Simple Profile (linux-vsp)
Version: 0.1.0
Target:  local://

  ×  filetester: Testing files (2 failed)
     ×  File /tmp/foo/sixhundred owner is expected to eq "root"

     expected: "root"
          got: nil

     (compared using ==)

     ×  File /tmp/foo/sevenhundred group is expected to eq "root"

     expected: "root"
          got: nil

     (compared using ==)

  ✔  inetd: Do not install inetd
     ✔  System Package inetd is expected not to be installed
  ↺  auditd: Check auditd configuration (1 skipped)
     ✔  System Package auditd is expected to be installed
     ↺  Can't find file: /etc/audit/auditd.conf


Profile Summary: 1 successful control, 1 control failure, 1 control skipped
Test Summary: 2 successful, 2 failures, 1 skipped

cincauditor@trace1:~$ find . -name "linux-vsp.1*" -exec awk -F '"' '{print $2}' {} \; | sort -u > linux-vsp_files-accessed.txt

root@trace1:~# diff --suppress-common-lines -y linux-vsp_files-accessed.txt /home/cincauditor/linux-vsp_files-accessed.txt | grep -v /opt/cinc-auditor
							      >	/home
							      >	/home/cincauditor
							      >	/home/cincauditor/.dpkg.cfg
							      >	/home/cincauditor/.gem/ruby/2.7.0
							      >	/home/cincauditor/.gem/ruby/2.7.0/specifications
							      >	/home/cincauditor/.inspec
							      >	/home/cincauditor/.inspec/cache
							      >	/home/cincauditor/.inspec/config.json
							      >	/home/cincauditor/.inspec/gems/2.7.0/specifications
							      >	/home/cincauditor/.inspec/plugins
							      >	/home/cincauditor/.inspec/plugins.json
							      >	/home/cincauditor/linux-vsp
/root							      <
/root/.dpkg.cfg						      <
/root/.gem/ruby/2.7.0					      <
/root/.gem/ruby/2.7.0/specifications			      <
/root/.inspec						      <
/root/.inspec/cache					      <
/root/.inspec/config.json				      <
/root/.inspec/gems/2.7.0/specifications			      <
/root/.inspec/plugins					      <
/root/.inspec/plugins.json				      <
/root/linux-vsp						      <
							      >	/tmp/foo/sevenhundred
							      >	/tmp/foo/sixhundred
							      >	linux-vsp/controls/filetest.rb
root@trace1:~#

The end of that previous block's output shows compiling the list of files accessed when the cincauditor user runs the profile in the same way we did for the root user, then a diff of the two files. Looking at that output, it's fairly obvious that the profile is trying to access the newly created files which are in a directory we made inaccessible to the cincauditor user (with chmod 700 /tmp/foo), and when we give cinc-auditor access to that directory with chmod 750 /tmp/foo the profile is able to check those files. A manufactured replication of the use case, but it does show that it's possible to use the output to accomplish the task. Whether chmod is the right way to give an least-privilege user access to the files is a question best left up to the implementer, their organization, and their auditors - the purpose of this exercise is to demonstrate the potential value of the strace debugger.

It is important to note that file permissions aren't the only reason why a program wouldn't run. If you're not able to use the information strace gives you to get an application to run as a user with restricted privileges, at least you can get more information about what is happening under the hood and can communicate about why a program is not suitable for your environment. If a program needs to run anyway, you can profile the application's behavior (perhaps a tool built on eBPF would be more suitable than strace for ongoing monitoring in a production environment) and notify when its behavior changes.

Closing thoughts

Over the past few years I've had a lot of thoughts about how do get things done in modern environments, and I've come to the conclusion that it's okay to write shell scripts to get something like this done. Since in this case I'm wrapping arbitrary tasks so I can extract information about what happens when they're running, and I won't be able to predict where I'll need it I figured it was a good idea to use bash and awk as those will be available via package manager where I want to do this sort of thing.

You might not agree, and wish to see something like this implemented in something like Ruby, Python, or Rust (I have to admit that I thought about trying to do this using Rust so as to get better at it), and you're of course welcome to do so. Again, I chose shell since it's something many folks can easily run, look at, comprehend, modify, and re-implement in the way that suits them.

Lastly, thanks very much to Julia Evans. A note about the power of storytelling in one of her posts made me think "I should write a story about solving this problem so I can be sure I learned something from it", and I hope I've done a decent job of emulating her empathy towards folks learning these concepts for the first time. 

Day 16 - Inspec gives insight

By: Jamie Andrews (@vorsprungbike)

Edited By: Ben Cotton (@FunnelFiasco)

What is Inspec?

How often to do want to scan your servers, all of them, to check a library has the correct version or that the right disk clone is mounted?

Configuration control systems like Puppet, Chef and Ansible have become de rigueur in running even modest sized groups of servers. Configuration control offers centralised control over a state which is set by policy.

However, sometimes something happens to one or more servers which challenges this happy state of affairs.

There might be a change requirement that doesn't fit in with the way that the configuration system works. You might decide that part of the system should be managed in a different way. Someone in your team might "temporarily" disable the configuration agent and then not turn it back on again - configuration drift happens

When it does happen, inevitably there are hard to trace and puzzling to understand problems.

This might be a job for a quick shell script and some ssh magic. But often I've done this and then realised that it's difficult to get just right. Inspec is a framework for systematically carrying out these types of tests.

Inspec can be very useful in these circumstances. It works like a configuration management tool like Puppet or Chef in that it holds a model of what the correct configuration should be. However, it does not modify the configuration. Instead it tests the targeted systems for compliance to the correct configuration.

I have found this to be useful in the situation where there are many systems, just a few of which have a problem. Inspec can pinpoint the problem systems, given a configuration profile.

And of course Inspec can be used to proactively check servers to detect problems before they occur

Inspec is based on an earlier system called "serverspec" and it uses the ruby classes from the rspec package.

Although it is promoted by Chef as part of their wider product offering, it works just fine standalone and is fully open source

What I'm covering in the article

Below, I'll look at installing Inspec, making a simple project "profile", how it works in action and installing a third party security checking profile

Setting up and installing

The easiest way of installing it is to use a package. Packages for Redhat and varients, Ubuntu, Suse, MacOSX and MS Windows are available here https://downloads.chef.io/inspec

Inspec has a dependency on Ruby. The packages include a bundled version of Ruby that avoids compatiblity problems. If you already have ruby installed and want to use it then "gem install inspec" is available. See the github repo https://github.com/chef/inspec for more details

To check Inspec is installed try this command

inspec version

Which will come back with the current installed version

Inspec does not have to be installed on all the target nodes. I have it installed on one admin host with ssh access to everything else. This allows any profile rule sets to be tested on anything it can ssh to. No agents, install of ruby or anything else is required

Make your own profile

inspec works by using a profile directory with a a set of control files that contain tests The "init profile" cli command is used to make a new profile

To see a generic blank profile do

inspec init profile example

The output from this command is something like

$ inspec init profile example
WARN: Unresolved specs during Gem::Specification.reset:
      rake (>= 0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Create new profile at /home/jan/temp/example
 * Create directory libraries
 * Create file inspec.yml
 * Create directory controls
 * Create file controls/example.rb
 * Create file README.md

It has made a set of files. The most interesting is "example/controls/example.rb"

This is very simple test that checks if /tmp exists, take a look at it

# encoding: utf-8
# copyright: 2017, The Authors

title 'sample section'

# you can also use plain tests
describe file('/tmp') do
  it { should be_directory }
end

# you add controls here
control 'tmp-1.0' do                  # A unique ID for this control
  impact 0.7                          # The criticality, if this control fails.
  title 'Create /tmp directory'       # A human-readable title
  desc 'An optional description...'
  describe file('/tmp') do            # The actual test
    it { should be_directory }
  end
end

The tests can be declared as "plain tests" or "controls". Being a control adds some metadata which makes it easier to track the test within a set_con

The actual test assertions "it { should be_directory }" follow the rspec syntax. The tests operate on a resource type, in this case "file". There are many useful built in test resourse types, including

• apache_conf
• crontab
• docker
• etc_fstab
• http

And a lot more, see https://www.inspec.io/docs/reference/resources/

A more complex example

Here's a real test I wrote a couple of weeks ago to deal with a DNS configuration drift.

The old DNS servers had been retired but we noticed that some servers still mentioned servers on the old network.

# encoding: utf-8

title 'DNS'

# some crude way to build a list of network interfaces
eth_files= ['/etc/sysconfig/network-scripts/ifcfg-eth0']
eth_files << '/etc/sysconfig/network-scripts/ifcfg-eth1' if file('/etc/sysconfig/network-scri
pts/ifcfg-eth1').exist?
eth_files << '/etc/sysconfig/network-scripts/ifcfs-ens32' if file('/etc/sysconfig/network-scr
ipts/ifcfg-ens32').exist?

control 'resolv' do                        # 
  impact 0.7                               # 
  title 'check old dns is not present'     # 
  desc 'old dns'
  describe file('/etc/resolv.conf') do     # The actual test
    its ('content')  { should_not match /193/ }
  end

  eth_files.each do|ef| 
  describe file(ef) do
    its ('content') { should_not match /^DOMAIN=193/ }
    its ('content') { should_not match /^DNS[123]=193/ }
  end
  end

end

I won't explain exactly how it works, you can see that there are regexps in there and that a ruby "each do" construct is used.

To run the tests do

inspec exec tc-config myuser@myhostname

As mentioned above, Inspec does not correct these problems. It is great at one job: checking compliance. Once the problem is found then you will have to devise a good method for fixing it.

I when looking at large numbers of live servers I usually run it in a shell and then redirect the output to a file. Once the time consuming checking is done I look at the file The colourizing makes it easy to spot the non-compliant areas

In the above example, I found 12 servers non-compliant out of 146. Some problems were found where Puppet conflicted with Redhat system policy I devised a simple, non-idempotent bash script and applied it to the affected servers only. This was quicker and a more certain result than running it on all the servers. After the correction, I reran the Inspec profile to see that everything was ok

Check to check the check

Once you start trying to use your own tests there is always scope for typos or syntax error or other sorts of mayhem. Inspec tries to help with a static checker.

inspec check example

Comes back with a report of how many controls there are plus if everything is valid.

This feature is a great idea, especially for those of us that are only using this tool occassionally.

Ready to use sets of tests

As the Inspec system is supported and promoted by Chef there are a set of profiles ready made that perform various types of compliance. These can be downloaded and used, see https://supermarket.chef.io/tools?type=compliance_profile

Installing and using the CIS DIL

One really useful profile is the CIS Distribution Independent Linux Benchmark https://github.com/dev-sec/cis-dil-benchmark

To try it, clone that github repo and then

 inspec check cis-dil-benchmark

It has 88 controls, many of which check multiple resources. On our LAN running it against a host took over 3 minutes.

The report it generates is interesting reading.

We will be using this profile for testing images generated with Packer and Puppet. The issues it reports will act as feedback for security improvements to our configuration scripts in Puppet and Packer

Further features

I am just scratching the surface with the many features that Inspec offers.
Please do look at http://inspec.io for a fuller run down!

Thanks

Thanks to Chef for supporting this excellent software and keeping it open source

Thanks to my employers, Institute of Physics Publishing for enabling me to try out cool stuff like this as part of my job.

Day 7 - Running InSpec as a Push Job…or…The Nightmare Before Christmas

By: Annie Hedgepeth (@anniehedgie) Edited By: Jan Ivar Beddari (@beddari)

Bored with his Halloween routine, Jack Skellington longs to spread Christmas joy, but his antics put Santa and the holiday in jeopardy! - Disney

I feel a kindred spirit with Jack Skellington. I, too, wanted to spread some holiday-InSpec joy with my client, but the antics of their air-gapped environment almost put InSpec and my holiday joy in jeopardy. All my client wanted for Christmas was to be able to run my InSpec profile in the Jenkins pipeline to validate configuration of their nodes, and I was eager to give that to them.

Sit back and let me tell the holiday tale of how I had no other choice but to use Chef push jobs to run InSpec in an air-gapped environment and why it almost ruined Christmas.

Nothing would have brought me more holiday cheer than to be able to run run the tests as a winrm or ssh command from the Jenkins server directly from a profile in a git repository, not checked out. However, my soul sank as I uncovered reason after reason for the lack of joy for the season:

Scroogey Problems:

1. Network Connectivity: The nodes are in an air-gapped environment, and we needed InSpec to run every time a node was added.
2. Jumpbox Not an Option: I could have PowerShell remoted into the jumpbox and run my InSpec command remotely, but this was, again, not an option for me. You see, my InSpec profile required an attributes file. An attribute is a specific detail about a node, so I had to create an attributes file for my InSpec profile by using a template resource in the Chef cookbook. This is because I needed node attributes and data-bag information for my tests that were specific to that particular Chef environment.
3. SSL Verification: There is an SSL error when trying to access the git repo in order to run the InSpec profile remotely. Chef is working on a feature to disable SSL verification. When that is ready, we can access InSpec via a git link but not now.

Because we were already using push jobs for other tasks, I finally succumbed to the idea that I would need to run my InSpec profiles as ::sigh:: push jobs.

Let me tell you real quickly what a push job is. Basically, you run a cookbook on your node that allows you to push a job from the Chef server onto your node. When you run the push jobs cookbook, you define that job with a simple name like “inspec” and what it does, for example: inspec exec .. Then you run that job with a knife command, like knife job start inspec mynodename.

Easy, right? Don’t get so cheery yet.

This was the high level of what would have to happen, or what you might call the top-of-the-Christmas-tree view:

1. The InSpec profile is updated.
2. The InSpec profile is zipped up into a .tar.gz file using inspec archive [path] and placed in the files/default folder of a wrapper cookbook to the cookbook that we were testing. The good thing about using the archive command is that it versions your profile in the file name.
3. The wrapper cookbook with the new version of the zipped up InSpec profile is uploaded to the Chef server.
4. Jenkins runs the wrapper cookbook as a push job when a new node is added, and the zipped up InSpec profile is added to the node using Chef’s file resource.
5. During the cookbook run, an attributes file is created from a template file for the InSpec profile to consume. The push jobs cookbook has a whitelist attribute to which you add your push job. You’re just telling Chef that it’s okay to run this job. Because my InSpec command was different each time due to the version of the InSpec profile, I had to create basically make the command into a variable, so that meant I had to nest my attributes, which looks like this:

   node['push_jobs']['whitelist'] = {
    'chef-client' => 'chef-client',
    'inspec' => node['mycookbook']['inspec_command']
   }

   The inspec_command attribute was defined like like this (more nesting):

   "C:/opscode/chef/embedded/bin/inspec exec #{Chef::Config[:file_cache_path]}/cookbooks/mycookbook/files/default/mycookbook-inspec-#{default['mycookbook']['inspec_profile_version']}.tar.gz --attrs #{default['mycookbook']['inspec_attributes_path']}"

6. Another Jenkins stage is added that runs the “inspec” push job.

And all of that needs to be automated so that it actually stays updated. Yay…

I will not get into the details of automating this process, but here is the basic idea. It is necessary to leverage a build that is kicked off in Jenkins by a pull request made in git. That build, which is a Jenksinsfile in my InSpec profile, does this: - archives the profile after it merges into master - checks out the wrapper cookbook and creates a branch - adds to new version of the profile to the files/default directory - updates the InSpec profile version number in the attributes file - makes a pull request to the wrapper cookbook’s master branch that also has a pull request build which ensures that Test Kitchen passes before it is merged

So…this works, but it’s not fun at all. It’s definitely the Nightmare Before Christmas and the Grinch Who Stole Christmas wrapped up into one. It takes a few plugins in both Jenkins and BitBucket, which can be difficult to pull off if you don’t have admin rights. I used this blog post as a reference.

I battled internally with a simpler way to do this. A couple of nice alternatives could have been Saltstack and Chef Automate, but neither of those were an option for me. I’m not familiar with Saltstack, but I’m told that its remote execution feature would be able to run InSpec in an air-gapped environment. Likewise, Chef Automate has the Chef Compliance feature which runs all of your InSpec profiles from the Compliance server that you can put in your network. I’m still on the fence about whether those would have been easier to implement, though, because of the heavy dependence I had on the node attributes and data-bags that are stored on the Chef server.

As ugly as this process is, every time I see those all successful test results displayed in the Jenkins output, I can’t help but put a big ol' jolly smile on my face. Sure, it super sucks to jump through all these hoops to get InSpec to work in this environment, but it when the automation works, it just works and no one knows what I had to go through to get it there. It’s like a Christmas miracle.

Do I recommend doing it this way if you don’t have to? No. Is this a great workaround if you have no other way to validate your configuration? Absolutely.

And if you need further convincing of the Christmas magic of InSpec, be sure to read my post last year about how InSpec builds empathy across organizations.

I hope you enjoyed my post! Many special thanks to Jan Ivar Beddari for editing this post and to Chris Webber for organizing this very merry blog for all of us! You can follow me on Twitter @anniehedgie. If you’d like to read more about InSpec, I wrote a whole tutorial series for you to follow here. And if you’d like me and my team at 10th Magnitude to help you out with all things Azure, give us a shout!

Day 2 - Shifting Left Securely With inSpec

Written By: Matt Stratton (@mattstratton)
Edited By: Rich Cassara (@rjcassara)

sta·bil·i·ty
stuh-bil-i-tee

1. the state or quality of being stable.
2. firmness in position.
3. continuance without change; permanence.
4. resistance to change, especially sudden change or deterioration.
5. steadfastness; constancy, as of character or purpose.

Stability is something that we attempt to ensure by reducing the vectors that can make changes to a system. In practice, that means “deployments are executed by trusted individuals with admin access”.

The problem with the “separation of duties” approach to solve all your stablity issues is that everyone can make mistakes. A job title doesn’t imply infallibilty. Every sysadmin who thinks that their title means they won’t make a mistake will tell me a story over a couple of pints about when they totally messed up production.

It’s also a huge burden on one group to be able to understand all the options and variance that steps in a “run book” will result in.

How do we usually do this? We might have a project that is 8 sprints long, but we don’t think about security until the 8th sprint which is the “hardening” sprint.

Naturally, we fail all of this hardening because we haven’t been thinking about security until now. Security and compliance need to be a consideration from the very beginning. We do this with tooling AND with culture. In this post, I’ll be mostly focusing on how to leverage tooling to assist in supporting the cultural changes required. Specifically, I will be focusing on using inSpec to verify your infracode (in this case I’ll be using Chef, but this will work just as well with other infracode tools).

Shift left

What is “shifting left” anyway? The idea is that we move our testing further “to the left” in our pipeline. The closer to the introduction of the defect that it is discovered, the cheaper and easier it is to fix.

Of course, this introduces some challenges, right? If my tests don’t pass, I can change the tests. We need to structure our systems to prevent this.

How does this help with security?

Compliance and security are just another aspect of quality. It would sound ludicrous to only QA test our applications right before we deploy to production. It should sound just as ridiculous to wait until a “hardening sprint” to start thinking about security and compliance.

Another problem is that often times, these tests we do at the end are heavy. That is to say, they are figuratively or literally expensive; they require large resources, or expensive per-seat licenses, so we don’t use them as often as we should.

We need something better.

Trying to prevent specific behaviors is a losing battle

• You spend time keeping people from doing x, y, or z
• They do a, b, or c to get the outcome they want
• No communication happens. Insights are lost.

No matter how much you try to block the “how”, you should focus on the “what”. Consider the outcome, not the mechanism that lead to that outcome.

Perceived problem with distributed configuration management

When I suggest to sysadmins that developers should write Chef cookbooks, this is generally what they think is going to happen:

• Developer reads on Stack Overflow that disabling selinux will make his Node app work better
• Developer updates his cookbook to disable selinux
• Sysadmins get fired because of 3viL haxx0rz

The better way

• Developer reads on Stack Overflow that disabling selinux will make his Node app work better
• Developer updates his cookbook to disable selinux
• Developer runs local tests which include compliance checks
• Compliance checks test for state of selinux
• Tests fail. Developer says “Welp, I guess I can’t do that.”

What if the developers don’t run those local tests?

The pipeline catches them.

They’ll do better next time.

Even organizations that have a high-trust culture test everything. Take a page from Ronald Reagan - “Trust, but verify”. I don’t even trust myself to remember to test all the time. Remember, we enable local testing, but we don’t count on it.

If you truly care about a thing, you care enough to write a test

Often times, the excuse given is “I don’t have time to write a test for this thing”. That’s a cop-out, and I’m here to tell you that it won’t fly. This goes back to the point of lightweight tools for compliance and testing - if it’s too onerous to write these tests, nobody will do them. The good news is, it’s not that hard.

When you are writing these tests, think about this: we are testing for outcomes. Outcomes are what matter. Our pipeline is creating an artifact (in the case of infracode, this artifact is a converged node). What matters is the state of that outcome. How you got it there is not the question. We should be testing compliance and security against artifacts, and the outcome we are testing is “is this thing the way it should be, or is it a scary nightmare that should never see production”?

Democratize your testing

Remember when I talked about the hubris of sysadmins? Infosec folks do it too. Tools are kind of the least important thing to think about, but make sure it’s not a tool that can only run tests from the security folks. If you care about compliance, move that stuff to the left. If your tool can’t do that, it’s time to find another tool.

That doesn’t mean you need to throw out what you have, but consider adding something. The more that you can emulate whatever you care about testing in production into your pipeline, the happier you will be. Monitoring is just testing with a time dimension; this applies to compliance as well.

What you do not want to do is have a test in the pipeline that cannot be run by the local developer. Basically, if you do this, you’re a big meanie[1].

Enter inSpec

InSpec is compliance as code. We write controls that test for the state of the system, and report back on this state. It’s also pretty easy to read, as it’s written in a spec-style langauge. Consider the difference between these two ways of checking the SSH protocol version on a system:

Shell

> grep "^Protocol" /etc/ssh/sshd_config | sed 's/Protocol //'
 2
 ```
 ### inSpec
```ruby
control 'ssh-1234' do
  impact 1.0
  title 'Server: Set protocol version to SSHv2'
  desc "
    Set the SSH protocol version to 2. Don't use legacy
    insecure SSHv1 connections anymore...
  "

  describe sshd_config do
   its('Protocol') { should eq 2 }
  end
end

As you can see, with inSpec, not only is if fairly human-readable, it provides context. We understand the impact of the control, and we can even add a nice description to explain what this is all about. The user experience[2] is better.

Examples

(All code for this post can be found at github.com/mattstratton/sysadvent-2017)

Cookbook

We start by taking a look at a basic cookbook that creates a user and gives it ownership of the /var/log directory (which we probably don’t want to happen):

user 'sa2017-app' do
  comment 'SysAdvent User'
  system true
  shell '/bin/false'
end

directory '/var/log' do
  owner 'sa2017-app'
end

Compliance profile

Our compliance profile for confirming that /var/log is owned by root looks like this:

title 'Directory Tests'

control 'log-1.0' do                        
  impact 0.7                                
  title 'Ownership of /var/log'             
  desc 'The /var/log directory should be owned by root'
  describe file('/var/log') do
    it { should be_directory }
    it { should be_owned_by 'root' }
  end
end

Tying it together

In this case, I’ve created a Jenkins pipeline that responds to changes on the cookbook’s GitHub repo. The Jenkinsfile defines the following stages:

1. Lint - Runs foodcritic and cookstyle against the cookbook
2. Smoke - Runs kitchen test using the tests in the cookbook itself
3. Compliance - Runs kitchen test, but includes our compliance profile, loaded for a git URL.

In order to accomplish the Compliance stage, we generate a new kitchen.yml file, which includes the path to the test we want to include. There are other ways to accomplish this, and it will depend upon how you build your pipelines. But hopefully this illustrates one way it can be done.

To prove the point, here’s the output in Jenkins when the Compliance stage is run:

Profile: My Compliance (sa2017-compliance)
Version: 0.1.0
Target:  docker://4967508ea0b8f431b161edf561164ab8a49eba780b58ed85673f64c60b3bb8bd

 [38;5;9m  ×  log-1.0: Ownership of /var/log (1 failed) [0m
 [38;5;41m     ✔  File /var/log should be directory [0m
 [38;5;9m     ×  File /var/log should be owned by "root"
     expected `File /var/log.owned_by?("root")` to return true, got false [0m

Profile Summary: 0 successful controls,  [38;5;9m1 control failure [0m, 0 controls skipped

[1] - Sorry for the strong language.
[2] - And I bet you thought “user experience” was just for front-end developers.

Day 3 - Building Empathy: a devopsec story

Written By: Annie Hedgpeth (@anniehedgie)
Edited By: Kerim Satirli (@ksatirli)

’Twas the night before Christmas, and all through the office not a creature was stirring … except for the compliance auditors finishing up their yearly CIS audits.

Ahh, poor them. This holiday season, wouldn’t you love to give your security and compliance team a little holiday cheer? Wouldn’t you love to see a bit of peace, joy, and empathy across organizations? I was lured into technology by just that concept, and I want to share a little holiday cheer by telling you my story.

I’m totally new to technology, having made a pretty big leap of faith into a career change. The thing that attracted me to technology was witnessing this display of empathy firsthand. My husband works for a large company who specializes in point-of-sale software, and he’s a very effective driver of DevOps within his organization. He was ready to move forward with automating all of the things and bringing more of the DevOps cheer to his company, but his security and compliance team was, in his eyes, blocking his initiatives - and for good reason!

My husband’s year-long struggle with getting his security and compliance team on board with automation was such an interesting problem to solve for me. He was excited about the agile and DevOps methodologies that he had adopted and how they would bring about greater business outcomes by increasing velocity. But the security and compliance team was still understandably hesitant, especially with stories of other companies experiencing massive data breaches in the news with millions of dollars lost. I would remind my husband that they were just trying to do their jobs, too. The security and compliance folks aren’t trying to be a grinch. They’re just doing their job, which is to defend, not to intentionally block.

So I urged him to figure out what they needed and wanted (ENTER: Empathy). And what he realized is that they needed to understand what was happening with the infrastructure. I can see how all of the automated configuration management could have caused a bit of hesitation on behalf of security and compliance. They wanted to be able to inspect everything more carefully and not feel like the automation was creating vulnerability issues that were out of their control.

But the lightbulb turned on when they realized that they could code their compliance controls with a framework called InSpec. InSpec is an open-source framework owned by Chef but totally platform agnostic. The cool thing about it is that you don’t even need to have configuration management to use it, which makes it a great introduction to automation for those that are new to DevOps or any sort of automation.

(Full-disclosure: Neither of us works for Chef/InSpec; we’re just big fans!)

You can run it locally or remotely, with nothing needing to be installed on the nodes being tested. That means you can store your InSpec test profile on your local machine or in version control and run it from the CLI to test your local machine or a remote host.

# run test locally
inspec exec test.rb

# run test on remote host on SSH
inspec exec test.rb -t ssh://user@hostname

# run test on remote Windows host on WinRM
inspec exec test.rb -t winrm://Administrator@windowshost --password 'your-password'

# run test on Docker container
inspec exec test.rb -t docker://container_id

# run with sudo
inspec exec test.rb --sudo [--sudo-password ...] [--sudo-options ...] [--sudo_command ...]

# run in a subshell
inspec exec test.rb --shell [--shell-options ...] [--shell-command ...]

The security and compliance team’s fears were finally allayed. All of the configuration automation that my husband was doing had allowed him to see his infrastructure as code, and now the security and compliance team could see their compliance as code, too.

They began to realize that they could automate a huge chunk of their PCI audits and verify every time the application or infrastructure code changed instead of the lengthy, manual audits that they were used to!

Chef promotes InSpec as being human-readable and accessible for non-developers, so I decided to learn it for myself and document on my blog whether or not that was true for me, a non-developer. As I learned it, I became more and more of a fan and could see how it was not only accessible, but in a very simple and basic way, it promoted empathy between the security and compliance teams and the DevOps teams. It truly is at the heart of the DevSecOps notion. We know that for DevOps to deliver on its promise of creating greater velocity and innovation that silos must be broken down. These silos being torn down absolutely must include those of the security and compliance teams. The InSpec framework does that in such a simple way that it is easy to gloss over. I promise you, though, it doens’t have to be complicated. So here it is…metadata. Let me explain.

If you’re a compliance auditor, then you’re used to working with PDFs, spreadsheets, docs, etc. One example of that is the CIS benchmarks. Here’s what a CIS control looks like.

And this is what that same control looks like when it’s being audited using InSpec. Can you see how the metadata provides a direct link to the CIS control above?

control "cis-1-5-2" do
  impact 1.0
  title "1.5.2 Set Permissions on /etc/grub.conf (Scored)"
  desc "Set permission on the /etc/grub.conf file to read and write for root only."
  describe file('/etc/grub.conf') do
    it { should be_writable.by('owner') }
    it { should be_readable.by('owner') }
  end
end

And then when you run a profile of controls like this, you end up with a nice, readable output like this.

When security and compliance controls are written this way, developers know what standards they’re expected to meet, and security and compliance auditors know that they’re being tested! InSpec allows them to speak the same language. When someone from security and compliance looks at this test, they feel assured that “Control 1.5.1” is being tested and what its impact level is for future prioritization. They can also read plainly how that control is being audited. And when a developer looks at this control, they see a description that gives them a frame of reference for why this control exists in the first place.

And when the three magi of Development, Operations, and Security and Compliance all speak the same language, bottlenecks are removed and progress can be realized!

Since I began my journey into technology, I have found myself at 10th Magnitude, a leading Azure cloud consultancy. My goal today is to leverage InSpec in as many ways as possible to add safety to 10th Magnitude’s Azure and DevOps engagements so that our clients can realize the true velocity the cloud makes possible.

I hope this sparked your interest in InSpec as it is my holiday gift to you! Find me on Twitter @anniehedie, and find much more about my journey with InSpec and technology on my blog.