A most trustworth.ee corner of the Internet.

Writing My First Sigma Rule: Container Residence Discovery

2023-08-24

Contents

• Background
• Getting Started
  • What The Blog Suggests We Look For
  • Formulating Our Approach
• Looking Around At Other Information
  • Putting it all together
• Open Source Community and Conformance
  • The Sigma Rules
• Learning More and Thanks

Background

One of the things that I love most about the wandering path of my career journey is that I get to do lots of new things. I enjoy building on things that I know to do things that I’ve never done before.

I’ve been involved in one form or another with detection engineering (DE) for the last 10 years. My detection engineering team at my day job uses git and a CI/CD pipeline to manage and deploy our detection rules into our SIEM. So from that perspective, this blog is about “things I know”.

But I don’t have much history contributing technical content to open source projects, and I’ve never authored a Sigma rule. So that will be “things I’ve never done before”.

By and large, the experience of writing and submitting my first Sigma rule was really painless and enjoyable. For anybody involved or looking to get involved with DE, I can recommend the project as a great way to learn more and give back to the community.

Getting Started

Back in July, I noticed a post from @Skyper@fosstodon.org on detecting if your commands were executing from a shell that’s resident in a container or not. I took a few minutes to read their blog, and I learned some new things about container residence discovery.

I had also been working through a training course on writing Sigma rules, and it felt like a great opportunity to try out my new learnings alongside this new rule format, so let’s begin!

You should take some time to read through the blog, it’s well-done. But I’m going to summarize what I took away from it:

• Given command execution on a Linux host, how can an attacker determine from their environment whether they are in a native Linux host, or if they are containerized?
• Thinking like a defender, I want to expand this a little bit:
  • If I have containers in my environment, this could lead to some benign positive cases, eg. developers or admins doing the same activity
  • In most cases, I would expect that people working with containers would know that they’re operating in a container for much of the work they do
  • Much like other system discovery activity (whoami, id, arp/ping interrogation, etc.) this is probably going to catch some folks who need to act like tourists
• The blog is a great summary of the author’s approach and thought process, but I’m not deeply versed in containers myself. I’ll need to do a bit more research to see if there are good or bad sides to what they’ve discussed.

What The Blog Suggests We Look For

1. On a native Linux system, the output of ls -id / will emit the inode number for the root directory. If it’s low (like 1 or 2) you’re native; if it’s large you’re probably containerized.
2. On a native Linux system, the contents of /proc/1/cgroup likely end in / but they’re much different under Docker.
3. On a Docker containerized system, the presence of /.dockerenv is expected.

Formulating Our Approach

With this information in hand, I’m starting to piece together a plan:

1. Each of the approaches will rely on command-line logs like from MDE on Linux, auditd, or similar.
2. None of the activity requires that we, as detection engineers, know the contents of the files. If it did, we could use something like osquery file integrity monitoring. But in our case, we don’t have to care if a container exists in our environment. We just need to see if someone is looking for a container. If they try to find one, we want to catch them.
3. Viewing the world through the lens of the MITRE ATT&CK enterprise matrix:
   1. This is definitely ‘TA0007 - Discovery’ activity. We’re trying to find an attacker who is discovering the environment they reside in.
   2. While you might think to categorize this as T1613 - Container and Resource Discovery, it doesn’t feel like discovering Container assets exactly. It’s using system information to infer containerization status. As a result, I’d call this T1082 - System Discovery.
4. Some other thoughts about the command side of things:
   1. A command like ls -id / could be written a lot of different ways, because switch positions are interchangeable. So something like ls -id / and ls -di / and even ls -pdqi / would all give the attacker what they’re looking for.
   2. The /proc virtual file system exposes system information as files. Since the blog describes what the attacker is looking for, we should imagine all of the default or common tools for examining files. We want to cover cat /proc/1/cgroup as well as grep docker /proc/1/cgroup or a number of creative combinations. Strategically, anything that attempts to emit or match the contents of that virtual file will provide an attacker with their goal of knowing about the environment.

One other important thing: I need to take a look at the Sigma rule repository and see if anyone else has done anything else like this. If they have, I should check there work and see if it needs updated. If they haven’t, then it’d be a good candidate for a new rule.

To do so, let’s interrogate the repository:

(I submitted the initial version of this rule on Jul 31, 2023. If you want to follow along, checkout the Jul 28 commit of the SigmaHQ rules repository, with id 8dca7aa1ba3c119a6aa6f6bad49252eaf2fd5514).

    git clone https://github.com/SigmaHQ/sigma.git  
    git checkout 8dca7aa1ba3c119a6aa6f6bad49252eaf2fd5514  

And let’s do some light reconnaissance on whether something like what we’ve described already exists:

    cd sigma/rules/linux  
    grep -rl T1082 ./*  
    grep -rl T1613 ./*  
    grep -rl ontainer ./*  

Looking at that output, we should check the contents of:

• auditd/lnx_auditd_system_info_discovery.yml
• auditd/lnx_auditd_system_info_discovery2.yml
• process_creation/proc_creation_lnx_system_info_discovery.yml
• auditd/lnx_auditd_modify_system_firewall.yml

None of these match what we’re trying to detect, so that’s good.

Looking Around At Other Information

Given my relatively rudimentary level of experience with containers, I hunted around for more answers to “How can I tell if my process is in a container?”. I found some great info on Stack Overflow that both confirmed the’s approach and uncovered a few new techniques:

How to determine if a process runs inside LXC / Docker

• Outside of a container environment, we can expect /proc/2 to be kthreadd.
• Outside of a container environment, /proc/*/sched should emit a pid that matches the pid we interrogate (so /proc/1/sched should show pid 1, /proc/7/sched should show pid 7, etc.)

Putting it all together

So I ended up with an initial rule that:

• Looked for inode interrogation of /
• Looked for someone finding kthreadd at /proc/2
• Looked for various attempts to emit or interrogate file content of /proc/*/cgroup and /proc/*/sched for mismatches
• Looked for finding /.dockerenv

Putting all of this into a sigma rule looked like this: Lnx container discovery #4380

Open Source Community and Conformance

As instructed, I tried to make what I was doing conform to the SigmaHQ standards and Rule Creation Guide.

The ultimate outcomes? A really straightforward process! My rule got picked up after PR and sat waiting a short time for review. Once the review was completed, since it was fairly complicated, it was divided into three distinct rules.

The Sigma Rules

1. Container Residence Discovery Via Proc Virtual FS
2. Docker Container Discovery Via Dockerenv Listing
3. Potential Container Discovery Via Inodes Listing

Learning More and Thanks

So there we are! A few hours of research, some formatting, and a PR later… then wait some short few days to get a review… and voila! Three Sigma rules by me! :)

• Thanks to my reviewers on the Sigma rule(s) - nasbench and phantinuss
• I’ve been enjoying my review of this Detection Engineering with Sigma training from Applied Network Defense by Josh Brower, and I’m liking it a lot so far! You should check out @DefensiveDepth’s work, IMO.
• Cheers to Skyper for their great work on the blog; I’m gonna have to look at more things over there as well, and of course I followed on Mastodon as well.
• I finally got around to toying with Docker in the homelab. Of course I’m sure there’s more of that on the way as well.