Detect Container Orchestration Job in Splunk
Adversaries may abuse task scheduling functionality provided by container orchestration tools such as Kubernetes to schedule deployment of containers configured to execute malicious code. Container orchestration jobs run these automated tasks at a specific date and time, similar to cron jobs on a Linux system. Deployments of this type can also be configured to maintain a quantity of containers over time, automating the process of maintaining persistence within a cluster. In Kubernetes, a CronJob may be used to schedule a Job that runs one or more containers to perform specific tasks. An adversary may utilize a CronJob to schedule deployment of a Job that executes malicious code in various nodes within a cluster.
MITRE ATT&CK
- Technique
- T1053 Scheduled Task/Job
- Sub-technique
- T1053.007 Container Orchestration Job
- Canonical reference
- https://attack.mitre.org/techniques/T1053/007/
SPL Detection Query
index=kubernetes sourcetype="kube:audit" OR sourcetype="kube:apiserver:audit"
| spath input=_raw
| search (objectRef.resource="cronjobs" OR objectRef.resource="jobs")
(verb="create" OR verb="update" OR verb="patch")
| eval user=spath(_raw, "user.username")
| eval namespace=spath(_raw, "objectRef.namespace")
| eval resource_name=spath(_raw, "objectRef.name")
| eval source_ip=spath(_raw, "sourceIPs{0}")
| eval user_agent=spath(_raw, "userAgent")
| eval request_body=spath(_raw, "requestObject")
| eval IsSuspiciousImage=if(match(request_body, "(alpine|busybox|ubuntu|debian|kali|python|perl|ruby|php|ncat|netcat|nmap|masscan)"), 1, 0)
| eval IsSuspiciousCommand=if(match(request_body, "(curl|wget|bash\s+-i|/dev/tcp|nc\s+|ncat|python\s+-c|perl\s+-e|ruby\s+-e|chmod\s+\+x|base64\s+-d|eval|/bin/sh\s+-c|/bin/bash\s+-c|mkfifo|socat)"), 1, 0)
| eval IsHostPathMount=if(match(request_body, "hostPath"), 1, 0)
| eval IsPrivileged=if(match(request_body, "privileged"), 1, 0)
| eval IsHostNetwork=if(match(request_body, "hostNetwork"), 1, 0)
| eval IsSensitiveMount=if(match(request_body, "(/etc|/var/run/docker\.sock|/proc|/sys)"), 1, 0)
| eval SuspicionScore=IsSuspiciousImage + IsSuspiciousCommand + IsHostPathMount + IsPrivileged + IsHostNetwork + IsSensitiveMount
| where SuspicionScore > 0
| table _time, objectRef.resource, verb, namespace, resource_name, user, user_agent, source_ip, IsSuspiciousImage, IsSuspiciousCommand, IsHostPathMount, IsPrivileged, IsHostNetwork, IsSensitiveMount, SuspicionScore
| sort - SuspicionScore - _timeDetects suspicious Kubernetes CronJob and Job creation or modification via Kubernetes API audit logs in Splunk. Parses JSON audit log fields to identify dangerous container configurations including suspicious base images, shell injection commands, privileged containers, host path mounts, and sensitive volume mounts. A suspicion score aggregates risk indicators for analyst prioritization.
Data Sources
Required Sourcetypes
False Positives & Tuning
- Legitimate CI/CD pipeline jobs that use base images like alpine or ubuntu for build tasks
- Cluster maintenance CronJobs that use curl or wget to check service health endpoints
- Log rotation or data cleanup jobs that use shell commands like /bin/sh -c
- Security scanning jobs (Falco, Trivy, kube-bench) that mount host paths for vulnerability assessment
- Operators and controllers that create jobs programmatically as part of normal cluster operations
Other platforms for T1053.007
Testing Methodology
Validate this detection against 4 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Create Malicious Kubernetes CronJob with Reverse Shell Command
Expected signal: Kubernetes API audit log: verb=create, objectRef.resource=cronjobs, objectRef.namespace=default, objectRef.name=argus-test-cronjob. Request body will contain image=alpine:latest, command /bin/sh -c, args with nc reverse shell. Subsequent Job and Pod creation events will appear as the schedule triggers. Pod logs will show connection failure.
- Test 2Create Privileged Kubernetes Job with Host Path Mount
Expected signal: Kubernetes API audit log: verb=create, objectRef.resource=jobs, request body contains image=busybox:latest, securityContext.privileged=true, hostPath.path=/etc. Pod creation event follows. Container process execution visible in node container runtime logs and Falco alerts if deployed.
- Test 3Create Kubernetes CronJob Using Docker Socket Mount for Container Escape
Expected signal: Kubernetes API audit log: verb=create, objectRef.resource=cronjobs, request body contains hostPath.path=/var/run/docker.sock. Schedule */5 * * * * triggers Jobs periodically. Container execution attempts to access the Docker socket. Falco rule 'Write below monitored dir' or 'Container with sensitive mount' may trigger.
- Test 4Deploy Cryptomining CronJob via kubectl
Expected signal: Kubernetes API audit log: verb=create, objectRef.resource=cronjobs, request body contains image=ubuntu:20.04, command with curl download, chmod +x, and execution of downloaded binary. Subsequent pod creation when schedule triggers. Container logs show download failure. Sysmon/auditd on node would show curl process and chmod if running.
References (11)
- https://attack.mitre.org/techniques/T1053/007/
- https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
- https://kubernetes.io/docs/concepts/workloads/controllers/job/
- https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/
- https://falco.org/docs/rules/
- https://github.com/falcosecurity/falco/blob/master/rules/falco_rules.yaml
- https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1053.007/T1053.007.md
- https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
- https://docs.microsoft.com/en-us/azure/aks/monitor-aks
- https://www.cncf.io/blog/2021/12/06/cloud-native-security-whitepaper/
- https://github.com/SigmaHQ/sigma/tree/master/rules/cloud/kubernetes
Unlock Pro Content
Get the full detection package for T1053.007 including response playbook, investigation guide, and atomic red team tests.