Monitoring Docker Containers

with Splunk

Marc Chéné | Product Manager

Sept 27, 2017 | Washington, DC

 Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
 Who I am

Marc Chéné
• Product Manager, Engineer, APMer
• Dad/ super fan/ coach to 3, loves skiing, golfing,
music and a good drink
@marcchene
https://www.linkedin.com/in/marcchene
slack id: mchene
 Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
 Agenda

▶ Monitoring Options
▶ Analytical Insight – Tips & Tricks
▶ The World of Metrics
▶ (coming soon!) Docker Moby - v2 logging Plugin
Monitoring Options
logs, events and perf stats
 Splunk and Docker – At A Glance

Visibility in your
Monitoring for your Delivering Splunk as
Container
Cloud Environments Containers
Environments
Splunk Logging Driver for Deep Visibility in Amazon Make getting Splunk as easy
Docker Web Services (AWS) and in as a single Docker pull
• Built into Docker – no EC2 Container Services command from the Docker
extra software required (ECS) Hub/Store
• Insight into container and
apps running in containers Splunk provides support for Forwarders and Splunk
Google Cloud Platform (GCP) Enterprise pre-configured to
Docker Universal Control collect machine data from
Plane Container Host and Docker
• Insight into administration, API
changes, and composition
 Splunk Collection Options for Docker

• Docker Native Logging – Splunk logging driver, Syslog, JSON, AWS CloudWatch, etc.

• Forwarders – App Logs, Syslog UDP forwarding, Performance, etc.

• Logging libraries in .NET, Java and node.js

• Custom (e.g., Kafka with HTTP Event Collector)

• Cloud – AWS, GCP, Azure

Use the option that is right for you!

 Log Streaming - Splunk Logging Driver for Docker

▶ Secure—supports TLS/SSL and

tokens
▶ Simple – config-based setup and
collect data
▶ Buffering

▶ Scale – Based on HTTP Data

Collector Based on Splunk HTTP
▶ Configurable - Supports
container labels, environment
variables
 Log Streaming - Splunk Logging Driver for Docker
v1.13+
▶ Skip verification for the valid
splunk url
▶ Raw data collection from
the native log driver
▶ Embedded json format
support
▶ Performance Improvements
 Docker Hub/Store

▶ Splunk container images available

• Splunk Enterprise 6.6.3
• Splunk Universal Forwarder 6.6.3
▶ Includes configuration and Docker Add-On for container
monitoring out-of-the-box

docker pull store/splunk/enterprise

docker pull store/splunk/universalforwarder:6.6.3
 Deep Dive: What’s Do We Monitor?

▶ Docker Hub: https://hub.docker.com/r/splunk/universalforwarder/ tag: 6.5.3-

monitor
▶ GitHub: https://github.com/splunk/docker-itmonitoring
• Docker logs (ta-dockerlogs_fileinput) under “/host/containers/*/”
• [a-f0-9]+-json.log
• config.v2.json
• hostconfig.json
• hostname
• hosts
• resolv.conf
• Docker stats (ta-dockerstats)
• UCP logs (ta-ucplogs-sysloginput)
Demo Monitoring!
Analytical Insight –
Tips & Tricks
 Analytical Insight – Tips & Tricks

▶ Sample Docker Compose file

▶ Correlations
• Docker SWARM mode
• Amazon Web Services (AWS)
▶ Log Options
• --log-opt tag="{{.Name}}/{{.FullID}}"
The World of Metrics
 Terminology - What is a Measurement?
Treated natively as metrics, not log files

ABC.XYZ

​Time ​Metric Name ​Measure ​Dimensions

​Host (10.1.1.100,
​system.cpu.idle ​numeric data point, web01.splunk.com)
different types such as ​Region (e.g., us-east-1, us-

count, gauge, timing, west-1, us-west-2, us-central1)

sample, etc ​ IntanceTypes (e.g., t2.medium,

t2.large, m3.large)
 © 2017 SPLUNK INC.

“Splunk provides ONE platform

to analyze and investigate
across both Logs and Metrics
 Metrics Data Shape
Field Required Description
_time Y Microseconds since epoch
metric_name Y metric name
_value Y Value of the metric (numeric values only)
_dims Y Dimension names

host Y Origination Host

index Y Index to store the data
metric_type N Counter|Gauge – assume Gauge if not specified.
source N the source of the data
point, https://docs.splunk.com/Splexicon:Source
sourcetype Y Used for defining groupings of metrics and defining input time
rules
<fieldA>..<fieldZ> N Arbitrary number of dimensions

19
 Key Features

SPL

​Metric Store ​mstats ​Metrics Catalog

​Ability to ingest and store ​tstats equivalent to ​REST APIs to query lists
metric measurements at query time series from of ingested metrics and
scale metrics indexes dimensions
 Metrics Store

▶ Based on splunkd
▶ Dedicated Indexes for Metrics and Logs

▶ Full part of the platform

• RBAC
• Clustering
• Index Management
• Central Administration
▶ Optimized for fast time series queries and ingestion of metrics at scale
 SPL: mstats
▶ mstats
• New SPL command
• Built off of tstats,
http://docs.splunk.com/Documentation/Splunk/6.6.1/SearchReference/Tstats
• Syntax
• | mstats <stats-fun>…
[WHERE index=<mymetricindex> metric_name=<metricname>…]
[BY <dimension-list> [span=<timespan>] ]
• Sample
• Stats:
| mstats avg(_value), count(_value)
WHERE metric_name="*.cpu.percent" by metric_name span=30s
• Time Series Visualization:
| mstats avg(_value), count(_value)
WHERE metric_name="*.cpu.percent" by metric_name span=30s
| timechart first(avg(_value)) as "avg" span=30s by metric_name
 Metrics Catalog: Discovery & Search

▶ GET /services/catalog/metricstore/metrics • List all the dimension names that are

• List all metric names compatible with a given metric name
curl -k -u admin/pass "mem.free":
https://localhost:8089/services/catalog/metricstore/ curl -k -u admin/pass
metrics https://localhost:8089/services/catalog/metricstore/
dimensions?metric=mem.free
• List all metric names that apply to a given
dimension name "dc” • List all the dimension values for a given
curl -k -u admin/pass dimension name "dc”
https://localhost:8089/services/catalog/metricstore/ curl -k -u admin/pass
metrics?dimension=dc https://localhost:8089/services/catalog/metricstore/
dimensions/dc/values

▶ GET /services/catalog/metricstore/dimensions • List all the dimension values for a given

dimension name "dc" and metric name
• List all dimension names "mem.free”
curl -k -u admin/pass
curl -k -u admin/pass
https://localhost:8089/services/catalog/metricstore/
dimensions https://localhost:8089/services/catalog/metricstore/
dimensions/dc/values?metric=mem.free

23
 GDI - Metric Ingestion Protocol: Collectd – Write
HTTP plugin
▶ Collectd, https://collectd.org - ~100 frontend plugins
▶ Scheduled push interval: 30secs
▶ # of metrics collected: ~350 (~1M measurements per day per server)
▶ Enabled plugins configurations, collectd.conf

1. csv 8. Logfile 15. tcpconns

2. cpu 9. memory 16. thermal
3. df 10. Network 17. ptime
4. disk 11. processes
5. Interface 12. protocols
6. irq 13. Syslog
7. load 14. swap
 GDI: collectd write_http plugin
Sample write_http event
– {"values":[98.9363841194414],"dstypes":["derive"],"dsnames":["val
ue"],"time":1474401106.556,"interval":10.000,"host":"C5819124-
66AE-4B28-8E13-
914C3961E46C","plugin":"cpu","plugin_instance":"0","type":"cpu","
type_instance":"idle"}
Sample Result
– metric_name=cpu.idle.value
– _value=98.9363841194414
– Host=C5819124-66AE-4B28-8E13-914C3961E46C
 GDI Deployment Options: Collectd & HEC

Cloud

HTTPS - HEC
Splunk CollectD Package
write_http
plugin

Server Farm
Splunk Indexing Tier
 cAdvisor

▶ Provides container users an understanding of the resource usage and

performance characteristics of their running containers
▶ It is a running daemon that collects, aggregates, processes, and exports
information about running containers

27
DEMO Docker
Metrics!
Docker Moby - V2
Logging Plugin
Section subtitle goes here
 Docker Moby - v2 logging Plugin

▶ Docker Hub: https://github.com/splunk/docker-logging-plugin

▶ Running the logging plugin

docker run --log-driver=splunk-log-driver:next \

--log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 \
--log-opt splunk-url=https://splunkhost:8088 \
--log-opt splunk-capath=/path/to/cert/cacert.pem \
--log-opt splunk-caname=SplunkServerDefaultCert \
--log-opt tag="{{.Name}}/{{.FullID}}" \
--log-opt labels=location \
--log-opt env=TEST \
--env "TEST=false" \
--label location=west \ your/application
Demo
 © 2017 SPLUNK INC.

1. Docker Monitoring – You have options!

Key
Takeaways 2. Analytical Driven Insight
3. Metrics
4. Docker v2 logging API plugin
 © 2017 SPLUNK INC.

Thank You
Don't forget to rate this session in the
.conf2017 mobile app