Learning YAML

David Shepard

Software Engineering Institute

Carnegie Mellon University
Pittsburgh, PA 15213

Title of the Presentation Goes Here [DISTRIBUTION STATEMENT Please copy and paste the appropriate distribution statement into
[Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University this space.]

release and unlimited distribution.

1
Document Markings
Copyright 2021 Carnegie Mellon University.
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.
The view, opinions, and/or findings contained in this material are those of the author(s) and should not be construed as an official Government position, policy, or decision, unless designated by other documentation.
References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY
MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH
RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.
GOVERNMENT PURPOSE RIGHTS – Technical Data
Contract No.: FA8702-15-D-0002
Contractor Name: Carnegie Mellon University
Contractor Address: 4500 Fifth Avenue, Pittsburgh, PA 15213

The Government's rights to use, modify, reproduce, release, perform, display, or disclose these technical data are restricted by paragraph (b)(2) of the Rights in Technical Data—Noncommercial Items clause contained in the above identified contract. Any reproduction of technical data or portions
thereof marked with this legend must also reproduce the markings.
This material is distributed by the Software Engineering Institute (SEI) only to course attendees for their own individual study.
Except for any U.S. government purposes described herein, this material SHALL NOT be reproduced or used in any other manner without requesting formal permission from the Software Engineering Institute at [email protected].
Although the rights granted by contract do not require course attendance to use this material for U.S. Government purposes, the SEI recommends attendance to ensure proper understanding.
DM21-0158

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
2
Purpose of this talk
Learn enough about YAML, JSON, and a
developer’s workflow, to effectively apply
these to various DevOps/systems
configuration problems

Have some fun learning it

Learning YAML [Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University
release and unlimited distribution.
3
What is a YAML?
Pronounced “yam ul”
“Yet Another Markup Language”
“A strict superset of JSON”
Commonly-used configuration file language
User-friendly focus
Something you should have a passable familiarity with, if you will be doing
DevOps stuff

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
4
What is a JSON?
 Pronounced “jay son”

 “JavaScript Object Notation”

 Something that must be understood to understand YAML

 “an extremely simple data-interchange format... for humans to read and write and for machines to parse and
generate.”

 A logical alternative to XML (for humans)

 Comes in a few varieties, such as BSON (binary)

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
5
What is JSON really?
JSON is useful for
many tasks,
especially where
you are transferring
data between
software systems.

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
6
What is JSON really?
Document Root

JS Objects

JS Arrays
The legacy of JavaScript in
JSON is real, but JS has
nothing to do with JSON
today.
Learning YAML [Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University
release and unlimited distribution.
7
What is YAML really?
YAML is mostly a
more human-
readable variant of
JSON, but there are
some differences /
enhancements

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
8
What is YAML really?
Document Root
Like Python, whitespace
matters. Indent new levels
with two spaces.
JS Objects

JS Arrays

The important things to

notice are the different
ways of expressing the
same things as JSON.

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
9
JSON / YAML Data Types
Common to both:
 Object, AKA: Map/Associate Array/Struct: A sort of container data structure that
can contain all other types
 Array/List: Ordered Set of items, a sort of container data structure that can contain
all other types.
 String: “This is a string”
 Number: 1, .01, 1.2e+100
 Boolean: true, false
 Null: null
Unique to YAML:
 Date/Time Stamps, in ISO-8601 format: datetime: 2001-12-15T02:59:43.1Z
 Date, also ISO format: date: 2002-12-14
 Binary data, typically Base64-encoded: !!binary |
 R0lGODlhDAAMAIQAAP//9/X17unp5WZmZgAAAOfn515eXvPz7Y6OjuDg4J+fn5
 Unordered Set, implemented as a Map with null values
Learning YAML [Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University
release and unlimited distribution.
10
What can I use JSON/YAML for?
YAML is most often used for
 Docker configuration management,
but like JSON, is a general
 Kubernetes specification for managing
structured data.
 AWS/Azure configs
 Other common scripting tasks
 Web communications are predominantly JSON
 Mostly, anything that needs input data

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
11
How commonly used is JSON/YAML?

In my personal projects folder, JSON is an order-of-magnitude

more prevalent than YAML.

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
12
Are there good alternatives?
Absolutely! For many tasks, you might have your choice of solution:

 TOML: Arguably, even simpler for configuration management than YAML.

 https://github.com/toml-lang/toml
 Dhall: A much more powerful, “configuration language”
 https://dhall-lang.org#
 Pulumi: Powerful configuration language, made specifically for IaC needs. Uses a
configuration back-end
 https://www.pulumi.com
 Many others.

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
13
Working with JSON/YAML Data
 You will want a proper developer’s text editor
 VS Code
 Notepad++
 Jetbrains
 Vim/Emacs/Spacemacs/Atom/...

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
14
Any Text Editor Will Work, But...

 Without proper editor support,

you get only basic support for
reading, writing, and validation
 It’s common to see JSON that
looks like this –>
 I use VS Code throughout this
presentation

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
15
Useful VS Code Extensions

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
16
How can I use them from my scripts?

Many tools support data binding for scripts.

 “shyaml”: Useful for pulling YAML data into a shell script
 PyYAML: Python library for reading/writing YAML
 “json”: Built-in Python library for reading/writing JSON
 JSON is literally the native data format for JavaScript
 Pretty much every programming language will have native/library support for these
formats
Learning YAML [Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University
release and unlimited distribution.
17
Is my JSON/YAML Valid?
 Good question!
 Editors can perform basic syntax checks
 Data validation requires a schema
 The Python library ‘jsonschema’ can help with validation
 You will need to provide the schema

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
18
JSON Schema
The idea: create schema
items for each item name
and type in your data file.

It does work for validation.

Validation is limited, since

the data format itself only
provides limited support
for types.
Learning YAML [Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University
release and unlimited distribution.
19
Oddities in working with JSON
 Sometimes, when passing JSON data around, you will need to escape double
quotes (VERY common)

Text with
“escapes”
will upset
your text
editor. It’s
normal.

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
20
YAML schema validators
NodeJS Package:
Yaml-schema-
validator

As with JSON schema

validation, you must
provide a schema.

Undoubtedly, there are

other ways to do this.
This package seems
to work as advertised.
Learning YAML [Distribution Statement A] Approved for public
© 2020 Carnegie Mellon University
release and unlimited distribution.
21
Normalized YAML
 The YAML specification supports a number of features that you should probably never use, things
that will not universally work across all parsers, will make your YAML non-portable to JSON, and
will likely confuse people unfamiliar with these functions:

 Anchors

 References

 Extensions

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
22
...So let me tell you how they work
The ‘&’ is
The ‘<<’ is
the anchor
‘extends’ for
The ‘*’ is the an object
reference

It works like
a built-in More bizarre
macro macro-expansion
expansion.

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
23
Normalized YAML Continued...
 Using new YAML data types will not necessarily
directly transfer to JSON parsers, including:
 date/datetime
 set
 !!binary

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
24
Security Considerations
 Consider never placing usernames, passwords,
keys, and other authentication information in a plain-
text configuration file.
 If you really must do this thing, you must be able to
configure permissions in a way that only a privileged
account can access this data

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
25
Security Considerations Continued...
 Parsers don’t handle undefined behaviors uniformly.
 JSON and YAML are lightweight specifications that actually have
a number of undefined behaviors, for example as was seen in
CVE-2017-12635: https://cve.mitre.org/cgi-
bin/cvename.cgi?name=2017-12635

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
26
Anatomy of
a Parser
Vulnerability

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
27
Security Considerations Continued...
 Using a templating language can help with runtime injection of
credentials/keys
 Jinja2: Python-based templating language (DSL), allows you to write
your Python code in a way that will generate your JSON/YAML output
at runtime
 Dhall offers similar functionality to Jinja, in a different manner

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
28
Exercise!

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
29
Exercise Topics
- Access your editor & use it some (Hopefully...)

- Access the shell in the editor

- yamllint (find the error & fix it)

- Examine the various data structures (ask questions)

- shyaml (use tools to parse data)

- schema, jsonschema (data validation)

- yaml_to_json (automation scripts)

- validate_json (automation scripts)

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
30
 Thank you!

Learning YAML [Distribution Statement A] Approved for public

© 2020 Carnegie Mellon University
release and unlimited distribution.
31