yaml get Examples

1. Introduction
2. Reading Scalar Data Into Shell Scripts
   1. Scalar Hash Properties
   2. Scalar Array Elements
   3. All Scalar Array Elements
   4. Scalar Array-of-Hash (Object) Properties

Introduction

This page explores various real-world use-cases for the yaml-get command-line tool.

Mundane data access using YAML Paths via yaml-get is already well covered in the various discussions about Segments-of-a-YAML-Path and will not be repeated here. Rather, this page will present some tasks which can be solved by using yaml-get and in some cases, in concert with other YAML Path tools.

Reading Scalar Data Into Shell Scripts

Scalar data is any single, simple piece of data like a number, string (text), or boolean (true/false). In the simplest case, you want to read just one such piece of data from a YAML/JSON/EYAML/compatible file into your shell script. The following sections explore gathering a single Scalar value from different kinds of YAML data structures.

Scalar Hash Properties

Take for example this data file:

File: get1.yaml

this:
  is:
    a:
      hash: data
    structure: with
  several: levels

Suppose you're after whatever data is at this.several (also /this/several):

#!/bin/bash
# Read the Scalar value into a variable your script can use
thisSeveral=$(yaml-get --query=this.several get1.yaml)

# Do something with the value
echo "Got:  ${thisSeveral}"

This prints: Got: levels.

Scalar Array Elements

In some cases, the Scalar you want is exactly one element of an Array.

File: get2.yaml

another:
  hash:
    - with
    - its
    - own
  child:
    - nodes
    - and
    - data

Should you need the second entry of the Array at /another/child in your script, you would first remember that the first index of an Array is 0 (not 1) and then write something like:

#!/bin/bash
# Read the Scalar value into a variable your script can use
thisSeveral=$(yaml-get --query=/another/child[1] get2.yaml)

# Do something with the value
echo "Got:  ${thisSeveral}"

This prints Got: and.

What if you need the last array element and don't know how many elements there are? yaml-get permits using a negative index to draw from the end of an Array with -1 being the last element (because -0 is not different from 0), -2 being the second-to-last, and so on.

All Scalar Array Elements

When you need to read all elements of an Array into your script, ensure that the result will be printed vertically (one element per line). Should your query print a JSON Array where all elements are on one line, tack a * segment (shorthand for [.!=""]) onto the end of your YAML Path to switch the result to one element per line.

For this example, let's query some real-world data. benoitvallon has graciously provided a sample set of JSON data for anyone to query, 100 Best Books. With a simple shell script that uses yaml-get to search the data, we can demonstrate precisely how to capture all elements of an Array result from yaml-get. As a bonus to demonstrate further that yaml-get plays nicely with other common command-line tools, the output will be sorted and unique.

File: query-100-best-books.sh

#!/bin/bash
###############################################################################
# Perform simple queries against the "100 Best Books" offered publicly at:
# https://raw.githubusercontent.com/benoitvallon/100-best-books/master/books.json
###############################################################################
searchFor=${1:?"ERROR:  Please provide a YAML Path to query (author, title, language, etc.)."}
BOOKS_URL=https://raw.githubusercontent.com/benoitvallon/100-best-books/master/books.json

# Download the file if needed
booksFile=/tmp/books.json
if [ ! -f "$booksFile" ]; then
	wget --quiet --output-document="$booksFile" $BOOKS_URL
	if [ 0 -ne $? ]; then
		echo "ERROR:  Unable to download the books data file!" >&2
		exit 2
	fi
fi

# Capture all results into an Array, using \n to separate each answer so spaces
# in the results won't disrupt the Array structure.
IFS=$(echo -en "\n\b")
answerList=($(yaml-get --query="${searchFor}" "$booksFile" | sort | uniq))
if [ 1 -gt "${#answerList[@]}" ]; then
	echo "Query produced no matches:  ${searchFor}"
	exit 1
fi

# Print the results
echo "Matches:"
for answer in "${answerList[@]}"; do
	echo "* ${answer}"
done

cat <<EOM

This "100 Best Books" data was gracoiusly provided by
https://github.com/benoitvallon and is publicly available at
${BOOKS_URL}
EOM

This shell script accepts a YAML Path as its only input, enabling users to query for simple data like author to produce a lengthy report. It also allows for more interesting queries like, "Which books were printed before year 0?" Such a query produces:

$ ./query-100-best-books.sh '[year < 0].title'
Matches:
* Iliad
* Mahabharata
* Medea
* Odyssey
* Oedipus the King
* Ramayana
* The Aeneid
* The Book Of Job
* The Epic Of Gilgamesh

This "100 Best Books" data was gracoiusly provided by
https://github.com/benoitvallon and is publicly available at
https://raw.githubusercontent.com/benoitvallon/100-best-books/master/books.json

Scalar Array-of-Hash (Object) Properties

The data is sometimes comprised of an Array-of-Hashes (AKA: Array of Objects) and you won't always know precisely where in the Array the desired Hash is. With the immensely powerful YAML Path syntax, these cases are trivial for yaml-get to solve. Suppose you have this data:

File: get3.yaml

connections:
  database:
    - name: oltp
      host: oltp.domain1.tld
      port: 5280
      user: ENC[A1B2C3D4E5F67890]
      pass: ENC[ABCDEF0123456789]
    - name: reports
      host: reports.domain1.tld
      port: 5280
      user: ENC[0A1B2C3D4E5F6789]
      pass: ENC[0123456789ABCDEF]
    - name: consumer-db
      host: consumer1.domain2.tld
      port: 5280
      user: ENC[FEDCBA9876543210]
      pass: ENC[DEF0123ABD456789]

Note that because this example file holds secrets, is an EYAML file. All of the values encrypted with ENC[...some hex...] cannot be read without the correct encryption key. The yaml-get tool fully understands EYAML encryption but needs the key to decrypt the value for your shell script. Any attempt to access the value without the key will result in an error like, "CRITICAL: Unable to decrypt value! Please verify you are using the correct old EYAML keys and the value is not corrupt: ENC[0A1B2C3D4E5F6789]". Setting up and using EYAML encryption keys to protect secrets in YAML files is beyond the scope of this tutorial, so please research EYAML should you be interested in this extremely useful technology. For now, assume the correct keys are in use and are accessible to yaml-get in the expected default location.

A trivial query for a shell script might be, "What is the user-name for the reports database connection?" We could answer it like this:

#!/bin/bash
# Get the decrypted value from the YAML file
reportsUser=$(yaml-get --query=connections.database[name=reports].user get3.yaml)

# Use the value
some-db-client --user="$reportsUser" --other-params --etc

Note that using YAML Path enabled us to search the database connections for one with a known name (name=reports). YAML Path empowers many other ways to search for a value, like when you don't actually know the full name or need to use a RegEx to find it. You can even search against a property other than name in the same ways.

What if we needed a more complex answer like, "List the names of all database connections with host-names outside of domain1.tld." Try this:

#!/bin/bash
# We are expecting 0-N results, so gather them into an Array
externalHosts=($(yaml-get --query='/connections/database[!host$domain1.tld]/name' get3.yaml))

# List out the matches
if [ 0 -eq "${#externalHosts[@]}" ]; then
    echo "No external database connections."
else
    for connectionName in "${externalHosts[@]}"; do
        echo "External database connection name:  ${connectionName}"
    done
fi

With the example data, this script writes out:

External database connection name:  consumer-db