Output overhaul

[rgoldberg]

Output Overhaul Proposal

This output overhaul will resolve (or simplify subsequently resolving) the following:

• JSON output option #190: JSON output.
• --debug option to show JSON responses, other debug info #317: JSON debug output.
• list output column misalignment #412: Tabular formatting via column command.
• Consistent display justification of app IDs #598: Tabular formatting via column command.
• Output improvements & options #104: Filter, order & reformat JSON output.
• info: retrieve additional metadata about apps #395: Filter, order & reformat JSON output.
• outdated/info: display release notes  #60: Filter, order & reformat JSON output.
• list: display purchaser Apple ID #63: Filter, order & reformat JSON output.
• Shell completion overhaul #678: JSON output can be used to generate completions.
• Parallelize operations #586: Easier to parallelize when all output streamed thread-safe.
• Support multiple app ID arguments for all relevant commands #225: Multiple errors / warnings / successes handled.

Apple Raw Data Types

1. JSON (ordering & spacing): e.g., iTunes Search API results
2. Non-JSON text (ordering & ignorable spacing): e.g., Spotlight format (not currently used)
3. Ordered non-text (ordering): e.g., Swift array
4. Unordered non-text: e.g., Swift object

Output Categories

1. Normal: expected output
2. Ephemeral: e.g., progress bars for downloading or installing
3. Warning: nonfatal problems
4. Error: fatal problems

Exit Codes

Return 0 (success) unless 1 or more errors occur.

3 Output Formats

Tabular

mas will output tabular output by default. It will be an improved version of the existing output.

Tabular output can be explicitly requested via the --tabular flag.

mas might accept arguments to modify its tabular output.

Tabular would normally be used for most non-programmatic, non-debug uses.

Raw JSON

Raw JSON is a JSON representation of Apple raw data that's as similar to the raw data as possible.

Raw JSON output can be requested via the --raw-json flag.

All output categories (except possibly ephemeral) will be output as JSON.

Raw JSON would probably be used primarily for debugging.

Standard JSON

Standard JSON is a JSON representation of Apple raw data mapped to sensible mas-wide standards.

e.g., appID would be used as the standard property key for the following equivalent raw property keys from different data sources:

• trackId from the iTunes Search API
• kMDItemAppStoreAdamID from Spotlight
• CKSoftwareProduct.itemIdentifier from CommerceKit
• …

Standard JSON output can be requested via the --json flag.

All output categories (except possibly ephemeral) will be output as JSON.

Standard JSON should be preferred for programmatic use of mas (including for command-line use of custom output formats).

Messages

A message is a fragment of text representing the output of a self-contained event (like info for an app, a list of installed apps, an invalid argument error list, an unknown app ID argument, etc.).

Streaming

Messages will be streamed to the console; in both the JSON formats, each message will thus be its own JSON object in a stream of JSON objects.

Invalid Data

If any Apple raw data is corrupt, and thus cannot be represented as valid JSON (like text that should be JSON not being valid JSON), that data will be output as a properly escaped single JSON string in an error message.

External-to-Swift Implementation

Shell Wrapper Script Around Swift Binary Executable

It is impossible to properly align tabular console data in Swift.

The column executable, however, can properly align tabular console data.

mas will be restructured to have a zsh wrapper script named mas that will perform tabular & other formatting. mas will be in the executable search path. Users will normally interact with the new mas zsh wrapper just like they currently interact with the existing Swift mas, except the new mas will support new features, like outputting raw or standard JSON.

mas-json Swift Binary Executable

The existing mas Swift binary executable will be renamed to something like mas-json. It will remain in the executable search path. It will be called by the new mas wrapper script.

All the output from mas-json will be in JSON (except possibly ephemeral output), regardless of the format that the user has requested frommas.

Normal users will never directly interact with mas-json. They will only call mas, which will forward all arguments to mas-json, then format the JSON messages that it receives.

One Output Stream per Output Category

Each output category will have its own dedicated stream to which its messages will be written:

1. Normal: stdout
2. Ephemeral: file descriptor 4 (suppressed when not a TTY)
3. Warning: file descriptor 3
4. Error: stderr

Separate streams disambiguate output categories & allow piping/formatting per category (such formatting would be done by mas or a custom user script that calls mas-json). e.g.:

1. Normal: no formatting
2. Ephemeral: blue text
3. Warning: pink text
4. Error: red text

Formatting would be suppressed for any stream that isn't a TTY (might provide a per-stream setting to retain formatting for non-TTY).

Formatting could be user-configurable via one environment variable per stream.

If file descriptor 3 or 4 have not been redirected, then mas will redirect them to stderr. If either has been redirected, mas will not interfere with the redirection.

Testing

For testing, streams can be redirected to write to separate Swift Strings instead of to the console.

JSON Format Generation

mas-json will output raw JSON if it is called with the --raw-json flag, which mas will pass through to mas-json.

mas-json will otherwise output standard JSON.

Duplicate Keys Within an Object

If a JSON object contains duplicate keys, most JSON parsers either fail or only output the property for the last of the duplicate keys. mas will output all properties, including all duplicate-keyed properties.

Spacing

Spacing options for mas-json & mas raw & standard JSON:

1. As per raw data: only when data is JSON: as raw as possible
2. Minified: most efficient
3. Pretty printed with one property / element / value per line: easiest to read. Indents:
   • tabs?
   • 2 spaces?
4. User-configurable via some standard pretty printing format syntax: flexible, but should spacing be handled by piping mas JSON output to other programs?

Tabular Format Generation

If using the default tabular output format, mas will generate tabular output from standard (not raw) JSON returned by mas-json.

mas will use jq to generate tab-separated data from JSON, then column will be used to align the data like:

mas-json list | jq -r '(.[] | [.id, .name, .version]) | @tsv' | column -ts $'\t' -R 1

Configuring Tabular Format Generation

Given that the JSON output will preserve all data, and that JSON can be parsed & tabulated using jq & column, we could require that users who want anything other than the default tabular output use one of the 2 JSON outputs then format it themselves.

If that is too cumbersome for users, we could support options / flags like:

--verbose
--all-fields
--fields <comma-separated list of field names>

This isn't necessary for the initial release of the output overhaul, as it will be easy to implement afterward.

Internal-to-Swift Implementation

When Messages Are Written

Each message will be written to the appropriate stream when it occurs.

Should each JSON message be written to the stream as a complete JSON value? Or is it OK to write partial values in a streaming fashion? e.g., must a whole large JSON object be written at one time, or can parts of it be written sequentially?

Exit Code

mas-json will have an exit code singleton variable which will default to 0. Utility functions that write to stderr can optionally specify a non-zero value that will be bitwise ORed with the existing exit code to produce a new combined exit code before writing to stderr.

stderr will be observed such that, if anything is written to it outside the utility functions, the exit code will be bitwise ORed with 1 (or with some other power of 2).

Parallelism

If we allow parallel operations, then output (especially ephemeral output) could get jumbled. We can discuss this in more detail later, after all existing & planned parallelism has been identified.

Output Ordering

Raw JSON Output Ordering

Do not sort ordered raw data.

For unordered raw data, order as per Standard JSON.

Standard JSON Output Ordering

Keyed Data

By default, either sort standard JSON object properties alphabetically by key (retaining the order of properties with the same key), or use some logical ordering (primary ID first, followed by secondary IDs, etc., with the remaining properties sorted alphabetically by key). Properties with duplicate keys would retain their relative order from the raw data.

Sequential Data

Never sort sequential data where the raw ordering matters (e.g., mas search).

Sort some specific top-level sequential data (e.g., mas list) in a logical manner.

Possibly alphabetically / numerically sort specified JSON arrays that only contain scalar elements (i.e. numbers, strings, booleans, and nulls, but no arrays or objects).

Normally do not sort JSON arrays that are intended to contain arrays or objects.

Custom Ordering

If users want custom ordering, they can pipe output to other commands like jq & sort. The initial output overhaul needn't concern itself with custom ordering, even if we might support it in the future, since it would be easy to graft custom ordering onto any output overhaul implementation.

Location of Ordering Code

Assuming reordering JSON in zsh is simple, mas-json should output all standard JSON ordered as per the raw JSON data, while mas should reorder it for standard JSON or tabular output. If it will be much more difficult to reorder in zsh than in Swift, then reordering of standard JSON can be performed in Swift.

Configuration

If mas is to be heavily configurable, probably simplest to mainly use environment variables.

• All potential mas executables & scripts can easily read environment variables.
• Avoids creating & handling command-line flags, options, and/or arguments.
• Avoids reading config files from set locations or from locations from arguments, etc.
• Any files sourced before running mas executables or scripts can serve as config files.
• A command-line argument would override an equivalent environment variable.

Dependencies

The mas brew formula would depend on the jq & util-linux formulae, because jq doesn't come with macOS & column from macOS doesn't support right justification (at least the versions I've seen).

Tentative Roadmap

Before Version 1.9

• Draft standard JSON format specs to obtain feedback & to publish before JSON output availability.

Version 1.9

• Provide both raw & standard JSON output.
• Maybe provide some options to modify tabular output, but likely postpone them until 2.0.
• Likely postpone zsh wrapper until 2.0.

Version 2.0

• Use zsh wrapper to output updated default tabular output.
• Likely provide some options to modify tabular output.