Skip to content

Commit

Permalink
Resolve known tags in core schema (#172)
Browse files Browse the repository at this point in the history 
When using the core schema, if the parsed source includes one of the following explicit tags, parse it the same way as in the yaml-1.1 schema, and include the relevant tag in doc.schema.tags:
- !!binary
- !!omap
- !!pairs
- !!set
- !!timestamp

To disable, use the option { resolveKnownTags: false } in YAML.parse*()
  • Loading branch information
@eemeli
eemeli authored Aug 5, 2020
1 parent 9e6f644 commit 55593c4
Show file tree
Hide file tree
Showing 7 changed files with 115 additions and 47 deletions.
36 changes: 19 additions & 17 deletions docs/03_options.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,25 +20,27 @@ YAML.Document.defaults

The `version` option value (`'1.2'` by default) may be overridden by any document-specific `%YAML` directive.

| Option | Type | Description |
| --------------- | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| anchorPrefix | `string` | Default prefix for anchors. By default `'a'`, resulting in anchors `a1`, `a2`, etc. |
| customTags | `Tag[] ⎮ function` | Array of [additional tags](#custom-data-types) to include in the schema |
| indent | `number` | The number of spaces to use when indenting code. By default `2`. |
| indentSeq | `boolean` | Whether block sequences should be indented. By default `true`. |
| keepBlobsInJSON | `boolean` | Allow non-JSON JavaScript objects to remain in the `toJSON` output. Relevant with the YAML 1.1 `!!timestamp` and `!!binary` tags as well as BigInts. By default `true`. |
| keepCstNodes | `boolean` | Include references in the AST to each node's corresponding CST node. By default `false`. |
| keepNodeTypes | `boolean` | Store the original node type when parsing documents. By default `true`. |
| mapAsMap | `boolean` | When outputting JS, use Map rather than Object to represent mappings. By default `false`. |
| maxAliasCount | `number` | Prevent [exponential entity expansion attacks] by limiting data aliasing count; set to `-1` to disable checks; `0` disallows all alias nodes. By default `100`. |
| merge | `boolean` | Enable support for `<<` merge keys. By default `false` for YAML 1.2 and `true` for earlier versions. |
| prettyErrors | `boolean` | Include line position & node type directly in errors; drop their verbose source and context. By default `false`. |
| schema | `'core' ⎮ 'failsafe' ⎮` `'json' ⎮ 'yaml-1.1'` | The base schema to use. By default `'core'` for YAML 1.2 and `'yaml-1.1'` for earlier versions. |
| simpleKeys | `boolean` | When stringifying, require keys to be scalars and to use implicit rather than explicit notation. By default `false`. |
| sortMapEntries | `boolean ⎮` `(a, b: Pair) => number` | When stringifying, sort map entries. If `true`, sort by comparing key values with `<`. By default `false`. |
| version | `'1.0' ⎮ '1.1' ⎮ '1.2'` | The YAML version used by documents without a `%YAML` directive. By default `'1.2'`. |
| Option | Type | Description |
| ---------------- | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| anchorPrefix | `string` | Default prefix for anchors. By default `'a'`, resulting in anchors `a1`, `a2`, etc. |
| customTags | `Tag[] ⎮ function` | Array of [additional tags](#custom-data-types) to include in the schema |
| indent | `number` | The number of spaces to use when indenting code. By default `2`. |
| indentSeq | `boolean` | Whether block sequences should be indented. By default `true`. |
| keepBlobsInJSON | `boolean` | Allow non-JSON JavaScript objects to remain in the `toJSON` output. Relevant with the YAML 1.1 `!!timestamp` and `!!binary` tags as well as BigInts. By default `true`. |
| keepCstNodes | `boolean` | Include references in the AST to each node's corresponding CST node. By default `false`. |
| keepNodeTypes | `boolean` | Store the original node type when parsing documents. By default `true`. |
| mapAsMap | `boolean` | When outputting JS, use Map rather than Object to represent mappings. By default `false`. |
| maxAliasCount | `number` | Prevent [exponential entity expansion attacks] by limiting data aliasing count; set to `-1` to disable checks; `0` disallows all alias nodes. By default `100`. |
| merge | `boolean` | Enable support for `<<` merge keys. By default `false` for YAML 1.2 and `true` for earlier versions. |
| prettyErrors | `boolean` | Include line position & node type directly in errors; drop their verbose source and context. By default `false`. |
| resolveKnownTags | `boolean` | When using the `'core'` schema, support parsing values with these explicit [YAML 1.1 tags]: `!!binary`, `!!omap`, `!!pairs`, `!!set`, `!!timestamp`. By default `true`. |
| schema | `'core' ⎮ 'failsafe' ⎮` `'json' ⎮ 'yaml-1.1'` | The base schema to use. By default `'core'` for YAML 1.2 and `'yaml-1.1'` for earlier versions. |
| simpleKeys | `boolean` | When stringifying, require keys to be scalars and to use implicit rather than explicit notation. By default `false`. |
| sortMapEntries | `boolean ⎮` `(a, b: Pair) => number` | When stringifying, sort map entries. If `true`, sort by comparing key values with `<`. By default `false`. |
| version | `'1.0' ⎮ '1.1' ⎮ '1.2'` | The YAML version used by documents without a `%YAML` directive. By default `'1.2'`. |

[exponential entity expansion attacks]: https://en.wikipedia.org/wiki/Billion_laughs_attack
[yaml 1.1 tags]: https://yaml.org/type/

## Data Schemas

Expand Down
11 changes: 10 additions & 1 deletion src/doc/Schema.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,22 @@ import { getSchemaTags } from './getSchemaTags.js'
const sortMapEntriesByKey = (a, b) =>
a.key < b.key ? -1 : a.key > b.key ? 1 : 0

const coreKnownTags = {
'tag:yaml.org,2002:binary': tags.binary,
'tag:yaml.org,2002:omap': tags.omap,
'tag:yaml.org,2002:pairs': tags.pairs,
'tag:yaml.org,2002:set': tags.set,
'tag:yaml.org,2002:timestamp': tags.timestamp
}

export class Schema {
constructor({ customTags, merge, schema, sortMapEntries }) {
constructor({ customTags, merge, resolveKnownTags, schema, sortMapEntries }) {
this.merge = !!merge
this.name = schema
this.sortMapEntries =
sortMapEntries === true ? sortMapEntriesByKey : sortMapEntries || null
this.tags = getSchemaTags(schemas, tags, customTags, schema)
this.knownTags = resolveKnownTags ? coreKnownTags : {}
}

createNode(value, wrapScalars, tagName, ctx) {
Expand Down
1 change: 1 addition & 0 deletions src/options.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ export const documentOptions = {
'1.2': {
schema: 'core',
merge: false,
resolveKnownTags: true,
tagPrefixes: [
{ handle: '!', prefix: '!' },
{ handle: '!!', prefix: defaultTagPrefix }
Expand Down
9 changes: 8 additions & 1 deletion src/resolve/resolveTag.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { resolveScalar } from './resolveScalar.js'
import { resolveString } from './resolveString.js'

function resolveByTagName(doc, node, tagName) {
const { tags } = doc.schema
const { knownTags, tags } = doc.schema
const matchWithTest = []
for (const tag of tags) {
if (tag.tag === tagName) {
Expand All @@ -22,6 +22,13 @@ function resolveByTagName(doc, node, tagName) {
if (typeof str === 'string' && matchWithTest.length > 0)
return resolveScalar(str, matchWithTest, tags.scalarFallback)

const kt = knownTags[tagName]
if (kt) {
tags.push(Object.assign({}, kt, { default: false, test: undefined }))
const res = kt.resolve(doc, node)
return res instanceof Collection ? res : new Scalar(res)
}

return null
}

Expand Down
94 changes: 73 additions & 21 deletions tests/doc/YAML-1.2.spec.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -403,15 +403,79 @@ application specific tag: !something |
tgt: [
{
'not-date': '2002-04-28',
picture:
'R0lGODlhDAAMAIQAAP//9/X\n17unp5WZmZgAAAOfn515eXv\nPz7Y6OjuDg4J+fn5OTk6enp\n56enmleECcgggoBADs=\n',
picture: Buffer.from([
71,
73,
70,
56,
57,
97,
12,
0,
12,
0,
132,
0,
0,
255,
255,
247,
245,
245,
238,
233,
233,
229,
102,
102,
102,
0,
0,
0,
231,
231,
231,
94,
94,
94,
243,
243,
237,
142,
142,
142,
224,
224,
224,
159,
159,
159,
147,
147,
147,
167,
167,
167,
158,
158,
158,
105,
94,
16,
39,
32,
130,
10,
1,
0,
59
]),
'application specific tag':
'The semantics of the tag\nabove may be different for\ndifferent documents.\n'
}
],
warnings: [
[
'The tag tag:yaml.org,2002:binary is unavailable, falling back to tag:yaml.org,2002:str',
'The tag !something is unavailable, falling back to tag:yaml.org,2002:str'
]
],
Expand Down Expand Up @@ -467,18 +531,7 @@ application specific tag: !something |
? Mark McGwire
? Sammy Sosa
? Ken Griff`,
tgt: [
{
'Mark McGwire': null,
'Sammy Sosa': null,
'Ken Griff': null
}
],
warnings: [
[
'The tag tag:yaml.org,2002:set is unavailable, falling back to tag:yaml.org,2002:map'
]
]
tgt: [new Set(['Mark McGwire', 'Sammy Sosa', 'Ken Griff'])]
},

'Example 2.26. Ordered Mappings': {
Expand All @@ -490,12 +543,11 @@ application specific tag: !something |
- Sammy Sosa: 63
- Ken Griffy: 58\n\n`,
tgt: [
[{ 'Mark McGwire': 65 }, { 'Sammy Sosa': 63 }, { 'Ken Griffy': 58 }]
],
warnings: [
[
'The tag tag:yaml.org,2002:omap is unavailable, falling back to tag:yaml.org,2002:seq'
]
new Map([
['Mark McGwire', 65],
['Sammy Sosa', 63],
['Ken Griffy', 58]
])
]
}
},
Expand Down
7 changes: 2 additions & 5 deletions tests/doc/types.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -706,11 +706,8 @@ invoice:
})

test('no custom tag object', () => {
const doc = YAML.parseDocument(src)
const message =
'The tag tag:yaml.org,2002:binary is unavailable, falling back to tag:yaml.org,2002:str'
expect(doc.warnings).toMatchObject([{ message }])
expect(typeof doc.contents.value).toBe('string')
const bin = YAML.parse(src)
expect(bin).toBeInstanceOf(Uint8Array)
})
})
})
Expand Down
4 changes: 2 additions & 2 deletions tests/yaml-test-suite.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ testDirs.forEach(dir => {
}

describe(`${dir}: ${name}`, () => {
const docs = YAML.parseAllDocuments(yaml)
const docs = YAML.parseAllDocuments(yaml, { resolveKnownTags: false })
if (events) {
test('test.event', () => {
const res = testEvents(yaml)
Expand All @@ -90,7 +90,7 @@ testDirs.forEach(dir => {
if (!error) {
const src2 =
docs.map(doc => String(doc).replace(/\n$/, '')).join('\n...\n') + '\n'
const docs2 = YAML.parseAllDocuments(src2)
const docs2 = YAML.parseAllDocuments(src2, { resolveKnownTags: false })
trace: name,
'\nIN\n' + yaml,
'\nJSON\n' + JSON.stringify(docs[0], null, ' '),
Expand Down

0 comments on commit 55593c4

Please sign in to comment.