Faster paths for nested parsers

[tmcw]

Okay! Got another. Paths, specifically the pattern [...ctx.path, key] is a performance hotspot. This PR replaces that pattern with an object, ParseInputLazyPath, which is ParseInput, but instead of concatenating the path immediately, only creates the new array if it's needed (if there is an error). This yields a 20-30% performance boost on inputs like the realworld benchmark and the object benchmarks, and I can't easily profile it, but should reduce memory overhead a bit too.

• Refs Slow parsing performance #205

---

Before

% y benchmark
yarn run v1.22.4
$ ts-node src/benchmarks/index.ts
realworld: valid x 3,819 ops/sec ±0.97% (92 runs sampled)
z.enum: valid x 12,049,817 ops/sec ±0.72% (87 runs sampled)
z.enum: invalid x 65,717 ops/sec ±4.31% (88 runs sampled)
z.undefined: valid x 10,393,122 ops/sec ±0.98% (88 runs sampled)
z.undefined: invalid x 65,767 ops/sec ±3.98% (89 runs sampled)
z.literal: valid x 22,997,780 ops/sec ±0.59% (93 runs sampled)
z.literal: invalid x 69,284 ops/sec ±0.93% (89 runs sampled)
z.number: valid x 9,296,436 ops/sec ±1.02% (89 runs sampled)
z.number: invalid type x 66,951 ops/sec ±4.44% (88 runs sampled)
z.number: invalid number x 69,723 ops/sec ±1.09% (89 runs sampled)
z.string: empty string x 9,793,495 ops/sec ±0.82% (89 runs sampled)
z.string: short string x 9,464,260 ops/sec ±0.98% (88 runs sampled)
z.string: long string x 9,578,131 ops/sec ±0.99% (86 runs sampled)
z.string: optional string x 7,780,586 ops/sec ±1.08% (83 runs sampled)
z.string: nullable string x 6,718,051 ops/sec ±0.83% (88 runs sampled)
z.string: nullable (null) string x 9,268,075 ops/sec ±0.71% (88 runs sampled)
z.string: invalid: null x 65,552 ops/sec ±4.41% (88 runs sampled)
z.string: manual parser: long x 917,656,518 ops/sec ±1.39% (90 runs sampled)
z.object: empty: valid x 4,709,549 ops/sec ±1.17% (83 runs sampled)
z.object: empty: valid: extra keys x 4,065,901 ops/sec ±0.76% (90 runs sampled)
z.object: empty: invalid: null x 67,518 ops/sec ±0.84% (91 runs sampled)
z.object: short: valid x 2,259,276 ops/sec ±0.87% (88 runs sampled)
z.object: short: valid: extra keys x 2,019,209 ops/sec ±1.03% (86 runs sampled)
z.object: short: invalid: null x 62,770 ops/sec ±4.48% (86 runs sampled)
z.object: long: valid x 1,185,061 ops/sec ±1.02% (89 runs sampled)
z.object: long: valid: extra keys x 1,149,035 ops/sec ±1.12% (90 runs sampled)
z.object: long: invalid: null x 66,780 ops/sec ±0.78% (91 runs sampled)
z.union: double: valid: a x 1,765,709 ops/sec ±0.95% (88 runs sampled)
z.union: double: valid: b x 216,561 ops/sec ±7.73% (88 runs sampled)
z.union: double: invalid: null x 24,409 ops/sec ±1.17% (91 runs sampled)
z.union: double: invalid: wrong shape x 22,772 ops/sec ±4.70% (87 runs sampled)
z.union: many: valid: a x 1,792,144 ops/sec ±1.06% (86 runs sampled)
z.union: many: valid: c x 123,370 ops/sec ±6.10% (84 runs sampled)
z.union: many: invalid: null x 14,940 ops/sec ±4.33% (87 runs sampled)
z.union: many: invalid: wrong shape x 14,601 ops/sec ±1.10% (90 runs sampled)
z.discriminatedUnion: double: valid: a x 1,794,083 ops/sec ±0.97% (88 runs sampled)
z.discriminatedUnion: double: valid: b x 1,901,624 ops/sec ±0.94% (85 runs sampled)
z.discriminatedUnion: double: invalid: null x 62,635 ops/sec ±4.22% (83 runs sampled)
z.discriminatedUnion: double: invalid: wrong shape x 79,999 ops/sec ±0.99% (95 runs sampled)
z.discriminatedUnion: many: valid: a x 1,942,452 ops/sec ±1.02% (88 runs sampled)
z.discriminatedUnion: many: valid: c x 1,962,238 ops/sec ±0.96% (92 runs sampled)
z.discriminatedUnion: many: invalid: null x 84,518 ops/sec ±0.56% (92 runs sampled)
z.discriminatedUnion: many: invalid: wrong shape x 77,013 ops/sec ±0.82% (93 runs sampled)
✨  Done in 237.98s.

After

% y benchmark
yarn run v1.22.4
$ ts-node src/benchmarks/index.ts
realworld: valid x 4,934 ops/sec ±1.18% (88 runs sampled)
z.enum: valid x 12,020,432 ops/sec ±0.97% (89 runs sampled)
z.enum: invalid x 66,359 ops/sec ±4.15% (89 runs sampled)
z.undefined: valid x 10,118,294 ops/sec ±1.00% (88 runs sampled)
z.undefined: invalid x 64,990 ops/sec ±4.09% (88 runs sampled)
z.literal: valid x 22,881,671 ops/sec ±0.71% (95 runs sampled)
z.literal: invalid x 64,326 ops/sec ±1.02% (89 runs sampled)
z.number: valid x 7,213,872 ops/sec ±1.12% (87 runs sampled)
z.number: invalid type x 62,500 ops/sec ±4.11% (81 runs sampled)
z.number: invalid number x 66,110 ops/sec ±0.97% (91 runs sampled)
z.string: empty string x 7,727,690 ops/sec ±0.98% (89 runs sampled)
z.string: short string x 7,533,898 ops/sec ±1.01% (87 runs sampled)
z.string: long string x 7,424,665 ops/sec ±1.11% (90 runs sampled)
z.string: optional string x 6,298,061 ops/sec ±1.06% (88 runs sampled)
z.string: nullable string x 5,351,823 ops/sec ±1.27% (86 runs sampled)
z.string: nullable (null) string x 9,295,395 ops/sec ±1.00% (90 runs sampled)
z.string: invalid: null x 64,821 ops/sec ±4.40% (87 runs sampled)
z.string: manual parser: long x 898,213,431 ops/sec ±1.36% (87 runs sampled)
z.object: empty: valid x 4,163,427 ops/sec ±2.03% (89 runs sampled)
z.object: empty: valid: extra keys x 3,750,210 ops/sec ±1.11% (89 runs sampled)
z.object: empty: invalid: null x 67,258 ops/sec ±0.92% (91 runs sampled)
z.object: short: valid x 2,395,853 ops/sec ±1.11% (88 runs sampled)
z.object: short: valid: extra keys x 2,118,128 ops/sec ±0.99% (87 runs sampled)
z.object: short: invalid: null x 63,278 ops/sec ±4.17% (86 runs sampled)
z.object: long: valid x 1,436,958 ops/sec ±1.09% (89 runs sampled)
z.object: long: valid: extra keys x 1,330,563 ops/sec ±1.34% (90 runs sampled)
z.object: long: invalid: null x 65,808 ops/sec ±1.07% (89 runs sampled)
z.union: double: valid: a x 1,858,340 ops/sec ±0.98% (87 runs sampled)
z.union: double: valid: b x 214,908 ops/sec ±6.27% (91 runs sampled)
z.union: double: invalid: null x 23,372 ops/sec ±4.25% (87 runs sampled)
z.union: double: invalid: wrong shape x 23,212 ops/sec ±1.26% (90 runs sampled)
z.union: many: valid: a x 1,755,110 ops/sec ±1.17% (87 runs sampled)
z.union: many: valid: c x 117,378 ops/sec ±8.97% (86 runs sampled)
z.union: many: invalid: null x 15,048 ops/sec ±0.91% (91 runs sampled)
z.union: many: invalid: wrong shape x 14,277 ops/sec ±4.43% (88 runs sampled)
z.discriminatedUnion: double: valid: a x 1,952,220 ops/sec ±1.06% (88 runs sampled)
z.discriminatedUnion: double: valid: b x 1,975,703 ops/sec ±1.10% (89 runs sampled)
z.discriminatedUnion: double: invalid: null x 65,141 ops/sec ±0.94% (94 runs sampled)
z.discriminatedUnion: double: invalid: wrong shape x 78,921 ops/sec ±1.12% (90 runs sampled)
z.discriminatedUnion: many: valid: a x 1,931,200 ops/sec ±0.99% (86 runs sampled)
z.discriminatedUnion: many: valid: c x 1,922,676 ops/sec ±0.89% (86 runs sampled)
z.discriminatedUnion: many: invalid: null x 82,803 ops/sec ±1.04% (92 runs sampled)
z.discriminatedUnion: many: invalid: wrong shape x 76,801 ops/sec ±0.71% (90 runs sampled)
✨  Done in 238.26s.

[scotttrinh]

I wonder why number and string got slower in the benchmarks? Maybe the overhead of instantiating the ParseInputLazyPath class?

[tmcw]

Jitter :/. Running the benchmarks again yields this:

% y benchmark
yarn run v1.22.4
$ ts-node src/benchmarks/index.ts
realworld: valid x 5,450 ops/sec ±0.64% (95 runs sampled)
z.enum: valid x 14,557,054 ops/sec ±1.11% (90 runs sampled)
z.enum: invalid x 73,955 ops/sec ±3.90% (90 runs sampled)
z.undefined: valid x 11,573,460 ops/sec ±0.60% (96 runs sampled)
z.undefined: invalid x 71,706 ops/sec ±3.31% (93 runs sampled)
z.literal: valid x 27,433,991 ops/sec ±1.05% (94 runs sampled)
z.literal: invalid x 75,724 ops/sec ±0.62% (94 runs sampled)
z.number: valid x 11,273,030 ops/sec ±1.24% (90 runs sampled)
z.number: invalid type x 74,871 ops/sec ±4.17% (89 runs sampled)
z.number: invalid number x 75,297 ops/sec ±0.93% (94 runs sampled)
z.string: empty string x 11,148,330 ops/sec ±1.26% (90 runs sampled)
z.string: short string x 10,670,739 ops/sec ±1.80% (90 runs sampled)
z.string: long string x 11,017,787 ops/sec ±1.30% (88 runs sampled)
z.string: optional string x 9,269,393 ops/sec ±1.36% (91 runs sampled)
z.string: nullable string x 7,795,757 ops/sec ±0.74% (93 runs sampled)
z.string: nullable (null) string x 11,455,814 ops/sec ±0.96% (91 runs sampled)
z.string: invalid: null x 73,813 ops/sec ±4.15% (91 runs sampled)
z.string: manual parser: long x 872,885,496 ops/sec ±0.87% (87 runs sampled)
z.object: empty: valid x 4,643,618 ops/sec ±0.99% (94 runs sampled)
z.object: empty: valid: extra keys x 4,892,417 ops/sec ±0.58% (95 runs sampled)
z.object: empty: invalid: null x 72,944 ops/sec ±4.08% (91 runs sampled)
z.object: short: valid x 2,941,298 ops/sec ±0.56% (96 runs sampled)
z.object: short: valid: extra keys x 2,606,273 ops/sec ±0.92% (90 runs sampled)
z.object: short: invalid: null x 73,221 ops/sec ±1.08% (91 runs sampled)
z.object: long: valid x 1,643,766 ops/sec ±0.73% (91 runs sampled)
z.object: long: valid: extra keys x 1,589,414 ops/sec ±0.41% (95 runs sampled)
z.object: long: invalid: null x 71,422 ops/sec ±3.90% (92 runs sampled)
z.union: double: valid: a x 2,372,352 ops/sec ±0.91% (94 runs sampled)
z.union: double: valid: b x 268,907 ops/sec ±6.43% (86 runs sampled)
z.union: double: invalid: null x 26,471 ops/sec ±3.61% (91 runs sampled)
z.union: double: invalid: wrong shape x 26,852 ops/sec ±0.38% (92 runs sampled)
z.union: many: valid: a x 2,292,813 ops/sec ±0.54% (94 runs sampled)
z.union: many: valid: c x 144,190 ops/sec ±7.37% (88 runs sampled)
z.union: many: invalid: null x 16,547 ops/sec ±4.01% (91 runs sampled)
z.union: many: invalid: wrong shape x 16,374 ops/sec ±0.89% (94 runs sampled)
z.discriminatedUnion: double: valid: a x 2,482,874 ops/sec ±0.54% (95 runs sampled)
z.discriminatedUnion: double: valid: b x 2,418,194 ops/sec ±6.35% (86 runs sampled)
z.discriminatedUnion: double: invalid: null x 71,396 ops/sec ±3.65% (93 runs sampled)
z.discriminatedUnion: double: invalid: wrong shape x 83,699 ops/sec ±1.25% (96 runs sampled)
z.discriminatedUnion: many: valid: a x 2,448,872 ops/sec ±0.63% (91 runs sampled)
z.discriminatedUnion: many: valid: c x 2,464,820 ops/sec ±0.99% (93 runs sampled)
z.discriminatedUnion: many: invalid: null x 91,411 ops/sec ±0.52% (96 runs sampled)
z.discriminatedUnion: many: invalid: wrong shape x 82,014 ops/sec ±0.58% (90 runs sampled)
✨  Done in 236.96s.

Which has numbers a little faster than before. I suspect that these micro-benchmarks have more jitter than the others, but none of this compensates for variations in whatever else my macOS is doing a the same time as the bench.

[scotttrinh]

Yeah, I came away with similar results. I was hoping the benchmarking library itself doing a bunch of sampling runs would help alleviate this somewhat, but 🤷 I know it's common to disregard "microbenchmarks", but I have a lot of use cases that use single primitive Zod schemas, so I still personally care about the primitive performance apart from how they are composed into something like realworld.

As an aside: my dream here is to run benchmarks in CI and show diffs rather than the raw data. That would at least alleviate some of the issues with having other processes interfering with results.

Thanks for all of your great performance work @tmcw!