[PROJECT] New proto-backed Firestore structure

[jcqli]

Parent issue tracking the work needed to introduce protos as our db and Firestore schema.

• Implement proto converters for Android ([Code health] Introduce protos as structure for checking data in Firestore at compile-time ground-android#2049)
• Load survey list using new structure #1933
• Load survey and jobs using new structure #1934
• Use protos in create / edit survey flow #1859
• Use protos in import GeoJSON #1775
• Use protos when reading survey list ground-android#2500
• Use protos when reading survey ground-android#2501
• Use protos when reading LOIs #1778
• Use protos when reading submissions #1857
• Use protos when reading LOIs ground-android#2498
• Use protos when creating LOIs ground-android#2499
• Remove unused "edit submissions" code ground-android#2494
• Write submissions to remote db using new proto-based format ground-android#2548
• Use protos when exporting to CSV #1779
• Update Firestore and Storage rules to read from new schema #1780
• Delete code which uses old schema (Web) #1781
• Delete code which uses old schema ground-android#2468
• Delete code which uses old schema (Cloud Functions) #1858

Additional tasks related to this migration:

• [Edit survey] "Publish changes" doesn't write survey in new format #1884
• [Proto] Tasks in /job proto don't update on initial configuration #1911
• [Proto] Capture location proto doesn't serialize because minAccuracyMeters set to null #1922
• [Proto] LOI Metadata is swapped #1921
• [Proto] Time task is serialized with BOTH_DATE_AND_TIME type (implying both date and time input) which is not supported on the app #1925
• Write submissions in remote db using protos ground-android#2556

[gino-m]

Did some research on various protoc plugins for generating js+ts code from protodef. We need something which makes metadata accessible (esp. field numbers) in order to implement the representation described in #2455. Here are my findings:

protobufjs ✅

The most popular lib I found (16M weekly downloads), originally designed by Google. It can generate js and ts, but to get what we want you need to call it three times - once to generate the js, once to generate to ts bindings, and once to generate JSON with pb metadata.

npm i -D protobufjs protobufjs-cli
npx pbjs -t static-module ../proto/survey.proto -o dist/survey.js
npx pbts dist/survey.js -o dist/survey.d.ts
npx pbjs -t json ../proto/survey.proto -o dist/survey.proto.json

We could load the JSON at runtime to pick out field numbers we need.

Alternatives considered

@protobuf-ts/plugin

146k downloads/week. Generates pure ts code in a single command, and includes field numbers in the message metadata. But given this is a less "standard", less used library, prefer using protobufjs.

ts-proto

OTOO 300k weekly downloads, and doesn't encode the field numbers anywhere we can use at runtime, so we can exclude this one for now.

protoc-gen-ts

It seems the protoc packaged with the one didn't support my M1 Mac.

npm i -D @protobuf-ts/plugin
npx protoc --ts_out dist --proto_path ../proto survey.proto loi.proto submission.proto

It doesn't produce or consume JSON, but it's fairly easy to go from ts object<>JSON anyway, plus we'll prob want more control over the formats.

grpc_tools_node_protoc_ts

Only 100k downloads per week, but uses the standard google grpc-tools. Doesn't seem to embed the set of valid field numbers, though individual fields can be fetched by field number:

npm i -D grpc-tools grpc_tools_node_protoc_ts
npx grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./dist --grpc_out=grpc_js:./dist --plugin=protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin -I ../proto survey.proto
npx protoc --plugin=protoc-gen-ts=./node_modules/.bin/protoc-gen-ts --ts_out=grpc_js:./dist -I ../proto survey.proto

[gino-m]

Also, carrying forward proposal from google/ground-android#2455:

• Create lightweight converters in Kotlin and TypeScript which take a Map<Long, Any> and produce protobufs and vice-versa, where keys in Firestore correspond to a protobuf field number, and values primitive types supported by proto.

Example survey data before:

{
  "title": "My survey",
  "description": "An example survey",
   //...
}

Example survey data after:

{
  1: "My survey",
  2: "An example survey",
  ...
}

This has several advantages:

• Generated code ensures a consistent schema across web, Android, and Cloud Functions.
• Using field IDs instead of names allows field names can be changes without breaking clients.
• Storaging values as Firestore values rather than binary allows Firestore query filters to work without duplicating values in "index" fields, and it allows data to be viewed and debugged using standard tools.
• The implementation is simple, easy to write and debug.
• Using field numbers instead of names allows this sol'n to work with Android "lite" protobufs in which name bindings aren't available at runtime.
• Writing directly to generated code

This has a few disadvantages as well:

• Maintaining backwards compatibility with existing test surveys is very difficult. We would probably want to zap the dbs and start from scratch.
• Indirection in key names in Firestore; would need to check protobuf def to understand which field is which.

[jcqli]

Filed #1966 to track cleanup of old code @gino-m shall we mark this as closed?

[gino-m]

Yes, I think we can!