TextFormat specification seems to be undocumented

[endobson]

I'm trying to figure out how to print a protocol buffer in text format that is compatible with the existing implementations of text format. The issue is that I cannot find any specification of TextFormat and only implementations. Given that there are multiple implementations (C++,Java,Go) that are interoperable it seems that there is a spec even if it isn't yet documented.

The binary format and proto file format are all well documented on the developer site:
https://developers.google.com/protocol-buffers.

[liujisi]

The textformat isn't intended to be interoperable across languages/implementations, and I believe it already isn't fully portable. I'd recommend using binary format for any use cases that require portability.

[endobson]

I'm implementing support for protocol buffers in a language that doesn't already have support, even if text format isn't fully portable, it is used in many existing systems as a human readable version.

protoc seems to support some version of Text Format for use with the --encode and --decode options. Can that version at least be documented?

[xfxyjwf]

@endobson If you are implementing protobuf for a new language, can you support proto3 JSON format instead?

[endobson]

I'll investigate it. That probably is a better choice if protobuf is moving away from the old text format.

[hickford]

FWIW There exists internal documentation available to Googlers. It might make sense to publish it publically https://b.corp.google.com/issues/173410810 (Googlers only)

https://stackoverflow.com/q/18873924 "what does textproto look like?" is a popular question.

There are over two million textproto files on GitHub https://github.com/search?q=extension%3Apbtxt

[jvolkman]

FWIW There exists internal documentation available to Googlers. It might make sense to publish it publically https://b.corp.google.com/issues/173410810 (Googlers only)

https://stackoverflow.com/q/18873924 "what does textproto look like?" is a popular question

Yes. What's the status on that? I helped write that spec when I was there and would love to see it released.

[y3llowcake]

I, for one, don't love the ergonomics of JSON for human readable and editable applications and would greatly appreciate a documented specification for the text format. For those of us that maintain third party proto implementations and compiler plugins, this would be a useful reference. My use cases are largely human editable config for a single language. Having portability across implementations would be great, but I don't know that it's necessary, at least for the use cases I am thinking of. Also FWIW, it looks like there is now a conformance suite for the text implementation?

[PhilipMetzger]

I'm also a fan of textproto for human writeable and readable configuration files , since it doesn't have any of JSON downfalls, an actual number type. I and many others would appreciate a OSS Release of the textproto Spec.

There are currently about 2.3 Million files on Github which implicitly depend on this spec.

AFAIU there's an internal spec available at go/textproto-spec, which is under further discussion at b/173410810 (Googler only)

Queries:

• https://github.com/search?q=extension%3Apbtxt
• https://github.com/search?q=extension%3Apbtext
• https://github.com/search?q=extension%3Atxtpb
• https://github.com/search?q=extension%3Atextproto
• https://github.com/search?q=extension%3Atextpb
• https://github.com/search?q=extension%3Aasciipb
• https://github.com/search?q=extension%3Apbascii
• https://github.com/search?q=extension%3Aprotoascii
• https://github.com/search?q=extension%3Aprototxt

[tarndt]

The Go (golang) implementation seems to know how to marshal to and from the textual format. I presume they must have implemented this to some defined standard?

Old/v1:

• Doc: https://pkg.go.dev/github.com/golang/protobuf@v1.5.2/proto#MarshalText
• https://github.com/golang/protobuf/blob/v1.5.2/proto/text_encode.go
• https://github.com/golang/protobuf/blob/v1.5.2/proto/text_decode.go

Current/v2:

• Doc: https://pkg.go.dev/google.golang.org/protobuf@v1.27.1/encoding/prototext
• https://github.com/protocolbuffers/protobuf-go/blob/v1.27.1/encoding/prototext/encode.go
• https://github.com/protocolbuffers/protobuf-go/blob/v1.27.1/encoding/prototext/decode.go

The docs above do contain this warning:

Do not depend on the output being stable. It may change over time across different versions of the program.

[jvolkman]

The defacto spec is the c++ implementation. That's what everything else is based on, including the aforementioned internal spec (at least that was the case when I was there 3 years ago). The v2 go implementation was made in conjunction with the reverse-engineered internal spec.

[sxlijin]

Is there any chance we can get this reopened? It's unclear to me why this was closed.

Some of the C++ gRPC folks might be in the right place to get this moving again (unless there's a dedicated C++ proto team- I don't remember if there was one when I left last year).

[jvolkman]

There's a newer, open issue at #8636