Exporter README, receiver README updates (#29)

This flushes out the exporter README and makes a few corresponding fixes
to the receiver README.

README.md

@@ -9,7 +9,7 @@ still in transition, the primary OpenTelemetry Collector components
 are housed here:
 
 - [OTel Arrow Receiver](./collector/receiver/otelarrowreceiver/README.md)
-- OTel Arrow Exporter (TODO)
+- [OTel Arrow Exporter](./collector/exporter/otelarrowexporter/README.md)
 
 ## Reference implementation.
 

collector/exporter/otelarrowexporter/README.md

@@ -1,20 +1,48 @@
-# OTLP gRPC Exporter
+# OTel Arrow Exporter
 
-| Status                   |                       |
-| ------------------------ | --------------------- |
-| Stability                | traces [stable]       |
-|                          | metrics [stable]      |
-|                          | logs [beta]           |
-| Supported pipeline types | traces, metrics, logs |
-| Distributions            | [core], [contrib]     |
+<!-- status autogenerated section -->
+| Status        |           |
+| ------------- |-----------|
+| Stability     | [development]: metrics, traces, logs   |
+| Distributions | [] |
+| Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fotelarrow%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fotelarrow) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fotelarrow%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fotelarrow) |
 
-Export data via gRPC using [OTLP](
-https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md)
-format. By default, this exporter requires TLS and offers queued retry capabilities.
+[development]: https://github.com/open-telemetry/opentelemetry-collector#development
+<!-- end autogenerated section -->
+
+Exports telemetry data using the
+[OTel-Arrow](https://github.com/open-telemetry/otel-arrow) protocol and standard [OTLP](
+https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md)
+protocol via gRPC.
 
 ## Getting Started
 
-The following settings are required:
+The OTel-Arrow exporter is an extension of the core OpenTelemetry
+Collector [OTLP
+exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter)
+component with additional support for the
+[OTel-Arrow](https://github.com/open-telemetry/otel-arrow) protocol.
+
+OTel-Arrow supports column-oriented data transport using the Apache
+Arrow data format.  This component converts OTLP data into an
+optimized representation and then sends batches of data using Apache
+Arrow to encode the stream.  The [OTel-Arrow
+receiver](https://github.com/open-telemetry/otel-arrow/tree/main/collector/receiver/otelarrowreceiver)
+component contains logic to reverse the process used in this
+component.
+
+The use of an OTel-Arrow exporter-receiver pair is recommended when
+the network is expensive.  Typically, expect to see a 50% reduction in
+bandwidth compared with the same data being sent using standard
+OTLP/gRPC and gzip compression.
+
+This component includes all the features and configuration of the core
+OTLP exporter, making it possible to upgrade from the core component
+simply by replacing "otlp" with "otelarrow" as the component name in
+the collector configuration.
+
+To enable the OTel-Arrow exporter, include it in the list of exporters
+for a pipeline.  Two settings are required:
 
 - `endpoint` (no default): host:port to which the exporter is going to send OTLP trace data,
 using the gRPC protocol. The valid syntax is described
@@ -26,35 +54,140 @@ Example:
 
 ```yaml
 exporters:
-  otlp:
-    endpoint: otelcol2:4317
+  otelarrow/secure:
+    endpoint: external-collector:4317
     tls:
       cert_file: file.cert
       key_file: file.key
-  otlp/2:
-    endpoint: otelcol2:4317
+  otelarrow/insecure:
+    endpoint: internal-collector:4317
     tls:
       insecure: true
 ```
 
-By default, `gzip` compression is enabled. See [compression comparison](../../config/configgrpc/README.md#compression-comparison) for details benchmark information. To disable, configure as follows:
+By default, `zstd` compression is enabled at the gRPC level. See
+[compression
+comparison](../../config/configgrpc/README.md#compression-comparison)
+for details and benchmark information.  To disable gRPC-level
+compression, configure as follows:
 
 ```yaml
 exporters:
-  otlp:
-    ...
+  otelarrow:
     compression: none
+    endpoint: ...
+    tls: ...
 ```
 
-## Advanced Configuration
+## Configuration
 
 Several helper files are leveraged to provide additional capabilities automatically:
 
 - [gRPC settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md)
 - [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
 - [Queuing, retry and timeout settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
 
-[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta
-[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
-[core]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol
-[stable]: https://github.com/open-telemetry/opentelemetry-collector#stable
+### Arrow-specific Configuration
+
+In the `arrow` configuration block, the following settings enable and
+disable the use of OTel Arrow as opposed to standard OTLP.
+
+- `disabled` (default: false): disables use of Arrow, causing the exporter to use standard OTLP
+- `disable_downgrade` (default: false): prevents this exporter from using standard OTLP.
+
+The following settings determine the resources that the exporter will use:
+
+- `num_streams` (default: number of CPUs): the number of concurrent Arrow streams
+- `max_stream_lifetime` (default: unlimited): duration after which streams are recycled.
+
+### Network Configuration
+
+This component uses `round_robin` by default as the gRPC load
+balancer.  This can be modified using the `balancer_name` setting, for
+example, to configure the `pick_first` balancer:
+
+```yaml
+exporters:
+  otelarrow:
+    balancer_name: pick_first
+    endpoint: ...
+    tls: ...
+```
+
+When the server or an intermediate proxy uses a keepalive setting, the
+Arrow-specific `max_stream_lifetime` setting is critical to avoiding
+abrupt termination of Arrow streams, which causes retries of the
+in-flight requests.  The maximum stream lifetime should be set to a
+value less than the minimum of the server's keepalive parameter (and
+any of the intermediate proxies), plus the export timeout.
+
+```yaml
+exporters:
+  otelarrow:
+    timeout: 30s
+    arrow:
+      max_stream_lifetime: 9m30s
+    endpoint: ...
+    tls: ...
+```
+
+When this is configured, the stream will terminate cleanly without
+causing retries, with `OK` gRPC status.
+
+[The corresponding `otelarrowreceiver` keepalive setting, that is
+compatible with the one above,
+reads](../../receiver/otelarrowreceiver/README.md):
+
+```
+receivers:
+  otelarrow:
+    protocols:
+      grpc:
+        keepalive:
+          server_parameters:
+            max_connection_age: 1m
+            max_connection_age_grace: 10m
+```
+
+### Exporter metrics
+
+In addition to the the standard
+[exporterhelper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
+and
+[obsreport](https://pkg.go.dev/go.opentelemetry.io/collector/obsreport)
+metrics, this component provides network-level measurement instruments
+which we anticipate will become part of `exporterhelper` and/or
+`obsreport` in the future.  At the `normal` level of metrics detail:
+
+- `exporter_sent`: uncompressed bytes sent, prior to compression
+- `exporter_sent_wire`: compressed bytes sent, on the wire.
+
+Arrow's compression performance can be derived by dividing the average
+`exporter_sent` value by the average `exporter_sent_wire` value.
+
+At the `detailed` metrics detail level, information about the stream
+of data being returned to the exporter will be instrumented:
+
+- `exporter_recv`: uncompressed bytes received, prior to compression
+- `exporter_recv_wire`: compressed bytes received, on the wire.
+
+### Experimental Configuration
+
+The exporter supports configuring compression at the [Arrow
+columnar-protocol
+level](https://arrow.apache.org/docs/format/Columnar.html#format-ipc).
+
+- `payload_compression`: compression applied at the Arrow IPC level, "none" by default, "zstd" supported.
+
+The exporter uses the signal-specific Arrow stream methods (i.e.,
+`ArrowTraces`, `ArrowLogs`, and `ArrowMetrics`) by default.  There is
+an option to use the generic `ArrowStream` method instead.
+
+- `enable_mixed_signals` (default: false): Use `ArrowStream` instead of per-signal stream methods.
+
+This option has the potential to enable the future exporter to cross
+signals, meaning to allow traces, metrics and logs to refer to the
+same shared-data items across a single stream.  Presently, there is no
+cross-signal compression benefit, this option simply causes one method
+name instead of three method names to be used by the exporter
+instances of different signal types.

collector/exporter/otelarrowexporter/doc.go

@@ -1,5 +1,6 @@
 // Copyright The OpenTelemetry Authors
 // SPDX-License-Identifier: Apache-2.0
 
-// package otelarrowexporter exports data by using the OTLP format to a gPRC endpoint.
+//go:generate mdatagen metadata.yaml
+
 package otelarrowexporter // import "github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter"

collector/exporter/otelarrowexporter/metadata.yaml

@@ -0,0 +1,7 @@
+type: otelarrow
+
+status:
+  class: exporter
+  stability:
+    development: [metrics, traces, logs]
+  distributions: [otelarrow]

collector/receiver/otelarrowreceiver/README.md

@@ -26,15 +26,17 @@ component with additional support for the
 [OTel-Arrow](https://github.com/open-telemetry/otel-arrow) protocol.
 
 OTel-Arrow supports column-oriented data transport using the Apache
-Arrow data format.  The OTel-Arrow exporter converts OTLP data into an
-optimized representation and then sends batches of data using Apache
-Arrow to encode the stream.  This component contains logic to reverse
-the process used in the OTel-Arrow exporter.
+Arrow data format.  The [OTel-Arrow
+exporter](https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter/README.md)
+converts OTLP data into an optimized representation and then sends
+batches of data using Apache Arrow to encode the stream.  This
+component contains logic to reverse the process used in the OTel-Arrow
+exporter.
 
 The use of an OTel-Arrow exporter-receiver pair is recommended when
 the network is expensive.  Typically, expect to see a 50% reduction in
-bandwidth compared with the same data being sent using standard OTLP,
-unary gRPC, and zstd compression.
+bandwidth compared with the same data being sent using standard
+OTLP/gRPC and gzip compression.
 
 This component includes all the features and configuration of the core
 OTLP receiver, making it possible to upgrade from the core component
@@ -126,13 +128,33 @@ For example, an exporter with `9m30s` stream lifetime:
 ```
 exporters:
   otelarrow:
-    # other configuration (e.g., endpoint)
-    ...
+    timeout: 30s
     arrow:
-      # other configuration (e.g., num_streams)
       max_stream_lifetime: 9m30s
+    endpoint: ...
+    tls: ...
 ```
 
+### Receiver metrics
+
+In addition to the the standard
+[obsreport](https://pkg.go.dev/go.opentelemetry.io/collector/obsreport)
+metrics, this component provides network-level measurement instruments
+which we anticipate will become part of `obsreport` in the future.  At
+the `normal` level of metrics detail:
+
+- `receiver_recv`: uncompressed bytes received, prior to compression
+- `receiver_recv_wire`: compressed bytes received, on the wire.
+
+Arrow's compression performance can be derived by dividing the average
+`receiver_recv` value by the average `receiver_recv_wire` value.
+
+At the `detailed` metrics detail level, information about the stream
+of data being returned from the receiver will be instrumented:
+
+- `receiver_sent`: uncompressed bytes sent, prior to compression
+- `receiver_sent_wire`: compressed bytes sent, on the wire.
+
 ## HTTP-specific documentation
 
 To enable optional OTLP/HTTP support, the HTTP protocol must be

collector/receiver/otelarrowreceiver/metadata.yaml

@@ -4,4 +4,4 @@ status:
   class: receiver
   stability:
     development: [metrics, traces, logs]
-  distributions: [contrib]
+  distributions: [otelarrow]