Improve OC Receiver Documentation (#203)

* First round of receiver and opencensus receiver documentation.

* Undo go mod/sum changes

* Address initial set of comments.

* Address next set of comments.

* Address next set of comments.

* Fix use of server instead of receiver in comment and explain settings can be mix and matched.

* Merged master and fixed mispell error caugh with new tools

README.md

@@ -122,14 +122,29 @@ exporting will resume.
 ## <a name="config"></a>Configuration
 
 The OpenTelemetry Service (both the Agent and Collector) is configured via a
-YAML file. In general, you need to configure one or more receivers as well as
-one or more exporters. In addition, diagnostics can also be configured.
+YAML file. In general, at least one enabled receiver and one enabled exporter
+needs to be configured.
+
+*Note* This documentation is still in progress. For any questions, please reach out in the
+[OpenTelemetry Gitter](https://gitter.im/open-telemetry/opentelemetry-service) or
+refer to the [issues page](https://github.com/open-telemetry/opentelemetry-service/issues).
+
+There are four main parts to a config:
+```yaml
+receivers:
+  ...
+exporters:
+  ...
+processors:
+  ...
+pipelines:
+  ...
+```
 
 ### <a name="config-receivers"></a>Receivers
 
-A receiver is how you get data into the OpenTelemetry Service. One or more
-receivers can be configured. By default, the `opentelemetry` receiver is enabled
-on the Collector and required as a defined receiver for the Agent.
+A receiver is how data gets into OpenTelemetry Service. One or more receivers
+must be configured.
 
 A basic example of all available receivers is provided below. For detailed
 receiver configuration, please see the [receiver
@@ -182,6 +197,33 @@ exporters:
     endpoint: "http://127.0.0.1:9411/api/v2/spans"
 ```
 
+### <a name="config-pipelines"></a>Pipelines
+Pipelines can be of two types:
+
+- metrics: collects and processes metrics data.
+- traces: collects and processes trace data.
+
+A pipeline consists of a set of receivers, processors, and exporters. Each
+receiver/processor/exporter must be specified in the configuration to be
+included in a pipeline and each receiver/processor/exporter can be used in more
+than one pipeline.
+
+*Note:* For processor(s) referenced in multiple pipelines, each pipeline will
+get a separate instance of that processor(s). This is in contrast to
+receiver(s)/exporter(s) referenced in multiple pipelines, one instance of
+a receiver/exporter is reference by all the pipelines.
+
+The following is an example pipeline configuration. For more information, refer
+to [pipeline documentation](docs/pipelines.md)
+```yaml
+pipelines:
+  traces:
+    receivers: [examplereceiver]
+    processors: [exampleprocessor]
+    exporters: [exampleexporter]
+
+```
+
 ### <a name="config-diagnostics"></a>Diagnostics
 
 zPages is provided for monitoring running by default on port ``55679``.
@@ -208,7 +250,10 @@ zpages:
     disabled: true
 ```
 
+
 ### <a name="global-attributes"></a> Global Attributes
+**TODO** Remove this once processors have been documented since that handles
+these features now.
 
 The OpenTelemetry Service also takes some global configurations that modify its
 behavior for all receivers / exporters. This configuration is typically applied
@@ -301,7 +346,6 @@ sampling:
 ```
 
 > Note that an exporter can only have a single sampling policy today.
-
 ## <a name="collector-usage"></a>Usage
 
 > It is recommended that you use the latest [release](https://github.com/open-telemetry/opentelemetry-service/releases).

config/config.go

@@ -55,6 +55,8 @@ const (
 	errPipelineExporterNotExists
 	errMetricPipelineCannotHaveProcessors
 	errUnmarshalError
+	errMissingReceivers
+	// TODO(ccaraman): Add an error for missing Processors with corresponding test cases.
 )
 
 type configError struct {
@@ -175,6 +177,19 @@ func loadReceivers(v *viper.Viper, factories map[string]receiver.Factory) (confi
 	// Get the map of "receivers" sub-keys.
 	keyMap := v.GetStringMap(receiversKeyName)
 
+	// Currently there is no default receiver enabled. The configuration must specify at least one receiver to enable
+	// functionality.
+	if len(keyMap) == 0 {
+		return nil, &configError{
+			code: errMissingReceivers,
+			msg:  "no receivers specified in config",
+		}
+	}
+
+	// This boolean is used to track if there are any enabled receivers. If there are none at the end of loading
+	// all of the receivers, throw an error.
+	enabledReceiver := false
+
 	// Prepare resulting map
 	receivers := make(configmodels.Receivers, 0)
 
@@ -211,6 +226,9 @@ func loadReceivers(v *viper.Viper, factories map[string]receiver.Factory) (confi
 			err = customUnmarshaler(subViper, key, receiverCfg)
 		} else {
 			// Standard viper unmarshaler is fine.
+			// TODO(ccaraman): UnmarshallExact should be used to catch erroneous config entries.
+			// 	This leads to quickly identifying config values that are not supported and reduce confusion for
+			// 	users.
 			err = subViper.UnmarshalKey(key, receiverCfg)
 		}
 
@@ -228,9 +246,20 @@ func loadReceivers(v *viper.Viper, factories map[string]receiver.Factory) (confi
 			}
 		}
 
+		// Or'ing the enabled flag for all receivers will return true if at least one is enabled.
+		enabledReceiver = enabledReceiver || receiverCfg.IsEnabled()
+
 		receivers[fullName] = receiverCfg
 	}
 
+	// There must be at least one enabled receiver for the config to be valid.
+	if !enabledReceiver {
+		return nil, &configError{
+			code: errMissingReceivers,
+			msg:  "no enabled receivers specified in config",
+		}
+	}
+
 	return receivers, nil
 }
 
@@ -496,6 +525,7 @@ func validatePipelineReceivers(
 				zap.String("receiver", ref))
 		}
 	}
+
 	pipeline.Receivers = rs
 
 	return nil
@@ -594,6 +624,9 @@ func validatePipelineProcessors(
 	return nil
 }
 
+// TODO(ccaraman): Determine if there a way to consolidate the validate receivers apart of the loadReceivers method.
+//	 Currently, validateReceivers needs to be invoked after validatePipelineReceivers because that bit of
+//	 code checks if a receiver is enabled prior to finalizing the pipelines.
 func validateReceivers(cfg *configmodels.Config) {
 	// Remove disabled receivers.
 	for name, rcv := range cfg.Receivers {

config/config_test.go

@@ -176,7 +176,8 @@ func TestDecodeConfig_Invalid(t *testing.T) {
 	}{
 		{name: "empty-config"},
 		{name: "missing-all-sections"},
-		{name: "missing-receivers"},
+		{name: "missing-enabled-receivers", expected: errMissingReceivers},
+		{name: "missing-receivers", expected: errMissingReceivers},
 		{name: "missing-exporters"},
 		{name: "missing-processors"},
 		{name: "invalid-receiver-name"},

config/configmodels/configmodels.go

@@ -142,11 +142,19 @@ const (
 // ReceiverSettings defines common settings for a single-protocol receiver configuration.
 // Specific receivers can embed this struct and extend it with more fields if needed.
 type ReceiverSettings struct {
-	TypeVal             string `mapstructure:"-"`
-	NameVal             string `mapstructure:"-"`
-	Disabled            bool   `mapstructure:"disabled"`
-	Endpoint            string `mapstructure:"endpoint"`
-	DisableBackPressure bool   `mapstructure:"disable-backpressure"`
+	TypeVal string `mapstructure:"-"`
+	NameVal string `mapstructure:"-"`
+	// Configures if the receiver is disabled and doesn't receive any data.
+	// The default value is false(meaning the receiver is enabled by default), and it is expected that receivers
+	// continue to use the default value of false.
+	Disabled bool `mapstructure:"disabled"`
+	// Configures the endpoint in the format 'address:port' for the receiver.
+	// The default value is set by the receiver populating the struct.
+	Endpoint string `mapstructure:"endpoint"`
+	// Configures if the back pressure functionality is disabled for this receiver.
+	// The default value is false, and it is expected that receivers
+	// continue to use the default value of false.
+	DisableBackPressure bool `mapstructure:"disable-backpressure"`
 }
 
 // Name gets the receiver name.

config/testdata/missing-enabled-receivers.yaml

@@ -0,0 +1,23 @@
+receivers:
+  examplereceiver:
+    disabled: true
+  examplereceiver/disabled:
+    disabled: true
+
+processors:
+  exampleprocessor:
+  exampleprocessor/disabled:
+    disabled: true
+
+exporters:
+  exampleexporter/myexporter:
+    extra: "some export string 2"
+  exampleexporter/disabled:
+    disabled: true
+  exampleexporter:
+
+pipelines:
+  traces:
+    receivers: [examplereceiver, examplereceiver/disabled]
+    processors: [exampleprocessor, exampleprocessor/disabled]
+    exporters: [exampleexporter/disabled, exampleexporter]

docs/pipelines.md

@@ -0,0 +1,4 @@
+# Pipelines
+*Note* This documentation is still in progress. For any questions, please reach
+out in the [OpenTelemetry Gitter](https://gitter.im/open-telemetry/opentelemetry-service)
+or refer to the [issues page](https://github.com/open-telemetry/opentelemetry-service/issues).

internal/zpagesserver/zpagesserver.go

@@ -25,6 +25,7 @@ import (
 
 const (
 	// ZPagesHTTPPort is the name of the flag used to specify the zpages port.
+	// TODO(ccaraman): Move ZPage configuration to be apart of global config/config.go, maybe under diagnostics section.
 	ZPagesHTTPPort = "zpages-http-port"
 )
 

processor/README.md

@@ -0,0 +1,4 @@
+# Processors
+*Note* This documentation is still in progress. For any questions, please reach
+out in the [OpenTelemetry Gitter](https://gitter.im/open-telemetry/opentelemetry-service)
+or refer to the [issues page](https://github.com/open-telemetry/opentelemetry-service/issues).

receiver/README.md

@@ -1,11 +1,28 @@
-A variety of receivers are available to the OpenCensus Service (both Agent and Collector)
+**Note** This documentation is still in progress. For any questions, please
+reach out in the [OpenTelemetry Gitter](https://gitter.im/open-telemetry/opentelemetry-service)
+or refer to the [issues page](https://github.com/open-telemetry/opentelemetry-service/issues).
 
-__Currently there are some inconsistencies between Agent and Collector configuration, those will be addressed by issue
-[#135](https://github.com/census-instrumentation/opencensus-service/issues/135).__ 
+# Receivers
+A receiver is how data gets into OpenTelemetry Service. Generally, a receiver
+accepts data in a specified format and can support traces and/or metrics. The
+format of the traces and metrics supported are receiver specific.
 
-## OpenCensus
+Supported receivers (sorted alphabetically):
+- [Jaeger Receiver](#jaeger)
+- [OpenCensus Receiver](#opencensus)
+- [Prometheus Receiver](#prometheus)
+- [VM Metrics Receiver](#vmmetrics)
+- [Zipkin Receiver](#zipkin)
 
-This receiver receives spans from OpenCensus instrumented applications and translates them into the internal span types that are then sent to the collector/exporters.
+## Configuring Receiver(s)
+TODO - Add what a fullname is and how that is referenced in other parts of the
+configuration. Describe the common receiver settings: endpoint, disabled and
+disable-backpressure.
+
+## <a name="opencensus"></a>OpenCensus Receiver
+**Traces and metrics are supported.**
+
+This receiver receives spans from [OpenCensus](https://opencensus.io/) instrumented applications and translates them into the internal format sent to processors and exporters in the pipeline.
 
 Its address can be configured in the YAML configuration file under section "receivers", subsection "opencensus" and field "address". The syntax of the field "address" is `[address|host]:<port-number>`.
 
@@ -39,7 +56,12 @@ receivers:
     - https://*.example.com  
 ```
 
+### Deprecated YAML Configurations
+**Note**: This isn't a full list of deprecated OpenCensus YAML configurations. If something is missing, please expand the documentation
+or open an issue.
+
 ### Collector Differences
+TODO(ccaraman) - Delete all references to opencensus-service issue 135 in follow up pr.
 (To be fixed via [#135](https://github.com/census-instrumentation/opencensus-service/issues/135))
 
 By default this receiver is ALWAYS started on the OpenCensus Collector, it can be disabled via command-line by
@@ -97,9 +119,12 @@ receivers:
         permit-without-stream: true
 ```
 
-## Jaeger
+
+## <a name="jaeger"></a>Jaeger Receiver
+**Only traces are supported.**
 
 This receiver receives spans from Jaeger collector HTTP and Thrift uploads and translates them into the internal span types that are then sent to the collector/exporters.
+Only traces are supported. This receiver does not support metrics.
 
 Its address can be configured in the YAML configuration file under section "receivers", subsection "jaeger" and fields "collector_http_port", "collector_thrift_port".
 
@@ -124,32 +149,8 @@ receivers:
     jaeger-thrift-http-port: 14268
 ```
 
-## Zipkin
-
-This receiver receives spans from Zipkin (V1 and V2) HTTP uploads and translates them into the internal span types that are then sent to the collector/exporters.
-
-Its address can be configured in the YAML configuration file under section "receivers", subsection "zipkin" and field "address".  The syntax of the field "address" is `[address|host]:<port-number>`.
-
-For example:
-
-```yaml
-receivers:
-  zipkin:
-    address: "127.0.0.1:9411"
-```
-
-### Collector Differences
-(To be fixed via [#135](https://github.com/census-instrumentation/opencensus-service/issues/135))
-
-On the Collector Zipkin reception at the port 9411 can be enabled via command-line `--receive-zipkin`. On the Collector only the port can be configured, example:
-
-```yaml
-receivers:
-  zipkin:
-    port: 9411
-```
-
-## Prometheus
+## <a name="prometheus"></a>Prometheus Receiver
+**Only metrics are supported.**
 
 This receiver is a drop-in replacement for getting Prometheus to scrape your services. Just like you would write in a
 YAML configuration file before starting Prometheus, such as with:
@@ -180,3 +181,37 @@ receivers:
             static_configs:
               - targets: ['localhost:9777']
 ```
+
+## <a name="vmmetrics"></a>VM Metrics Receiver
+**Only metrics are supported.**
+
+<Add more information - I'm lonely.>
+
+## <a name="zipkin"></a>Zipkin Receiver
+**Only traces are supported.**
+
+This receiver receives spans from Zipkin (V1 and V2) HTTP uploads and translates them into the internal span types that are then sent to the collector/exporters.
+
+Its address can be configured in the YAML configuration file under section "receivers", subsection "zipkin" and field "address".  The syntax of the field "address" is `[address|host]:<port-number>`.
+
+For example:
+
+```yaml
+receivers:
+  zipkin:
+    address: "127.0.0.1:9411"
+```
+
+### Collector Differences
+(To be fixed via [#135](https://github.com/census-instrumentation/opencensus-service/issues/135))
+
+On the Collector Zipkin reception at the port 9411 can be enabled via command-line `--receive-zipkin`. On the Collector only the port can be configured, example:
+
+```yaml
+receivers:
+  zipkin:
+    port: 9411
+```
+
+## Common Configuration Errors
+<Fill this in as we go with common gotchas experienced by users. These should eventually be made apart of the validation test suite.>