New component: Google Cloud Logging Receiver

[abhishek-at-cloudwerx]

The purpose and use-cases of the new component

Presently, the recommended method for retrieving Google Cloud Logs is to use Log Router, Google PubSub and the googlepubsubreceiver. However, this would involve creation and setup of infrastructure for routing the logs. Each time we need a new log, the router infrastructure would be added/changed.

Proposing a component that would allow retrieving logs directly using GCP Cloud Logging API. This would allow clients with lower log volumes to easily retrieve logs for specific services and resources.

Example configuration for the component

googlecloudlogging:
  collection_interval: 120s
  region: us-central1
  project_id: my-project-id
  service_account_key: "path/to/service_account.json"
  resources:
    - resource_type: "cloudfunctions"
      additionalFilters: 
        - name: "resource.labels.service_name"
          value: "service-name"
        - name: "resource.labels.location"
          value: "us-east-1"
      resource_names:
        - "projects/my-project-id1"
        - "projects/my-project-id2"
    - resource_type: "another-resource-type"
      resource_names:
        - "projects/my-project-id3"
        - "projects/my-project-id4"

Telemetry data types supported

logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@abhishek-at-cloudwerx, @TylerHelmuth

Sponsor (optional)

@dashpole

Additional context

This proposal seeks community opinion on how useful/necessary the component is. We are open to do the development for this component. However, the component is not intended to be used at extremely high volumes - use of Log router and GCP Pub Sub is the right approach for that case.

[dashpole]

To clarify, the reason why the current recommendation is to use pub-sub is because it is difficult to shard log scraping between multiple collectors when querying the API. You either need to only use a single collector, which is what this proposal suggests, or design a sharding method by which collectors can coordinate to avoid querying the same logs. I worry most users won’t be aware of the path through pub-sub, and will run into issues with higher volumes.

Here is the current documentation for routing logs to pub-sub: https://cloud.google.com/logging/docs/export/pubsub#integrate-thru-pubsub.

We (Google) would prefer if there was a single blessed way to export logs to a collector, and can add or update documentation to make the current recommended solution via pub-sub more approachable for users. But if others still think this is necessary, I am willing to sponsor it, and help review the implementation.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

Marking as accepted since @dashpole is ok sponsoring.