Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #18298 to 7.x: [Filebeat][New Input] Http Input #18719

Merged
merged 1 commit into from
May 25, 2020

Conversation

kvch
Copy link
Contributor

@kvch kvch commented May 25, 2020

Cherry-pick of PR #18298 to 7.x branch. Original message:

What does this PR do?

This filebeat input configures a HTTP port listener, accepting JSON formatted POST requests, which again is formatted into a event, initially the event is created with the "json." prefix and expects the ingest pipeline to mutate the event during ingestion.

The initial set of features is based on the Logstash input plugin, but implemented differently: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html

Why is it important?

This idea is based on a few different scenarios:

  • The user already has a large beats installation and no Logstash, and do not want to install Logstash solely for a single feature.
  • HTTP Input allows applications to be directly integrated with Elastic, without needing connectivity to Elasticsearch directly (or Logstash).
  • Allows us to integrate and create modules for any product that supports HTTP POST events like SOAR, cloud applications, ticketing systems etc etc.

Features currently implemented

HTTP Basic Auth On/Off
HTTP/HTTPS configurable
Listening interface and port configurable
Response code on success configurable
Response body on success configurable
Response header on success configurable
Proper HTTP codes on both success and error responses
Message prefix configurable
URL to post to is configurable
SSL path to cert, key and CA is configurable.

TODO

  • Clean up code
  • Import configuration options from common
  • Write documentation
  • Add TLS mutual auth support
  • Config validation
  • Tests

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## What does this PR do?

This filebeat input configures a HTTP port listener, accepting JSON formatted POST requests, which again is formatted into a event, initially the event is created with the "json." prefix and expects the ingest pipeline to mutate the event during ingestion.

The initial set of features is based on the Logstash input plugin, but implemented differently: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html

## Why is it important?

This idea is based on a few different scenarios:

- The user already has a large beats installation and no Logstash, and do not want to install Logstash solely for a single feature.
- HTTP Input allows applications to be directly integrated with Elastic, without needing connectivity to Elasticsearch directly (or Logstash).
- Allows us to integrate and create modules for any product that supports HTTP POST events like SOAR, cloud applications, ticketing systems etc etc.

## Features currently implemented

- HTTP Basic Auth On/Off
- HTTP/HTTPS configurable
- Listening interface and port configurable
- Response code on success configurable
- Response body on success configurable
- Response header on success configurable
- Proper HTTP codes on both success and error responses
- Message prefix configurable
- URL to post to is configurable
- SSL path to cert, key and CA is configurable.

(cherry picked from commit 0b84f0a)
@kvch kvch added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels May 25, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 25, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 25, 2020
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18719 opened]

  • Start Time: 2020-05-25T12:28:01.106+0000

  • Duration: 54 min 46 sec

Test stats 🧪

Test Results
Failed 0
Passed 2647
Skipped 417
Total 3064

Copy link
Contributor

@michalpristas michalpristas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok

@kvch kvch merged commit 873500a into elastic:7.x May 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Services (Deprecated) Label for the former Integrations-Services team [zube]: Done
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants