Support granular search of AWS services #122

kaiyan-sheng · 2021-02-02T00:07:09Z

Introduction

Instead of only exposing the AWS package in Fleet/Integrations list, all supported services inside AWS package should also have their own icons and displayed in Fleet/Integrations list in Kibana UI. Using EC2 as an example, under Integrations, we should have both AWS and EC2 icons:

Design

Here is my initial design. I split them into three parts for three repos I believe we need to make changes on: elastic/integrations, elastic/package-registry, and elastic/kibana.

Part 1(integration):

In elastic/integrations repo, for packages, we should introduce two new config options into manifest.yml files.
At the integration package level, we have a manifest.yml file that includes the icon that UI is using for the whole integration. We should add a config parameter expand_data_stream at the package level to indicate this package should be expanded to display all data streams.
For example, manifest.yml should look like this:

icons:
 - src: /img/logo_aws.svg
   title: logo aws
   size: 32x32
   type: image/svg+xml
expand_data_stream: true
policy_templates:
 - name: aws
   title: AWS logs and metrics
   description: Collect logs and metrics from AWS services
   inputs:
     - type: s3
	...
- type: aws/metrics
...

At the data stream level, we also have an existing manifest.yml file. We should add a section for icons and a section for description for each data stream so when expand_data_stream is true, these icons at the data stream level will be displayed in Kibana Fleet/Integrations list.

For example, manifest.yml file should look like this:

title: AWS EC2 metrics
release: beta
type: metrics
description: AWS EC2 Integration
icons:
 - src: /img/logo_aws_ec2.svg
   title: logo aws ec2
   size: 32x32
   type: image/svg+xml
streams:
 - input: aws/metrics
 ...

Part 2(Elastic Package Registry):

In the Elastic Package Registry(EPR), we should adjust the code to read the new config option expand_data_stream. When expand_data_stream is set to true, abstract data stream info and store them into the /search API endpoint result. I created a draft PR for this to show the changes to the /search API response.

For example with expand_data_stream: true in AWS package, /search API response will look like this:

[
  {
    "name": "aws",
    "title": "AWS",
    "version": "0.3.18",
    "release": "beta",
    "description": "AWS Integration",
    "type": "integration",
    "download": "/epr/aws/aws-0.3.18.zip",
    "path": "/package/aws/0.3.18",
    "icons": [
      {
        "src": "/img/logo_aws.svg",
        "path": "/package/aws/0.3.18/img/logo_aws.svg",
        "title": "logo aws",
        "size": "32x32",
        "type": "image/svg+xml"
      }
    ],
    "base_data_streams": [
      ...
      {
        "type": "metrics",
        "dataset": "aws.ec2_metrics",
        "title": "AWS EC2 metrics",
        "release": "beta",
        "description": "AWS EC2 Integration",
        "icons": [
          {
            "src": "../img/logo_aws_ec2.svg",
            "path": "",
            "title": "logo aws ec2",
            "size": "32x32",
            "type": "image/svg+xml"
          }
        ]
      },
      ...
    ]
  }
]

Part 3(Kibana Fleet UI):

Continue to take the response from the EPR's /search endpoint, also take into account the icons section under base_data_streams.

Follow up

Once we achieve the granular search for AWS, we can split README.md into each data stream/service. This way when the user clicks on an icon for a specific service, some dedicated documentation will show up instead of the full documentation.

@hbharding @jasonrhodes WDYT about this approach?

cc @sorantis @masci

The text was updated successfully, but these errors were encountered:

mtojek · 2021-02-02T09:31:13Z

@kaiyan-sheng I reviewed the draft PR (elastic/package-registry#681) to follow the changes you're proposing. I think you should start with a PR to the package-spec suggesting modifications to the package spec (I noticed that you'd like to introduce icons/screenshots per data streams and expand_data_streams property).

ruflin · 2021-02-02T13:32:54Z

I'm not sure if the info you are interested in is actually coming from the data stream level. Lets take EC2 above: There are at least 2 data streams here, one for logs and one for metrics. Instead I think each integration (like EC2) is an additional entry in the policy_template list: https://github.com/elastic/integrations/blob/master/packages/aws/manifest.yml#L126 If that is the case, we would need additional entries for icons in policy_templates.

mtojek · 2021-02-02T16:36:05Z

I had a "short" call with @ycombinator today regarding enabling this feature. Let's call them temporarily - promoted data streams.

I tried to prepare a sample test package with promoted data streams. It looks like like promotion requires extra elements to be present like icons (with new directory), READMEs (with new directory), READMEs templates. Not sure if all data stream READMEs should be glued together into a package README or do we want to keep them consistent.

You can consider this sample package as an input for tests for this feature: https://github.com/mtojek/package-spec/tree/promote-data-stream-tests/code/go/internal/validator/test/packages/promoted_data_streams

A data stream can be highlighted as promoted with a manifest flag: promoted: true.

I'm not sure about description and title properties, which are not part of the data stream, but part of every data stream->streams[0]->input. Do we always have one of them here?

title: Second data stream
type: logs
streams:
  - input: logfile
    title: Second data stream title
    description: Second data stream description
icons:
  - src: /img/first_icon.svg
    title: Second data stream icon
    size: 32x32
    type: image/svg+xml
promoted: true

mostlyjason · 2021-02-02T16:39:28Z

I'm not sure these are streams or policy templates. I agree with @ruflin that these are not streams since there are multiple streams per service, and we only want to show services on the search page. That means that the streams for EC2 logs and metrics should create a single tile for EC2.

I'm not sure they are policy templates either because @sorantis proposed that we'd have only two policies for AWS, one for logs and one for metrics. This means the policy template alone cannot group by service.

I think we need a way for each integration to provide a list of services as metadata at the top of the integration. Each service will have a title, description, icon, etc. The service tiles are just links back to the parent integration. They are virtual integrations in the sense that they don't exist on their own, they just provide easily searched shortcuts back the parent.

This issue has nothing to do with how the integration policy editor is built or displayed. I think we should open a separate issue for how to specify which service each stream is assigned to, and how that builds the policy editor.

kaiyan-sheng · 2021-02-02T17:09:54Z

I'm not sure if the info you are interested in is actually coming from the data stream level. Lets take EC2 above: There are at least 2 data streams here, one for logs and one for metrics.

@ruflin Good point! We do want EC2 to be displayed as one icon instead of two (one for metrics, one for logs).
@mtojek This means we probably shouldn't add the icons section and promoted option inside the manifest.yml under each data stream.

Can we add a separate section called promoted_data_streams in the top-level manifest.yml file? For example, the new integrations/packages/aws/manifest.yml will look like this:

format_version: 1.0.0
name: aws
title: AWS
version: 0.3.18
license: basic
description: AWS Integration
type: integration
categories:
  - aws
  - cloud
  - network
  - security
release: beta
icons:
  - src: /img/logo_aws.svg
    title: logo aws
    size: 32x32
    type: image/svg+xml
promoted_data_streams:
  - name: aws_ec2
    release: beta
    description: AWS EC2 Integration
    icons:
      - src: /img/logo_aws_ec2.svg
        title: logo aws ec2
        size: 32x32
        type: image/svg+xml
  - name: aws_rds
    release: beta
    description: AWS RDS Integration
    icons:
      - src: /img/logo_aws_rds.svg
        title: logo aws rds
        size: 32x32
        type: image/svg+xml
  - name: aws_sns
    release: beta
    description: AWS SNS Integration
    icons:
      - src: /img/logo_aws_sns.svg
        title: logo aws sns
        size: 32x32
        type: image/svg+xml
policy_templates:
  - name: aws
    title: AWS logs and metrics
    description: Collect logs and metrics from AWS services
    inputs:
      - type: s3
      ...
      - type: aws/metrics
      ...

ycombinator · 2021-02-02T17:21:38Z

Thanks @mostlyjason, your comment simplifies a few things, in particular:

The service tiles are just links back to the parent integration.

This was not the requirement I was made aware of; instead my understanding was that clicking on the service tiles was supposed to take the user to a service detail page which would be similar to the integration detail page in layout but contain service information instead. Additionally, I was told clicking on the "Add Integration" button from that page should take the user to the parent integration configuration page but with only the selected service's data streams enabled.

But given what you've just said, we can design something much simpler. I'm imagining an optional services field in a package's manifest.yml, something like:

services:
- name: ec2
  title: EC2
  description: ...
  icons:
    - src: /img/logo_aws_ec2.svg
      title: AWS EC2 logo
      size: 32x32
      type: image/svg+xml
- name: s3
  title: S3
  ...
...

When the package registry starts up and builds an index of packages, it could parse the services section, if present, from the package's manifest.yml file and index virtual packages with those details, one virtual package for each service. All other details would come from the parent package.

mtojek · 2021-02-02T17:30:20Z

@ruflin Good point! We do want EC2 to be displayed as one icon instead of two (one for metrics, one for logs).
@mtojek This means we probably shouldn't add the icons section and promoted option inside the manifest.yml under each data stream.

Out of curiosity, did we consider introducing a new integration called just "EC2 Integration" that combine logs and metrics instead?

kaiyan-sheng · 2021-02-02T17:34:49Z

@mtojek We did consider that in elastic/integrations#577 (comment). With this design, we will have to maintain all these individual service integrations, which will be a nightmare (versioning and etc). Also when users want to enable multiple services under AWS, they have to duplicate the onboarding process multiple times, which is not user friendly. cc @masci

kaiyan-sheng · 2021-02-02T18:07:14Z

@mostlyjason @sorantis When a user clicks on the AWS EC2 icon and clicks add integration, the same page for adding AWS integration should show up but with only EC2 metrics and EC2 logs enabled by default, correct?

Will Kibana have a way to identify EC2 metrics and EC2 logs that should be enabled or this requires changes in EPR as well? Maybe we need a data_stream section under services in the top level manifest.yml like this:

icons:
  - src: /img/logo_aws.svg
    title: logo aws
    size: 32x32
    type: image/svg+xml
services:
  - name: aws_ec2
    release: beta
    description: AWS EC2 Integration
    icons:
      - src: /img/logo_aws_ec2.svg
        title: logo aws ec2
        size: 32x32
        type: image/svg+xml
    data_stream:
      - ec2_logs
      - ec2_metrics
  - name: aws_sns
    release: beta
    description: AWS SNS Integration
    data_stream:
      - sns      
    icons:
      - src: /img/logo_aws_sns.svg
        title: logo aws sns
        size: 32x32
        type: image/svg+xml
policy_templates:
  - name: aws
    title: AWS logs and metrics
    description: Collect logs and metrics from AWS services
    inputs:
      - type: s3
      ...
      - type: aws/metrics
      ...

ph · 2021-02-02T21:00:56Z

@jen-huang It would be good that you keep on eye on this, this will require probably changes in our UI.

mostlyjason · 2021-02-02T22:11:57Z

my understanding was that clicking on the service tiles was supposed to take the user to a service detail page which would be similar to the integration detail page in layout but contain service information instead. Additionally, I was told clicking on the "Add Integration" button from that page should take the user to the parent integration configuration page but with only the selected service's data streams enabled.

Yes that sounds right. I was thinking of the search page as a separate issue from the integration detail and integration policy editor pages. They are all related to the same user journey and need to link to each other. So based on which tile the user selects on the search page, the integration detail could either be specific to each service, or have a section that is activated for each service. I don't remember seeing a mockup from @sorantis on what the detail page would show. It'd be nice to know what that looks like though. @hbharding and I can help with the nginx use case if desired.

+1 on having a services section, and associating the services with streams.

ycombinator · 2021-02-02T22:19:16Z

Thanks @mostlyjason.

@kaiyan-sheng Based on Jason's comment above, I think what you proposed in #122 (comment) is good. The only tweaks I'd suggest are:

call the field under services.N as data_streams (plural) instead of data_stream (singular), and
add a field, title under services.N which is separate from name. This mirrors what's going on for a package as well.

I think there's sufficient consensus at this point to move forward with a PR to the package spec. We can iterate on further changes once you have the PR in review. Thanks!

kaiyan-sheng · 2021-02-02T22:30:12Z

@ycombinator Thank you for the suggestions! Please let me know if the section below looks better now! I can start a PR in package spec and discuss more there!

icons:
  - src: /img/logo_aws.svg
    title: logo aws
    size: 32x32
    type: image/svg+xml
services:
  - name: aws_ec2
    title: AWS EC2
    release: beta
    description: AWS EC2 Integration
    icons:
      - src: /img/logo_aws_ec2.svg
        title: logo aws ec2
        size: 32x32
        type: image/svg+xml
    data_streams:
      - ec2_logs
      - ec2_metrics
  - name: aws_sns
    title: AWS SNS
    release: beta
    description: AWS SNS Integration
    data_streams:
      - sns      
    icons:
      - src: /img/logo_aws_sns.svg
        title: logo aws sns
        size: 32x32
        type: image/svg+xml
policy_templates:
  - name: aws
    title: AWS logs and metrics
    description: Collect logs and metrics from AWS services
    inputs:
      - type: s3
      ...
      - type: aws/metrics
      ...

ycombinator · 2021-02-02T22:49:18Z

@kaiyan-sheng Yup, LGTM. PR time!

ruflin · 2021-02-03T07:00:17Z

The discussion internally is still ongoing on what the exact concept should be so it is impossible to map to a spec yet. Lets hold this discussion until we all agreed on the mocks and the user flow.

ycombinator · 2021-02-03T13:56:14Z

The discussion internally is still ongoing on what the exact concept should be so it is impossible to map to a spec yet.

I thought we had agreed that the package-spec repo is where all discussions that impact the spec, even if they impact other parts of the stack, should happen. So I want to note here that it's quite frustrating to not be privy to all discussions and think that this issue is where all the discussion is happening.

At the very least it would be helpful if discussions happening in other issues were linked from here or vice versa (in case of private repo issues). And if discussions are happening in Zoom, Slack, etc. — that's fine but please bring them back to issues. Otherwise its very frustrating to those of us not privy to all the discussion and not a very open way of collaboration. 😞

mostlyjason · 2021-02-03T15:37:41Z

+1 on holding off development until we agree on the definition. @sorantis is leading an effort to define this initiative in more detail and it is still WiP. We need approval from drivers and key stakeholders before proceeding.

ycombinator · 2021-02-03T16:17:28Z

++ in the future lets not file GH issues until definitions/design have settled elsewhere. Otherwise it leads to confusion and churn like we saw here.

kaiyan-sheng · 2021-07-12T03:17:49Z

Closing this issue since PRs on both integrations side and Kibana side are merged.

kaiyan-sheng self-assigned this Feb 2, 2021

kaiyan-sheng added Team:Integrations Label for the Integrations team discuss Issue needs discussion labels Feb 2, 2021

kaiyan-sheng mentioned this issue Feb 2, 2021

[AWS] Add support for granular search of cloud service integrations elastic/integrations#526

Closed

kaiyan-sheng mentioned this issue Feb 3, 2021

Add services field into package manifest.yml #125

Closed

2 tasks

mtojek mentioned this issue Feb 10, 2021

[Discuss] Proposal: support integration sets #132

Closed

kaiyan-sheng mentioned this issue Mar 8, 2021

Adjust aws package to use input groups elastic/integrations#767

Merged

3 tasks

kaiyan-sheng closed this as completed Jul 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support granular search of AWS services #122

Support granular search of AWS services #122

kaiyan-sheng commented Feb 2, 2021 •

edited

Loading

mtojek commented Feb 2, 2021

ruflin commented Feb 2, 2021

mtojek commented Feb 2, 2021

mostlyjason commented Feb 2, 2021 •

edited

Loading

kaiyan-sheng commented Feb 2, 2021

ycombinator commented Feb 2, 2021 •

edited

Loading

mtojek commented Feb 2, 2021 •

edited

Loading

kaiyan-sheng commented Feb 2, 2021

kaiyan-sheng commented Feb 2, 2021

ph commented Feb 2, 2021

mostlyjason commented Feb 2, 2021 •

edited

Loading

ycombinator commented Feb 2, 2021 •

edited

Loading

kaiyan-sheng commented Feb 2, 2021

ycombinator commented Feb 2, 2021

ruflin commented Feb 3, 2021

ycombinator commented Feb 3, 2021 •

edited

Loading

mostlyjason commented Feb 3, 2021

ycombinator commented Feb 3, 2021

kaiyan-sheng commented Jul 12, 2021

Support granular search of AWS services #122

Support granular search of AWS services #122

Comments

kaiyan-sheng commented Feb 2, 2021 • edited Loading

Introduction

Design

Part 1(integration):

Part 2(Elastic Package Registry):

Part 3(Kibana Fleet UI):

Follow up

mtojek commented Feb 2, 2021

ruflin commented Feb 2, 2021

mtojek commented Feb 2, 2021

mostlyjason commented Feb 2, 2021 • edited Loading

kaiyan-sheng commented Feb 2, 2021

ycombinator commented Feb 2, 2021 • edited Loading

mtojek commented Feb 2, 2021 • edited Loading

kaiyan-sheng commented Feb 2, 2021

kaiyan-sheng commented Feb 2, 2021

ph commented Feb 2, 2021

mostlyjason commented Feb 2, 2021 • edited Loading

ycombinator commented Feb 2, 2021 • edited Loading

kaiyan-sheng commented Feb 2, 2021

ycombinator commented Feb 2, 2021

ruflin commented Feb 3, 2021

ycombinator commented Feb 3, 2021 • edited Loading

mostlyjason commented Feb 3, 2021

ycombinator commented Feb 3, 2021

kaiyan-sheng commented Jul 12, 2021

kaiyan-sheng commented Feb 2, 2021 •

edited

Loading

mostlyjason commented Feb 2, 2021 •

edited

Loading

ycombinator commented Feb 2, 2021 •

edited

Loading

mtojek commented Feb 2, 2021 •

edited

Loading

mostlyjason commented Feb 2, 2021 •

edited

Loading

ycombinator commented Feb 2, 2021 •

edited

Loading

ycombinator commented Feb 3, 2021 •

edited

Loading