Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Infra] Make apm synthtrace kibana client a service available to functional and integration tests #175064

Closed
neptunian opened this issue Jan 17, 2024 · 1 comment · Fixed by #179764
Closed

[Infra] Make apm synthtrace kibana client a service available to functional and integration tests #175064

neptunian opened this issue Jan 17, 2024 · 1 comment · Fixed by #179764
Assignees
@neptunian
Labels
apm:synthtrace APM Synthtrace Topics chore Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team technical debt Improvement of the software architecture and operational architecture

Comments

@neptunian
Copy link
Contributor

neptunian commented Jan 17, 2024 •
edited
Loading

Create a service provider for apm kibana synthtrace client to make it shareable across other apps. It will make apm package management accessible within tests so that the tests can handle installing/uninstalling of the package instead of doing it in the plugin config and not cleaning up.

With this in place, the apm elasticsearch synthtrace client can be used directly in tests alongside it to avoid having to add it as a service in the plugin config.

This will help plugins to avoid causing an error for apps (infra) that are included in the FF/Edge tests because unshared services on an app config will not be available in that environment.

Unlike the other synthtrace es clients (logs, metrics, assets), the APM es client depends on the installation of the APM package. Speaking with appEx team it's advised not to do this when loading a service and the appropriate place is when the first test that is using it is loaded.

Given both integration and functional tests need this service it should be added here https://github.com/elastic/kibana/tree/main/x-pack/test/common/services.

ACE

  • Create a service for the apm synthtrace kibana client
  • Use the service within infra only for now
  • install/uninstall the apm package within the infra tests where needed
  • Replace infra's current usage in favor of the service
  • Use the service in Infra's integration tests and revert custom changes to the config

Usages of apm elasticsearch and kibana synthtrace clients service within plugin config

Potential Follow up issues

  • Update other app's configs using the apm es synthtrace to use the service and the same pattern. Undo the redfined FtrProviderContext type in the configs and move the apm package installation and uninstall it during cleanup
  • Create a service for infra synthtrace es client
  • Create a service for the logs synthtrace es client
@neptunian neptunian added apm:synthtrace APM Synthtrace Topics Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Jan 17, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@neptunian neptunian mentioned this issue Jan 18, 2024
Merged
neptunian added a commit that referenced this issue Feb 5, 2024
@neptunian @kibanamachine
[Infra] infra services api (#173875)
6fc6950 
## Summary
Creation of a new endpoint within Infra to get services from APM indices
that are related to a give host through `host.name`. These services will
be listed in the Host Detail view in another PR. This endpoint queries
apm transaction metrics and apm logs to get services.

Closes #171661

### Test
The easiest way to test this api is to visit it directly using a host
that has some services attached to it using our test cluster

URL: http://localhost:5601/api/infra/services
eg usage:
`http://localhost:5601/api/infra/services?from=now-15m&to=now&filters={"host.name":"gke-edge-oblt-edge-oblt-pool-5fbec7a6-nfy0"}&size=5`

response:

```
{
    "services": [
        {
            "service.name": "productcatalogservice",
            "agent.name": "opentelemetry/go"
        },
        {
            "service.name": "frontend",
            "agent.name": "opentelemetry/nodejs"
        }
    ]
}
```



### Follow up 
- Have APM server collect host.name as part of service_summary metrics
and query that instead. Service summary aggregates transaction, error,
log, and metric events into service-summary metrics. This would simplify
the query.

- `added apm-synthtrace` to `metrics_ui` api tests and created follow up
PR for removing the code i needed to duplicate
#175064

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
fkanout pushed a commit to fkanout/kibana that referenced this issue Feb 7, 2024
@neptunian @kibanamachine @fkanout
[Infra] infra services api (elastic#173875)
92c6d9a 
## Summary
Creation of a new endpoint within Infra to get services from APM indices
that are related to a give host through `host.name`. These services will
be listed in the Host Detail view in another PR. This endpoint queries
apm transaction metrics and apm logs to get services.

Closes elastic#171661

### Test
The easiest way to test this api is to visit it directly using a host
that has some services attached to it using our test cluster

URL: http://localhost:5601/api/infra/services
eg usage:
`http://localhost:5601/api/infra/services?from=now-15m&to=now&filters={"host.name":"gke-edge-oblt-edge-oblt-pool-5fbec7a6-nfy0"}&size=5`

response:

```
{
    "services": [
        {
            "service.name": "productcatalogservice",
            "agent.name": "opentelemetry/go"
        },
        {
            "service.name": "frontend",
            "agent.name": "opentelemetry/nodejs"
        }
    ]
}
```



### Follow up 
- Have APM server collect host.name as part of service_summary metrics
and query that instead. Service summary aggregates transaction, error,
log, and metric events into service-summary metrics. This would simplify
the query.

- `added apm-synthtrace` to `metrics_ui` api tests and created follow up
PR for removing the code i needed to duplicate
elastic#175064

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@neptunian neptunian changed the title [synthtrace] Make synthtrace available across functional tests apis [synthtrace] Make synthtrace service available across functional and integration tests Feb 12, 2024
@neptunian neptunian added the technical debt Improvement of the software architecture and operational architecture label Feb 12, 2024
@neptunian neptunian mentioned this issue Feb 12, 2024
Merged
@neptunian neptunian changed the title [synthtrace] Make synthtrace service available across functional and integration tests [Obs UX] Make apm synthtrace es client a service available to functional and integration tests Feb 13, 2024
@neptunian neptunian assigned neptunian and unassigned neptunian Feb 13, 2024
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@neptunian @kibanamachine @CoenWarmer
[Infra] infra services api (elastic#173875)
a8cbd51 
## Summary
Creation of a new endpoint within Infra to get services from APM indices
that are related to a give host through `host.name`. These services will
be listed in the Host Detail view in another PR. This endpoint queries
apm transaction metrics and apm logs to get services.

Closes elastic#171661

### Test
The easiest way to test this api is to visit it directly using a host
that has some services attached to it using our test cluster

URL: http://localhost:5601/api/infra/services
eg usage:
`http://localhost:5601/api/infra/services?from=now-15m&to=now&filters={"host.name":"gke-edge-oblt-edge-oblt-pool-5fbec7a6-nfy0"}&size=5`

response:

```
{
    "services": [
        {
            "service.name": "productcatalogservice",
            "agent.name": "opentelemetry/go"
        },
        {
            "service.name": "frontend",
            "agent.name": "opentelemetry/nodejs"
        }
    ]
}
```



### Follow up 
- Have APM server collect host.name as part of service_summary metrics
and query that instead. Service summary aggregates transaction, error,
log, and metric events into service-summary metrics. This would simplify
the query.

- `added apm-synthtrace` to `metrics_ui` api tests and created follow up
PR for removing the code i needed to duplicate
elastic#175064

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@neptunian @kibanamachine @fkanout
[Infra] infra services api (elastic#173875)
d06f087 
## Summary
Creation of a new endpoint within Infra to get services from APM indices
that are related to a give host through `host.name`. These services will
be listed in the Host Detail view in another PR. This endpoint queries
apm transaction metrics and apm logs to get services.

Closes elastic#171661

### Test
The easiest way to test this api is to visit it directly using a host
that has some services attached to it using our test cluster

URL: http://localhost:5601/api/infra/services
eg usage:
`http://localhost:5601/api/infra/services?from=now-15m&to=now&filters={"host.name":"gke-edge-oblt-edge-oblt-pool-5fbec7a6-nfy0"}&size=5`

response:

```
{
    "services": [
        {
            "service.name": "productcatalogservice",
            "agent.name": "opentelemetry/go"
        },
        {
            "service.name": "frontend",
            "agent.name": "opentelemetry/nodejs"
        }
    ]
}
```



### Follow up 
- Have APM server collect host.name as part of service_summary metrics
and query that instead. Service summary aggregates transaction, error,
log, and metric events into service-summary metrics. This would simplify
the query.

- `added apm-synthtrace` to `metrics_ui` api tests and created follow up
PR for removing the code i needed to duplicate
elastic#175064

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@neptunian neptunian changed the title [Obs UX] Make apm synthtrace es client a service available to functional and integration tests [Obs UX] Make apm synthtrace kibana client a service available to functional and integration tests Apr 1, 2024
@neptunian neptunian self-assigned this Apr 1, 2024
@neptunian neptunian changed the title [Obs UX] Make apm synthtrace kibana client a service available to functional and integration tests [Infra] Make apm synthtrace kibana client a service available to functional and integration tests Apr 1, 2024
@neptunian neptunian mentioned this issue Apr 1, 2024
Merged
neptunian added a commit that referenced this issue Apr 8, 2024
@neptunian @kibanamachine @weltenwort
[Infra] add apm synthtrace kibana service and cleanup package install (…
069d814 
…#179764)

## Summary

Closes #175064

- Creates a service for ApmSynthtraceKibanaClient to easily access in
tests and other plugins for managing the installation of the APM package
needed for indexing apm documents with synthtrace's elasticsearch client
- Updates the Infra api integration and functional tests to use the
service
- Updates Infra tests to cleanup and uninstall the apm package 
- Updates ApmSynthtraceKibanaClient.installApmPackage to install the
latest version if no version was passed in
- Updates ApmSynthtraceKibanaClient.installApmPackage to return the
version that was installed
- Updates ApmSynthtraceKibanaClient to have an uninstallApmPackage
method

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5599

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@neptunian neptunian

Labels
apm:synthtrace APM Synthtrace Topics chore Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team technical debt Improvement of the software architecture and operational architecture
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Infra] add apm synthtrace kibana service and cleanup package install neptunian/kibana
2 participants
@neptunian @elasticmachine