[APM/Design] Code integration design proposal

[formgeist]

Summary

Continuing from #101, @daveyholler and I have put together a design proposal that is meant to give some context to how we imagine APM UI would integrate with Code in order to first and foremost be able to serve source code references to stack traces.

Why

While APM agents today have the option to send up stack traces for spans and exceptions, it's at a cost of the payload and at the end of the day more expensive for the customer. Now with the Code Kibana app launched, we're excited by the potential replacement solution. Being able to have a seamless link between the executed code on the client-side and the source code is invaluable information for debugging performance problems.

An added benefit is that we'll also be able to drive further adoption of Code by enabling our customers to integrate.

How

The initial integration is currently only related to the APM UI. We need to be able to look up source code references via the Code API and be able to map source code repositories to services. If a mapping doesn't exist, we'll enable the user to add the mapping manually either by mapping it to an existing Code repository or a new one, imported by URL.

TODO: We need to investigate the Code API for source code lookups, mapping or adding repositories.

Design proposal

Links

👉 Marvel prototype

Code integration in Span detail

• Show an embedded Code element that displays the source code for the stack trace reference by filename and line number.
• The ability to go to the source code repository in Code either by the project name link or the Code icon in the top of the embedded Code element.
• Technically, we imagine this is a mirror Code element that is embedded into APM UI with its abilities like function reference lookups, syntax highlighting.
  • NOTE: Still to be discussed how this Code embed is shared between APM UI and Code UI teams.

👉 Direct link to screens

Code integration not setup in Span detail

• If there's no current mapping for a source code repository, we'll nudge the user to set it up.
• By displaying a "View in Code" button enables the user to instantly map or import a repository to the Service.
• This takes the user through a mini-setup flow where they can pick from a list of already imported Code repositories, or import a new.
• We'll show a status badge for the Service that the repository is indexing.

👉 Direct link to screens

Manage source code repository for Service

• We don't only allow for mapping or importing inside the Span detail view, but also add an option to our existing Integrations menu which also contains Machine Learning and Watcher integrations.
• This enables the user to manage their mapped repository; map, import or remove the repository.

👉 Direct link to screens

APM Setup instructions Code integration

• If we nudge the user early enough in their process, like the Setup Instructions page for APM, they will have the mapping setup early on and once their data comes in, it'll be available with source code references.
• Enables the user to map to an existing repository or import a new one to Code.

👉 Direct link to screens

Open questions

• Code embed: How would we share a component like the Code embed between Code and APM UI?
• Background tasks: Wondering if there's a way that we can notify the user with a toast once the repo has been cloned and indexed, letting them know the process is complete. That would require some running background task to check for the status of the mapped or imported repository via Code API and create a toast in APM UI.
• Language servers: If a user imports a repository that demands a specific language server that needs installation or simply isn't supported, do we tell them upfront or will we still be able to serve the source code references just without the rich function definitions?

---

@elastic/apm-ui @roncohen @tbragin @nehaduggal @chongzhe Thoughts on this proposal? I imagine we will want to have a sync between the APM and Code teams so we can figure out an ideal way to proceed.

It's possible that many of the setup interactions can be scoped to a later implementation, so ideally we'd investigate the embedded component first by creating a POC of the UI element and it querying the Code API for source code references.

[mikker]

Glorious!

[dgieselaar]

Oh man, this stuff is amazing. Should the user be able to deeplink to a file? The flyout is going to make that a little more difficult. Back & forth navigation using the URL history is bound to be hard as well, not sure if we can pull that off with a flyout.

[formgeist]

@dgieselaar I understand the concern of adding another param to the URL for sharing. Instead I imagine a simpler solution would be to add a "Share" option on that file preview which enables the user to copy a link to the file in Code UI, which I suppose is also a more preferred way to share source code files instead of inside the APM UI.

[spacedragon]

Some early thoughts on how to embed code in APM:

1. Contribute to EUI, make the Monaco editor as a EUI component, so APM and other products can use it as an editor.
2. The code team can maintain a front-end library that connects the Monaco editor to the back end of the code, basically providing:
   1. Load the files in the code backend API into the Monaco editor
   2. Configure code intelligence in the Monaco editor
   3. The state of the repository and language server
3. APM team can use this library and Monaco in the APM UI

I haven’t discussed these with @elastic/code. It’s just a preliminary proposal.

[formgeist]

@spacedragon I think that's a very good proposal - wondering how feasible this is within the short-term timeline, but something we can discuss in our upcoming kick off. Thanks for your input!

[sorenlouv]

FYI: The changes suggested in elastic/kibana#40375 (comment) might simplify the Code Search integration.

[mw-ding]

FYI: The changes suggested in elastic/kibana#40375 (comment) might simplify the Code Search integration.

That looks pretty good. We had internal team sync this week chatting about the design as well. Will send out a cross-team sync this week or early next week to sync on some progresses

[formgeist]

Design update

I've been updating the user experience flows for when a user wants to map a repository to a service from deep inside the context of a service and transactions. There was also a need for an overview page that would display all the mappings between services and repositories, which is now added to the Settings page within APM. The user is able to manage the mappings and set up new ones from one central view.

➡️ Marvel prototype

Updated flyouts with new Code component

Since we landed on a redesign of the stack strace experience in APM, I've updated the designs in the Code integration to reflect the new design.

Mapping a repository to a service within APM views

Instead of adding an option to map for each stackframe (Codeblock) component, we opted to design a banner at the top of the content to entice users to add the mapping. They'll be able to dismiss the message, which is saved to local storage for each Kibana user separately.

Direct link to prototype

List of mappings within Settings

As mentioned previously, we've designed a list page for all mappings within APM so the user can get a full overview and be able to manage mappings from one place.

Direct link to prototype

cc @nehaduggal @roncohen @chongzhe @mw-ding @sqren

[dgieselaar]

This looks great Casper! Can the user map multiple repositories for a service?

[formgeist]

Can the user map multiple repositories for a service?

Yes, although it's not exactly demoed in the above prototypes. One enhancement might be to be able to add multiple "new" repositories in that modal at a time. I'll give it some thought