[Dataset Quality] Provide field limit mitigation option with rollover

[achyutjhunjhunwala]

📓 Summary

Add UI to be able to update the Field Limit. Things to consider

• Validations around how high the limit can be set, should the limit be proposed
• Proper messages around, how they can fix this otherwise like reducing the number of fields sent.
• Rollover the Datastream to see the affected changes.

Acceptance Criteria

• Section to take user input to increase Field Limit as per Wireframes
• The Degraded Fields table on the Main page should start referring to last backing index in the DS
• Show user a message to Rollover if they have done this. (P.S. - Its not possible to preserve this state, meaning if the user did update the size and did not do a rollover, comes later, we will not be able to show this message)

🎨 Design

Figma

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[achyutjhunjhunwala]

cc: @gbamparop @LucaWintergerst @flash1293

As discussed during the weekly sync, updating the discussions here

1. The option to apply a new limit will only be available to integrations. Non integration dataset will only have the text for field limit but not provide option to do it.
2. After careful consideration, i propose we suggest Current Limit + 30% to the user for New Field Limit to start with.
3. If the user selects a new limit and clicks on Apply the following will happen

• Update the custom component template associated with the index template of the integration. If Custom component does not exist, same will be created.
• Update the last backing index with the new setting as well.
  • If this part is successful, do nothing
  • if updating the last backing index fails, show the user an option to do Rollover. @patpscal Can you help me here with some mock up for triggering the Rollover. This would basically be a second action.
    @mdbirnstiehl I would need a copy as well for the text here. As this would be considered as partial success. That component template was successfully updated but not the last backing index. Hence a manual rollover is required

Text i used for which i need copy is

4. One of the Mitigations is the link to update the Pipeline for Integrations. This should ideally take them to a page where they can either edit or create a @custom pipeline. This solution currently does not exists and a feature request has been created.

An alternate solution to this would be to display the user the name of the custom pipeline they should create with a copy icon next to it.
There should be a link which would take the users to the current create Pipeline Page.

A text must be presented to them, that they can copy the pipeline name and create a custom pipeline to handle this situation.
@mdbirnstiehl @patpscal Can you both help me here with this point 4 in terms of design and copy.

[mdbirnstiehl]

@mdbirnstiehl I would need a copy as well for the text here. As this would be considered as partial success. That component template was successfully updated but not the last backing index. Hence a manual rollover is required

How about:

Changes not applied to new data

The component template was successfully updated with the new field limit, but the changes were not applied to the most recent backing index.
Perform a rollover to apply your changes to new data.

4. One of the Mitigations is the link to update the Pipeline for Integrations. This should ideally take them to a page where they can either edit or create a @custom pipeline. This solution currently does not exists and a feature request has been created.

An alternate solution to this would be to display the user the name of the custom pipeline they should create with a copy icon next to it. There should be a link which would take the users to the current create Pipeline Page.

Do we have a design for this one? Not sure exactly what it will look like and what text we will need here.

[achyutjhunjhunwala]

@mdbirnstiehl Thanks for the copy.

I would need one more copy from you based on the discussion we had today.

When a user fixes a Field Limit issue, we apply the fix on the current backing index as well. This means that the issue still exists and now when the user open the pop up, they would see our fallback Potential Cause which is called ignore malformed .

Hence in the call today @flash1293 had a great suggestion that we can display a text similar to this when ignore_malformed happened.

@patpscal Can you help me and Mike here with a change in design as well.

Previously we were displaying this

But now for the Pipeline part, we won't be displaying a Link, but as mentioned above, we will display the Pipeline name with a copy icon next to it and there will be some text stating that please go here and update the pipeline

[mdbirnstiehl]

I would need one more copy from you based on the discussion we had today.

When a user fixes a Field Limit issue, we apply the fix on the current backing index as well. This means that the issue still exists and now when the user open the pop up, they would see our fallback Potential Cause which is called ignore malformed .

@achyutjhunjhunwala Will this only apply for field limit changes? Can we be specific or do we want to avoid mentioning the specific settings?

How about something like this:

If you've recently updated your field limit settings, this quality issue may not be relevant. Rollover the data stream to verify.

[achyutjhunjhunwala]

This makes more sense

[achyutjhunjhunwala]

@mdbirnstiehl As Patri is on PTO, i came with a rough design to explain what i meant

For Non Integrations we would recommend them a very generic custom pipeline based on datastreamType@custom

For Integrations, we would recommend them with explicit custom pipeline name which is datastreamType-datastreamDataset@custom

cc: @flash1293 @LucaWintergerst

[mdbirnstiehl]

@achyutjhunjhunwala would the user be taken to the page to create a pipeline when they copy the name or would there be another link for them to follow?

In this scenario, would users always be creating the custom pipeline? If they are always creating the pipeline, it would make sense to make the title "Add custom ingest pipeline" and remove the edit.

Do we want to give a little summary of what/why the user would add a custom pipeline? I'm not sure that would be obvious to the user.

We could possibly organize it like this:

Add a custom ingest pipeline to...(I'm not sure of the mechanics of it, but we could give a reason here as to why they need to create the pipeline):

1. Copy the following pipeline name:
   logs@custom
2. [Create a custom pipeline](link to create pipeline page) using the name you copied.

[achyutjhunjhunwala]

@mdbirnstiehl
When they click on that text with copy icon, at the moment i am only copying the pipeline name.
The Title is a link to the Pipeline page, where they can either search for that pipeline if it exists and edit.
If not, they can create it. We are only aiding them with the name of the pipeline here as it is important to have this particular naming convention.

Why the user need to modify a pipeline ?

At the moment we are only offering a fix it flow for Field Limit Issues that too only for integrations. Other issues like character limit reached or malformed values still need to be fixed manually by the user. Editing/Adding Pipeline processor is also one of the way

[achyutjhunjhunwala]

@patpscal As you are back, can you help me with the UX for this Pipeline panel. I just randomly built this post @LucaWintergerst comment.

[LucaWintergerst]

How hard would it be for us to check if the pipeline already exists?

GET _ingest/pipeline/<name>

if it exists, we display

[link icon] Edit ingest pipeline

if it does not exist, we do

[link icon] Create ingest pipeline
---
1. Copy the following pipeline name:
    [copy icon] logs@custom
2. [link icon] [Create a custom pipeline](link to create pipeline page) using the name you copied.

this would make for a cleaner UX as we'd have a nicer solution in the case that it's there already

if we can't easily check for the presence of the pipeline I'd suggest

[link icon] Create or edit ingest pipeline
---
1. Copy the following pipeline name:
    [copy icon] logs@custom
2. [link icon] [Create or edit the pipeline](link to create pipeline page) using the name you copied.

[achyutjhunjhunwala]

@LucaWintergerst In favour of this issue - #183992
I would propose i got by the last option you provided where we don't check and display the copy and navigate option.

Reason being once the above ticket is implemented, we can always point to that URL and Management will automatically will take care of opening Edit or Create page.

[LucaWintergerst]

Ok, that makes sense to me

[patpscal]

It seems the original design did not account for the intermediate "copy" step, so it links directly to the create pipeline page, hence the external link icon. We can disregard the link for this card title and use an accordion instead for the content, as in the Increase field mapping limit card.

A variant to highlight the action/link:

Figma link