Failing test: Observability AI Assistant API Integration tests (enterprise).x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base_user_instructions·spec·ts - Observability AI Assistant API tests knowledge_base/knowledge_base_user_instructions.spec.ts Knowledge base user instructions when creating private and public user instructions "editor" can retrieve their own private instructions and the public instruction

[kibanamachine]

A test failed on a tracked branch

Error: expected [ { doc_id: 'private-doc-from-editor',
    text: 'Private user instruction from "editor"',
    public: false },
  { doc_id: 'public-doc-from-editor',
    text: 'Public user instruction from "john"',
    public: true } ] to sort of equal [ { doc_id: 'private-doc-from-editor',
    public: false,
    text: 'Private user instruction from "editor"' },
  { doc_id: 'public-doc-from-editor',
    public: true,
    text: 'Public user instruction from "editor"' },
  { doc_id: 'public-doc-from-john',
    public: true,
    text: 'Public user instruction from "john"' } ]
    at Assertion.assert (expect.js:100:11)
    at Assertion.eql (expect.js:244:8)
    at Context.<anonymous> (knowledge_base_user_instructions.spec.ts:99:46)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  actual: '[\n' +
    '  {\n' +
    '    "doc_id": "private-doc-from-editor"\n' +
    '    "public": false\n' +
    '    "text": "Private user instruction from \\"editor\\""\n' +
    '  }\n' +
    '  {\n' +
    '    "doc_id": "public-doc-from-editor"\n' +
    '    "public": true\n' +
    '    "text": "Public user instruction from \\"john\\""\n' +
    '  }\n' +
    ']',
  expected: '[\n' +
    '  {\n' +
    '    "doc_id": "private-doc-from-editor"\n' +
    '    "public": false\n' +
    '    "text": "Private user instruction from \\"editor\\""\n' +
    '  }\n' +
    '  {\n' +
    '    "doc_id": "public-doc-from-editor"\n' +
    '    "public": true\n' +
    '    "text": "Public user instruction from \\"editor\\""\n' +
    '  }\n' +
    '  {\n' +
    '    "doc_id": "public-doc-from-john"\n' +
    '    "public": true\n' +
    '    "text": "Public user instruction from \\"john\\""\n' +
    '  }\n' +
    ']',
  showDiff: true
}

First failure: kibana-on-merge - main

[kibanamachine]

New failure: kibana-on-merge - 8.x

[kibanamachine]

New failure: kibana-on-merge - main

[kibanamachine]

New failure: kibana-on-merge - main

[mistic]

Skipped.

main: 550015b

[kibanamachine]

New failure: kibana-on-merge - 8.x

[viduni94]

I've looked into this issue before, and haven't been able to reproduce this test failure/flakiness.
All tests succeeded locally and via the flaky test runner:

Via local tests:

Via flaky test runner:

• [Obs AI Assistant] Attempt to resolve flaky knowledge based user instructions test #196026 (comment)
• [Obs AI Assistant] Attempt to resolve flaky knowledge based user instructions test #196026 (comment)

Hey @sorenlouv
Do you have an idea as to why this test would fail sometimes in the kibana-ci?

[sorenlouv]

It's very weird to me how stable the flaky test runner is, and then it still fails the week after. I suggest adding the retry to the failing test around the request GET /internal/observability_ai_assistant/kb/user_instructions and ensure it returns 3 instructions.

[viduni94]

It's very weird to me how stable the flaky test runner is, and then it still fails the week after. I suggest adding the retry to the failing test around the request GET /internal/observability_ai_assistant/kb/user_instructions and ensure it returns 3 instructions.

Thanks @sorenlouv
I've created a PR - #200022
Hope you could take a look.
Thank you

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x