Fix alert summary prompt

[qianheng-aws]

Description

Fix alert summary prompt, it needs to start prompt with \n\nHuman: for model claude v1

Related Issues

Resolves #955

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.57%. Comparing base (35b171e) to head (9ce16b3).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #958   +/-   ##
=========================================
  Coverage     77.57%   77.57%           
  Complexity      993      993           
=========================================
  Files            99       99           
  Lines          4714     4714           
  Branches        431      431           
=========================================
  Hits           3657     3657           
  Misses          877      877           
  Partials        180      180           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

LGTM (feel free to revert the changelog entry, those are just there as linked from documentation page).

Have you looked at any other templates for similar issue?

[qianheng-aws]

LGTM (feel free to revert the changelog entry, those are just there as linked from documentation page).

Have you looked at any other templates for similar issue?

No. These 2 prompt starts with Human: instead of \n\nHuman:, but still works fine per the authors testing.

flow-framework/sample-templates/anomaly-detector-suggestion-agent-claude.json

Line 77 in 35b171e

"prompt": "Human:\" turn\": Here are some examples of the create anomaly detector API in OpenSearch: Example 1. POST _plugins/_anomaly_detection/detectors, {\"time_field\":\"timestamp\",\"indices\":[\"ecommerce\"],\"feature_attributes\":[{\"feature_name\":\"feature1\",\"aggregation_query\":{\"avg_total_revenue\":{\"avg\":{\"field\":\"total_revenue_usd\"}}}},{\"feature_name\":\"feature2\",\"aggregation_query\":{\"max_total_revenue\":{\"max\":{\"field\":\"total_revenue_usd\"}}}}]}, Example 2. POST _plugins/_anomaly_detection/detectors, {\"time_field\":\"@timestamp\",\"indices\":[\"access_log*\"],\"feature_attributes\":[{\"feature_name\":\"feature1\",\"feature_enabled\":true,\"aggregation_query\":{\"latencyAvg\":{\"sum\":{\"field\":\"responseLatency\"}}}}]} and here are the mapping info containing all the fields in the index ${indexInfo.indexName}: ${indexInfo.indexMapping}, and the optional aggregation methods are value_count, avg, min, max and sum, note that value_count can perform on both numeric and keyword type fields, and other aggregation methods can only perform on numeric type fields. Please give me some suggestion about creating an anomaly detector for the index ${indexInfo.indexName}, you need to give the key information: the top 3 suitable aggregation fields which are numeric types(long, integer, double, float, short etc.) and the suitable aggregation method for each field, you should give at most 3 aggregation fields and corresponding aggregation methods, if there are no numeric type fields, both the aggregation field and method are empty string, and also give at most 1 category field if there exists a keyword type field whose name is just like region, country, city or currency, if not exist, the category field is empty string, note the category field must be keyword type. Show me a format of keyed and pipe-delimited list wrapped in a curly bracket just like {category_field=the category field if exists|aggregation_field=comma-delimited list of all the aggregation field names|aggregation_method=comma-delimited list of all the aggregation methods}. \n\nAssistant:\" turn\""

flow-framework/sample-templates/query-assist-data-summary-agent-claude-tested.json

Line 72 in 35b171e

"prompt": "Human: You are an assistant that helps to summarize the data and provide data insights.\nThe data are queried from OpenSearch index through user's question which was translated into PPL query.\nHere is a sample PPL query: `source=<index> | where <field> = <value>`.\nNow you are given ${parameters.sample_count} sample data out of ${parameters.total_count} total data.\nThe user's question is `${parameters.question}`, the translated PPL query is `${parameters.ppl}` and sample data are:\n```\n${parameters.sample_data}\n```\nCould you help provide a summary of the sample data and provide some useful insights with precise wording and in plain text format, do not use markdown format.\nYou don't need to echo my requirements in response.\n\nAssistant:"

[dbwiddis]

(Re-) approving. Thanks for fixing this!

[yuye-aws]

@qianheng-aws I've tested with your new template, but I still meet with the same error:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Error from remote service: {\"message\":\"prompt must start with \\\"\\n\\nHuman:\\\" turn after an optional system prompt\"}"
      }
    ],
    "type": "status_exception",
    "reason": "Error from remote service: {\"message\":\"prompt must start with \\\"\\n\\nHuman:\\\" turn after an optional system prompt\"}"
  },
  "status": 400
}

Can you please check the code here:

flow-framework/sample-templates/alert-summary-log-pattern-agent.json

Line 29 in b32f98b

"request_body": "{\"prompt\":\"\\n\\nHuman: ${parameters.prompt}\\n\\nAssistant:\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature}, \"anthropic_version\":\"${parameters.anthropic_version}\" }",

[qianheng-aws]

@qianheng-aws I've tested with your new template, but I still meet with the same error:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Error from remote service: {\"message\":\"prompt must start with \\\"\\n\\nHuman:\\\" turn after an optional system prompt\"}"
      }
    ],
    "type": "status_exception",
    "reason": "Error from remote service: {\"message\":\"prompt must start with \\\"\\n\\nHuman:\\\" turn after an optional system prompt\"}"
  },
  "status": 400
}

Can you please check the code here:

flow-framework/sample-templates/alert-summary-log-pattern-agent.json

Line 29 in b32f98b

"request_body": "{\"prompt\":\"\\n\\nHuman: ${parameters.prompt}\\n\\nAssistant:\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature}, \"anthropic_version\":\"${parameters.anthropic_version}\" }",

Refact as your suggestion, can you help verify it again.

[owaiskazi19]

Good to merge once @yuye-aws verifies

[yuye-aws]

Good to merge once @yuye-aws verifies

Verified.

[yuye-aws]

Both the agent and model work. LGTM!