Search stats for coordinator node misses total search request time

[jainankitk]

Currently, the coordinator node only publishes the phase breakdown metrics. It should also publish the total search request time as part of node stats on the coordinator node

Related #8381, #8386

[jainankitk]

@dzane17 - Are you able to take stab at this one?

[msfroh]

Is this likely to be different from the took time? If so, how?

[msfroh]

Is this likely to be different from the took time? If so, how?

Oh... This is for the stats as opposed to the debugging info reported in SearchResponse?

[jainankitk]

Oh... This is for the stats as opposed to the debugging info reported in SearchResponse?

Yeah, I missed that in the initial PR!

[dzane17]

Yes, I will take care of it

[jainankitk]

@dzane17 - Have we already addressed this issue?

[sandeshkr419]

@dzane17 Are you still working on this?

[dzane17]

No, the team has decided to deprioritize this for now. I will unassign myself.

[jainankitk]

As per offline discussion, @dzane17 should be able to prioritize this for 2.17. Reassigning

[dzane17]

@jainankitk Can you confirm what the final output should look like. I have added the took section below as a proposal. I think it should be within the "request" object since this is where we keep all coordinator-level metrics. But the name should distinguish it from other metrics within "request" which are per phase.

...
      "indices" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 4,
          "query_time_in_millis" : 22,
          "query_current" : 0,
          "concurrent_query_total" : 0,
          "concurrent_query_time_in_millis" : 0,
          "concurrent_query_current" : 0,
          "concurrent_avg_slice_count" : 0.0,
          "fetch_total" : 4,
          "fetch_time_in_millis" : 3,
          "fetch_current" : 0,
          "scroll_total" : 0,
          "scroll_time_in_millis" : 0,
          "scroll_current" : 0,
          "point_in_time_total" : 0,
          "point_in_time_time_in_millis" : 0,
          "point_in_time_current" : 0,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0,
          "search_idle_reactivate_count_total" : 0,
          "request" : {
            "took" : {
              "time_in_millis" : 71,
              "current" : 0,
              "total" : 2
            },
            "dfs_pre_query" : {
              "time_in_millis" : 0,
              "current" : 0,
              "total" : 0
            },
            "query" : {
              "time_in_millis" : 68,
              "current" : 0,
              "total" : 2
            },
            "fetch" : {
              "time_in_millis" : 3,
              "current" : 0,
              "total" : 2
            },
            "dfs_query" : {
              "time_in_millis" : 0,
              "current" : 0,
              "total" : 0
            },
            "expand" : {
              "time_in_millis" : 0,
              "current" : 0,
              "total" : 2
            },
            "can_match" : {
              "time_in_millis" : 0,
              "current" : 0,
              "total" : 0
            }
          }
        }

[jainankitk]

took sounds correct to me since it correlates directly with the client search request took time