CCS controlling node returns a 500 when the data nodes return a 404

[chrisronline]

I have a CCS environment setup, using the guide here. It contains two data nodes and one controlling node. Both data nodes contain the same index, test, with a single document in each. One data node is running off :9201, the other off :9202. The controlling node is running off :9220.

This request is good:

curl -XPOST localhost:9201/t/_search -i
HTTP/1.1 404 Not Found

However, this request is unexpected:

curl -XPOST localhost:9220/*:t/_search -i
HTTP/1.1 500 Internal Server Error

(curl -XPOST localhost:9220/cluster_one:t/_search -i and curl -XPOST localhost:9220/cluster_two:t/_search -i return the same 500 too.)

I'm assuming the connecting node is detecting the 404 from the data nodes and is converting that into a 500, but I'm not sure it should do that. The connecting node should probably inspect the type of error and return the same error type instead of the 500.

[javanna]

@chrisronline can you please post the complete error that you are getting back?

[chrisronline]

@javanna Sure, should have included that in the issue.

{
  "error": {
    "root_cause": [{
      "type": "index_not_found_exception",
      "reason": "no such index",
      "index_uuid": "_na_",
      "resource.type": "index_or_alias",
      "resource.id": "t",
      "index": "t"
    }],
    "type": "transport_exception",
    "reason": "unable to communicate with remote cluster [cluster_two]",
    "caused_by": {
      "type": "index_not_found_exception",
      "reason": "no such index",
      "index_uuid": "_na_",
      "resource.type": "index_or_alias",
      "resource.id": "t",
      "index": "t"
    }
  },
  "status": 500
}

[s1monw]

I removed the discuss lable @javanna can you follow up on this?!

[andyb-elastic]

This still occurs on master

# ccs-cluster0-node0/config/elasticsearch.yml
cluster.name: cluster0
network.host: '127.0.0.1'
http.port: 9200
transport.tcp.port: 9300
search.remote.cluster1.seeds: '127.0.0.1:8300'

# ccs-cluster1-node0/config/elasticsearch.yml
cluster.name: cluster1
network.host: '127.0.0.1'
http.port: 8200
transport.tcp.port: 8300

Local search on the "remote" cluster

[andy@andy-macbook elasticsearch ]$ curl -i localhost:8200/noexist/_search?pretty
HTTP/1.1 404 Not Found
content-type: application/json; charset=UTF-8
content-length: 500

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_or_alias",
        "resource.id" : "noexist",
        "index_uuid" : "_na_",
        "index" : "noexist"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "resource.type" : "index_or_alias",
    "resource.id" : "noexist",
    "index_uuid" : "_na_",
    "index" : "noexist"
  },
  "status" : 404
}

Search for the same index on the remote cluster

[andy@andy-macbook elasticsearch ]$ curl -i localhost:9200/cluster1:noexist/_search?pretty
HTTP/1.1 500 Internal Server Error
content-type: application/json; charset=UTF-8
content-length: 645

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : "noexist",
        "index" : "noexist"
      }
    ],
    "type" : "transport_exception",
    "reason" : "unable to communicate with remote cluster [cluster1]",
    "caused_by" : {
      "type" : "index_not_found_exception",
      "reason" : "no such index",
      "index_uuid" : "_na_",
      "resource.type" : "index_or_alias",
      "resource.id" : "noexist",
      "index" : "noexist"
    }
  },
  "status" : 500
}

@elastic/es-search-aggs