Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java API: IndicesOptions.lenientExpandOpen() required index names to contain "*" #23557

Closed
jjbursik opened this issue Mar 12, 2017 · 1 comment

Comments

@jjbursik
Copy link

jjbursik commented Mar 12, 2017

Elasticsearch version: 5.1.1

Description of the problem including expected versus actual behavior:
Using aliases to manage indices used for search, I'm unable to ignore closed indices without specifying a wildcard search char ("*") in the index name. Here are my sample SearchRequestBuilders that works and doesn't work:

Works:

SearchRequestBuilder srb=elasticsearch.getClient().prepareSearch("reportsource*")
                .setIndicesOptions(IndicesOptions.lenientExpandOpen())
                .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                .setQuery(functionScoreQueryBuilder)
                .addSort("_score", SortOrder.DESC)
                .setFrom(page * size).setSize(size);

SearchResponse searchResponse = srb.execute().actionGet();

Doesn't work:

SearchRequestBuilder srb=elasticsearch.getClient().prepareSearch("reportsource")
                .setIndicesOptions(IndicesOptions.lenientExpandOpen())
                .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                .setQuery(functionScoreQueryBuilder)
                .addSort("_score", SortOrder.DESC)
                .setFrom(page * size).setSize(size);

SearchResponse searchResponse = srb.execute().actionGet();

I would have expected that I don't need the "*" in the index name to ignore closed indices having specified IndicesOptions.lenientExpandOpen(). Using Sense, I don't see the same requirement:

GET reportsource/_search?ignore_unavailable=true,allow_no_indices=true,expand_wildcards_open=true,expand_wildcards_closed=false,allow_alisases_to_multiple_indices=true,forbid_closed_indices=false&pretty&search_type=dfs_query_then_fetch
{
  "from" : 0,
  "size" : 10,
  "query" : {
    "function_score" : {
      "query" : {
        "bool" : {
          "should" : [
            {
              "match" : {
                "name.raw" : {
                  "query" : "Sub Agreement Id",
                  "operator" : "OR",
                  "prefix_length" : 0,
                  "max_expansions" : 50,
                  "fuzzy_transpositions" : true,
                  "lenient" : false,
                  "zero_terms_query" : "NONE",
                  "boost" : 1.0
                }
              }
            }
          ],
          "disable_coord" : false,
          "adjust_pure_negative" : true,
          "minimum_should_match" : "2",
          "boost" : 1.0
        }
      },
      "functions" : [
        {
          "filter" : {
            "match_all" : {
              "boost" : 1.0
            }
          },
          "field_value_factor" : {
            "field" : "rating",
            "factor" : 1.0,
            "missing" : -0.3499999940395355,
            "modifier" : "log2p"
          }
        }
      ],
      "score_mode" : "multiply",
      "boost_mode" : "multiply",
      "max_boost" : 2.0,
      "boost" : 1.0
    }
  },
  "ext" : { }
}

Steps to reproduce:

  1. Create 2 indexes, both with aliases "reportsource"
  2. Close 1 index
  3. Use the SearchRequestBuilder code above, you can remove the query if needed, and you'll either get an error or not. Error for index closed pasted below.

Provide logs (if relevant):
Error log:

2017-03-12 16:05:44.608 ERROR 72028 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is ClusterBlockException[blocked by: [FORBIDDEN/4/index closed];]] with root cause

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/4/index closed];
	at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:178) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.search.TransportSearchAction.buildPerIndexAliasFilter(TransportSearchAction.java:80) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1343) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1242) ~[elasticsearch-5.1.1.jar:5.1.1]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-client-5.1.1.jar:5.1.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) ~[netty-common-4.1.6.Final.jar:4.1.6.Final]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
@javanna
Copy link
Member

javanna commented Mar 21, 2017

Hi @jjbursik ,
the indices options api can definitely be improved, especially how it is exposed in our Java API.

That said, your REST call doesn't doesn't provide any indices options after all. The right separator between the different arguments is & and not ,. You end up passing in a huge string (ignore_unavailable=true,allow_no_indices=true,expand_wildcards_open=true,expand_wildcards_closed=false,allow_alisases_to_multiple_indices=true,forbid_closed_indices=false) for the ignore_unavailable option, which gets ignored as it is not a boolean value (this will cause an error in 6.0, see #22200).

The only valid arguments in your REST call are pretty&search_type=dfs_query_then_fetch. That has the same effect as not passing in any indices options, hence using the default indices options for SearchRequest : IndicesOptions#strictExpandOpenAndForbidClosed. The forbid closed is the important part here, which is an internal flag that defines whether closed indices are to be treated as unavailable or not (varies api by api). This bit is implicitly set to false in IndicesOptions#lenientExpandOpen which causes the error you are seeing. You could create lenient indices options with forbid closed set to true this way: IndicesOption.fromOptions(true, true, true, false, IndicesOption.strictExpandOpenAndForbidClosed).

Your REST call though should return an error when performed against a closed index, or an alias that points to at least one closed index, something like this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_closed_exception",
        "reason" : "closed",
        "index_uuid" : "hcUaFRjURxO6Yu6RvqY63g",
        "index" : "test2"
      }
    ],
    "type" : "index_closed_exception",
    "reason" : "closed",
    "index_uuid" : "hcUaFRjURxO6Yu6RvqY63g",
    "index" : "test2"
  },
  "status" : 403
}

Also, when you say that you didn't expect to have to use the wildcard in your index name, that is correct. Do keep in mind though that one thing is expanding wildcard expressions, controlled via expand_wildcards option, and another thing is ignoring unavailable indices when provided explicitly, controlled via ignore_unavailable.

Conclusion:

  1. check your REST call, it should return an error as a closed index is being hit and indices options are not lenient.
  2. Provide ignore_unavailable=true to the REST call with explicit index names to make it lenient, meaning that closed indices will not be hit.
  3. Update your Java api code to do the same by initializing indices options this way: IndicesOption.fromOptions(true, true, true, false, IndicesOption.strictExpandOpenAndForbidClosed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants