[CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection failure #37356

iverase · 2019-01-11T11:37:26Z

I noticed some failures in the test above. For example:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+multijob-unix-compatibility/os=fedora/46/console

It can be reproduced without adding the method name:

./gradlew :server:unitTest
-Dtests.seed=506AA3004455CABB
-Dtests.class=org.elasticsearch.search.geo.GeoShapeQueryTests
-Dtests.method="testQueryRandomGeoCollection"
-Dtests.security.manager=true
-Dtests.locale=it
-Dtests.timezone=Asia/Vladivostok
-Dcompiler.java=11
-Druntime.java=8

I had a look into it and I could reproduce the issue at Lucene level.

The problem is that we are generating degenerated polygons in one dimension, for example:

polygon ((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 1.401298464324817E-45 22.0))

Then when indexing this polygon and perform an intersect query using the same one, they do not match. This is probably due to the encoding Lucene is using.

elasticmachine · 2019-01-11T11:44:39Z

Pinging @elastic/es-analytics-geo

iverase · 2019-01-11T12:09:33Z

I have opened https://issues.apache.org/jira/browse/LUCENE-8634

jkakavas · 2019-01-17T08:50:36Z

I muted the test as it failed again today on
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/185/console

imotov · 2019-01-17T19:45:44Z

@jkakavas did you mute it only in 6.x? I think it should be failing in all branches, I opened #37579 that should ignore the failing part of the test if a tiny polygon is generated,

imotov · 2019-01-23T19:54:40Z

I unmuted the test and added assumeTrue to ignore tests with random polygons that are causing issues because of LUCENE-8634. The changes are merged into 6.6, 6.x and 7.x.

nknize · 2019-01-25T17:42:45Z

I've added some feedback on this issue at https://issues.apache.org/jira/browse/LUCENE-8634

tldr; subcentimeter polygons are not supported w/ the encoding we use in the WGS84 projection. Polygons this small should be reserved for an alternate projection (e.g., mercator, UTM). This will eventually be supported with a new XYShape / geometry field type (probably ES 8.0).

talevy · 2019-02-20T22:28:00Z

related issue popped up today

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+release-tests/209/console

./gradlew :server:unitTest \
  -Dtests.seed=4E97A47EAD7E2AC3 \
  -Dtests.class=org.elasticsearch.search.geo.GeoShapeQueryTests \
  -Dtests.method="testRandomGeoCollectionQuery" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=es-HN \
  -Dtests.timezone=SystemV/HST10 \
  -Dcompiler.java=11 \
  -Druntime.java=8

ERROR   0.08s J6 | GeoShapeQueryTests.testRandomGeoCollectionQuery <<< FAILURES!
   > Throwable #1: MapperParsingException[failed to parse field [location] of type [geo_shape]]; nested: InvalidShapeException[Cannot determine orientation: edges adjacent to (-6.7745978568024175E-133,-2.8287277918521384E-246) coincide];
   > 	at __randomizedtesting.SeedInfo.seed([4E97A47EAD7E2AC3:26EAAADDFB5DF54B]:0)
   > 	at org.elasticsearch.index.mapper.GeoShapeFieldMapper.parse(GeoShapeFieldMapper.java:116)
   > 	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488)
   > 	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505)
   > 	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395)
   > 	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
   > 	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96)
   > 	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69)
   > 	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281)
   > 	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:763)
   > 	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:739)
   > 	at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:718)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:463)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:486)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:461)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
   > 	at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1051)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1029)
   > 	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:425)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:371)
   > 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
   > 	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
   > 	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
   > 	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2352)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:988)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:370)
   > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:325)
   > 	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:312)
   > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
   > 	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:687)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759)
   > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: org.locationtech.spatial4j.exception.InvalidShapeException: Cannot determine orientation: edges adjacent to (-6.7745978568024175E-133,-2.8287277918521384E-246) coincide
   > 	at org.elasticsearch.common.geo.builders.PolygonBuilder.getOrientation(PolygonBuilder.java:763)
   > 	at org.elasticsearch.common.geo.builders.PolygonBuilder.ring(PolygonBuilder.java:716)
   > 	at org.elasticsearch.common.geo.builders.PolygonBuilder.createEdges(PolygonBuilder.java:698)
   > 	at org.elasticsearch.common.geo.builders.PolygonBuilder.coordinates(PolygonBuilder.java:215)
   > 	at org.elasticsearch.common.geo.builders.PolygonBuilder.buildLucene(PolygonBuilder.java:238)
   > 	at org.elasticsearch.common.geo.builders.GeometryCollectionBuilder.buildLucene(GeometryCollectionBuilder.java:193)
   > 	at org.elasticsearch.index.mapper.GeoShapeFieldMapper.parse(GeoShapeFieldMapper.java:110)
   > 	... 39 more

imotov · 2019-02-21T15:03:53Z

That looks like a different issue. The original issue was caused by lucene indexing and this issue is caused by parsing. I will take a look.

Moves the check for tiny polygons earlier in the test. It turned out that polygons can be so tiny that we cannot even figure out their orientation. Relates to elastic#37356

imotov · 2019-02-21T22:06:05Z

So. It turned out it is a different issue, but with the same cause. GeoTestUtil.nextPolygon() can generate such a small polygon that it causes arithmetic underflow when we try to calculate orientation. In this case the polygon that is causing the issue is this:

{
  "type" : "polygon",
  "orientation" : "right",
  "coordinates" : [
    [
      [
        -6.7745978568024175E-133,
        -2.8287277918521384E-246
      ],
      [
        -6.7745978568024175E-133,
        0.0
      ],
      [
        -3.3164115140693285E-230,
        0.0
      ],
      [
        -6.7745978568024175E-133,
        -2.8287277918521384E-246
      ]
    ]
  ]
}

I opened #39275 to ignore the test if such a small polygon is generated for now.

However, I am wondering if we should change GeoTestUtil.nextPolygon() to generate more practical polygons. While precision is important, in this particular case we are talking about a polygon smaller than 10^-127m if my math is correct. I cannot really think for any practical applications for that besides implementing geo search for small objects in the Arquillian Galaxy.

Moves the check for tiny polygons earlier in the test. It turned out that polygons can be so tiny that we cannot even figure out their orientation. Relates to #37356

imotov · 2019-04-06T17:59:21Z

I cannot find any failures since the last fix was merged. Closing.

iverase changed the title ~~[CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection fails occas~~ [CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection failure Jan 11, 2019

iverase added >bug :Analytics/Geo Indexing, search aggregations of geo points and shapes v7.0.0 labels Jan 11, 2019

iverase added the v6.7.0 label Jan 11, 2019

jasontedor added v8.0.0 and removed v7.0.0 labels Feb 6, 2019

danielmitterdorfer added v7.2.0 and removed v6.7.0 labels Feb 7, 2019

imotov self-assigned this Feb 21, 2019

imotov mentioned this issue Feb 21, 2019

[CI] Fixes testQueryRandomGeoCollection failure again #39275

Merged

imotov closed this as completed Apr 6, 2019

gwbrown mentioned this issue Jul 26, 2019

GeoShapeQueryTests.testRandomGeoCollectionQuery failures on CI #44915

Closed

iverase mentioned this issue Aug 19, 2019

ShapeQueryTests#testFieldAlias returns no hits, fails assertion #45628

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection failure #37356

[CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection failure #37356

iverase commented Jan 11, 2019 •

edited

Loading

elasticmachine commented Jan 11, 2019

iverase commented Jan 11, 2019

jkakavas commented Jan 17, 2019

imotov commented Jan 17, 2019

imotov commented Jan 23, 2019

nknize commented Jan 25, 2019

talevy commented Feb 20, 2019

imotov commented Feb 21, 2019

imotov commented Feb 21, 2019

imotov commented Apr 6, 2019

[CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection failure #37356

[CI] org.elasticsearch.search.geo.GeoShapeQueryTests. testQueryRandomGeoCollection failure #37356

Comments

iverase commented Jan 11, 2019 • edited Loading

elasticmachine commented Jan 11, 2019

iverase commented Jan 11, 2019

jkakavas commented Jan 17, 2019

imotov commented Jan 17, 2019

imotov commented Jan 23, 2019

nknize commented Jan 25, 2019

talevy commented Feb 20, 2019

imotov commented Feb 21, 2019

imotov commented Feb 21, 2019

imotov commented Apr 6, 2019

iverase commented Jan 11, 2019 •

edited

Loading