Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Switch mapping/aggregations over to java time #36363

Merged
merged 202 commits into from
Jan 23, 2019
Merged

Core: Switch mapping/aggregations over to java time #36363

merged 202 commits into from
Jan 23, 2019

Conversation

spinscale
Copy link
Contributor

This is the PR of the java-time feature branch, so review can start. The cut-over to java time is a requirement so that we can support nanoseconds properly. Most of the code is replacing the joda time classes with the java time ones - however this cannot be done gradually, thus this one big feature-branch behind this PR.

This mainly converts the field mapper classes plus aggregations and their tests, but also some code in x-pack that relied on joda time.

Note: This cannot be merged yet because we want to push a BWC layer in master/6.x first and then merge this one. I will keep this branch up-to-date until this happens.

Relates #27330

This commit moves the aggregation and mapping code from joda time to
java time.

This includes field mappers, root object mappers, aggregations with date
histograms, query builders and a lot of changes within tests.
Copy link
Contributor

@polyfractal polyfractal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@spinscale
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/2

@spinscale spinscale merged commit daa2ec8 into master Jan 23, 2019
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jan 23, 2019
…ead-de-duplication

* elastic/master: (24 commits)
  [TEST] Mute MlMappingsUpgradeIT testMappingsUpgrade
  Streamline skip_unavailable handling (elastic#37672)
  Only bootstrap and elect node in current voting configuration (elastic#37712)
  Ensure either success or failure path for SearchOperationListener is called (elastic#37467)
  Target only specific index in update settings test
  Add a note how to benchmark Elasticsearch
  Don't use Groovy's `withDefault` (elastic#37726)
  Adapt SyncedFlushService (elastic#37691)
  Mute FilterAggregatorTests#testRandom
  Switch mapping/aggregations over to java time (elastic#36363)
  [ML] Update ML results mappings on process start (elastic#37706)
  Modify removal_of_types.asciidoc (elastic#37648)
  Fix edge case in PutMappingRequestTests (elastic#37665)
  Use new bulk API endpoint in the docs (elastic#37698)
  Expose sequence number and primary terms in search responses (elastic#37639)
  Remove LicenseServiceClusterNotRecoveredTests (elastic#37528)
  Migrate SpecificMasterNodesIT to Zen2 (elastic#37532)
  Fix MetaStateFormat tests
  Use plain text instead of latexmath
  Fix a typo in a warning message in TestFixturesPlugin (elastic#37631)
  ...
@jsoriano
Copy link
Member

jsoriano commented Jan 25, 2019

Hi @spinscale, there is a test in filebeat failing with ES snapshot and a bisect points to this change.

The failure seen is like this:

{"type":"mapper_parsing_exception","reason":"failed to parse field [suricata.eve.flow.start] of type [date]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2018-10-03T14:42:44.613469+0000] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2018-10-03T14:42:44.613469+0000' could not be parsed, unparsed text found at index 26"}}

Find the ES trace here.

It seems unable to parse 2018-10-03T14:42:44.613469+0000, is it expected?

The test validates this ingest pipeline with these log files.

jsoriano added a commit to jsoriano/beats that referenced this pull request Jan 25, 2019
These dates fail to parse after recent changes in Elasticsearch for date
formatting, see elastic/elasticsearch#36363

Information is still stored in parsed ECS fields.
jsoriano added a commit to elastic/beats that referenced this pull request Jan 25, 2019
These dates fail to parse after recent changes in Elasticsearch for date
formatting, see elastic/elasticsearch#36363

Information is still stored in parsed ECS fields.
@colings86 colings86 deleted the java-time branch May 27, 2020 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Core Core issues without another label >refactoring v7.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.