Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

strftime fails to zero pad year < 1000 on AWS Lambda #3310

Closed
dsotirho-ucsc opened this issue Aug 9, 2021 · 4 comments
Closed

strftime fails to zero pad year < 1000 on AWS Lambda #3310

dsotirho-ucsc opened this issue Aug 9, 2021 · 4 comments
Assignees
Labels
bug [type] A defect preventing use of the system as specified code [subject] Production code no demo [process] Not to be demonstrated at the end of the sprint orange [process] Done by the Azul team spike:2 [process] Spike estimate of two points

Comments

@dsotirho-ucsc
Copy link
Contributor

Issue #2562 added inner entity fields submission_date and update_date as datetime objects, and a null_datetime value to serve as a valid datetime value that can represent None/null while indexed in Elasticsearch.

Originally the value datetime(1, 1, 1, tzinfo=timezone.utc) (converted to the string 0001-01-01T00:00:00.000000Z) was to be used to represent null, however when indexed in sandbox a mapper_parsing_exception occurred.

error.caused_by.reason  Invalid format: "1-01-01T00:00:00.000000Z" is malformed at "-01-01T00:00:00.000000Z"
error.caused_by.type    illegal_argument_exception
error.reason            failed to parse field [contents.files.update_date] of type [date] in document with id '948b7062-15d6-5f13-9b8c-0365f4aac067_42a861e6-2e8a-5470-945a-5e3604d3eb96_2021-05-05T21:24:26.174274Z_exists'
error.root_cause.0.reason   failed to parse field [contents.files.update_date] of type [date] in document with id '948b7062-15d6-5f13-9b8c-0365f4aac067_42a861e6-2e8a-5470-945a-5e3604d3eb96_2021-05-05T21:24:26.174274Z_exists'
error.root_cause.0.type mapper_parsing_exception

The same error was encountered when a 2 and 3 digit year was tested, and the error did not occur when a 4 digit year was used.

As a workaround, the value datetime(1970, 1, 1, tzinfo=timezone.utc) (1970-01-01T00:00:00.000000Z) was used as the None/null value.

Determine why Elasticsearch on sandbox cannot parse the year if it has one or more zeros as a prefix, and why this issue does not occur when testing locally on a developer's machine.

@dsotirho-ucsc dsotirho-ucsc added the orange [process] Done by the Azul team label Aug 9, 2021
@hannes-ucsc
Copy link
Member

Ticket needs reproduction.

@melainalegaspi melainalegaspi added the spike:2 [process] Spike estimate of two points label Aug 9, 2021
@melainalegaspi
Copy link

@danielsotirhos spike for reproduction.

@melainalegaspi melainalegaspi added bug [type] A defect preventing use of the system as specified code [subject] Production code labels Aug 9, 2021
@dsotirho-ucsc
Copy link
Contributor Author

From https://bugs.python.org/issue13305

on linux
>>> datetime(1,  2, 10, 11, 41, 23).strftime("%Y")
'1'

on osx
>>> datetime(1,  2, 10, 11, 41, 23).strftime("%Y")
'0001'

>>> datetime.strptime('0001', '%Y')
datetime.datetime(1, 1, 1, 0, 0)

>>> datetime.strptime('1', '%Y')
ValueError: time data '1' does not match format '%Y'

@dsotirho-ucsc
Copy link
Contributor Author

Workaround added to #2562 (PR #3026) to enforce format_dcp2_datetime outputs a string with a 4-digit year. Also added doctests to format_dcp2_datetime and parse_dcp2_datetime to make sure they support year '0001'.

@hannes-ucsc hannes-ucsc changed the title Sandbox ES fails to parse dates with a less than 4 digit year datetime.strftime doesn't zero-pad years < 1000 on AWS Lambda Aug 20, 2021
@hannes-ucsc hannes-ucsc changed the title datetime.strftime doesn't zero-pad years < 1000 on AWS Lambda strftime fails to zero pad year < 1000 on AWS Lambda Aug 20, 2021
@hannes-ucsc hannes-ucsc added the no demo [process] Not to be demonstrated at the end of the sprint label Aug 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug [type] A defect preventing use of the system as specified code [subject] Production code no demo [process] Not to be demonstrated at the end of the sprint orange [process] Done by the Azul team spike:2 [process] Spike estimate of two points
Projects
None yet
Development

No branches or pull requests

4 participants