-
Notifications
You must be signed in to change notification settings - Fork 166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support decimal, date, time, timestamp with time zone and timestamp python types in prepared statements and result sets #160
Conversation
8c316b4
to
d2ef96c
Compare
As mentioned in #32, this change will impact the behavior on existing codebases. Should we provide a feature flag for this improved type detection? |
d2ef96c
to
af6fd32
Compare
Time with time zone (utc offset) doesn't really exist as a Python type. It's possible to have time with a timezone, but however without an actual moment in time, it's impossible to know the UTC offset. So I don't think this is something we can support for now. |
67c104d
to
a02bc6d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This introduces regression regarding range of values that can be represented and hence needs a toggle somewhere - maybe add an alternative Cursor implementation?
Left some other comments regarding tests.
6d3f47b
to
a804244
Compare
To convert named timezones i have added pytz as dependency. As we support older python versions, it seems easier to just use pytz, which was already a test dependency. Maybe we should make this an optional dependency as it's only required for when the flag is activated. Please advice. Starting from python 3.9 we could replace this with the standard zoneinfo (however seems still need an additional library on Windows: https://tzdata.readthedocs.io/en/latest/). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should make this an optional dependency as it's only required for when the flag is activated.
Let's leave it as a required dependency since e.g I want to continue to use old python types and then switch to the new one.
48a19eb
to
bf99688
Compare
bf99688
to
9ebab39
Compare
9ebab39
to
1f33e2a
Compare
trino/client.py
Outdated
elif "timestamp" in raw_type: | ||
return datetime.strptime(value, "%Y-%m-%d %H:%M:%S.%f") | ||
elif "time with time zone" in raw_type: | ||
return datetime.strptime(value, "%H:%M:%S.%f").time() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this include timezone information?, i,.e., lines 537 and 539 are the same.
Note the dateutil.parser.parse
method could be useful and replace the logic in lines 527–539, somewhat ignoring the specific raw_type
, i.e., value
should contain the timezone information if the raw_type
is timestamp with time zone
or time with time zone
, i.e.,
from dateutil.parser import parse
if raw_type in ("time", "time with time zone", "timestamp", "timestamp with time zone"):
result = parse(value)
if raw_type in ("time", "time with time zone"):
result = result.time().replace(tzinfo=result.tzinfo)
return result
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's indeed a gap in the testing as well. On the library: currently we use pytz, and the format we will get is predictable so maybe no need to swap out the library.
trino/client.py
Outdated
elif "timestamp" in raw_type: | ||
return datetime.strptime(value, "%Y-%m-%d %H:%M:%S.%f") | ||
elif "time with time zone" in raw_type: | ||
return datetime.strptime(value, "%H:%M:%S.%f").time() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this include timezone information?
Note the dateutil.parser.parse
method could be useful and replace the logic in lines 527–539, somewhat ignoring the specific raw_type
, i.e., value
should contain the timezone information if the raw_type
is timestamp with time zone
or time with time zone
, i.e.,
from dateutil.parser import parse
if raw_type in ("time", "time with time zone", "timestamp", "timestamp with time zone"):
result = parse(value)
if raw_type in ("time", "time with time zone"):
result = result.time().replace(tzinfo=result.tzinfo)
return result
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few remaining comments. Thanks for the work on this @mdesmet.
Looks almost good to go.
1f33e2a
to
2ed780c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % last batch of comments.
trino/client.py
Outdated
matches = re.match(r'^(.*)([\+\-])(\d{2}):(\d{2})$', value) | ||
assert matches is not None | ||
if matches.group(2) == '+': | ||
tz = timedelta(hours=int(matches.group(3)), minutes=int(matches.group(4))) | ||
else: | ||
tz = -timedelta(hours=int(matches.group(3)), minutes=int(matches.group(4))) | ||
return datetime.strptime(matches.group(1), "%H:%M:%S.%f").time().replace(tzinfo=timezone(tz)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nineinchnick / @hovaesco Can you please double check this? It looks correct to me - not sure of simpler alternatives.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, we could avoid checking if matches.group(2) == '+'
and just add +/- and don't have a conditional here. But I don't have a strong opinion about either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. It seems for values like 2022-01-01 01:01:01 05:30
we'd convert this to negative offset which seems wrong.
@mdesmet can you passthrough the sign here? Or explicitly compare to -
as well. the default should be positive offset AFAIK.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point about the default. Will make that change now.
2ed780c
to
1ab53ad
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % a few more comments @mdesmet
1ab53ad
to
840ed53
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you.
Fixes #150 , #32