added static typing to data_utils.py #662
Conversation
Sanketh7
requested review from
JGSweets,
ksneab7,
taylorfturner,
micdavis and
tyfarnan
as code owners
|
@Sanketh7 looks like the branch needs updating as well. |
There was a problem hiding this comment.
comments mainly around formatting and +1 to @JGSweets comment around not casting
Sanketh7
force-pushed
the
data_utils_typing_new
branch
from
f08ecf5 to
bc44f0d
Compare
Head branch was pushed to by a user without write access
| @@ -90,7 +105,7 @@ def unicode_to_str(data, ignore_dicts=False): | |||
| # if data is a dictionary | |||
| if isinstance(data, dict) and not ignore_dicts: | |||
| return { | |||
| unicode_to_str(key, ignore_dicts=True): unicode_to_str( | |||
| cast(str, unicode_to_str(key, ignore_dicts=True)): unicode_to_str( | |||
There was a problem hiding this comment.
there's no guarantee this is a string, this could be an int, right?
There was a problem hiding this comment.
Removed the cast (not needed anymore after I changed JSONType).
There was a problem hiding this comment.
awesome -- thx, @Sanketh7
There was a problem hiding this comment.
is it removed @Sanketh7 ? looks like it is still there ...
There was a problem hiding this comment.
Should be removed by 8de0cef
| @@ -99,7 +114,11 @@ def unicode_to_str(data, ignore_dicts=False): | |||
| return data | |||
|
|
|||
|
|
|||
| def json_to_dataframe(json_lines, selected_columns=None, read_in_string=False): | |||
| def json_to_dataframe( | |||
| json_lines: List[JSONType], | |||
There was a problem hiding this comment.
We should actually validate this, @taylorfturner can df = pd.DataFrame(json_lines) take in any list of JSON?
There was a problem hiding this comment.
It works assuming List[JSONType] but should it be Dict[str, List[JSONType]]?
from typing import Dict, List, Union
import pandas as pd
JSONType = Union[str, int, float, bool, None, List, Dict]
test_iter_list = []
test_iter_list.append(['test', 'test'])
test_iter_list.append([1,2,3])
test_iter_list.append([1.02, 2.02, 3.03])
test_iter_list.append([True, False])
test_iter_list.append([None, None])
test_iter_list.append([['test', 'test'], ['test', 'test']])
test_iter_list.append([{'test': 'test', 'test': 'test'}])
for test_iter in test_iter_list:
pd.DataFrame(test_iter)
There was a problem hiding this comment.
@Sanketh7, I think the docstring for this on L126 :type json_lines: list(dict) needs updating
There was a problem hiding this comment.
This does work. So I think we are GTG
for test_iter in test_iter_list: json_to_dataframe(test_iter)
There was a problem hiding this comment.
List[list] works
In [307]: data = data_generator("""[["test", "test"], ["test", "test"]]""".splitlines())
In [308]: read_json(data)
Out[308]: [[['test', 'test'], ['test', 'test']]]
Head branch was pushed to by a user without write access
#610