CLN: Lint for lists instead of generators in built-in Python functions #18335

mroeschke · 2017-11-17T02:20:45Z

tests passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff

Continuation of #18276 plus a lint check for list comprehension in built-in Python functions. The grep will only catch list comprehensions on single lines though. I'm all ears for greps that can catch multi-line list comprehensions or a better grep in general.

mroeschke · 2017-11-17T06:12:37Z

~~I am not exactly sure why cython won't let me compile either:~~

~~_str_reso_map = dict((v, k) for k, v in _reso_str_map.items())~~

or

~~_freq_reso_map = dict((v, k) for k, v in _reso_freq_map.items())~~

~~in resolution.pyx when _reso_str_map and _reso_freq_map are defined a few lines above them. It will compile if I pass in the list comprehension though.~~

When I changed the dict function call to {}, cython compiled successfully.

max-sixty · 2017-11-18T17:57:46Z

asv_bench/benchmarks/frame_ctor.py

@@ -132,7 +132,7 @@ def setup(self, offset, n_steps):
        offset = getattr(offsets, offset)
        self.idx = get_index_for_offset(offset(n_steps, **kwargs))
        self.df = DataFrame(np.random.randn(len(self.idx), 10), index=self.idx)
-        self.d = dict([(col, self.df[col]) for col in self.df.columns])
+        self.d = dict((col, self.df[col]) for col in self.df.columns)


could have all these be dict (or set, below) comprehensions now that py2.6 is unsupported

So you mean {foo for foo in bar} instead of set(foo for foo in bar)? Sure I don't see why not. Maybe that can be a followup to this.

dict(df.items()) should work here

codecov · 2017-11-18T18:56:18Z

Codecov Report

Merging #18335 into master will decrease coverage by 0.01%.
The diff coverage is 96%.

@@            Coverage Diff             @@
##           master   #18335      +/-   ##
==========================================
- Coverage   91.38%   91.36%   -0.02%     
==========================================
  Files         164      164              
  Lines       49790    49790              
==========================================
- Hits        45501    45492       -9     
- Misses       4289     4298       +9

Flag	Coverage Δ
#multiple	`89.17% <88%> (ø)`	⬆️
#single	`39.49% <42%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/clipboards.py	`100% <ø> (ø)`	⬆️
pandas/plotting/_core.py	`82.45% <0%> (ø)`	⬆️
pandas/tseries/holiday.py	`93.1% <100%> (ø)`	⬆️
pandas/plotting/_converter.py	`65.25% <100%> (ø)`	⬆️
pandas/core/indexing.py	`92.8% <100%> (ø)`	⬆️
pandas/core/generic.py	`95.73% <100%> (ø)`	⬆️
pandas/core/sparse/frame.py	`94.78% <100%> (ø)`	⬆️
pandas/core/indexes/interval.py	`92.85% <100%> (ø)`	⬆️
pandas/io/sql.py	`94.8% <100%> (ø)`	⬆️
pandas/core/internals.py	`94.54% <100%> (ø)`	⬆️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cfad581...c91410f. Read the comment docs.

codecov · 2017-11-18T18:56:42Z

Codecov Report

Merging #18335 into master will decrease coverage by 0.01%.
The diff coverage is 96%.

@@            Coverage Diff             @@
##           master   #18335      +/-   ##
==========================================
- Coverage   91.36%   91.34%   -0.02%     
==========================================
  Files         164      164              
  Lines       49790    49790              
==========================================
- Hits        45489    45480       -9     
- Misses       4301     4310       +9

Flag	Coverage Δ
#multiple	`89.14% <88%> (ø)`	⬆️
#single	`39.53% <42%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/clipboards.py	`100% <ø> (ø)`	⬆️
pandas/plotting/_core.py	`82.45% <0%> (ø)`	⬆️
pandas/tseries/holiday.py	`93.1% <100%> (ø)`	⬆️
pandas/io/parsers.py	`95.59% <100%> (ø)`	⬆️
pandas/stats/moments.py	`71.19% <100%> (ø)`	⬆️
pandas/io/sql.py	`94.8% <100%> (ø)`	⬆️
pandas/core/sparse/frame.py	`94.78% <100%> (ø)`	⬆️
pandas/core/reshape/concat.py	`97.58% <100%> (ø)`	⬆️
pandas/core/panelnd.py	`98.21% <100%> (ø)`	⬆️
pandas/core/common.py	`93% <100%> (ø)`	⬆️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1798c9d...3572626. Read the comment docs.

jreback · 2017-11-19T00:34:25Z

can you rebase

jreback · 2017-11-19T00:35:17Z

asv_bench/benchmarks/frame_ctor.py

@@ -132,7 +132,7 @@ def setup(self, offset, n_steps):
        offset = getattr(offsets, offset)
        self.idx = get_index_for_offset(offset(n_steps, **kwargs))
        self.df = DataFrame(np.random.randn(len(self.idx), 10), index=self.idx)
-        self.d = dict([(col, self.df[col]) for col in self.df.columns])
+        self.d = dict((col, self.df[col]) for col in self.df.columns)


dict(df.items()) should work here

jreback · 2017-11-19T00:37:31Z

asv_bench/benchmarks/io_bench.py

@@ -202,7 +202,7 @@ class read_json_lines(object):
    def setup(self):
        self.N = 100000
        self.C = 5
-        self.df = DataFrame(dict([('float{0}'.format(i), randn(self.N)) for i in range(self.C)]))
+        self.df = DataFrame(dict(('float{0}'.format(i), randn(self.N)) for i in range(self.C)))
        self.df.to_json(self.fname,orient="records",lines=True)


could use the comprehension

{('float{0}'.format(i), randn(self.N)) for i in range(self.C))}

jreback · 2017-11-19T00:39:14Z

pandas/core/indexes/api.py

@@ -101,7 +101,7 @@ def conv(i):


 def _sanitize_and_check(indexes):
-    kinds = list(set([type(index) for index in indexes]))
+    kinds = list(set(type(index) for index in indexes))


this could be a set comprehension
{type(index) for index in indexes}

prob more of these as well

mroeschke · 2017-11-19T06:57:32Z

Rebased. I can create a follow up issue/PR to replace instances of set(foo for foo in iterator) with {foo for foo in iterator} where appropriate (likewise for lists and dicts)

jreback · 2017-11-19T15:05:38Z

thanks!

gfyoung added Clean Code Style Code style, linting, code_checks labels Nov 17, 2017

mroeschke force-pushed the use_generators_lint branch from 4936ffc to dabb815 Compare November 18, 2017 07:07

max-sixty reviewed Nov 18, 2017

View reviewed changes

jreback requested changes Nov 19, 2017

View reviewed changes

mroeschke added 4 commits November 18, 2017 22:22

CLN: Lint for lists instead of generators in built-in Python functions

4a90d9f

Change function call to symbol

28f5ca2

Fix join statement and more linting

8188319

Rebase and some followup

3572626

mroeschke force-pushed the use_generators_lint branch from c91410f to 3572626 Compare November 19, 2017 06:41

jreback added this to the 0.22.0 milestone Nov 19, 2017

jreback approved these changes Nov 19, 2017

View reviewed changes

jreback merged commit 410ad37 into pandas-dev:master Nov 19, 2017

mroeschke mentioned this pull request Nov 20, 2017

CLN: Replace comprehensions list/set/dict functions with corresponding symbols #18383

Merged

2 tasks

mroeschke deleted the use_generators_lint branch November 22, 2017 19:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLN: Lint for lists instead of generators in built-in Python functions #18335

CLN: Lint for lists instead of generators in built-in Python functions #18335

mroeschke commented Nov 17, 2017

mroeschke commented Nov 17, 2017 •

edited

Loading

max-sixty Nov 18, 2017 •

edited

Loading

mroeschke Nov 18, 2017

jreback Nov 19, 2017

codecov bot commented Nov 18, 2017 •

edited

Loading

codecov bot commented Nov 18, 2017 •

edited

Loading

jreback commented Nov 19, 2017

jreback Nov 19, 2017

jreback Nov 19, 2017

jreback Nov 19, 2017

mroeschke commented Nov 19, 2017

jreback commented Nov 19, 2017

CLN: Lint for lists instead of generators in built-in Python functions #18335

CLN: Lint for lists instead of generators in built-in Python functions #18335

Conversation

mroeschke commented Nov 17, 2017

mroeschke commented Nov 17, 2017 • edited Loading

max-sixty Nov 18, 2017 • edited Loading

Choose a reason for hiding this comment

mroeschke Nov 18, 2017

Choose a reason for hiding this comment

jreback Nov 19, 2017

Choose a reason for hiding this comment

codecov bot commented Nov 18, 2017 • edited Loading

Codecov Report

codecov bot commented Nov 18, 2017 • edited Loading

Codecov Report

jreback commented Nov 19, 2017

jreback Nov 19, 2017

Choose a reason for hiding this comment

jreback Nov 19, 2017

Choose a reason for hiding this comment

jreback Nov 19, 2017

Choose a reason for hiding this comment

mroeschke commented Nov 19, 2017

jreback commented Nov 19, 2017

mroeschke commented Nov 17, 2017 •

edited

Loading

max-sixty Nov 18, 2017 •

edited

Loading

codecov bot commented Nov 18, 2017 •

edited

Loading

codecov bot commented Nov 18, 2017 •

edited

Loading