Add more local pipeline operations #24
Conversation
replomancer
added
the
Type: New Feature ➕
replomancer
force-pushed
the
local_operations_adapter
branch
from
dc5181b
to
de90115
Compare
| def count_per_element(self, col, stage_name: str): | ||
| pass | ||
| def count_per_element(self, col, stage_name: Optional[str] = None): | ||
| yield from collections.Counter(col).items() |
There was a problem hiding this comment.
I like idea with Counter!
One question, in this line, is 'col' accessed the first time only when iterations over count_per_element() started?
What do you think about adding the 'lazyness' test:
Let 'exception_generator' is a generator which raise an exception on the first access to it
and then
def test_lazyness():
self.ops.map(exception_generator) # should be no exception
self.ops.map_tuple(exception_generator)
....
on implementing a new method it would be easy to add to the test and to ensure that the method is lazy
There was a problem hiding this comment.
Cool idea for testing!
I added a test like this and yes, count_per_element works as expected.
|
@replomancer there are some conflicts. Can you please resolve them? |
replomancer
force-pushed
the
local_operations_adapter
branch
from
de90115
to
d9fbc71
Compare
replomancer
force-pushed
the
local_operations_adapter
branch
from
d9fbc71
to
9cb4e9c
Compare
| try: | ||
| operator(exceptions_generator_function(), *args) | ||
| except ZeroDivisionError: | ||
| self.fail(f"local {operator.__name__} is not lazy") |
There was a problem hiding this comment.
This function exists for tests to fail properly with AssertionError (assertNotRaises does not exists in unittest).
e.g. when filter implementation is broken:
return list(filter(fn, col))the fail looks like this:
tests/pipeline_operations_test.py:122: in assert_laziness
self.fail(f"local {operator.__name__} is not lazy")
E AssertionError: local filter is not lazy
There was a problem hiding this comment.
Thanks for explanation. I like that approach!
| try: | ||
| operator(exceptions_generator_function(), *args) | ||
| except ZeroDivisionError: | ||
| self.fail(f"local {operator.__name__} is not lazy") |
There was a problem hiding this comment.
Thanks for explanation. I like that approach!
Description
Adds every other remaining method to
LocalPipelineOperations(we split work with Zachery).Affected Dependencies
None.
How has this been tested?
Unit tests added 🧹
Checklist