Refactor to separate concerns and facilitate new behavior

[jaraco]

• Extract freeze_data and load_pip_packages to separate concerns.
• Extract load_pip_package function.
• Use degenerate form for fallback.
• Extract package_item behavior.
• Extract is_valid filter on pip package entries with no URL.

[hoodmane]

Thanks @jaraco! Looks good to me.

[hoodmane]

I'm curious how you feel about the readability of map and filter in Python where we don't have any way to do right function application. What I'd really like to see is

importlib.metadata.distributions().map(load_pip_package).filter(None).map(package_item)

so that the logic flows from left to right and top to bottom instead of in a spiral. I tend to use comprehensions instead but they are arguably just as bad if not worse. Combining filter and map does make sense but list comprehensions do it in a strange order. I do like Rust's filter_map quite a lot.

Anyways, code is clear I'm happy to do it this way.

[jaraco]

Personally, I'm a big fan of the general concept of filter and map (and reduce). Part of my affinity comes from my first formal language training being in Scheme. But my reason for sticking with it is how it incentivizes stateless functional programming. Comprehensions and generator expressions achieve the same thing, but often by adding unnecessary variables. If I'm working with a comprehension and it has the form [do(x) for x in input if check(x)], I will almost certainly translate that to map(do, filter(check, input)) and avoid creating the name x. I will sometimes separate the two lines into filtered = filter(check, input); map(do, filtered) if the name filtered improves readability (which it could in the case above).

I do avoid map/filter when their use feels contrived or convoluted for the application.

What I'd really like to see is

importlib.metadata.distributions().map(load_pip_package).filter(None).map(package_item)

I agree - that reads better. And other languages (Node, Ruby, ...) design that paradigm into their language. Python, on the other hand, has historically discouraged these chains of operations. So my pragmatic approach is to accept that the language constructs are somewhat inelegant, but still preferable to the alternative of eager-evaluated for/append loops.

[hoodmane]

Yeah a lot of my early programming experience was also with lisps. I particularly enjoy programming with only immutable data structures. But it's not always the best for performance.

I'm really excited about the iterator helpers spec in JavaScript which will make doing all of this lazily very ergonomic too:
https://github.com/tc39/proposal-iterator-helpers

[hoodmane]

This is interesting, often the need to write a function like this instead of a lambda dissuades me from using map. The combination of spiral control flow with in line lambdas tends to be a bit unmanageable.

[jaraco]

Yes. I wouldn't object to inlining this behavior in a generator expression in load_pip_packages.

[hoodmane]

Is filter(None just a no-op? It's there to make the diff look better?

[jaraco]

filter(None, ...) is the same as filter(bool, ...). It was there to suppress the distributions that returned None. I included that to maintain the prior expectation that if the url is None, no package entry is generated (the continue in the prior implementation).

I've been using that form since Python 1.5, before bool was introduced in Python 2.3. It occurs to me I should start using filter(bool, ...) for better readability.

[hoodmane]

Suggested change

	return entry["url"] is not None
	return entry.get("url", None) is not None

[hoodmane]

@jaraco if we do this change we can merge it. I'll rebase and apply this and then do a rebase merge.

[hoodmane]

Would still be interested for answers to my questions about your view on the ergonomics and readability of filter, map and other functional constructs in Python.

[jaraco]

Oh, whoops. This was a mistake. The proper key should be "file_name". Specifically,

return entry["file_name"] is not None

Since load_pip_package sets file_name unconditionally, calling .get() is unnecessary; the code can rely on that key being present, but it may be set to None.

With the patch as suggested, entry.get("url", None) will always be None and no entry will ever be valid.

[jaraco]

Oh, and it appears you recognized that in the rewritten commit. Thanks!

[hoodmane]

Yeah, thankfully we have test coverage for this =)

[hoodmane]

Thanks @jaraco!