Add additional methods to itertools

[rhagenson]

Previously there were methods for Iter within itertools which, while originally planned, where not implemented. These methods have now been implemented.

The added methods are:

• dedup which removes local duplicates from consecutive identical elements via a provided hash function
• interleave which alternates values between two iterators until both run out
• interleave_shortest which alternates values between two iterators until one of them runs out
• intersperse which yields a given value after every n elements of the iterator
• step_by which yields every nth element of the iterator
• unique which removes global duplicates of identical elements via a provided hash function

[ponylang-main]

Hi @rhagenson,

The changelog - added label was added to this pull request; all PRs with a changelog label need to have release notes included as part of the PR. If you haven't added release notes already, please do.

Release notes are added by creating a uniquely named file in the .release-notes directory. We suggest you call the file 3992.md to match the number of this pull request.

The basic format of the release notes (using markdown) should be:

## Title

End user description of changes, why it's important,
problems it solves etc.

If a breaking change, make sure to include 1 or more
examples what code would look like prior to this change
and how to update it to work after this change.

Thanks.

[rhagenson]

Only missing method left from the original ones commented out is:

fun ref dedup[H: HashFunction[A] val = HashIs[A]](): Iter[A!]^ =>
    """
    Return an iterator that removes duplicates from consecutive identical
    elements. Equality is determined by the HashFunction `H`.

    ## Example

    ```pony
    Iter[I64]([as I64: 1; 1; 2; 3; 3; 2; 2].values())
      .dedup()
    ```
    `1 2 3 2`
    """

I tried implementing this one first, but could not get it correct. I think I can use a similar structure to what I have already by having a _prev_hash variable to compare against, but my issue there is that has_next() can give the wrong result. has_next() should logically give whether there is another dedupped value, but as I currently have it written it gives the presence of a possibly duplicated value. Given that I expect there is a way to lag only one value behind the underlying iterator, I do not want to have to traverse all values just to fix this has_next() problem.

[rhagenson]

My plan on this is: 1) to take another look at what methods I have here already to see if I can clean anything up, 2) look at other similar libraries (e.g., the ones in the Zulip thread) for related methods we may want to add.

My initial look at other libraries yielded possible step_by and sort methods.

[rhagenson]

I am foregoing implementing a sort method as I cannot determine the "best" way to make that work using the existing Sort primitive and the possibility of an infinite iterator. My initial thought was to have a buffered sort which read in n elements, keeps those n elements sorted and returns the lowest/highest based on user preference.

let iter = Iter[USize](Range(0, 4)).interleave(Range(4, 6))  // 0 4 1 5 2 3
iter.sort(2)  // 0 1 4 2 3 5
/*
Steps:
1. Create buffer of size 2, load values 0 4 in buffer
2. Yield 0
3. 4 1 in buffer
4. Yield 1
5. 4 5 in buffer
6. Yield 4
7. 5 2 in buffer
8. Yield 2
9. 5 3 in buffer
10. Yield 3
11. 5 _ in buffer
12. Yield 5
/*

This seems like a "bad" idea as it breaks how I would expect sort to work, but is the idea I had for making it work under the possible infinite iterator condition.

[jemc]

To me it seems somewhat "off" that we are only using H.hash and not also using H.eq to confirm true value equality (instead of just hash-equivalence).

Some hash functions may have a non-negligible collision rate for non-identical elements, and because HashFunction has both a hash method and an eq method, users may be relying/assuming that the H.eq function will be used to check for true value equivalence after the H.hash equivalence has been checked, so that the overall check is resilient to false collisions.

[rhagenson]

I had to look at the source again around HashFunction, but I see what you mean and will make the necessary change(s) to use H.eq where needed.

[rhagenson]

@jemc Please check this again and make suggestions for improvement. In order to use H.eq I needed to keep both the previous hash and previous value. I initially tried to solve this with a var _prev: ((A!, USize) | None) = None but unpacking the tuple was causing me issues with ponyc insisting _iter.next()? having a return type of (A #any ! | None val^) so I split the value and hash into separate variables. I presume based on your comment here and the documentation around H.eq that the order of equality should be checking the hash match then using H.eq but am not 100% certain it should not be the other way around.

[rhagenson]

I also think this code is a bit messy right now with cur_{value,hash}, _prev_{value,hash}, and prev_value so if you have advice on making these names less redundant or the code less repetitive, please let me know!

[rhagenson]

@SeanTAllen I made the changes you suggested in release notes.

[SeanTAllen]

@rhagenson awsome.