Use a sorted set for unique list functionality

[lewispb]

I benchmarked an implementation of a unique list using a Redis Sorted Set as opposed to the current implementation which uses a Redis list.

https://gist.github.com/lewispb/bcb190cf72c93b67f57671e0287fa7a7

I think this shows potential performance improvements of at least 5x, or depending on the size of the list more than that.

If we want to move towards using a sorted set, we'd need to think about the migration. Maybe we maintain unique list for backwards compatiblity but add a new data structure, Kredis.sorted_set?

[dhh]

Think the problem is that redis already has a sorted set with certain expectations tied to it (like being unordered). Think we might have to go with something like unique_fast_list, deprecate the old one, then set a cut-off for passing over.

Although I suppose it'd be even nicer if we could detect the old data format and convert?

[lewispb]

Although I suppose it'd be even nicer if we could detect the old data format and convert?

I think this would be possible actually. We could rescue the error: WRONGTYPE Operation against a key holding the wrong kind of value (Redis::CommandError). Then there'd be a one-off hit to migrate the data there and then from a list to a sorted set.

[dhh]

Seems worth exploring.

[lewispb]

👍 I'll look into it