Refactor deduper and write additional tests

[dianashk]

Refactored the deduper. Here's what got done:

• Break out the diff functions to make testing them easier
• Add tests for the newly separated diff functions
• In the diff function, compare the layer, parent.*_id, address_parts.*, and name.*
  • Only compare zip if both records have it set, otherwise ignore
  • When comparing the parent.*_id properties, skip the check where *===item.layer
  • When comparing name.* only expect name.default to match, but only check additional languages if both items have it set, otherwise ignore

Fixes #604

[dianashk]

TBD: I'd like to add a function in the deduper middleware to check which of the two items is prefered when a dupe is found. This would allow us to pick WOF records over Geonames, and OA over OSM, etc.

[missinglink]

the method signature of this function is not very intuitive, it's called isDiffLayer which I think is a great name if it returns a boolean value.

looking at the calling code it seems as though the return value is not actually used and the functions main purpose is validation via throw.

a bit nit-picky, but would this function name be better as validateDiffLayer (which returns nothing and throws on error) or isDiffLayer (which does not throw and returns bool so the caller can handle the error case)?

the same question applies to all the is* functions here.

[dianashk]

Thanks @missinglink for the helpful feedback. Right on and not at all nitpicky. 👍 I think I've addressed all your comments.

[trescube]

Very helpful description and I think this will solve our deduping issues.

[missinglink]

[dianashk]

Thanks all for feedback. Merging this and will create a separate issue for the isPreferred functionality mentioned in a comment above. 🚀