rec: dedup records

[omoerbeek]

Short description

This deduplicaties records in two places:

1. Records received from authoritative servers
2. Records sent out to clients (only on non-packet cache hits).

When testing this, I encountered a few cases where auths sent duplicate records.

dig @ns4.dnsv5.com getui.com txt 
dig @ord.geons.ftw.jiveip.net api.geoip.jive.com txt
dig @dnsd10.chatango.com chatango.com
dig @ns.sotoon53.com yektanet.com txt
dig @ns.viettelidc.com.vn ns9.viettelidc.com.vn
dig @ns1.cdnetdns.net flypgs.com txt

Nothing actually breaks if we don't dedup afaik. So it is questionable of we want this part.

But on the client side, this fixes #14120 and maybe other cases I do not know.

The big question is if we want this. Dedupping is fundamentally not cheap, although I tried to optimize the dedup code.

Originally I played with the idea to change the data structure building the reply vector to avoid duplicates, but that requires changes in many places. Still the idea is not completely off the table.

On the receiving side the dedup internals could also be weaved into the sanitize code, at the cost of increasing complexity. That would avoid the separate dedup() call.

This will remain a draft until I have some some speed measurements and pondered the alternative approaches some more. This PR is mainly to share thoughts.

The test changes are needed as a few of them use duplicate records.

Checklist

I have:

• read the CONTRIBUTING.md document
• compiled this code
• tested this code
• included documentation (including possible behaviour changes)
• documented the code
• added or modified regression test(s)
• added or modified unit test(s)

[coveralls]

Pull Request Test Coverage Report for Build 11515732828

Details

• 79 of 79 (100.0%) changed or added relevant lines in 9 files are covered.
• 43 unchanged lines in 10 files lost coverage.
• Overall coverage increased (+0.02%) to 64.64%

---

Files with Coverage Reduction	New Missed Lines	%
modules/gpgsqlbackend/gpgsqlbackend.cc	1	88.62%
pdns/dnsdistdist/dnsdist-crypto.cc	2	75.72%
pdns/rcpgenerator.cc	2	89.5%
pdns/dnsdistdist/dnsdist.cc	2	68.38%
pdns/packethandler.cc	3	72.7%
pdns/recursordist/rec-main.cc	3	61.98%
modules/gpgsqlbackend/spgsql.cc	3	67.7%
pdns/iputils.cc	6	54.99%
pdns/recursordist/test-syncres_cc1.cc	7	89.55%
modules/lmdbbackend/lmdbbackend.cc	14	72.69%

Totals
Change from base Build 11501457816:	0.02%
Covered Lines:	124819
Relevant Lines:	162316

---

💛 - Coveralls

[rgacogne]

The logic looks good to me. I guess we could try a few more heuristics before actually serializing the content, like checking if we have several records with the same (qtype, qname), but it might not be worth it. Having real-world numbers would indeed be useful.

[rgacogne]

If we go forward, we might want to consider a small refactoring to avoid duplicating code between serialize and wireFormatContent.

[rgacogne]

Perhaps dedupRecords instead of dedup?

[omoerbeek]

Rebased to fix conflict.

[omoerbeek]

Some observations:
quad1 does dedup in some cases (only when the dups are adjacent?)
quad8 does not dedup
OpenDNS does dedup

Software tested in default config
unbound does not do dedup
bind does dedup
knot-resolver does dedup

[omoerbeek]

Speedtest results:

'2 DedupRecords (generate only)' 0.10 seconds: 1230169.1 runs/s, 0.81 us/run
'2 DedupRecords' 0.10 seconds: 504897.9 runs/s, 1.98 us/run
'2 DedupRecords (with dup)' 0.10 seconds: 628577.4 runs/s, 1.59 us/run
'256 DedupRecords (generate only)' 0.10 seconds: 9248.3 runs/s, 108.13 us/run
'256 DedupRecords' 0.10 seconds: 3867.5 runs/s, 258.57 us/run
'256 DedupRecords (with dup)' 0.10 seconds: 3823.4 runs/s, 261.55 us/run
'4096 DedupRecords (generate only)' 0.11 seconds: 561.7 runs/s, 1780.35 us/run
'4096 DedupRecords' 0.10 seconds: 241.7 runs/s, 4137.49 us/run
'4096 DedupRecords (with dup)' 0.10 seconds: 239.3 runs/s, 4178.26 us/run

The measured slowdown is about 2.5 and is uniform over the various test case sizes.

So the dedupping takes time as expected, but for the already pretty extreme case of 256, records, its absolute value is not a lot compared to the expected network latency. For the 4096 case we spent time that comes closer to the expected network latency.

[paddg]

Can we rule out that deduping is an attack vector?

[omoerbeek]

Can we rule out that deduping is an attack vector?

Not completely, in the extreme case spending even a few CPU ms on a single auth result is quite a lot.

[paddg]

Not completely, in the extreme case spending even a few CPU ms on a single auth result is quite a lot.

Are you considering an on/off switch for it?

[omoerbeek]

Not completely, in the extreme case spending even a few CPU ms on a single auth result is quite a lot.

Are you considering an on/off switch for it?

Yes, that would be one of the options. Another alternative would be to not do the dedupping on large answers as we already refuse to cache them anyway.

[omoerbeek]

The logic looks good to me. I guess we could try a few more heuristics before actually serializing the content, like checking if we have several records with the same (qtype, qname), but it might not be worth it. Having real-world numbers would indeed be useful.

I played a bit with a pre-scan on qtype and name only, but saw no speedup