IPNS record TTL/caching improvements

[zot]

Namesys creates a cache with no eviction callback. This means that Namesys does nothing when records are dumped from the cache because their TTL expires.

So, if an IPNS record's TTL has expired, the peer dumps the record and fetches from the DHT. This means that even if the record hasn't actually changed the peer still has to wait for the fetch from the DHT. There's not a clear way for requesting peers to determine TTL or lifetime for IPNS records so that makes implementing alternate behavior difficult.

What about this behavior for ResolveAsync (maybe activated with a special ResolveOpt)?

When a record's TTL has expired but lifetime has not:

1. output the expired cache record
2. fetch the record from the DHT
   1. cache the result
   2. output the result only if it has changed (meaning there can be 0, 1, or 2 results from the channel)
   3. close the channel

I think it'd just need a secondary cache based on lifetime for this and an eviction callback on the current lru that transfers records with expired TTLs to the secondary cache. The alternative ResolveAsync behavior could consult the secondary cache for records with expired TTL but not expired lifetime.

[aschmahmann]

@zot there are some good thoughts here, but I'd like to separate them out to make sure we're on the same page.

The issue here is that the TTL serves two purposes:

1. We should not do a network query within the TTL as it's the duration which it is "safe" to assume the result we have is the latest value
2. We have a time based cache of IPNS records, and in order to not keep them all around forever we throw out ones that are older than receiveTime + EOL

This dual usage is what's causing the confusion, since output the expired cache record is impossible as we've already cleared it from cache.

By the way it's worth noting that:

output the result only if it has changed (meaning there can be 0, 1, or 2 results from the channel)

There's actually an unlimited number number of results that can come through the channel. ResolveAsync for IPNS calls routing.SearchValue which returns the "best" (i.e. latest) values it finds as it comes across them.

There are a number of IPNS/Namesys API things to fix, especially now that IPNS over PubSub can function independently from the DHT.

---

IMO what would be better for our TTL problems here is to:

1. Store Namesys (maybe just IPNS?) records that we've received in the datastore for future use, and have some command to garbage collect them
2. Make sure ipfs repo gc also garbage collects the IPNS records
   a) Implement something like IPNS pinning: ipfs name follow #4435 to implementing "pinning" of IPNS records so they do not get garbage collected
3. Implement TTL-based caching that is dependent on the resolution mechanism
   a) One way would be to allow resolvers like IPNS over PubSub to perform cache invalidation upon receiving an update
   b) Another would be to have one TTL based cache for DNSLink and another for IPNS over DHT, and leave the other systems alone.
4. Allow cache overriding from ipfs name resolve

[lidel]

if an IPNS record's [cache] TTL has expired, the peer dumps the record and fetches from the DHT. This means that even if the record hasn't actually changed the peer still has to wait for the fetch from the DHT. There's not a clear way for requesting peers to determine TTL or lifetime for IPNS records so that makes implementing alternate behavior difficult.

iirc in the past namesys cache was hardcoded to keep things for 1 minute.

Kubo 0.24 improves things and leverages TTL of IPNS record for cache eviction (release notes).

IPNS publisher now can control both --lifetime and the --ttl.

@zot does this provide enough flexibility you've been looking for?

Or are you looking into having an option to resolve with equivalent of stale-while-revalidate behavior from HTTP, where the best valid result is returned from cache immediately, but if TTL expired, the cache is updated asynchronously, without blocking requests?