Is kv_cache being read for anything other than cross_attn with GreedyDecoder?

[mmmkkaaayy]

Based on my debugging, I've seen that kv_cache is written to during self.attn and self.cross_attn calls of the TextDecoder forward pass. However, I can only see values from the kv_cache being read during self.cross_attn (by entering this branch, which is not possible for self.attn because xa is not passed as an arg).

Otherwise, the only other time I found the kv_cache being accessed is here, where I feel an int counter would be sufficient instead of having to actually save any inferred values.

Am I missing some other part of the code/conditions under which the kv_cache is being read? I see this line in BeamSearchDecoder, but I'm interested in the GreedyDecoder path.

[mmmkkaaayy]

Couple other notes:

• Appending to the cache indeed reads the previous value. But is the final result used anywhere?
• kv_cache is only available for cross attention blocks? #489 confirms that the kv_cache is written to in both self.attn and self.cross_attn, but doesn't directly answer my question about whether the values are being read.

[alexterminator88]

.github

[mmmkkaaayy]

Oh, I get it now... the return value of save_to_cache is actually overriding the return value of the layer it's hooked into, so that's how the values are being read.