[Backport maintenance/3.3.x] Fix manager.clear_cache() not fully clearing the module cache (#2572)

[Backport maintenance/3.3.x] Fix manager.clear_cache() not fully clearing the module cache (#2572) #5284