[CI] Fix caching the env in between jobs

[blefaudeux]

What does this PR do?

Minor, properly cache the env in between jobs, speeding up CI a little.
I compared that with not using a cache at all, using a cache is actually faster:

• with a warm cache ~26 minutes
• prior to this PR ~35 minutes
• no cache ~32 minutes

(timings give or take a couple of minutes, but a proper cache is clearly better)

Before submitting

• Did you have fun?
  • Make sure you had fun coding 🙃
• Did you read the contributor guideline?
• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • N/A
• Did you make sure to update the docs?
  • N/A
• Did you write any new necessary tests?
  • N/A
• Did you update the changelog? (if needed)
  • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[blefaudeux]

insta land @dianaml0 @fmassa, probably not worth too much of your time :)