[CON-229] - Fix content caching

[vicky-g]

Description

We only want to cache on 404, but for some reason, content was still cached. Turns out it is because we were setting the cache-control header to public, max-age=2592000 even if content was not servable (e.g. 404-ing). The keyword public implies that content can be cached on client or server side.

So, this PR is to only set the the download/cache headers if content is definitely served.

Tests

Before, if the ipfs/<cid> route was getting a 404, it would result in a cache miss, then a cache hit.

After, all 404s will only result in a cache miss.

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

We will still monitor nginx logs in the logs file, and can observe the x-cache-status header

[SidSethi]

great find!! one Q

i think we'll need to make same change to dirCID route right? but can test this on staging first before making that change

[vicky-g]

great find!! one Q

i think we'll need to make same change to dirCID route right? but can test this on staging first before making that change

actually, this is for the DIR scenario bc it sets the cache-control header, and then if the flow errs, the cache-control header is removed

  // Set the CID cache-control so that client cache the response for 30 days
  res.setHeader('cache-control', 'public, max-age=2592000, immutable')

  // Attempt to stream file to client
  try {
    req.logger.info(`Retrieving ${storagePath} directly from filesystem`)
    return await streamFromFileSystem(req, res, storagePath)
  } catch (e) {
    req.logger.info(`Failed to retrieve ${storagePath} from FS`)
  }

  // Attempt to find and stream CID from other content nodes in the network
  try {
    // CID is the file CID, parse it from the storagePath
    const CID = storagePath.split('/').slice(-1).join('')
    const libs = req.app.get('audiusLibs')
    await findCIDInNetwork(storagePath, CID, req.logger, libs)
    return await streamFromFileSystem(req, res, storagePath)
  } catch (e) {
    req.logger.error(
      `Error calling findCIDInNetwork for path ${storagePath}`,
      e
    )
    // Unset the cache-control header so that a bad response is not cached
    res.removeHeader('cache-control')
    return sendResponse(req, res, errorResponseServerError(e.message))
  }

[SidSethi]

great find!! one Q
i think we'll need to make same change to dirCID route right? but can test this on staging first before making that change

actually, this is for the DIR scenario bc it sets the cache-control header, and then if the flow errs, the cache-control header is removed

  // Set the CID cache-control so that client cache the response for 30 days
  res.setHeader('cache-control', 'public, max-age=2592000, immutable')

  // Attempt to stream file to client
  try {
    req.logger.info(`Retrieving ${storagePath} directly from filesystem`)
    return await streamFromFileSystem(req, res, storagePath)
  } catch (e) {
    req.logger.info(`Failed to retrieve ${storagePath} from FS`)
  }

  // Attempt to find and stream CID from other content nodes in the network
  try {
    // CID is the file CID, parse it from the storagePath
    const CID = storagePath.split('/').slice(-1).join('')
    const libs = req.app.get('audiusLibs')
    await findCIDInNetwork(storagePath, CID, req.logger, libs)
    return await streamFromFileSystem(req, res, storagePath)
  } catch (e) {
    req.logger.error(
      `Error calling findCIDInNetwork for path ${storagePath}`,
      e
    )
    // Unset the cache-control header so that a bad response is not cached
    res.removeHeader('cache-control')
    return sendResponse(req, res, errorResponseServerError(e.message))
  }

nvm, i see this is already done