Appellate: Free docs not uploaded?

[mlissner]

In PACER, courts can mark documents as official opinions. If they do, those opinions become free.

In at least one case, when opinions are free, users don't get shown the receipt page. When that's the case, we don't upload the doc to RECAP. Opinions are the outcome of the case, so the inability to upload them is a huge deal.

STR:

1. Go to NVLSP v. U.S.: https://www.courtlistener.com/docket/8034892/nvlsp-v-united-states/
2. Click the "View on PACER" link.
3. Click docket entry number 8, which is an opinion.

Results:

1. The PDF is downloaded immediately.
2. The PDF is not uploaded.

Expected results:

1. The PDF should be uploaded to RECAP.

[litewarp]

I actually have an implementation of this done, except it's slightly different (better?) than your process.

In short -- rather than wait till the user clicks on the free opinion, we can identify it on the docket page (or at least as of two years ago) and then grab it automagically and upload it in the background, regardless of whether the user downloads that particular document -- any visit to the docket page will grab the free opinion.

The code below is obviously written for my fork, but you can probably extract out the dom traversing logic and dispatch the upload using the (barf) ajax callbacks.

import PACER from '../pacer';
import {
  courtListenerURL,
  authHeader,
  getBrowserFetch,
  blobToDataURL,
  updateTabStorage,
  dispatchBackgroundFetch,
  dispatchNotifier,
  searchParamsURL,
} from '../utils';

// check if the opinion is free to download and if so
// fetch it and upload it to recap in the background

export async function checkForAndUploadOpinion({ pacerCaseId }) {
  const trs = [...document.querySelectorAll('tr')];
  const opinionTr = trs.find((tr) => {
    if ([...tr.children].length > 0) {
      const match = [...tr.children].find(
        (td) => td.textContent.match(/OPINION/) && td.width === '90%'
      );
      if (match) {
        return true;
      }
    }
  });
  const link = opinionTr && opinionTr.querySelector('a');
  if (!link) return console.info('RECAP: No opinion link found. Not uploading.');

  const params = {
    caseId: pacerCaseId,
    dls_id: link.href.match(/docs1\/(\d+)/)[1],
    servlet: 'ShowDoc',
    dktType: 'dktPublic',
  };

  const url = searchParamsURL({ base: document.URL.replace(/\?.*$/, ''), params });

  const contentScriptFetch = getBrowserFetch();
  const blob = await contentScriptFetch(url).then((res) => res.blob());

  if (!blob || !blob.type.includes('pdf'))
    return console.info('RECAP: Not uploading. No blob or incorrect blob type found.');

  const dataUrl = await blobToDataURL(blob);

  const stashedInStorage = await updateTabStorage({
    [this.tabId]: { ['file_blob']: dataUrl },
  });

  if (!stashedInStorage) return console.error('RECAP: Blob not stashed in storage. Not uploading.');

  const uploaded = dispatchBackgroundFetch({
    url: courtListenerURL('recap'),
    options: {
      method: 'POST',
      headers: { ...authHeader },
      body: {
        court: PACER.convertToCourtListenerCourt(this.court),
        pacerCaseId: pacerCaseId,
        pacerDocId: params.dls_id,
      },
    },
  });

  if (!uploaded) return console.error('RECAP: Opinion not uploaded. Something went wrong.');

  const notified = dispatchNotifier({
    title: 'successful_free_opinion_upload',
    action: 'showUpload',
    message: 'Case Opinion automatically uploaded to the public RECAP Archive.',
  });

  if (notified) console.info('RECAP: User notified of successful upload.');
}

[litewarp]

Pasting a link to the unit test for the code above ☝️

[mlissner]

Thanks Nick. That's actually really interesting as an approach. I think we need to wait for users to click things though, because otherwise we violate the "quazi-contract" we have with them, which says, "You click things, they get uploaded." I wouldn't want to surprise our users, some of which are really sensitive about what they upload.

On a technical front, I'm not sure you can trust this either:

(td) => td.textContent.match(/OPINION/) && td.width === '90%'

Though I know that Eduardo is struggling a bit to figure out which of the docs on each page are opinions. Maybe that's the best we can do, but I'm hopeful there's a clue in the code somewhere that he'll uncover.

[litewarp]

I hear ya. There's not a really good or well-labeled distinction between the different documents. What I have in my notes is:

Document links are contained in a table taking the following shape
------------------------------------------------------------------
<tr>
  <td valign="top">09/01/2020</td>
  <td valign="top" nowrap>
    <a
      href="https://ecf.cafc.uscourts.gov/docs1/01301646325"
      onclick="return doDocPostURL('01301646325','16239');"
      title="Open Document"
    >      
      &nbsp;1&nbsp;
    </a>
  </td>
  <td width="90%" valign="top">      
     Appeal docketed. Received: 08/31/2020. [718915]
     <br>
     Entry of Appearance is due on 09/15/2020. Certificate of Interest is due on 09/15/2020. [...]
  </td>
</tr>

So searching the text of the td cell is probably the cleanest way. The 90% attribute is just to make sure that it's in the 3rd column in the row. In other words, the text will be the first filter, and then you can verify it by checking to see if the 90% attribute is there, or alternatively, whether it is a 'td' that is a sibling that contains an "a" element. Unfortunately there's no specific class, id, or data-attribute to identify the opinion.