Generate crosswalk and import cap pdf #4442

jtmst · 2024-09-11T14:35:36Z

Import Harvard Case Law Access Project (CAP) PDFs to CourtListener

Issues Addressed:

Changes Implemented:

Created a management command to generate crosswalk files between CAP and CL data.
Developed a management command to import CAP PDFs to CL using the generated crosswalk.
Implemented S3 storage integration for PDF storage (configurable for local storage in development).
Added error handling and logging for better debugging and monitoring.
Added settings for CAP R2 env variables
Added tests for both commands

Testing Instructions:

Prerequisites:

Ensure you have the necessary environment variables set for R2 and S3 access.
For local testing, configure the storage to use local file system instead of S3.

Steps:

Generate the crosswalk:

 docker exec -it cl-django python /opt/courtlistener/manage.py generate_cap_crosswalk

This command will create crosswalk files in cl/search/crosswalks/.

Import CAP PDFs:
```
 docker exec -it cl-django python /opt/courtlistener/manage.py import_harvard_pdfs
```
This command will use the generated crosswalk to fetch and store PDFs.

To test with specific parameters:

 docker exec -it cl-django python /opt/courtlistener/manage.py generate_cap_crosswalk --reporter "A.2d" --volume 100

Verification:

Check the cl/search/crosswalks/ directory for generated crosswalk files.
Verify that PDFs are stored either in S3 or local storage (based on configuration).
Examine the logs for any errors or warnings during the process.

Screenshots:

CAP Crosswalk File (Sample Data):

Imported PDFs (Local Storage):

for more information, see https://pre-commit.ci

FRodriguez18

Looks good! Didn't approve it because I think John proposed some changes, but this looks good to me 👍

cl/search/tests/test_generate_cap_crosswalk.py

mlissner · 2024-09-17T22:53:18Z

@flooie I just put this on your backlog to prioritize. The background is that we want to have the Harvard PDFs imported into CL and we want to do so regularly. Grab me when you have a sec, and I can give the details before you review or assign to somebody else to review.

quevon24

In general, it is necessary to add typing to the functions in addition to updating the docstrings of the functions so that they are in accordance with the format used in courtlistener.

cl/search/management/commands/generate_cap_crosswalk.py

cl/search/management/commands/import_harvard_pdfs.py

cl/search/tests/test_generate_cap_crosswalk.py

cl/search/tests/test_import_harvard_pdfs.py

…d-import-cap-pdf

for more information, see https://pre-commit.ci

…mst/courtlistener into generate-crosswalk-and-import-cap-pdf

for more information, see https://pre-commit.ci

quevon24

Everything looks good, you've addressed the comments I left, one function is still missing the docstrings, and a comment I left regarding the find_matching_case() function is still pending.

After solving what I mentioned before, I think it will be ready.

quevon24 · 2024-09-19T21:35:10Z

cl/search/management/commands/generate_cap_crosswalk.py

+                    f"Single match found: {matched_case.case_name} (CL ID: {matched_case.id})"
+                )
+                return matched_case
+            elif match_count > 1:


What would happen if there were several cases on the same page (with the same citation)? Wouldn't the PDF always be added to the same cluster (the first one that matches the citation)?

In this case I suppose that would be the result, but I was under the impression that when this occurs the citation would be differentiated by page number as "100 A.2d 36a" or "100 A.2d 36b". Is this not the case?

If they were the exact same citation starting on the same page then there would be multiple matches and a logged warning with the citations.

If this is a more common occurrence than I thought then I'll need to find a way to be more specific, how would CL differentiate between the two in this case, by docket number?

Addressed in this comment bellow: #4442 (comment)

cl/search/management/commands/generate_cap_crosswalk.py

…mst/courtlistener into generate-crosswalk-and-import-cap-pdf

cl/search/management/commands/generate_cap_crosswalk.py

quevon24

Everything looks very good, using the cap id greatly simplified the match, I have no more comments

for more information, see https://pre-commit.ci

mlissner · 2024-10-01T22:18:27Z

I think we're just waiting for @flooie's final review here, but he's out sick today. Hopefully we'll get this merged shortly. Sorry for the delay.

flooie

This looks good to me- Thanks for taking the time to rewrite the matching code.

mlissner

Sorry to chime in a bit late. I came in here to see if there were any DB migrations before merging, but I noticed a couple of last minute things I think we should tweak.

@flooie or @quevon24, do you think we could make these changes so Josh can consider his part of this done?

cl/search/crosswalks/example_crosswalk.json

cl/search/management/commands/generate_cap_crosswalk.py

cl/settings/third_party/r2.py

move cap env variables to misc.py

…in the command

…-pdf' into generate-crosswalk-and-import-cap-pdf

flooie · 2024-10-03T15:53:09Z

@mlissner Thanks for your suggestions. The failing test passed on a re-run and appears to be related to an error in the meeting I missed.

mlissner · 2024-10-03T16:02:50Z

Woot! We're merged. @quevon24, do you want to file an issue to get this deployed and work with Ramiro on landing the changes?

quevon24 · 2024-10-03T16:14:23Z

get

Woot! We're merged. @quevon24, do you want to file an issue to get this deployed and work with Ramiro on landing the changes?

Yes, I'll get to it, I think we would only need the values of the environment variables necessary for the command, or are they already set?

mlissner · 2024-10-03T17:09:18Z

They're not, I don't think. I'll get them to Ramiro.

Harvard's Caselaw Access Project has been sunset. For projects which have existing references to CAP cases, there's a need to identify a CAP case's corresponding CL opinion cluster. An indexed `harvard_id` column is added to `OpinionCluster`. The field is also added to the `fields` of `OpinionClusterFilter`. For migration, this patch builds on work done in freelawproject#4284 and freelawproject#4442 and extends `import_harvard_pdfs` to populate the `harvard_id` column using CAP crosswalk file. Fixes: freelawproject#4313

jtmst and others added 8 commits September 10, 2024 09:41

crosswalk and import script testing and wip

e345a3b

Testing work for import and crosswalk scripts

480ccf9

import logic, comments, and crosswalk test

bfbc594

comments for crosswalk script

53178c7

[pre-commit.ci] auto fixes from pre-commit.com hooks

d91fb44

for more information, see https://pre-commit.ci

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

05d5690

modify test env vars

3b6703e

sample crosswalk for test

bf462c3

FRodriguez18 reviewed Sep 12, 2024

View reviewed changes

cl/search/tests/test_generate_cap_crosswalk.py Outdated Show resolved Hide resolved

Updated vars/tests, comments for new vars

2d1a6ff

jtmst requested review from FRodriguez18 and mlissner and removed request for FRodriguez18 September 16, 2024 16:34

jtmst marked this pull request as ready for review September 16, 2024 16:35

quevon24 reviewed Sep 19, 2024

View reviewed changes

jtmst and others added 3 commits September 20, 2024 10:07

Merge remote-tracking branch 'origin/main' into generate-crosswalk-an…

67eca14

…d-import-cap-pdf

PR Feedback, cleanup, simplify tests for errors

f9869a2

[pre-commit.ci] auto fixes from pre-commit.com hooks

12aa4e1

for more information, see https://pre-commit.ci

jtmst requested a review from quevon24 September 23, 2024 20:39

flooie and others added 5 commits September 23, 2024 17:17

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

81d8b22

move storage and cleanup imports

e8d0c87

Merge branch 'generate-crosswalk-and-import-cap-pdf' of github.com:jt…

d75d66a

…mst/courtlistener into generate-crosswalk-and-import-cap-pdf

[pre-commit.ci] auto fixes from pre-commit.com hooks

3108f2e

for more information, see https://pre-commit.ci

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

1c5ee58

quevon24 reviewed Sep 24, 2024

View reviewed changes

jtmst added 2 commits September 25, 2024 09:33

missing doc string

c78db4a

Merge branch 'generate-crosswalk-and-import-cap-pdf' of github.com:jt…

1212771

…mst/courtlistener into generate-crosswalk-and-import-cap-pdf

flooie reviewed Sep 27, 2024

View reviewed changes

cl/search/management/commands/generate_cap_crosswalk.py Outdated Show resolved Hide resolved

update find_matching_case to utilize filepath cap id

d5db307

jtmst requested review from quevon24 and flooie September 30, 2024 17:42

quevon24 approved these changes Sep 30, 2024

View reviewed changes

quevon24 and others added 4 commits September 30, 2024 15:19

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

836cb03

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

7ee2caf

[pre-commit.ci] auto fixes from pre-commit.com hooks

7921a6e

for more information, see https://pre-commit.ci

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

dd60166

flooie approved these changes Oct 2, 2024

View reviewed changes

mlissner requested changes Oct 2, 2024

View reviewed changes

cl/search/crosswalks/example_crosswalk.json Outdated Show resolved Hide resolved

cl/search/management/commands/generate_cap_crosswalk.py Outdated Show resolved Hide resolved

cl/settings/third_party/r2.py Outdated Show resolved Hide resolved

quevon24 added 5 commits October 2, 2024 17:59

fix(cap): make output-dir arg required and remove default

697ef68

move cap env variables to misc.py

Merge branch 'main' into generate-crosswalk-and-import-cap-pdf

90cd41d

refactor(cap): delete example_crosswalk.json and add it as a comment …

8aa66d8

…in the command

Merge remote-tracking branch 'jtmst/generate-crosswalk-and-import-cap…

f4f5ca4

…-pdf' into generate-crosswalk-and-import-cap-pdf

test(cap): update crosswalk test

08358d8

mlissner approved these changes Oct 3, 2024

View reviewed changes

mlissner merged commit ea77adb into freelawproject:main Oct 3, 2024
9 checks passed

cweider mentioned this pull request Oct 25, 2024

feat(harvard_merger): add harvard_id field to OpinionCluster #4622

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate crosswalk and import cap pdf #4442

Generate crosswalk and import cap pdf #4442

jtmst commented Sep 11, 2024 •

edited

Loading

FRodriguez18 left a comment

mlissner commented Sep 17, 2024

quevon24 left a comment

quevon24 left a comment

quevon24 Sep 19, 2024

jtmst Sep 25, 2024

jtmst Sep 30, 2024 •

edited

Loading

quevon24 left a comment

mlissner commented Oct 1, 2024

flooie left a comment

mlissner left a comment

flooie commented Oct 3, 2024

mlissner commented Oct 3, 2024

quevon24 commented Oct 3, 2024

mlissner commented Oct 3, 2024

Generate crosswalk and import cap pdf #4442

Generate crosswalk and import cap pdf #4442

Conversation

jtmst commented Sep 11, 2024 • edited Loading

Import Harvard Case Law Access Project (CAP) PDFs to CourtListener

Issues Addressed:

Changes Implemented:

Testing Instructions:

Prerequisites:

Steps:

Verification:

Screenshots:

CAP Crosswalk File (Sample Data):

Imported PDFs (Local Storage):

FRodriguez18 left a comment

Choose a reason for hiding this comment

mlissner commented Sep 17, 2024

quevon24 left a comment

Choose a reason for hiding this comment

quevon24 left a comment

Choose a reason for hiding this comment

quevon24 Sep 19, 2024

Choose a reason for hiding this comment

jtmst Sep 25, 2024

Choose a reason for hiding this comment

jtmst Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

quevon24 left a comment

Choose a reason for hiding this comment

mlissner commented Oct 1, 2024

flooie left a comment

Choose a reason for hiding this comment

mlissner left a comment

Choose a reason for hiding this comment

flooie commented Oct 3, 2024

mlissner commented Oct 3, 2024

quevon24 commented Oct 3, 2024

mlissner commented Oct 3, 2024

jtmst commented Sep 11, 2024 •

edited

Loading

jtmst Sep 30, 2024 •

edited

Loading