Let's use CiteURL to hyperlink statutory citations to Cornell, etc!

[raindrum]

Hi everyone! I recently wrote a python library, CiteURL, which can detect (theoretically) all kinds of legal citations in text, and insert hyperlinks to view them on sites like Cornell.

By default, the library recognizes an incomplete mishmash of federal bodies of law, but it's easy to extend it with YAML, including replacing the built-in schemas altogether.

I'd love to help integrate it into CourtListener, but there are a few big decisions to make first.

First, what bodies of should it support before it's worth running it against the whole database? You can find the currently-supported formats here. Where multiple sites publish a body of law, which one is preferred? By default, CiteURL uses mostly the Cornell website, but a lot of federal materials are available on government sites, for instance.

The biggest question is, if we want to implement CiteURL for statutory and other citations, is it worth using it for linking intra-site court case citations as well? This would be a much more complicated change, but it could be worthwhile, since CiteURL supports shortform citations and pincites.

What do folks think?

EDIT: After reading through #1521 and learning a bit more about how CiteURL already handles court opinion matching, I hesitate to suggest replacing it. But I do think CiteURL would be very useful for other kinds of citations.

[mlissner]

This is great, @raindrum, sorry for the slow reply. So your list of supported materials looks pretty great. Some replies:

• Continuing to use eyecite (the new name for our citation parser), is probably our preference for case law citations, since it has many years of tweaking and enhancement.
• Using cornell is generally ideal for us. They're great, and we'd be happy to link to them.
• I don't think I saw links to the constitution in here.
• More state statutes would probably be good. Justia is doing a lot with these right now. I think we might even be able to link to specific editions of the statutes, which would be nice. I.e., link to the 1977 version instead of the 2021 version, if that makes sense and it exists.
• A part of me wishes that this code were integrated into eyecite instead of its own library. It feels pretty clunky to have two really big citation libraries running against the same text. For example, I could see id cites working better in a system where all types of citations were being parsed at the same time. Would you consider merging this into eyecite? I'm curious how hard it would be, but I haven't looked at your code.
• As far as getting this into CL goes, it actually shouldn't be that hard. We have an ingestion pipeline, and so we'd want to just add this as part of the pipeline. The only thing that concerns me there is performance. Yanking out citations is already a big use of CPU time, and I imagine this would make it worse, but perhaps it wouldn't be too bad.
• I wonder how this would affect our SEO. Something to think about. Maybe we just would want to put noreferrer on all the outbound links. If we're not careful, we might look like a link farm and get heavily demoted.
• One final thing to think about is that we're not currently finding citations in the RECAP DB. it's on our list to do that eventually. Probably not as part of this, but it's something to have in mind.

cc: @jcushman, @mattdahl, @flooie for any additional thoughts.

[mlissner]

Oh, one other thing: This is really neat! I think this would be really helpful for folks.

[mattdahl]

To chime in, I agree that making this play nicely with our existing parsing logic would be ideal. As @mlissner says, doing so would do wonders for helping CL resolve Id. citations. Right now we just use a dummy class to keep track of when possible "non-opinion citations" appear (https://github.com/freelawproject/eyecite/blob/master/eyecite/models.py#L241) -- so we don't erroneously link Id. citations to them -- but this implementation is very brittle.

@raindrum your code looks very nice to me and has some nice abstractions. However, I fear that it is paradigmatically different from the approach that eyecite takes, for better or worse. eyecite iterates over each token in the string one at a time, building out a citation by scanning left and right; CiteURL seems to work (on my reading) by matching pre-defined regexes to the string using finditer(). I fear that this makes the packages difficult to combine.

As I wrote in #931 (comment), I once attempted to do something similar, by way of integrating all these regexes here: https://github.com/freelawproject/citation-regexes/. The approach I took was to make two passes through each opinion, one using our existing citation logic (now eyecite) and one doing a bunch of regex matching (which I believe is your approach). In order to combine the results of each pass, I kept a record of the character index in the opinion string at which each extracted citation (by either method) was found. I then combined the results of each pass and sorted the extracted citations by those indices, so everything was in proper order, and so Id. citations would resolve to their proper antecedents. However, I never merged this code, as it was either not performant or the regexes I were using were not robust enough. (I can't remember, maybe I encountered some other problem too.)

I hope that's helpful! There are surely other strategies to pursue as well. Your package looks really interesting and I would love to see it used on CL!

[jcushman]

For what it's worth I believe eyecite should end up using regexes to extract pages and volumes as well, and then scan tokens only for the court/year/case name stuff. That'll make its architecture closer to CiteURL's.

There's a bunch of edge cases that are hard to handle with the current approach -- dashes like 2020-NMSC-01 (currently only handled in the trivial case with no other text), multiword volume and page numbers like25 Supp. Tex. 1 (currently not handled), out-of-order cites like T. C. Memo. 2020-01 (some handled with special cases in post-processing, some not). If the tokenizer starts tokenizing "foo 2020-NMSC-01 bar" as [Token("foo"), CiteToken(reporter=Reporter(), volume="2020", reporter_str="NMSC", page="01", raw_cite="2020-NMSC-01"), Token("bar")], extracted via regex, then the edge cases I know of can be handled via regex configuration instead of code ... and it gets a lot easier to throw in non-case extraction.

I believe this architecture will be slower when using the vanilla python regex library, but it'll actually be >10x faster than the current library when using hyperscan on x86, where throwing in additional regexes for additional citation formats is essentially free. (There might also be a performance middle ground using other libraries like re2 or pyahocorasick, though I'm not sure if those end up working out.) I'll work on trying to nail down the performance implications.

[raindrum]

Thanks for the feedback, all! I just put out an update to CiteURL that addresses some of @mlissner's concerns, and others:

• Added support for US constitution
• It's now possible to specify a rel attribute (e.g. "noreferrer") for generated links
• You can provide CiteURL's insert_links function with a list of indices in the string, where chains of id. citations should be interrupted. You can also provide a regex that does the same thing. Unfortunately this won't help eyecite skip non-opinion citations, but it at least should let CiteURL skip eyecite citations if you run CiteURL as a postprocessor.

I'm open to the idea of merging CiteURL into eyecite if folks think it's feasible, but I'm not really familiar with how eyecite works, apart from @mattdahl's summary. In case it helps, CiteURL's basic process is this:

1. Create a citator from a set of citation schemas (i.e. regexes and tools to generate URLs and shortform citations from the matches)
2. Run each schema against the input text to get a list of all long-form citations.
3. Generate shortform schemas (but not id. form schemas) from each long-form citation.
4. Match each shortform schema against all text after its associated longform citation, and add each resulting citation to the main list of citations.
5. Sort the citation list into index order, and remove any that overlap.
6. Loop through the list of all full or shortform citations, generating id. citations for each one.
7. While there are any id. schemas, do the following:
   1. Match the first id. schema in the list against all the text between its parent citation and the index of the next longform citation, and any results to the list of citations.
   2. Delete that id. schema
   3. Generate one or more id. schemas from any citation it produced, and add them to the beginning of the list.

The purpose of the recursive id. schemas is to let each id. schema match only once, and create the context for the next one. However, that feature doesn't seem to be working, so right now CiteURL just fudges it and uses the context from the most-recent full or shortform citation. Unfortunately that leads to errors where if someone cites 42 USC 1983, then id. at 1988, then id. at (b), it'll generate a link to § 1983(b). I'll try and fix the issue soon.

As for performance, it might be possible to migrate CiteURL to use Hyperscan, but I'm not really familiar enough with the latter to know.

[raindrum]

I've since refactored the code a bit to fix the id. chaining bug mentioned above.

[mlissner]

I just want to write a quick comment, since I'm mostly on vacation this week.

I think we have two really interesting big things happening simultaneously right now, and I think we should make them into just one. Over in freelawproject/eyecite#10, @jcushman has a big PR that I've briefly looked at which overhauls the approach of eyecite. I think it accomplishes a lot of what he describes above and that it will make eyecite a better, more flexible tool.

I haven't thought carefully yet about how this project could get folded into eyecite, but I'm happy to hear that you're willing and interested, @raindrum. I think my suggestion would be that we merge Jack's big PR, and that we then do the work of integrating this. Between the two, I think we could have something wildly neat.