Add Client (external API) Module For Enhanced Metadata

[mskarlin]

This adds a new module: paperqa.clients which provides base classes for both providing (MetadataProvider) and processing (MetadataPostProcessor) Doc metadata. Two metadata providers are included: CrossrefProvider and SemanticScholarProvider and one processor: JournalQualityPostProcessor (which adds in a quality score for each extracted journal). DocMetadataClient then orchestrates all the usage of these providers, so a user doesn't need to worry about where metadata comes from, they just need to request it. Usage examples make this simpler:

async with aiohttp.ClientSession() as session:
    client = DocMetadataClient(session)
    doc_details = await client.query(title="PaperQA: Retrieval-Augmented Generative Agent for Scientific Research")

The doc_details (a DocDetails object) is a subclass of a Doc with extra metadata and the ability to be merged with the __add__ operator. In this way, users can use whichever MetadataProvider's they want and they will get a resultant DocDetails object which fills in whatever fields they can get. DocDetails have rich metadata like:

print(doc_details.formatted_citation) 
>>>Jakub L'ala, Odhran O'Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, and Andrew D. White. Paperqa: retrieval-augmented generative agent for scientific research. ArXiv, Dec 2023. doi:10.48550/arxiv.2312.07559. This article has 21 citations.

The DocMetadataClient.query method will accept any valid ClientQuery supported by the MetadataProviders you have selected (by default all). But we've only implemented a DOIQuery and a TitleAuthorQuery for now. It will return all fields by default, but you can also filter for whatever you'd like:

async with aiohttp.ClientSession() as session:
        # now get with authors just from one source
        s2_client = DocMetadataClient(session, clients=[SemanticScholarProvider])
        s2_details = await s2_client.query(
            title="Augmenting large language models with chemistry tools",
            fields=["title", "doi", "authors"],
        )

The above example will only query SemanticScholar (not Crossref), and it'll only return the title, doi, and authors for that paper. This is useful if you need to a small lookup for DOI or something along those lines. The processors run by default, but if you don't specify them in the clients argument they won't run, like in the above example. So to run with JournalQualityPostProcessor enabled:

async with aiohttp.ClientSession() as session:
        # now get with authors just from one source
        s2_client = DocMetadataClient(session, clients=[SemanticScholarProvider, JournalQualityPostProcessor])
        s2_details = await s2_client.query(
            title="Augmenting large language models with chemistry tools",
            fields=["title", "doi", "authors", "journal"],
        )

Note we had to add "journal" as a field, otherwise the JournalQualityPostProcessor would have had no journal to work from.

The metadata client has been added into Docs.aadd method -- so if you set the use_doc_details flag to true, your other inputs will be used to get rich metadata using a DocDetails object rather than the standard Doc object.

Tests are included which use the pytest-vcr module and cassettes to avoid needing network access. Secrets are filtered from the cassettes by default.

[jamesbraza]

Bonus points for comments documenting why each of these were chosen

[jamesbraza]

Wdyt of moving this to module level?

[mskarlin]

I think this is specific to bibtex, no?

[whitead]

Should we just change these names? Now would be the time

[mskarlin]

I don't know if if we want to break backwards compatibility quite yet-- I'm for keeping and having extra duplicative fields.

[whitead]

So fucking cool - great work