Full text crawlers

[stefan-kolb]

This PR enables automatic PDF fulltext downloads.

Current catalogs:

• ACS
• arXiv
• Springer
• Sciencedirect (Elsevier)
• Google Scholar

Questions:

• Do we want to include more Crawlers? If yes, which?
• Action was in the Tools menu. Imho it should be either auto downloaded or included with the downloador auto button in the detailed entry view.

TODO:

• Make it nonblocking and keep the progress bar!
• Add an option to delete the local file if a wrong file was downloaded
• Tests on Travis & CircleCI are blocked by Google by 403 Forbidden (Bot detection)

[mlep]

About "Do we want to include more Crawlers? If yes, which?":
You have a gold mine in the section "Search (and import) from my specific database, please." in the sorted list of feature requests (this morning email).

[stefan-kolb]

@mlep Thanks 👍

[simonharrer]

Your feature works pretty well. But there are cases in which a file can be downloaded but it is not the file that is intended. Thus, there must be a way to preview the pdf before adding it.

[stefan-kolb]

We need to evaluate how often this will happen. Can you give me an example?

I think with a reasonable amount of crawlers this will work with a very high accuracy.
I would rather have the user open and check the file and then delete it and manually download it if it is not the correct file, as this will happen very seldom. If we use your approach we need to preview every PDF that will be anyway in > 90% of the times the correct one.

[simonharrer]

Ok, deletion after download is OK when it is done easily. At the moment, I cannot easily delete the file entry from the bibtex entry AND also delete the file from the hard drive. When we fix this, we do not need the check feature in the first place. Good idea!

My example was: I wanted to download the conference article Barros, Service Interaction Patterns, but got the techrep instead. Another issue was that I got a PDF of slides in some language I do not understand, probably czech when attempting to download a very old version of the BPMN standard.

[koppor]

Regarding, "clean all used fields from Latex stuff, e.g. {~ etc.", this should be one of the "Edit -> Cleanup entries" functionality, isn't it? If not, this functionality should be added there, too.

Regarding "check for duplicates", JabRef offers "Search -> find duplicates". This functionality should be reused here.

[koppor]

A similar approach might have been taken by Christoph Lehner: https://sourceforge.net/p/jabref/discussion/318824/thread/6e5fea64/ Is it possible to synchronize somehow?
I think, this makes https://github.com/wbrenna/LocalCopy obsolete, doesn't it?

[stefan-kolb]

The approach of C.Lehner is the base repository https://github.com/lehner/LocalCopy.
This PR is a native replacement for this and all forks.
Also, we don't support plugins anymore.

[lenhard]

What is missing to get this PR functional and integrated into master?

[simonharrer]

This is just too messy to understand right now. Fixing this would require fixing the separation of GUI Event Thread with other Code parts - a major effort.

The issue is how to implement swing actions that require multiple user interactions during their task taking place. Normally, this would require a nesting of swing worker classes, one for each step and starting the next step within the EDT update method.

What is more, sometimes SwingWorker, and sometimes the spin Framework is used. All of this makes this issue even more complicated.

Someone else may take a look if they see this issue more clearly than me.

[lenhard]

So you are saying that you will not complete the PR? I cannot see anybody else who will. There is no point of having this hanging around in limbo until it deprecates. So we can close this PR without merging and close related open issues as won't fix.

@simonharrer: Please confirm!

[simonharrer]

Merge as good enough.

[lecc0r]

Hi, I've just tested the newly implemented "Full text article download" function, however for several articles it downloaded a version from ResearchGate but I want to have the one from Elsevier. Dependent who you are working for, you have special access privileges on certain publisher sites and it would be great to specify in the preferences which crawler the download function should prefer. Perhaps even define a priority list.

[stefan-kolb]

There is a hard-coded priority list right know which prefers the official publishers over google scholar for example. If you want to file a new feature request or enhancement please create a separate issue.

[koppor]

Done at #435.