Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Worldcat fetcher #6035

Closed
wants to merge 58 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
7d72fd8
Add performSearch test case
GlennOlsson Feb 24, 2020
ee96fdc
Add basics of fetcher
GlennOlsson Feb 25, 2020
61a45fe
Start work on importer
GlennOlsson Feb 25, 2020
b257085
Create worldcat importer test
GlennOlsson Feb 26, 2020
27069f4
Clean up and document worldcat fetcher and importer
GlennOlsson Feb 27, 2020
caa7076
Add worldcat fetcher to webFetchers
GlennOlsson Feb 27, 2020
d303999
Fix tests
GlennOlsson Feb 27, 2020
e0ab537
Change to java.net.URL and fix formatting
GlennOlsson Mar 29, 2020
8503720
Move XML tests to own files
GlennOlsson Mar 29, 2020
c6b3e26
Add preferences field
GlennOlsson Mar 29, 2020
2e631c3
Fick checkstyle for main
GlennOlsson Mar 29, 2020
6a5db50
Fix checkstyle test
GlennOlsson Mar 29, 2020
6db97d7
Add to eng. language file
GlennOlsson Mar 29, 2020
f80f2fc
Update CHANGELOG.md
GlennOlsson Mar 29, 2020
b78b387
Update src/main/java/org/jabref/logic/importer/fileformat/WorldcatImp…
GlennOlsson Mar 31, 2020
bf6cb11
Update style and pass worldcat key directly
GlennOlsson Mar 31, 2020
85f01e0
Merge branch 'worldcat-fetcher' of github.com:augustjanse/jabref into…
GlennOlsson Mar 31, 2020
676f8e4
Update to non-static
GlennOlsson Mar 31, 2020
c6e2ea4
Update src/main/java/org/jabref/logic/importer/fetcher/WorldcatFetche…
GlennOlsson Apr 1, 2020
c7ef5f2
Update src/main/java/org/jabref/logic/importer/fetcher/WorldcatFetche…
GlennOlsson Apr 1, 2020
a6ed4e7
Update src/main/java/org/jabref/logic/importer/fetcher/WorldcatFetche…
GlennOlsson Apr 1, 2020
b8b91a4
Update src/main/java/org/jabref/logic/importer/fetcher/WorldcatFetche…
GlennOlsson Apr 1, 2020
f9fd6b7
Change name of apiKey
GlennOlsson Apr 1, 2020
ca3fd6e
Fix checkstyle
GlennOlsson Apr 1, 2020
8056dec
Fix changelog duplicate
GlennOlsson Apr 1, 2020
c1ed7cd
Parse XML in fetcher and send new XML to importer
GlennOlsson Apr 7, 2020
a8cb706
Change test-xml for importer
GlennOlsson Apr 7, 2020
5563170
Fix checkstyle
GlennOlsson Apr 7, 2020
bdf544c
Merge branch 'master' into worldcat-fetcher
GlennOlsson Apr 7, 2020
40f7f16
Fix duplication in CHANGELOG
GlennOlsson Apr 7, 2020
6725481
Wrapp bufferedReader and Stringwriter, and remove TODO comment
GlennOlsson Apr 7, 2020
0e73f64
Add descritpion to preferences
GlennOlsson Apr 10, 2020
434a00e
Fix traling spaces
GlennOlsson Apr 10, 2020
65f6de8
Change to https
GlennOlsson Apr 10, 2020
69da970
Remove null test
GlennOlsson Apr 10, 2020
f8d989a
Fix locality issue
GlennOlsson Apr 10, 2020
959f64a
Add link to worldcat as hyperlink
GlennOlsson Apr 12, 2020
2fa6fc4
Merge remote-tracking branch 'upstream/master' into worldcat-fetcher
calixtus Feb 1, 2021
fbd3003
Fixed merge errors and minor ide warnings
calixtus Feb 1, 2021
35b03ce
Fixed checkstyle errors and minor ide warnings
calixtus Feb 1, 2021
f32931d
Fixed test
calixtus Feb 1, 2021
025350b
Fix typo
koppor Feb 1, 2021
c0b5ab9
Fix changelog
koppor Feb 1, 2021
c134fb6
Merge remote-tracking branch 'upstream/master' into worldcat-fetcher
calixtus Feb 1, 2021
1f31ca8
Updated class packages
calixtus Feb 1, 2021
4b607a3
Merge remote-tracking branch 'augustjanse/worldcat-fetcher' into worl…
calixtus Feb 1, 2021
e3e1c6e
Merge remote-tracking branch 'upstream/master' into worldcat-fetcher
calixtus Feb 10, 2021
3ff24fb
Added WorldCat key secret to BuildInfo and fixed some minor IDE sugge…
calixtus Feb 10, 2021
194e3a8
Fixed http 505 response from server
calixtus Feb 11, 2021
6138fc6
Fixed dom import error
calixtus Feb 12, 2021
6503f87
Changed to use secret api key as default
calixtus Feb 13, 2021
97ec0f2
Merge branch 'master' into worldcat-fetcher
calixtus Mar 8, 2021
5612216
Merge remote-tracking branch 'upstream/main' into worldcat-fetcher
calixtus Jun 22, 2021
07d70d2
Fixed merge error
calixtus Jun 22, 2021
ea35891
Merge remote-tracking branch 'augustjanse/worldcat-fetcher' into worl…
calixtus Jul 7, 2021
9b95add
Merge remote-tracking branch 'upstream/main' into worldcat-fetcher
calixtus Jul 7, 2021
bb6257e
Work in progress
calixtus Jul 7, 2021
c59235d
Work in Progress
calixtus Jul 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/deployment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ env:
SpringerNatureAPIKey: ${{ secrets.SpringerNatureAPIKey }}
AstrophysicsDataSystemAPIKey: ${{ secrets.AstrophysicsDataSystemAPIKey }}
IEEEAPIKey: ${{ secrets.IEEEAPIKey }}
WorldCatAPIKey: ${{ secrets.WorldCatAPIKey }}
OSXCERT: ${{ secrets.OSX_SIGNING_CERT }}
GRADLE_OPTS: -Dorg.gradle.daemon=false -Dorg.gradle.vfs.watch=false

Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Note that this project **does not** adhere to [Semantic Versioning](http://semve
- We added some symbols and keybindings to the context menu in the entry editor. [#7268](https://github.com/JabRef/jabref/pull/7268)
- We added keybindings for setting and clearing the read status. [#7264](https://github.com/JabRef/jabref/issues/7264)
- We added two new fields to track the creation and most recent modification date and time for each entry. [koppor#130](https://github.com/koppor/jabref/issues/130)
- We added a fetcher for [WorldCat](https://www.worldcat.org/). [#1065](https://github.com/JabRef/jabref/issues/1065) [#2581](https://github.com/JabRef/jabref/issues/2581)
- We added a feature that allows the user to copy highlighted text in the preview window. [#6962](https://github.com/JabRef/jabref/issues/6962)
- We added a feature that allows you to create new BibEntry via paste arxivId [#2292](https://github.com/JabRef/jabref/issues/2292)
- We added support for conducting automated and systematic literature search across libraries and git support for persistence [#369](https://github.com/koppor/jabref/issues/369)
Expand Down
1 change: 1 addition & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,7 @@ processResources {
"springerNatureAPIKey": System.getenv('SpringerNatureAPIKey') ? System.getenv('SpringerNatureAPIKey') : '',
"astrophysicsDataSystemAPIKey": System.getenv('AstrophysicsDataSystemAPIKey') ? System.getenv('AstrophysicsDataSystemAPIKey') : '',
"ieeeAPIKey": System.getenv('IEEEAPIKey') ? System.getenv('IEEEAPIKey') : '',
"worldCatAPIKey": System.getenv('worldCatAPIKey') ? System.getenv('worldCatAPIKey') : '',
"scienceDirectApiKey": System.getenv('SCIENCEDIRECTAPIKEY') ? System.getenv('SCIENCEDIRECTAPIKEY') : ''
)
filteringCharset = 'UTF-8'
Expand Down
3 changes: 2 additions & 1 deletion docs/advanced-reading/fetchers.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ Fetchers are the implementation of the [search using online services](https://do
| [SAO/NASA Astrophysics Data System](https://docs.jabref.org/collect/import-using-online-bibliographic-database#sao-nasa-astrophysics-data-system) | [ADS UI](https://ui.adsabs.harvard.edu/user/settings/token) | `AstrophysicsDataSystemAPIKey` | 5000 calls/day |
| [ScienceDirect](https://www.sciencedirect.com/) | | `ScienceDirectApiKey` | |
| [Springer Nature](https://docs.jabref.org/collect/import-using-online-bibliographic-database#springer) | [Springer Nature API Portal](https://dev.springernature.com/) | `SpringerNatureAPIKey` | 5000 calls/day |
| [WorldCat](https://www.worldcat.org/) | [OCLC Developer Network](https://www.oclc.org/developer/develop/authentication/how-to-request-a-wskey.en.html) | `worldCatAPIKey` | \(none\) |
| [Zentralblatt Math](https://www.zbmath.org/) | \(none\) | \(none\) | Depending on the current network |

"Depending on the current network" means that it depends whether your request is routed through a network having paid access. For instance, some universities have subscriptions to MathSciNet.
"Depending on the current network" means that it depends on whether your request is routed through a network having paid access. For instance, some universities have subscriptions to MathSciNet.

On Windows, you have to log-off and log-on to let IntelliJ know about the environment variable change. Execute the gradle task "processResources" in the group "others" within IntelliJ to ensure the values have been correctly written. Now, the fetcher tests should run without issues.

Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/jabref/gui/entryeditor/EntryEditor.java
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ private void setupToolBar() {

// Add menu for fetching bibliographic information
ContextMenu fetcherMenu = new ContextMenu();
for (EntryBasedFetcher fetcher : WebFetchers.getEntryBasedFetchers(preferencesService.getImportFormatPreferences())) {
for (EntryBasedFetcher fetcher : WebFetchers.getEntryBasedFetchers(preferencesService.getImportFormatPreferences(), preferencesService.getApiKeyPreferences())) {
MenuItem fetcherMenuItem = new MenuItem(fetcher.getName());
fetcherMenuItem.setOnAction(event -> fetchAndMerge(fetcher));
fetcherMenu.getItems().add(fetcherMenuItem);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import org.jabref.gui.preferences.entryeditortabs.CustomEditorFieldsTab;
import org.jabref.gui.preferences.exporter.ExportCustomizationTab;
import org.jabref.gui.preferences.external.ExternalTab;
import org.jabref.gui.preferences.fetcher.FetcherTab;
import org.jabref.gui.preferences.file.FileTab;
import org.jabref.gui.preferences.general.GeneralTab;
import org.jabref.gui.preferences.groups.GroupsTab;
Expand Down
42 changes: 42 additions & 0 deletions src/main/java/org/jabref/gui/preferences/fetcher/FetcherTab.fxml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>

<?import javafx.scene.control.Button?>
<?import javafx.scene.control.CheckBox?>
<?import javafx.scene.control.Label?>
<?import javafx.scene.control.TextField?>
<?import javafx.scene.layout.ColumnConstraints?>
<?import javafx.scene.layout.GridPane?>
<?import javafx.scene.layout.HBox?>
<?import javafx.scene.layout.RowConstraints?>
<?import javafx.scene.layout.VBox?>
<?import org.jabref.gui.icon.JabRefIconView?>
<fx:root spacing="10.0" type="VBox" prefWidth="650.0"
xmlns="http://javafx.com/javafx" xmlns:fx="http://javafx.com/fxml"
fx:controller="org.jabref.gui.preferences.fetcher.FetcherTab">

<Label styleClass="titleHeader" text="%Fetcher API keys"/>

<Label text="Use custom API keys"/>
<GridPane hgap="10.0" vgap="4.0">
<columnConstraints>
<ColumnConstraints hgrow="SOMETIMES" percentWidth="30.0"/>
<ColumnConstraints hgrow="SOMETIMES"/>
</columnConstraints>
<rowConstraints>
<RowConstraints minHeight="10.0" />
<RowConstraints minHeight="10.0" />
</rowConstraints>
<CheckBox fx:id="useWorldcatKey" text="%WorldCat" GridPane.rowIndex="0" GridPane.columnIndex="0"/>
<TextField fx:id="worldcatKey" disable="${!useWorldcatKey.selected}"
GridPane.rowIndex="0" GridPane.columnIndex="1"/>
<HBox alignment="BASELINE_LEFT" spacing="4.0" onMouseClicked="#openWorldcatWebpage"
GridPane.rowIndex="1" GridPane.columnIndex="1">
<Button styleClass="icon-button,narrow" onAction="#openWorldcatWebpage" text="%Request a key on the WorldCat platform."
prefHeight="15.0" GridPane.columnIndex="2">
<graphic>
<JabRefIconView glyph="LINK_VARIANT"/>
</graphic>
</Button>
</HBox>
</GridPane>
</fx:root>
41 changes: 41 additions & 0 deletions src/main/java/org/jabref/gui/preferences/fetcher/FetcherTab.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
package org.jabref.gui.preferences.fetcher;

import javafx.fxml.FXML;
import javafx.scene.control.CheckBox;
import javafx.scene.control.TextField;

import org.jabref.gui.preferences.AbstractPreferenceTabView;
import org.jabref.gui.preferences.PreferencesTab;
import org.jabref.logic.l10n.Localization;

import com.airhacks.afterburner.views.ViewLoader;

public class FetcherTab extends AbstractPreferenceTabView<FetcherTabViewModel> implements PreferencesTab {

@FXML private CheckBox useWorldcatKey;
@FXML private TextField worldcatKey;

public FetcherTab() {
ViewLoader.view(this)
.root(this)
.load();
}

@FXML
private void initialize() {
viewModel = new FetcherTabViewModel(preferencesService);

useWorldcatKey.selectedProperty().bindBidirectional(viewModel.getUseWorldcatKeyProperty());
worldcatKey.textProperty().bindBidirectional(viewModel.getWorldcatKeyProperty());
}

@Override
public String getTabName() {
return Localization.lang("Fetcher API keys");
}

@FXML
private void openWorldcatWebpage() {
viewModel.openWorldcatWebpage();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
package org.jabref.gui.preferences.fetcher;

import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import javafx.beans.property.BooleanProperty;
import javafx.beans.property.SimpleBooleanProperty;
import javafx.beans.property.SimpleStringProperty;
import javafx.beans.property.StringProperty;

import org.jabref.gui.desktop.JabRefDesktop;
import org.jabref.gui.preferences.PreferenceTabViewModel;
import org.jabref.logic.importer.FetcherApiPreferences;
import org.jabref.model.strings.StringUtil;
import org.jabref.preferences.PreferencesService;

public class FetcherTabViewModel implements PreferenceTabViewModel {

private final String WORLDCAT_REQUEST_URL = "https://platform.worldcat.org/wskey";

private final BooleanProperty useWorldcatKeyProperty = new SimpleBooleanProperty();
private final StringProperty worldcatKeyProperty = new SimpleStringProperty();

private final PreferencesService preferencesService;
private final FetcherApiPreferences initialFetcherApiPreferences;

public FetcherTabViewModel(PreferencesService preferencesService) {
this.preferencesService = preferencesService;
this.initialFetcherApiPreferences = preferencesService.getApiKeyPreferences();
}

@Override
public void setValues() {
if (StringUtil.isNotBlank(initialFetcherApiPreferences.getWorldcatKey())) {
useWorldcatKeyProperty.setValue(true);
worldcatKeyProperty.setValue(initialFetcherApiPreferences.getWorldcatKey());
}
}

@Override
public void storeSettings() {
Map<String, String> keys = new HashMap<>();

keys.put("worldcat", useWorldcatKeyProperty.getValue() ? worldcatKeyProperty.getValue() : "");

preferencesService.storeApiKeyPreferences(new FetcherApiPreferences(
keys.get("worldcat"))
);
}

public void openWorldcatWebpage() {
JabRefDesktop.openBrowserShowPopup(WORLDCAT_REQUEST_URL);
}

@Override
public boolean validateSettings() {
return true;
}

@Override
public List<String> getRestartWarnings() {
return Collections.emptyList();
}

public BooleanProperty getUseWorldcatKeyProperty() {
return useWorldcatKeyProperty;
}

public StringProperty getWorldcatKeyProperty() {
return worldcatKeyProperty;
}
}
13 changes: 13 additions & 0 deletions src/main/java/org/jabref/logic/importer/FetcherApiPreferences.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
package org.jabref.logic.importer;

public class FetcherApiPreferences {
private final String worldcatKey;

public FetcherApiPreferences(String worldcatKey) {
this.worldcatKey = worldcatKey;
}

public String getWorldcatKey() {
return worldcatKey;
}
}
4 changes: 3 additions & 1 deletion src/main/java/org/jabref/logic/importer/WebFetchers.java
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.jabref.logic.importer.fetcher.SpringerFetcher;
import org.jabref.logic.importer.fetcher.SpringerLink;
import org.jabref.logic.importer.fetcher.TitleFetcher;
import org.jabref.logic.importer.fetcher.WorldCatFetcher;
import org.jabref.logic.importer.fetcher.ZbMATH;
import org.jabref.model.entry.field.Field;
import org.jabref.model.entry.field.StandardField;
Expand Down Expand Up @@ -133,13 +134,14 @@ public static SortedSet<IdBasedFetcher> getIdBasedFetchers(ImportFormatPreferenc
/**
* @return sorted set containing entry based fetchers
*/
public static SortedSet<EntryBasedFetcher> getEntryBasedFetchers(ImportFormatPreferences importFormatPreferences) {
public static SortedSet<EntryBasedFetcher> getEntryBasedFetchers(ImportFormatPreferences importFormatPreferences, FetcherApiPreferences fetcherApiPreferences) {
SortedSet<EntryBasedFetcher> set = new TreeSet<>(Comparator.comparing(WebFetcher::getName));
set.add(new AstrophysicsDataSystem(importFormatPreferences));
set.add(new DoiFetcher(importFormatPreferences));
set.add(new IsbnFetcher(importFormatPreferences));
set.add(new MathSciNet(importFormatPreferences));
set.add(new CrossRef());
set.add(new WorldCatFetcher(fetcherApiPreferences.getWorldcatKey()));
set.add(new ZbMATH(importFormatPreferences));
return set;
}
Expand Down
Loading