Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for issue #4652: Add Find Unlinked Files Filter based on Date #7846

Merged
merged 67 commits into from
Jun 28, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
afd9ba6
Display last edited date along with filename
gliargovas May 5, 2021
3d5c737
Create DateRange object
gliargovas Jun 13, 2021
318e79d
Add ExternalFilesDateViewModel
gliargovas Jun 13, 2021
59d5bfd
Add getDisplayTextWithEditDate method
gliargovas Jun 13, 2021
aed71d4
Create File Date Combobox
gdrosos Jun 21, 2021
be11dab
Fix typo
gdrosos Jun 21, 2021
3b9d286
Add Functionality of FileDateCombo box
gdrosos Jun 21, 2021
3f7aa41
Add ViewModel Functionality for the Last Edited Filter
gdrosos Jun 21, 2021
cb303c5
Modify Unlinked Files Crawler to support Last Edited Filter
gdrosos Jun 21, 2021
04962f0
Add FileFilterUtils.java
gliargovas Jun 21, 2021
c6625bf
Add "Last Edited:" to l10n package
gliargovas Jun 21, 2021
ec54bc3
add External File Sorter Class
gdrosos Jun 21, 2021
c099ce0
Add FileSortViewModel
gliargovas Jun 22, 2021
e6e15d5
Add Sort by combo box
gliargovas Jun 22, 2021
1e9ec13
Add fileSortCombo view side code
gliargovas Jun 22, 2021
639aa8e
Add sort by date functionality methods
gliargovas Jun 22, 2021
291d645
Add View Model implementation for sort by date field
gdrosos Jun 22, 2021
ba7116e
Add Sort Support to Crawler
gdrosos Jun 22, 2021
87c0111
Rename filteredFiles variable in Crawler
gdrosos Jun 22, 2021
7cc1c50
Add "Sort by:" to en localiztion properties
gliargovas Jun 22, 2021
4b3038e
Fix syntax error
gliargovas Jun 22, 2021
7a53d2e
Fix typo in File Filter Utils
gdrosos Jun 22, 2021
18b6df6
Fix incorrect reverse sorting method call
gliargovas Jun 22, 2021
757a8a7
Convert days to months in isDuringLastMonth method
gdrosos Jun 22, 2021
22aef66
Add Unit Tests for isDuring methods in FileFilterUtils
gdrosos Jun 22, 2021
f49a0ae
Improve code quality of FileFilterUtilsTest
gdrosos Jun 22, 2021
07d5174
make variables of FileFilterUtilsTest final
gdrosos Jun 22, 2021
6791053
Leave new line to all files
gdrosos Jun 22, 2021
02017c3
Update Changelog
gdrosos Jun 22, 2021
7fcfff6
Merge branch 'main' into fix-for-issue-4652
gdrosos Jun 22, 2021
91235b4
Improve code quality according to Codacy Code Analysis
gdrosos Jun 23, 2021
e191344
Merge branch 'fix-for-issue-4652' of https://github.com/gdrosos/jabre…
gdrosos Jun 23, 2021
076f1db
Further improve code quality according to Codacy Code Analysis
gdrosos Jun 23, 2021
1e6cca7
Minor Code quality improvement
gdrosos Jun 23, 2021
66db608
Fix Typo
gdrosos Jun 23, 2021
0998abd
Use Logger to log exceptions
gdrosos Jun 25, 2021
fde0180
Use method reference operator
gdrosos Jun 25, 2021
7d83221
Use localization to show localized strings
gdrosos Jun 25, 2021
87d6b9f
Apply suggested changes from Codacy
Jun 25, 2021
aad00fc
Revert "Use method reference operator "
gdrosos Jun 25, 2021
998d52d
Add unlinked files filter fields localisation
gliargovas Jun 25, 2021
6a02336
Merge branch 'fix-for-issue-4652' of https://github.com/gdrosos/jabre…
gliargovas Jun 25, 2021
425c258
Fix typo
gliargovas Jun 25, 2021
482212d
Use meaningful names in lambda expressions
gliargovas Jun 25, 2021
234943a
Refactor code to use DateRange enum instead of strings
gliargovas Jun 26, 2021
4f812a9
Remove wrongly commited code
gliargovas Jun 26, 2021
e949db9
Use ExternalFileSorter Enums instead of Strings
gdrosos Jun 26, 2021
7295535
Ensure the code passes LocalizationConsistencyTests
gdrosos Jun 26, 2021
9b001f6
Add tests for FileFilterUtils sort methods
gliargovas Jun 27, 2021
ccd8e92
Fix code format issues
gliargovas Jun 27, 2021
c2c7ae6
Update FileFilterUtilsTests with minor fixes
gdrosos Jun 27, 2021
bd9dbf7
Add negative sort tests and improve code format
gliargovas Jun 27, 2021
1a58342
Use Path.of() instead of Paths.get()
gliargovas Jun 27, 2021
08f0f19
Fix FileFilterUtilsTest code format
gliargovas Jun 27, 2021
4db03ee
Add documentation to UnlinkedFilesCrawler and FileFilterUtils
gliargovas Jun 27, 2021
3825ee9
Add comment descriptions to FileFilterUtilsTest
gliargovas Jun 27, 2021
7762cb9
Replace minusMonths() and minusYears() with minusDays()
gliargovas Jun 27, 2021
6e721c5
Use TempDir at Unit Tests
gdrosos Jun 27, 2021
a495c16
Remove the use of java.io.File
gliargovas Jun 27, 2021
a6f91c0
Use streams for sort tests
gliargovas Jun 27, 2021
5136b1f
Remove unused imports
gliargovas Jun 27, 2021
0b459a6
Use enums directly for filter and sort fields
gliargovas Jun 27, 2021
226010b
Remove redundant default cases from FileFilterUtils.java
gliargovas Jun 27, 2021
1e2f8c4
replace DateRange enums with values() method
gdrosos Jun 27, 2021
3334398
Replace ExternalFileSorter enums with values() method in UnlinkedFile…
gdrosos Jun 27, 2021
4ae48a1
Place the correct order in DateTime enums
gdrosos Jun 27, 2021
4a00080
Make sure a value is returned when an IOException occurs in getFileTi…
gdrosos Jun 28, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ Note that this project **does not** adhere to [Semantic Versioning](http://semve
- We added a progress counter to the title bar in Possible Duplicates dialog window. [#7366](https://github.com/JabRef/jabref/issues/7366)
- We added new "Customization" tab to the preferences which includes option to choose a custom address for DOI access. [#7337](https://github.com/JabRef/jabref/issues/7337)
- We added zbmath to the public databases from which the bibliographic information of an existing entry can be updated. [#7437](https://github.com/JabRef/jabref/issues/7437)
- We showed to the find Unlinked Files Dialog the date of the files' most recent modification. [#4652](https://github.com/JabRef/jabref/issues/4652)
- We added to the find Unlinked Files function a filter to show only files based on date of last modification (Last Year, Last Month, Last Week, Last Day). [#4652](https://github.com/JabRef/jabref/issues/4652)
- We added to the find Unlinked Files function a filter that sorts the files based on the date of last modification(Sort by Newest, Sort by Oldest First). [#4652](https://github.com/JabRef/jabref/issues/4652)
- We added the possibility to add a new entry via its zbMath ID (zbMATH can be chosen as ID type in the "Select entry type" window). [#7202](https://github.com/JabRef/jabref/issues/7202)
- We added the extension support and the external application support (For Texshow, Texmaker and LyX) to the flatpak [#7248](https://github.com/JabRef/jabref/pull/7248)
- We added some symbols and keybindings to the context menu in the entry editor. [#7268](https://github.com/JabRef/jabref/pull/7268)
Expand Down
21 changes: 21 additions & 0 deletions src/main/java/org/jabref/gui/externalfiles/DateRange.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package org.jabref.gui.externalfiles;

import org.jabref.logic.l10n.Localization;

public enum DateRange {
ALL_TIME(Localization.lang("All time")),
YEAR(Localization.lang("Last year")),
MONTH(Localization.lang("Last month")),
WEEK(Localization.lang("Last week")),
DAY(Localization.lang("Last day"));

private final String dateRange;

DateRange(String dateRange) {
this.dateRange = dateRange;
}

public String getDateRange() {
return dateRange;
}
}
19 changes: 19 additions & 0 deletions src/main/java/org/jabref/gui/externalfiles/ExternalFileSorter.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package org.jabref.gui.externalfiles;

import org.jabref.logic.l10n.Localization;

public enum ExternalFileSorter {
DEFAULT(Localization.lang("Default")),
DATE_ASCENDING(Localization.lang("Newest first")),
DATE_DESCENDING(Localization.lang("Oldest first"));

private final String sorter;

ExternalFileSorter(String sorter) {
this.sorter = sorter;
}

public String getSorter() {
return sorter;
}
}
112 changes: 112 additions & 0 deletions src/main/java/org/jabref/gui/externalfiles/FileFilterUtils.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
package org.jabref.gui.externalfiles;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.FileTime;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class FileFilterUtils {

private static final Logger LOGGER = LoggerFactory.getLogger(FileFilterUtils.class);

/* Returns the last edited time of a file as LocalDateTime. */
public static LocalDateTime getFileTime(Path path) {
FileTime lastEditedTime = null;
try {
lastEditedTime = Files.getLastModifiedTime(path);
} catch (IOException e) {
LOGGER.error("Could not retrieve file time", e);
return LocalDateTime.now();
}
LocalDateTime localDateTime = lastEditedTime
.toInstant()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IntelliJ complains here, that toInstant may throw a NPE. This seems possible, since you don't return when a IOException occurs. Is that something we should think about?

Copy link
Contributor Author

@gdrosos gdrosos Jun 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @calixtus , so maybe we need to place the code in a try catch-block to handle the NPE or in the same try catch block so when an IO Exception occurs the toInstant() method call is omited.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, there must be a value returned from this method... Which one, if getLastModifiedTime fails?
Today? null?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sorry you are right, we will return LocalDateTime.now() when an IOException occurs

.atZone(ZoneId.systemDefault())
.toLocalDateTime();
return localDateTime;
}

/* Returns true if a file with a specific path
* was edited during the last 24 hours. */
public boolean isDuringLastDay(LocalDateTime fileEditTime) {
LocalDateTime NOW = LocalDateTime.now(ZoneId.systemDefault());
return fileEditTime.isAfter(NOW.minusHours(24));
}

/* Returns true if a file with a specific path
* was edited during the last 7 days. */
public boolean isDuringLastWeek(LocalDateTime fileEditTime) {
LocalDateTime NOW = LocalDateTime.now(ZoneId.systemDefault());
return fileEditTime.isAfter(NOW.minusDays(7));
}

/* Returns true if a file with a specific path
* was edited during the last 30 days. */
public boolean isDuringLastMonth(LocalDateTime fileEditTime) {
LocalDateTime NOW = LocalDateTime.now(ZoneId.systemDefault());
return fileEditTime.isAfter(NOW.minusDays(30));
}

/* Returns true if a file with a specific path
* was edited during the last 365 days. */
public boolean isDuringLastYear(LocalDateTime fileEditTime) {
LocalDateTime NOW = LocalDateTime.now(ZoneId.systemDefault());
return fileEditTime.isAfter(NOW.minusDays(365));
}

/* Returns true if a file is edited in the time margin specified by the given filter. */
public static boolean filterByDate(Path path, DateRange filter) {
FileFilterUtils fileFilter = new FileFilterUtils();
LocalDateTime fileTime = FileFilterUtils.getFileTime(path);
boolean isInDateRange = switch (filter) {
case DAY -> fileFilter.isDuringLastDay(fileTime);
case WEEK -> fileFilter.isDuringLastWeek(fileTime);
case MONTH -> fileFilter.isDuringLastMonth(fileTime);
case YEAR -> fileFilter.isDuringLastYear(fileTime);
case ALL_TIME -> true;
};
return isInDateRange;
}

/* Sorts a list of Path objects according to the last edited date
* of their corresponding files, from newest to oldest. */
public List<Path> sortByDateAscending(List<Path> files) {
return files.stream()
.sorted(Comparator.comparingLong(file -> FileFilterUtils.getFileTime(file)
.atZone(ZoneId.systemDefault())
.toInstant()
.toEpochMilli()))
.collect(Collectors.toList());
}

/* Sorts a list of Path objects according to the last edited date
* of their corresponding files, from oldest to newest. */
public List<Path> sortByDateDescending(List<Path> files) {
return files.stream()
.sorted(Comparator.comparingLong(file -> -FileFilterUtils.getFileTime(file)
.atZone(ZoneId.systemDefault())
.toInstant()
.toEpochMilli()))
.collect(Collectors.toList());
}

/* Sorts a list of Path objects according to the last edited date
* the order depends on the specified sorter type. */
public static List<Path> sortByDate(List<Path> files, ExternalFileSorter sortType) {
FileFilterUtils fileFilter = new FileFilterUtils();
List<Path> sortedFiles = switch (sortType) {
case DEFAULT -> files;
case DATE_ASCENDING -> fileFilter.sortByDateDescending(files);
case DATE_DESCENDING -> fileFilter.sortByDateAscending(files);
};
return sortedFiles;
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,16 @@ public class UnlinkedFilesCrawler extends BackgroundTask<FileNodeViewModel> {

private final Path directory;
private final Filter<Path> fileFilter;
private final DateRange dateFilter;
private final ExternalFileSorter sorter;
private final BibDatabaseContext databaseContext;
private final FilePreferences filePreferences;

public UnlinkedFilesCrawler(Path directory, Filter<Path> fileFilter, BibDatabaseContext databaseContext, FilePreferences filePreferences) {
public UnlinkedFilesCrawler(Path directory, Filter<Path> fileFilter, DateRange dateFilter, ExternalFileSorter sorter, BibDatabaseContext databaseContext, FilePreferences filePreferences) {
this.directory = directory;
this.fileFilter = fileFilter;
this.dateFilter = dateFilter;
this.sorter = sorter;
this.databaseContext = databaseContext;
this.filePreferences = filePreferences;
}
Expand All @@ -61,6 +65,9 @@ protected FileNodeViewModel call() throws IOException {
* For ensuring the capability to cancel the work of this recursive method, the first position in the integer array
* 'state' must be set to 1, to keep the recursion running. When the states value changes, the method will resolve
* its recursion and return what it has saved so far.
* <br>
* The files are filtered according to the {@link DateRange} filter value
* and then sorted according to the {@link ExternalFileSorter} value.
*
* @throws IOException if directory is not a directory or empty
*/
Expand Down Expand Up @@ -92,11 +99,19 @@ private FileNodeViewModel searchDirectory(Path directory, UnlinkedPDFFileFilter
parent.getChildren().add(subRoot);
}
}

parent.setFileCount(files.size() + fileCount);
parent.getChildren().addAll(files.stream()
.map(FileNodeViewModel::new)
.collect(Collectors.toList()));
// filter files according to last edited date.
List<Path> filteredFiles = new ArrayList<Path>();
for (Path path : files) {
if (FileFilterUtils.filterByDate(path, dateFilter)) {
filteredFiles.add(path);
}
}
// sort files according to last edited date.
filteredFiles = FileFilterUtils.sortByDate(filteredFiles, sorter);
parent.setFileCount(filteredFiles.size() + fileCount);
parent.getChildren().addAll(filteredFiles.stream()
.map(FileNodeViewModel::new)
.collect(Collectors.toList()));
return parent;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,12 @@

<Label text="%File type" GridPane.columnIndex="0" GridPane.rowIndex="1"/>
<ComboBox fx:id="fileTypeCombo" maxWidth="Infinity" GridPane.columnIndex="1" GridPane.rowIndex="1"/>
<Label text="%Last edited:" GridPane.columnIndex="0" GridPane.rowIndex="2"/>
<ComboBox fx:id="fileDateCombo" maxWidth="Infinity" GridPane.columnIndex="1" GridPane.rowIndex="2"/>
<Label text="%Sort by:" GridPane.columnIndex="0" GridPane.rowIndex="3"/>
<ComboBox fx:id="fileSortCombo" maxWidth="Infinity" GridPane.columnIndex="1" GridPane.rowIndex="3"/>
<Button fx:id="scanButton" onAction="#scanFiles" text="%Search"
GridPane.columnIndex="2" GridPane.rowIndex="1">
GridPane.columnIndex="2" GridPane.rowIndex="3">
<tooltip>
<Tooltip text="%Searches the selected directory for unlinked files."/>
</tooltip>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ public class UnlinkedFilesDialogView extends BaseDialog<Void> {

@FXML private TextField directoryPathField;
@FXML private ComboBox<FileExtensionViewModel> fileTypeCombo;
@FXML private ComboBox<DateRange> fileDateCombo;
@FXML private ComboBox<ExternalFileSorter> fileSortCombo;
@FXML private CheckTreeView<FileNodeViewModel> unlinkedFilesList;
@FXML private Button scanButton;
@FXML private Button exportButton;
Expand Down Expand Up @@ -141,11 +143,23 @@ private void initDirectorySelection() {
fileTypeCombo.setItems(viewModel.getFileFilters());
fileTypeCombo.valueProperty().bindBidirectional(viewModel.selectedExtensionProperty());
fileTypeCombo.getSelectionModel().selectFirst();
new ViewModelListCellFactory<DateRange>()
.withText(DateRange::getDateRange)
.install(fileDateCombo);
fileDateCombo.setItems(viewModel.getDateFilters());
fileDateCombo.valueProperty().bindBidirectional(viewModel.selectedDateProperty());
fileDateCombo.getSelectionModel().selectFirst();
new ViewModelListCellFactory<ExternalFileSorter>()
.withText(ExternalFileSorter::getSorter)
.install(fileSortCombo);
fileSortCombo.setItems(viewModel.getSorters());
fileSortCombo.valueProperty().bindBidirectional(viewModel.selectedSortProperty());
fileSortCombo.getSelectionModel().selectFirst();
}

private void initUnlinkedFilesList() {
new ViewModelTreeCellFactory<FileNodeViewModel>()
.withText(FileNodeViewModel::getDisplayText)
.withText(FileNodeViewModel::getDisplayTextWithEditDate)
.install(unlinkedFilesList);

unlinkedFilesList.maxHeightProperty().bind(((Control) filePane.contentProperty().get()).heightProperty());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ public class UnlinkedFilesDialogViewModel {
private final ImportHandler importHandler;
private final StringProperty directoryPath = new SimpleStringProperty("");
private final ObjectProperty<FileExtensionViewModel> selectedExtension = new SimpleObjectProperty<>();
private final ObjectProperty<DateRange> selectedDate = new SimpleObjectProperty<>();
private final ObjectProperty<ExternalFileSorter> selectedSort = new SimpleObjectProperty<>();

private final ObjectProperty<Optional<FileNodeViewModel>> treeRootProperty = new SimpleObjectProperty<>();
private final SimpleListProperty<TreeItem<FileNodeViewModel>> checkedFileListProperty = new SimpleListProperty<>(FXCollections.observableArrayList());
Expand All @@ -65,6 +67,9 @@ public class UnlinkedFilesDialogViewModel {

private final ObservableList<ImportFilesResultItemViewModel> resultList = FXCollections.observableArrayList();
private final ObservableList<FileExtensionViewModel> fileFilterList;
private final ObservableList<DateRange> dateFilterList;
private final ObservableList<ExternalFileSorter> fileSortList;

private final DialogService dialogService;
private final PreferencesService preferences;
private BackgroundTask<FileNodeViewModel> findUnlinkedFilesTask;
Expand All @@ -90,9 +95,13 @@ public UnlinkedFilesDialogViewModel(DialogService dialogService, ExternalFileTyp
stateManager);

this.fileFilterList = FXCollections.observableArrayList(
new FileExtensionViewModel(StandardFileType.ANY_FILE, externalFileTypes),
new FileExtensionViewModel(StandardFileType.BIBTEX_DB, externalFileTypes),
new FileExtensionViewModel(StandardFileType.PDF, externalFileTypes));
new FileExtensionViewModel(StandardFileType.ANY_FILE, externalFileTypes),
new FileExtensionViewModel(StandardFileType.BIBTEX_DB, externalFileTypes),
new FileExtensionViewModel(StandardFileType.PDF, externalFileTypes));

this.dateFilterList = FXCollections.observableArrayList(DateRange.values());

this.fileSortList = FXCollections.observableArrayList(ExternalFileSorter.values());

Predicate<String> isDirectory = path -> Files.isDirectory(Path.of(path));
scanDirectoryValidator = new FunctionBasedValidator<>(directoryPath, isDirectory,
Expand All @@ -104,11 +113,12 @@ public UnlinkedFilesDialogViewModel(DialogService dialogService, ExternalFileTyp
public void startSearch() {
Path directory = this.getSearchDirectory();
Filter<Path> selectedFileFilter = selectedExtension.getValue().dirFilter();

DateRange selectedDateFilter = selectedDate.getValue();
ExternalFileSorter selectedSortFilter = selectedSort.getValue();
progressValueProperty.unbind();
progressTextProperty.unbind();

findUnlinkedFilesTask = new UnlinkedFilesCrawler(directory, selectedFileFilter, bibDatabase, preferences.getFilePreferences())
findUnlinkedFilesTask = new UnlinkedFilesCrawler(directory, selectedFileFilter, selectedDateFilter, selectedSortFilter, bibDatabase, preferences.getFilePreferences())
.onRunning(() -> {
progressValueProperty.set(ProgressIndicator.INDETERMINATE_PROGRESS);
progressTextProperty.setValue(Localization.lang("Searching file system..."));
Expand Down Expand Up @@ -189,6 +199,14 @@ public ObservableList<FileExtensionViewModel> getFileFilters() {
return this.fileFilterList;
}

public ObservableList<DateRange> getDateFilters() {
return this.dateFilterList;
}

public ObservableList<ExternalFileSorter> getSorters() {
return this.fileSortList;
}

public void cancelTasks() {
if (findUnlinkedFilesTask != null) {
findUnlinkedFilesTask.cancel();
Expand Down Expand Up @@ -234,6 +252,14 @@ public ObjectProperty<FileExtensionViewModel> selectedExtensionProperty() {
return this.selectedExtension;
}

public ObjectProperty<DateRange> selectedDateProperty() {
return this.selectedDate;
}

public ObjectProperty<ExternalFileSorter> selectedSortProperty() {
return this.selectedSort;
}

public StringProperty directoryPathProperty() {
return this.directoryPath;
}
Expand Down
Loading