Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes system-files not being ignored by ZIP-native extractor when executing an import-job #2059

Merged
merged 11 commits into from
Dec 11, 2024
Merged

Fixes system-files not being ignored by ZIP-native extractor when executing an import-job #2059

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5f165cf
Implements a filter for __MACOSX-folder in native Mac archives
scireum-mbo Dec 9, 2024
8f1d9ab
Adds a list for filenames to ignore when extracting from archive
scireum-mbo Dec 9, 2024
f2fb2f8
Changes comparison logic to use list instead of single value
scireum-mbo Dec 9, 2024
917882b
Replaces contains with startsWith for a more precise search
scireum-mbo Dec 9, 2024
80a2212
Implements new methods isConsideredHidden and isConsideredMetadata
scireum-mbo Dec 10, 2024
6855a52
Removes the list of filenames due to it being moved to siri-kernel
scireum-mbo Dec 10, 2024
fbba26f
Merge branch 'develop' into feature/mbo/SIRI-1028
scireum-mbo Dec 11, 2024
19c709c
Implements a check for the list of imported files
scireum-mbo Dec 11, 2024
cf2847c
Bumps sirius-kernel
scireum-mbo Dec 11, 2024
0091520
Removes check to be used in a different place
scireum-mbo Dec 11, 2024
f024063
Implements check for hidden and metadata files for zip entries
scireum-mbo Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/main/java/sirius/biz/jobs/batch/file/ArchiveImportJob.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import sirius.biz.util.ExtractedZipFile;
import sirius.kernel.async.TaskContext;
import sirius.kernel.commons.Amount;
import sirius.kernel.commons.Files;
import sirius.kernel.commons.Producer;
import sirius.kernel.commons.Strings;
import sirius.kernel.health.Exceptions;
Expand Down Expand Up @@ -110,7 +111,7 @@ protected Optional<ExtractedFile> fetchEntry(String fileName) throws IOException
Enumeration<? extends ZipEntry> zipEntries = zipFile.entries();
while (zipEntries.hasMoreElements() && taskContext.isActive()) {
ZipEntry zipEntry = zipEntries.nextElement();
if (FILE_FILTER.stream().anyMatch(zipEntry.getName()::startsWith)) {
if (Files.isConsideredHidden(zipEntry.getName()) || Files.isConsideredMetadata(zipEntry.getName())) {
idlira marked this conversation as resolved.
Show resolved Hide resolved
continue;
}
if (Strings.areEqual(fileName, zipEntry.getName())) {
Expand Down
3 changes: 2 additions & 1 deletion src/main/java/sirius/biz/storage/layer3/VirtualFileSystemController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import sirius.biz.tycho.QuickAction;
import sirius.biz.tycho.UserAssistant;
import sirius.biz.web.BizController;
import sirius.kernel.commons.Files;
import sirius.kernel.commons.Limit;
import sirius.kernel.commons.Streams;
import sirius.kernel.commons.Strings;
Expand Down Expand Up @@ -380,7 +381,7 @@ private void outputChildren(WebContext webContext, JSONStructuredOutput output,
Arrays.stream(extensionString.asString().split(","))
.map(Strings::trim)
.filter(Strings::isFilled)
.map(string -> string.startsWith(".") ? string.substring(1) : string)
.map(string -> Files.isConsideredHidden(string) ? string.substring(1) : string)
.forEach(search::withFileExtension);
});

Expand Down
2 changes: 1 addition & 1 deletion src/main/java/sirius/biz/storage/s3/ObjectStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ protected ObjectStore(ObjectStores stores, String name, AmazonS3 client, String
.withS3Client(client)
.build();

if (Strings.isFilled(bucketSuffix) && bucketSuffix.contains(".") && !bucketSuffix.startsWith(".")) {
if (Strings.isFilled(bucketSuffix) && bucketSuffix.contains(".") && !Files.isConsideredHidden(bucketSuffix)) {
ObjectStores.LOG.WARN(
"The bucketSuffix '%s' contains a '.' but does not start with one. This might lead to errors!",
bucketSuffix);
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/sirius/biz/util/ArchiveExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ private Predicate<String> enhanceFileFilter(@Nullable Predicate<String> filter)

private boolean ignoreHiddenFiles(String path) {
String filename = Files.getFilenameAndExtension(path);
return Strings.isFilled(filename) && !filename.startsWith(".") && !filename.startsWith("__MACOSX");
return Strings.isFilled(filename) && !Files.isConsideredHidden(filename) && !Files.isConsideredMetadata(path);
}

private void extractZip(File archiveFile,
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/sirius/biz/util/LocalArchiveExtractCallback.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,15 +72,15 @@ public ISequentialOutStream getStream(int index, ExtractAskMode extractAskMode)
skipExtraction |= (attributes & PropID.AttributesBitMask.FILE_ATTRIBUTE_HIDDEN) != 0;
}
if (filePath != null) {
skipExtraction |= filePath.startsWith("__MACOSX");
skipExtraction |= Files.isConsideredMetadata(filePath);
if (filter != null) {
skipExtraction |= !filter.apply(filePath);
}
}
if (fileName != null) {
// need to filter hidden files (starting with dot), because some tar implementations create
// hidden index files (ending with xml, too)
skipExtraction |= fileName.startsWith(".");
skipExtraction |= Files.isConsideredHidden(fileName);
}

if (skipExtraction) {
Expand Down