Fix charset detection with utf16 and others #8947

Siedlerchr · 2022-07-03T13:56:57Z

Fixes #8895
Fixes #8870

Needs to be tested under Windows

Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

Fixes #8895 Fixes #8870

koppor

I assume following quote meant that I should run "BibtexImporterTest" on Windows?

Needs to be tested under Windows

Does not work here.

On both tests :-\

2022-07-03 22:35:03 [Test worker] org.jabref.logic.importer.fileformat.BibtexImporter.getSuppliedEncoding()
ERROR: Supplied encoding could not be determined: java.nio.charset.MalformedInputException: Input length = 2
	at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:176)
	at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
	at org.jabref.logic.importer.fileformat.BibtexImporter.getSuppliedEncoding(BibtexImporter.java:121)
	at org.jabref.logic.importer.fileformat.BibtexImporter.importDatabase(BibtexImporter.java:72)
	at org.jabref.logic.importer.fileformat.BibtexImporterTest.testParsingOfUtf816EncodedFileReadsUmlatCharacterCorrectly(BibtexImporterTest.java:179)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:226)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:204)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:139)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:107)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:107)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:42)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at jdk.proxy2/jdk.proxy2.$Proxy5.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

Input length = 2
java.nio.charset.MalformedInputException: Input length = 2
	at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:176)
	at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
	at java.base/java.io.BufferedReader.read(BufferedReader.java:183)
	at java.base/java.io.FilterReader.read(FilterReader.java:65)
	at java.base/java.io.PushbackReader.read(PushbackReader.java:91)
	at org.jabref.logic.importer.fileformat.BibtexParser.determineNewLineSeparator(BibtexParser.java:154)
	at org.jabref.logic.importer.fileformat.BibtexParser.parse(BibtexParser.java:136)
	at org.jabref.logic.importer.fileformat.BibtexImporter.importDatabase(BibtexImporter.java:97)
	at org.jabref.logic.importer.fileformat.BibtexImporter.importDatabase(BibtexImporter.java:81)
	at org.jabref.logic.importer.fileformat.BibtexImporterTest.testParsingOfUtf816EncodedFileReadsUmlatCharacterCorrectly(BibtexImporterTest.java:179)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:226)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:204)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:139)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:107)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:107)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:42)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at jdk.proxy2/jdk.proxy2.$Proxy5.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

This reverts commit 1082f8a.

# Conflicts: # src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

* upstream/main: Bump checkstyle from 10.1 to 10.3.1 (#8950) Bump mariadb-java-client from 2.7.5 to 2.7.6 (#8953) Add notification when adding previous entries to new group configuration (#8919) Remember Sidepane width after restart (#8936) move "Warn about duplicates on import" preferences option (#8937) Fix theme switching exception (#8939)

koppor · 2022-07-04T20:05:13Z

This is somehow blocked by unicode-org/icu#2127.

We intend:

if (detectedCharset== ASCII) {
  charset = UTF-8
}

Reason:

If a user starts with a simple BibEntry and then adds other bibEntries, JabRef should default to UTF-8 and not to some ISO thing.

* upstream/main: Disable ResearchGateTest on CI (#8955) Bump checkstyle from 10.1 to 10.3.1 (#8950)

…harset * 'detectCharset' of github.com:JabRef/jabref: Try jitpack

Siedlerchr · 2022-07-09T16:07:26Z

Jitpack failed to find a build file:

Build starting...
Start: Sat Jul 9 15:53:44 UTC 2022 2369baee1473
Git:
cldr/2022-06-27-5-gc661193
commit c66119346844b94506996a0896f151b3f4d8bb23
Author: Oliver Kopp 
Date:   Mon Jul 4 21:42:04 2022 +0200

    ICU-22080 Add ASCII Recognition
    

Init SDKMan

ERROR:
No build file found. Looking for Gradle/Maven/Sbt/Lein build file in the root of the project
See the documentation at https://docs.jitpack.io
Files: 
CONTRIBUTING.md
KEYS
README.md
WORKSPACE
docs
icu4c
icu4j
tools
vendor
2022-07-09T15:53:44.820961101Z
Exit code: 1

⚠️ ERROR: No build artifacts found

https://jitpack.io/com/github/Jabref/icu/addASciiTest-cldr~2022-06-27-gc661193-5/build.log

Siedlerchr · 2022-07-09T21:58:32Z

TODO: Build locally

* upstream/main: Keep UTF-8 encoding header if present (#8964) Cleanup index when opening a library (#8962) Refactor context menu entry types changing (#8957) Improvement on check for missing commas when importing or opening a .bib file (#8840) # Conflicts: # src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

koppor · 2022-07-10T18:00:12Z

I needed to update the test files. When the one with header is opened with UTF-16BE (in vs code), it reads as follows:

The test file without header had only the umlaut encoded with UTF-16 - the other chars with ASCII (UTF-8).

Note that UTF-16 always uses (at least) 2 bytes for one character (and not 1 as UTF-8 does)

The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (216) values, which would require 2 bytes (16 bits) per character.
[...]
The early 2-byte encoding was originally called "Unicode", but is now called "UCS-2".[7]
[...]
When it became increasingly clear that 216 characters would not suffice,[1] IEEE introduced a larger 31-bit space and an encoding (UCS-4) that would require 4 bytes per character.

I believe that the UTF-16 things come in because of a strange default in JabRef during some months . I don't think, there are any "real" files with UTF-16 of JabRef. And if there are, there is the encoding comment. With that comment, we don't have BOM issues (as our implementation ignores them when interpreting the encoding comment).

Workaround for the test: Supply % DBID:. Then, the BOM is also removed 😇

Small notes:

Issue: Java provides the BOM into the Reader: JDK-6959785 : UTF-8 encoding does not recognize initial BOM (and linked issue)
The byte-order-mark (BOM) is optional: https://softwareengineering.stackexchange.com/a/370095/52607 (even though, it should be recognized to have the correct order)

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

Source: http://biercoff.com/malformedinputexception-input-length-1-exception-solution-for-scala-and-java/ Co-authored-by: Christoph <siedlerkiller@gmail.com>

Additionally - Replace unknown characters - Remove obsolete wrapping classes in test

…failure-dialog * upstream/main: (76 commits) New Crowdin updates (#8972) New Crowdin updates (#8969) Fix .bat errorlevel handling with pwsh.exe check (#8965) revert jsoup version upgrade Fix charset detection with utf16 and others (#8947) Bump classgraph from 4.8.147 to 4.8.149 (#8968) Bump jsoup from 1.15.1 to 1.15.2 (#8967) Keep UTF-8 encoding header if present (#8964) Cleanup index when opening a library (#8962) Refactor context menu entry types changing (#8957) Improvement on check for missing commas when importing or opening a .bib file (#8840) Disable ResearchGateTest on CI (#8955) Bump checkstyle from 10.1 to 10.3.1 (#8950) Bump checkstyle from 10.1 to 10.3.1 (#8950) Bump mariadb-java-client from 2.7.5 to 2.7.6 (#8953) Add notification when adding previous entries to new group configuration (#8919) Remember Sidepane width after restart (#8936) move "Warn about duplicates on import" preferences option (#8937) Fix theme switching exception (#8939) fix merge conflict ... # Conflicts: # src/main/resources/l10n/JabRef_en.properties

* upstream/main: (28 commits) [Bot] Update CSL styles (JabRef#8978) New Crowdin updates (JabRef#8972) New Crowdin updates (JabRef#8969) Fix .bat errorlevel handling with pwsh.exe check (JabRef#8965) revert jsoup version upgrade Fix charset detection with utf16 and others (JabRef#8947) Bump classgraph from 4.8.147 to 4.8.149 (JabRef#8968) Bump jsoup from 1.15.1 to 1.15.2 (JabRef#8967) Keep UTF-8 encoding header if present (JabRef#8964) Cleanup index when opening a library (JabRef#8962) Refactor context menu entry types changing (JabRef#8957) Improvement on check for missing commas when importing or opening a .bib file (JabRef#8840) Disable ResearchGateTest on CI (JabRef#8955) Bump checkstyle from 10.1 to 10.3.1 (JabRef#8950) Bump checkstyle from 10.1 to 10.3.1 (JabRef#8950) Bump mariadb-java-client from 2.7.5 to 2.7.6 (JabRef#8953) Add notification when adding previous entries to new group configuration (JabRef#8919) Remember Sidepane width after restart (JabRef#8936) move "Warn about duplicates on import" preferences option (JabRef#8937) Fix theme switching exception (JabRef#8939) ... # Conflicts: # src/main/java/org/jabref/gui/groups/GroupTreeViewModel.java

* upstream/main: (25 commits) Update Gradle Wrapper from 7.4.2 to 7.5. (JabRef#8986) New translations JabRef_en.properties (Russian) (JabRef#8985) [Bot] Update CSL styles (JabRef#8978) New Crowdin updates (JabRef#8972) New Crowdin updates (JabRef#8969) Fix .bat errorlevel handling with pwsh.exe check (JabRef#8965) revert jsoup version upgrade Fix charset detection with utf16 and others (JabRef#8947) Bump classgraph from 4.8.147 to 4.8.149 (JabRef#8968) Bump jsoup from 1.15.1 to 1.15.2 (JabRef#8967) Keep UTF-8 encoding header if present (JabRef#8964) Cleanup index when opening a library (JabRef#8962) Refactor context menu entry types changing (JabRef#8957) Improvement on check for missing commas when importing or opening a .bib file (JabRef#8840) Disable ResearchGateTest on CI (JabRef#8955) Bump checkstyle from 10.1 to 10.3.1 (JabRef#8950) Bump checkstyle from 10.1 to 10.3.1 (JabRef#8950) Bump mariadb-java-client from 2.7.5 to 2.7.6 (JabRef#8953) Add notification when adding previous entries to new group configuration (JabRef#8919) Remember Sidepane width after restart (JabRef#8936) ...

Fix charset detection with utf16 and others

dc9b9cb

Fixes #8895 Fixes #8870

Siedlerchr added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Jul 3, 2022

checkstyöe

b2f9b4b

koppor reviewed Jul 3, 2022

View reviewed changes

Fix typo in method names

533f905

Siedlerchr removed the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Jul 3, 2022

Siedlerchr added 2 commits July 3, 2022 23:12

change newlines

ca72dff

get bytes

1082f8a

github-actions bot reviewed Jul 3, 2022

View reviewed changes

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java Outdated Show resolved Hide resolved

HoussemNasri and others added 8 commits July 3, 2022 23:03

Set newline character to LF

455ae72

Merge remote-tracking branch 'upstream/detectCharset' into detectCharset

c1c5eac

Revert "get bytes"

e3085b1

This reverts commit 1082f8a.

progress

8d7a38a

switch line sep to LF

59f3ffc

Please work

f93cfb8

Merge remote-tracking branch 'upstream/detectCharset' into detectCharset

ded49cd

# Conflicts: # src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java

ThiloteE added import unicode unicode related issues labels Jul 4, 2022

Merge remote-tracking branch 'upstream/main' into detectCharset

832a4bd

* upstream/main: Disable ResearchGateTest on CI (#8955) Bump checkstyle from 10.1 to 10.3.1 (#8950)

Siedlerchr added the status: depends-on-external A bug or issue that depends on an update of an external library label Jul 7, 2022

koppor and others added 2 commits July 7, 2022 21:20

Try jitpack

758de5b

Merge branch 'detectCharset' of github.com:JabRef/jabref into detectC…

6287508

…harset * 'detectCharset' of github.com:JabRef/jabref: Try jitpack

Siedlerchr added 2 commits July 10, 2022 12:11

Add manual build of icu4j

680de72

github-actions bot reviewed Jul 10, 2022

View reviewed changes

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java Show resolved Hide resolved

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java Outdated Show resolved Hide resolved

Check if we have ascii in the list of charsets

28e1616

fix checkstyle

787d594

Siedlerchr added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Jul 10, 2022

koppor added 4 commits July 10, 2022 18:10

Update external-libraries.md

b3795fd

Enocde with UTF-16BE

164c629

Fix umlaut

67ffee9

Hack to get test running

4d4a124

koppor approved these changes Jul 10, 2022

View reviewed changes

Also compare meta data

d79dfb7

github-actions bot reviewed Jul 10, 2022

View reviewed changes

src/test/java/org/jabref/logic/importer/fileformat/BibtexImporterTest.java Outdated Show resolved Hide resolved

koppor and others added 6 commits July 10, 2022 20:45

Add enforced ignorance of malformed characters

63e7882

Source: http://biercoff.com/malformedinputexception-input-length-1-exception-solution-for-scala-and-java/ Co-authored-by: Christoph <siedlerkiller@gmail.com>

checkstyle

b5e915d

IntelliJ now also renders the file correctly

f8df61b

Add test

2a4c1dd

Additionally - Replace unknown characters - Remove obsolete wrapping classes in test

Refine CHANGELOG.md

34c8210

Remove non-working jpackage reference

040ee18

Siedlerchr merged commit 76850e7 into main Jul 11, 2022

Siedlerchr deleted the detectCharset branch July 11, 2022 19:23

Siedlerchr mentioned this pull request Dec 25, 2022

BOM now missing at beginning of bibliography file -- causes JabRef to not recognize existing library #9496

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix charset detection with utf16 and others #8947

Fix charset detection with utf16 and others #8947

Siedlerchr commented Jul 3, 2022 •

edited

Loading

koppor left a comment

koppor commented Jul 4, 2022

Siedlerchr commented Jul 9, 2022

Siedlerchr commented Jul 9, 2022

koppor commented Jul 10, 2022

Fix charset detection with utf16 and others #8947

Fix charset detection with utf16 and others #8947

Conversation

Siedlerchr commented Jul 3, 2022 • edited Loading

koppor left a comment

Choose a reason for hiding this comment

koppor commented Jul 4, 2022

Siedlerchr commented Jul 9, 2022

Siedlerchr commented Jul 9, 2022

koppor commented Jul 10, 2022

Siedlerchr commented Jul 3, 2022 •

edited

Loading