Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add direct access option to avoid cachi...; Port [#18326] to branch-2.10 #18384

Closed

Conversation

alluxio-bot
Copy link
Contributor

What changes are proposed in this pull request?

Datalake formats such as Iceberg requires frequent changes to certain files, it is better not to cache them at all to avoid frequent invalidations.

Why are the changes needed?

for correct functionality using iceberg

Does this PR introduce any user facing changes?

one user property that is by default empty, alluxio.user.file.direct.access

[This is an auto-generated PR to cherry-pick committed PR #18326 into target branch branch-2.10]

### What changes are proposed in this pull request?

Datalake formats such as Iceberg requires frequent changes to certain files, it is better not to cache them at all to avoid frequent invalidations. 

### Why are the changes needed?

for correct functionality using iceberg

### Does this PR introduce any user facing changes?

one user property that is by default empty, alluxio.user.file.direct.access

			pr-link: #18326
			change-id: cid-25a1ee3c2876a4a4126727113342c24d40b06126
@alluxio-bot alluxio-bot added the type-feature This issue is a feature request label Nov 6, 2023
@Xenorith
Copy link
Contributor

Xenorith commented Nov 6, 2023

test failure looks real:

2023-11-06T19:38:53.1751152Z 19:38:53.170 [ERROR] alluxio.client.fs.DirectAccessIntegrationTest.writeDirect  Time elapsed: 2.307 s  <<< ERROR!
2023-11-06T19:38:53.1752401Z java.lang.ClassCastException: Cannot cast java.lang.String to alluxio.client.WriteType
2023-11-06T19:38:53.1753165Z 	at java.lang.Class.cast(Class.java:3369)
2023-11-06T19:38:53.1753923Z 	at alluxio.conf.InstancedConfiguration.getEnum(InstancedConfiguration.java:307)
2023-11-06T19:38:53.1754900Z 	at alluxio.conf.OverlayConfiguration.getEnum(OverlayConfiguration.java:128)
2023-11-06T19:38:53.1755953Z 	at alluxio.util.FileSystemOptionsUtils.createFileDefaults(FileSystemOptionsUtils.java:134)
2023-11-06T19:38:53.1757109Z 	at alluxio.util.FileSystemOptionsUtils.createFileDefaults(FileSystemOptionsUtils.java:114)
2023-11-06T19:38:53.1758179Z 	at alluxio.client.file.BaseFileSystem.lambda$createFile$3(BaseFileSystem.java:214)
2023-11-06T19:38:53.1759093Z 	at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:701)
2023-11-06T19:38:53.1760052Z 	at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:213)
2023-11-06T19:38:53.1841437Z 	at alluxio.client.fs.DirectAccessIntegrationTest.writeDirect(DirectAccessIntegrationTest.java:65)
2023-11-06T19:38:53.1848139Z 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-11-06T19:38:53.1851142Z 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-11-06T19:38:53.1855238Z 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-11-06T19:38:53.1857119Z 	at java.lang.reflect.Method.invoke(Method.java:498)
2023-11-06T19:38:53.1863353Z 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2023-11-06T19:38:53.1865081Z 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2023-11-06T19:38:53.1866461Z 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2023-11-06T19:38:53.1867964Z 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2023-11-06T19:38:53.1883027Z 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2023-11-06T19:38:53.1884210Z 	at alluxio.testutils.LocalAlluxioClusterResource$1.evaluate(LocalAlluxioClusterResource.java:202)
2023-11-06T19:38:53.1885475Z 	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
2023-11-06T19:38:53.1886765Z 	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
2023-11-06T19:38:53.1887745Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2023-11-06T19:38:53.1888365Z 	at java.lang.Thread.run(Thread.java:748)

@yuzhu could you help fix it?

@yuzhu
Copy link
Contributor

yuzhu commented Nov 8, 2023

@Xenorith can not repro locally for some reason, re-triggering to see if it is flakey. may be specific to github environment..

@Xenorith Xenorith closed this Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-feature This issue is a feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants