ci: Add test case for hdfs over gcs bucket #4145
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ArmandoZ
commented
Feb 5, 2024
ArmandoZ
commented
Feb 5, 2024
Xuanwo
reviewed
Feb 5, 2024
ArmandoZ
commented
Feb 5, 2024
ArmandoZ
force-pushed
the
add_hdfs_gcs
branch
2 times, most recently
from
c9491a9 to
9c1e273
Compare
Xuanwo
changed the title
ArmandoZ
force-pushed
the
add_hdfs_gcs
branch
2 times, most recently
from
4240700 to
7911458
Compare
ArmandoZ
commented
Feb 6, 2024
Xuanwo
reviewed
Feb 6, 2024
ArmandoZ
force-pushed
the
add_hdfs_gcs
branch
from
7911458 to
8dcb19d
Compare
ArmandoZ
commented
Feb 7, 2024
|
Workflow: https://github.com/apache/opendal/actions/runs/7811201570/job/21305944190?pr=4167 2024-02-07 07:15:21,938 ERROR conf.Configuration: error parsing conf core-site.xml
com.ctc.wstx.exc.WstxParsingException: Illegal processing instruction target ("xml"); xml (case insensitive) is reserved by the specs.
at [row,col,system-id]: [21,5,"file:/home/runner/hadoop-3.3.5/etc/hadoop/core-site.xml"]
at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:634)
at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:504)
at com.ctc.wstx.sr.BasicStreamReader.readPIPrimary(BasicStreamReader.java:4004)
at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2138)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1179)
at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3405)
at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3191)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3084)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3045)
at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2923)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2905)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1247)
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1301)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1527)
at org.apache.hadoop.fs.FileSystem$Cache.<init>(FileSystem.java:3615)
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:206)
could not find method getRootCauseMessage from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;
could not find method getStackTrace from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String; |
Got it. I'm looking into that. Will let you know when ready to run again. |
Xuanwo
reviewed
Feb 7, 2024
|
Test failed for: hdfsBuilderConnect(forceNewInstance=0, nn=gs://***, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
UnsupportedFileSystemException: No FileSystem for scheme "gs"org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "gs"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3546)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3569)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3673)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3624)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:278)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:275)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:275) |
Direct reason is hadoop didn't read the gcs-connector jar library. Probably need to set |
Xuanwo
reviewed
Feb 7, 2024
ArmandoZ
force-pushed
the
add_hdfs_gcs
branch
from
897cb95 to
6edb657
Compare
Xuanwo
reviewed
Feb 7, 2024
ArmandoZ
force-pushed
the
add_hdfs_gcs
branch
from
6edb657 to
3056386
Compare
|
Test passed! Thanks a lot for your work!
|
|
Sounds great! Thanks for the support! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue: #3504
Adds test case for hdfs over gcs bucket. Tested locally with my own gcs bucket.
TODO: Need to change how we obtain secret information from the test case (please see comments in code)