Implement getCompactionJob async RPC using Jetty and REST #5018
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change switches getCompactionJob() rpc call to be an async REST call using Jetty and Jersey so that we can use long polling and not block while waiting for jobs on the queue. Jetty is configured to use Jackson and a custom serializer/deserializer to handle Thrift objects being serialized to json. Authentication has been set up to work with username/password, SSL, and also Kerberos. Right now for the prototype the authentication is not quite complete and a couple shortcuts were taken to get it working with the IT set up so some more work needs to be done before it is finished.
dlmarion
reviewed
Oct 28, 2024
| // to prevent creating a new deserializer for every object? | ||
| private static <T extends TBase<?,?>> void deserialize(T obj, String json) throws IOException { | ||
| try { | ||
| final TDeserializer deserializer = new TDeserializer(new TJSONProtocol.Factory()); |
There was a problem hiding this comment.
The Factory is thread-safe, right? I can imagine that the Deserializer is not. You could put the TDeserializer and TSerializers in a ThreadLocal. It looks like they reset their internals for reuse.
Comment on lines
+1169
to
+1177
| String advertiseHost = getHostname(); | ||
| if (advertiseHost.equals("0.0.0.0")) { | ||
| try { | ||
| advertiseHost = InetAddress.getLocalHost().getHostName(); | ||
| } catch (UnknownHostException e) { | ||
| log.error("Unable to get hostname", e); | ||
| } | ||
| } | ||
| HostAndPort restHostAndPort = HostAndPort.fromParts(advertiseHost, restPort); |
There was a problem hiding this comment.
I think you can use the value of sa.address from the code above this. In the code above the Thrift server was started on an interface, I assume we want to use the same one, just a different port.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change switches getCompactionJob() rpc call to be an async REST call using Jetty and Jersey so that we can use long polling and not block while waiting for jobs on the queue. This is meant to demonstrate another alternative for async RPC as described in #5000
Jetty is configured to use Jackson and a custom serializer/deserializer to handle Thrift objects being serialized to json.
Authentication has been set up to work with username/password, SSL, and also Kerberos. Right now for the prototype the authentication is working but a couple shortcuts were taken to get it working with the IT set up so some more work needs to be done before it could work outside ITs if we wanted to use it.
Some notes about the prototype: