-
Notifications
You must be signed in to change notification settings - Fork 655
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Center deploy button #694
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merged
I think this PR was mostly just here as a reference for #719. As that is now completed, I will close this one as well. If I am mistaken, please feel free to reopen |
Lokiiiiii
pushed a commit
to Lokiiiiii/djl
that referenced
this pull request
Oct 10, 2023
* add neuronx new feature for generation * swap back
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
@roywei @anfee1
here is the deploy-action button in Central I mentioned yesterday on Slack.
this change is not yet finished, but it creates functionality similar to the download=button that @anfee1 is about to implement.
The button-list is defined in
/central/src/main/webapp/components/ModelViewActions.jsx
it should be quite simple to add another download button to this actions like:
import SaveIcon from '@material-ui/icons/Save';
.
.
.
>
In the deploy-function declared in the same file you can see an example how to call the backend providing values from the current selected model:
const deploy = () => {
**axios.post("http://"+window.location.host+"/serving/models?**modelName="+props.modelName+"&url="+encodeURIComponent(props.modelUri))
.then(function(response) {
console.log(response.data.message);
alert(response.data.message)
})
.catch(function(error) {
console.log(error);
})
.then(function() {
// always executed
});
};
"axios"-framework is used to do the Rest-call to the backend and trigger a function in the backend.
when we receive an response from backend then the call in ".then(function()...) is called.
(see /central/src/main/java/ai/djl/serving/central/handler/ModelDeploymentHandler.java which handles requests for the url /serving/models/...)
The models are already loaded into central-UI. modelZoo.listModel() gets invoked when open the UI.
see the useFetch-function in /central/src/main/webapp/components/ModelNavigator.jsx .
The the website already holds information for the models in browser-memory.
when you select a model in to Tree then
/central/src/main/webapp/components/ModelView.jsx
is used to display the model details.
The component ModelView has a property called "model" which is exactly the model-info of this model.
you can access values from the model using "{props.model.version}" or "{props.model.files.parameters.uri}" and so on.
So I think we do not have to call listModel in the backend again to implement the download function.
We just have to trigger the backend-download using a axios-Rest-Call to trigger the download. Or if we don't want to use axios a href.location=.... would do the trick as well. maybe we don't even need the backend in this case cause the browser can directly download the model from this url. But imo going through the backend is the cleaner option when we are thinking about clouds, firewalls, permissions,proxies and so on which can all be handled by the backend-host.
Then we need a Netty-ChannelInbound-Handler in the backend to download the data and stream to bytes to the "Response"-object Chunk-by-Chunks (netty can process large files in chunks without storing the whole file in memory first !!)
We just have to set the http-Header of the response correctly so that the file is downloaded and not interpreted as html content by the browser.
This can be done by setting Content-Disposition to attachment
response.setHeader("Content-Disposition", "attachment; filename=" + YOUR_FILE_NAME);
examples for the Netty-ChannelInbound-Handler can be found in
package ai.djl.serving.central.handler
you have to register a new handler in
/central/src/main/java/ai/djl/serving/central/handler/HttpStaticFileServerInitializer.java.
I think /central/src/main/java/ai/djl/serving/central/handler/HttpStaticFileServerHandler.java
is a good example of a Netty-ChannelInbound-Handler for download cause this handler sends static-file-resource from the folder to the client and it's quite similar to downloading a file when a particular url is called.
in line 196 you also can see that the filecontent is sent chunk-wise to the client.
} else {
sendFileFuture =
ctx.writeAndFlush(
new HttpChunkedInput(new ChunkedFile(raf, 0, fileLength, 8192)),
ctx.newProgressivePromise());
// HttpChunkedInput will write the end marker (LastHttpContent) for us.
lastContentFuture = sendFileFuture;
}
this handler is mainly inspired by the FileServer example I had. This fileserver is doing mostly what you want to do. It downloads a files ! :-)
maybe this resources can be helpful:
https://netty.io/wiki/user-guide-for-4.x.html
https://github.com/netty/netty/tree/4.1/example/src/main/java/io/netty/example/file
this would be a bit simpler when using a framework like spring,, but netty was already used in the project and we can avoid dependencies to huge 3rd party frameworks this way.
I didn't try this solution, these are just my first thought how I would access this problem.
Just ping me if you have any questions.
sorry if this is to detailed :-)
Brief description of what this PR is about