-
Notifications
You must be signed in to change notification settings - Fork 277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TFMA not rendering in JupyterLab #112
Comments
cc: @atn832 |
Thanks for sharing the steps. I haven't tried reproducing yet, but judging from the error you posted at #56 (comment), it seems like the TFMA extension is downloading vulcanized_tfma.js from the wrong URL. As a result, instead of actual Javascript, it gets some kind of markup, probably something like: <!doctype html>
<html>
...
file not found
...
</html> It reads the first
In the Network tab of the Chrome debugger tool, can you check the url of the request for vulcanized_tfma.js and share it here? |
@atn832 If I understand correctly, the url is: :31380/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js Is TFMA trying to download anything? We are running in an on-prem cluster with no external access so if TFMA needs external access that would be an issue. |
Getting the same issue in kubeflow/pipelines#5194, we use technique in #10 (comment) to visualize TFMA as html and then embed that HTML in a different place in iframe. The problem is similar to this reported issue that vulcanized_tfma.js doesn't load. I verified that TFMA 0.26.0 still works for us, so the regression happens between the two versions. |
@Bobgy Is the trick with embedding the html required? This is not something I usually use, for instance when rendering stats from TFDV. |
My use case I generate the html in a step in a pipeline, and the pipeline UI shows the html in an iframe. but that's not related to this issue |
Hi @ConverJens, besides what @atn832 asked, I have run the dockerfile with the notebook and data that you shared above (by In terms of
I don't know what the answer is. Maybe @atn832 can give the answer. I only know that running |
Upon running @ConverJens, can you share what you see in the Response tab for the vulcanized_tfma.js request? I expect it'll show some HTML with an error message that might tell us what is wrong. On my machine, since it worked, I can see a bunch of Javascript comments followed by Javascript code like this: |
@atn832 You are completely right! The response I get is basically Any idea how to proceed? @Bobgy You mentioned that you had this issue in KubeFlow as well. Is there some interaction with in KF that leads the browser into believing that JavaScript is disabled or otherwise may be causing this issue?
|
@fhuanming @Bobgy Note that I'm rendering TFMA works for me locally as well when running the image and data I supplied, but the exact same rendering fails in my KubeFlow hosted notebook. I'm don't know if this issue is because of something in KubeFlow or from the fact that our k8s cluster has no external access. @atn832 Do you know if TFMA needs to be able to reach NPM at runtime? @Bobgy Any ideas if KubeFlow itself is blocking something? |
@atn832 Hi, I'm also running into a similar issue with a hosted notebook solution (not Kubeflow). When I try to render, this happens I'd appreciate your help. |
@mwakaba2 the fact that you're getting a 404 makes me wonder about where it's trying to load vulcanized_tfma.js from. Could you hover over the filename in the network request and give us the whole URL? When I load that, it loads from: https://localhost:8080/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js which makes me wonder if this is a port problem, or a path problem, file permissions, or maybe an HTTPS problem (but I don't think that would be a 404). My guess is either a path or file permissions problem. |
@rcrowe-google Yeah that's the path.
I think the issue is that our hosted notebook solution is using jupyter_server in the backend. Jupyter_server by default doesn't support the I still get a 404 after that, so I think I also need to enable the extensions via |
Ok I got it working in Jupyterlab by installing and enabling the nbextensions.
You need that in order to access the /nbextensions path. |
Glad to hear you solved it Mariko! It seems so simple once you know the answer ... 😁 |
@rcrowe-google there's still an upstream problem here FYI. in the Jupyter ecosystem, there are two primary types of frontend extensions (this doc may be helpful):
nbextensions are effectively deprecated in favor of labextensions, because all modern notebook runtimes (incl AI Platform Notebooks etc are fronted with Jupyter Lab). so in order to enable nbextensions (Jupyter Classic Notebook extensions) on top of Jupyter Server (which replaces the Jupyter Notebook server) to support serving of this Javascript, we had to install a compatibility layer that otherwise wouldn't need to exist (called nbclassic). so tl;dr: the problem with TFMA's JupyterLab support is that it's not a pure JupyterLab extension. It's a labextension that depends on a nbextension to work. what we've employed here is a workaround vs a proper fix. the proper fix would involve making the labextension carry its own dependencies vs the existing hybrid model (which is unprecedented btw). this gap is likely where the other folks like @ConverJens are running into issues making this work - as it's non-obvious vs the way any other lab extension works. |
@mwakaba2 @rcrowe-google I tried the fix that you mentioned but there was no difference for me, still no output. Albeit, the error I had was different from the one @mwakaba2 experienced. @kwlzn I tried installing nbclassic but with no progress. I used to start my server by running |
@ConverJens
My tfma setup is probably different from yours. |
@mwakaba2 These are the commands I'm currently using:
along with pip installing the corresponding TFX version first. I've tested 0.26.0, 0.27.0 and 0.28.0 with exactly the same result. I still cannot load the vulcanized_tfma.js file. @kwlzn @rcrowe-google |
@atn832 @fhuanming @mwakaba2 @kwlzn @rcrowe-google If I check the sources tab in the developer console and right click the vulcanized_tfma.js, and choose "open in new tab" I'm directed to: If I manually concatenate these urls I get the actual js file: So it seems that this is actually a path issue. This also perfectly explains why it works when running the image locally but not with the re-written url. How do we proceed with this? |
I'm starting to reproduce, just verified using a local jupyter lab instance with instructions like #112 (comment). I can see tfma visualization in the notebook. In this case, vulcanized_tfma.js is downloaded from http://127.0.0.1:8888/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js. |
DISCLAIMER: the urls below are my private Kubeflow instance, so you won't be able to access them. I reproduced the issue. |
I'm getting closer, the correct download url should be "https://dev-5-14.endpoints.gongyuan-dev.cloud.goog/notebook/gongyuan/tfma4/nbextensions/tensorflow_model_analysis/vulcanized_tfma.js". |
NB_PREFIX env var is "/notebook/gongyuan/tfma4" for me case, so the correct URL is |
NB_PREFIX is passed to jupyterlab server in https://github.com/kubeflow/kubeflow/blob/f57be7ac6edc03561009eb3ae3f1344e16a05f09/components/example-notebook-servers/jupyter/s6/services.d/jupyterlab/run#L12
|
That's pretty much what I can figure out based on my knowledge. model-analysis/tensorflow_model_analysis/notebook/jupyter/js/lib/widget.js Lines 32 to 34 in e3bb17a
|
Just inspecting on jupyterlab HTML, I found some config data, probably we can directly query for content of the script, but I'm not sure if that's the official way to get the value.
EDIT: my guess was right, https://github.com/jupyterlab/jupyterlab/blob/ce2229e86fc1bf04fdd2ed4dab226a790ad9cd8d/packages/coreutils/src/pageconfig.ts#L25-L34 is code for reading the config data. |
PiperOrigin-RevId: 401267422
We think this should be fixed now, but would like someone to try it out. Special thanks to @Bobgy and @zijianjoy. |
@mdreves I can probably test this during next week. |
@mdreves Sorry for the late response! I've had multiple issues that I had to deal with. I'm ready to test this. What do I need to do to install your changes? |
@mdreves We can try this out as well. Will commit some time to testing the commit. |
The 0.35 release will be out shortly if you want to wait for that so you don't need to install from head, though you can also try with the nightly. |
@mdreves |
I've tried it and had the same error with 0.35 release. |
My apologies, but I don't have the cycles to work on this and we don't have anyone with expertise in this area at the moment. Just to clarify, it works except for when you go to display the results using |
We'd be very interested if someone can do a PR to fix this. |
@mdreves Do we have any update on this issue? Do you have the time to work on it now? |
@embr as FYI since he is the new TL for TFMA. Unfortunately we still don't have the cycles to work on this and we don't have anyone with expertise in this area at the moment. We would be very happy to see a community contribution in this case. |
This comment was marked as off-topic.
This comment was marked as off-topic.
@JanetVictorious That's not fixing this at all, you've just just jupyter notebook instead of JupyterLab and this entire issue is about JupyterLab support. |
Hopefully someone has the expertise to work on a PR, meanwhile we are actively working on DataFrame API as an alternative to provide pandas DataFrame for easier plotting. |
This may or may not help. I recently got the TFMA visualization working in JupyterLab by changing my browser settings. I'm using Chrome, and under Settings > Privacy and security > Security my setting was "Enhanced protection". Changing it to "Standard protection" fixed the issue, which was caused by Chrome stopping what it sees as a CORB request. I was using this code to display the visualization:
I hope that helps! |
Hi @ConverJens Can you please confirm on the comment from @rcrowe-google and suggest whether this can be closed. Thank you. |
System information
provided in TensorFlow Model Analysis): Yes, minor
Describe the problem
tfma.view.render_slicing_metrics shows no output.
Source code / logs
Slim docker image to reproduce the issue:
Below is an evaluation artifact from a small TFX pipeline and a minimal notebook to reproduce. Notebook consists of unzipping, install tfma, load eval result and try to display.
3053.zip
tfma-render-issue.ipynb.zip
Edit: I've also run the above with node 12 instead of 15 with the exact same result.
The text was updated successfully, but these errors were encountered: