Support JupyterLab

[astrojuanlu]

Edit: See current status in this comment.

At the moment, czml3.widget.CZMLWidget has a straightforward _repr_html_ that prints some HTML. However, this is not how custom widgets are supposed to be implemented [citation needed]

We should write our own custom Jupyter widget so it's compatible, at least, with modern JupyterLab.

Edit: Adding more information to this description for completeness

Rather than a JupyterLab extension, if I understand correctly we would need to create a custom Jupyter widget. Therefore, we should be using either https://github.com/jupyter-widgets/widget-cookiecutter (JavaScript, barebones) or https://github.com/jupyter-widgets/widget-ts-cookiecutter (TypeScript + bells & whistles), which are described in https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Low%20Level.html

However, we should wait for the TypeScript one to be updated: jupyter-widgets/widget-ts-cookiecutter#83

These cookiecutters already make use of https://github.com/jupyter/jupyter-packaging.

For inspiration, we could also have a look at https://github.com/bloomberg/bqplot, https://github.com/jovyan/pythreejs, https://github.com/ellisonbg/ipyleaflet or https://github.com/matplotlib/ipympl.

With this and https://www.npmjs.com/package/c137.js shared by @TJKoury above, we should have everything we need already.

[GorgiAstro]

According to this post, it could be necessary to get a webpack of Cesium: jupyterlab/jupyterlab#5566

A new webpack of Cesium will come soon: DigitalArsenal/webpack-cesium#7

[astrojuanlu]

Oh wow, Jon Mease already worked on it 😳 perhaps we'll have more luck this time!

[GorgiAstro]

Yes, but I could not find where he published his ipywidget. Maybe we can ask him in a chat or by tagging him here?

[astrojuanlu]

Hi @jonmmease! We noticed your comment from a couple of years ago about making a Cesium widget. @GorgiAstro found that there will be a new version of webpack-cesium soon. Do you have any code you can share so we can pick up from there and perhaps finish the work? Or do you think we should start from scratch?

[jonmmease]

Hi @astrojuanlu, thanks for reaching out. Sorry to say that I can't share the code, but that said it wasn't very far along. Most of the time spent was in tinkering to try to get cesium building as an ipywidget. Sounds like a webpack-cesium would make that much easier!

In terms of the widget itself, all that I got working was a proof-of-concept where I made a Python/JavaScript model that stored a single lat/lon/altitude coordinate. Updating that coordinate from python triggered the Cesium API to move the sphere around the earth.

Had I stuck with it, I was picturing adopting the approach that bqplot uses (https://github.com/bloomberg/bqplot), where you create a matching hierarchy of nested Python and JavaScript models. But that idea is as far as I got. Hope that helps a little. Good luck!

[TJKoury]

@jonmmease @astrojuanlu We have a new build that works great, and will be kept up to date with the latest in core Cesium.

As I am sure you are aware, the packing process is non-trivial, and in our case represents a significant investment to maintain. We are trying to keep it open for community use but have not decided on a licensing model yet.

Would any of the companies represented on this project consider sponsoring the work?

[astrojuanlu]

Thanks for chiming in @TJKoury! That's great news.

Would any of the companies represented on this project consider sponsoring the work?

poliastro and czml3 are a community effort, unfortunately the only support we can offer is integrating webpack-cesium in czml3 and try to make it work for a broader audience.

[TJKoury]

@astrojuanlu Understood. How do you think the community would respond to a 'non-commercial' license?

[astrojuanlu]

@TJKoury It's a good question. There are many forms of "non-commercial" licensing, for example what Qt does. If you decide use a copyleft license, we'll have to evaluate how to use it from czml3 (MIT, hence non-copyleft) without infringing the terms (and I might need a lawyer for this...). Other than that, I don't know how that can affect adoption.

[TJKoury]

Also, we are about to release a commercial package for the library behind Celestrak Orbit Viz.

[TJKoury]

After some consideration, I think we will probably just release it under MIT and ask for donations.

[astrojuanlu]

After some consideration, I think we will probably just release it under MIT and ask for donations.

That's great news :) Let me suggest https://opencollective.com/ for the donations part in case you don't know it. Looking forward for the release 💪

[TJKoury]

Here it is, C137.js.

Decided to go with CC BY 4.0 as freeware, since source-wise it is already open.

Left some crypto addresses at the bottom if people are feeling generous.

[astrojuanlu]

Awesome @TJKoury, thanks a lot! We'll soon try to figure out how to use, if @jonmmease doesn't beat us to it :)

[TJKoury]

It should be fairly straightforward, use an es6 import any bundler will take care of the rest. Let me know if you have any issues.

[jonmmease]

... if @jonmmease doesn't beat us to it :)

No risk of that for the foreseeable future 🙂

[astrojuanlu]

Seems like JupyterLab 3.0 and webpack 5.0 will change some things? See lengthy discussion at jupyterlab/jupyterlab#7468, keyword "Module Federation" https://module-federation.github.io/

[TJKoury]

@astrojuanlu It will not change the need to pack it up, or mess with our implementation. The issue is not dependency management, it is trying to load external resources like workers at runtime as if they were compile time assets.

[astrojuanlu]

I edited the first comment to add some extra information.

[petrushy]

Hi, was playing a bit with this, would like to get the old CesiumWidget up and running again.. Did a test with c137.js and cookiecutter widget, now showing the widget nothing else. https://github.com/petrushy/CesiumWidget3/tree/c137_version

I would however like to transition to the "official" cesium where all steps are reproducable, but c137.js was indeed very smooth to get running.

[astrojuanlu]

That's great news @petrushy ! I think it's pointless to have two widgets, so one option would be that we remove our widget.py from czml3 and we lend a hand with the maintenance of CesiumWidget3, what do you think?

[TJKoury]

Thanks for the vote of confidence @petrushy, and make sure to check out our soon-to-be decentralized notebook app, mapshot.app. Still in beta but you get the idea.

[petrushy]

@astrojuanlu I am very happy to collaborate on this, the one currently in czml is working but does not fully meet the needs of a widget for other purposes. I am thinking to also add a bit of higher level easy-plotting for quick visualizations.

@TJKoury , sorry I may express myself unclear, I do not see a future using closed source, your package is very nice and the example was a quick confirmation of that, but we need unfortunately to have transparency of the full process of the items that goes into it.

[TJKoury]

@petrushy No problem! The source is available, it’s just the build process that is proprietary. Seems like this project does not have issues using libraries built with closed-source IDE’s or operating systems, but if this is where you draw the line I understand.

[TJKoury]

It's open source now, Apache 2.0 licensed:

Github - c137.js

[astrojuanlu]

Thanks a lot @TJKoury !

By the way, this discussion is very relevant: jupyter/notebook#6210 basically we are in the same situation as many other extensions. Doing what czml3 does in Classic Notebook is easy, but it turns out that doing it in JupyterLab requires an entirely different set of skills. I added my two cents to that discussion, and it looks like there are some ideas to make tinkering with JupyterLab easier. But if we want this migration to happen in the short term, we will have to bite the bullet and figure out how to do it. Undoubtedly, having c137.js open sourced makes everything easier.

[TJKoury]

@astrojuanlu Sounds good, let us know if there's anything on our end that we can do to help.

[petrushy]

It's open source now, Apache 2.0 licensed:

Many Thanks!!!

[astrojuanlu]

PSA: I'm learning webpack. I might tackle this myself 😉

[astrojuanlu]

Inspiration: damianavila/RISE#605

[dfreeman06]

Started work here starting from a JupyterLab extension cookiecutter: https://github.com/dfreeman06/czml3/tree/gh61_support_jupyterlab with strong assistance from @bollwyvl.

The webpack integrates with the cesium source directly but takes care to keep the various modules loading asynchronously so try and keep the bundles separated. Currently there is a rough wrapper around the Cesium Viewer, Clock, Camera, and CZMLDataSource. But it should not be too much effort to pull in the data sources.

This also is using the main Cesium.d.ts typings to generate the entity json schema (CZML json spec) which could be used to help address #81.

I would love to get a PR back here so it would be great to get some more eyes on it and find the right path to integrate.

[bollwyvl]

@dfreeman06 you're a madman 🤯 glad you got it out there, happy to help move it forward.

unless the implementation shown is entirely different than the one we looked at the other day, here are some big ticket items to consider:

• replaces the attrs parameter system with ipywidgets and traitlets
  • it might be possible to emulate the existing API, but there would be new primitives, as described
  • more importantly, it becomes possible to get live data back from cesium (e.g. camera position, selections)
  • it might eventually be possible to integrate with other widget-based geometry systems such as pythreejs, jupyter-cadquery and ipyleaflet
    • this could be particularly interesting for large datasets, as some of the above already work over the binary buffer system that, at its core, efficiently bridges numpy arrays to the browser
• increases complexity of the build
  • there is some dark magic in the .d.ts->schema.json as well as webpack stuff in general
  • we've been using doit to handle these inter-language dependencies
• ties more closely to the jupyterlab/lumino stack, but this includes
  • retrolab (soon too be notebook)
  • jupyterlite
• unlike the RISE work, which has to deal with the very ill-tempered reveal.js, modern cesiumjs (~1.90) is very hygenic aside from one window-level global URL, which can be forgiven)
  • what you're seeing in that screenshot are not iframes, but rather all in-dom, which is what enables properly sharing cameras, etc.
    • there is some complexity when rendering the same viewer twice, but we'll be able to figure something out
  • confidently upgrading to a new cesium would be a matter of re-solving the js packages and re-building

[TJKoury]

Some of those issues are alleviated if you swap out our single-file open source distro, c137.js, for Cesium. We made it in part to deal with the issues you mentioned. You can clone the repo and use the build process with the Cesium code, if you want to customize.

[astrojuanlu]

This is awesome, thanks a lot @dfreeman06 and @bollwyvl! Happy to review whenever you're ready to submit a PR.

[bollwyvl]

c137.js

We had a look at c137, and indeed may emulate some of it in separating the build into several packages, with one just being the cesium assets. Through no shortcoming of c137, a lab extension is going to go through the lab-specific build chain, and would still need webpack tricks. While a bigger build-time download, by depending on cesium directly, we also gain access to the raw upstream cesium assets on disk, which is helpful for codegen, gathering licenses, and the ability to provide/manage a shared/federated cesium for other extensions. So _also_ depending on c137 would actually end up being even bigger, and more fragile vs version drift.

whenever you're ready to submit a PR.

Well, I didn't know we were going to be sharing this in this state just yet... So there are still a lot of hanging bits, and will take some doing to get it into shape!

[TJKoury]

Makes sense, let us know if you have any questions about the parts of the build process you may decide to use.

[petrushy]

Looks great!

[astrojuanlu]

Well, JupyterLab 4.0 is out and Classic Notebook 7.0 will be out soon based on JLab, which means that, if I understood correctly, czml3 is about to stop working everywhere.

To this day I still don't understand the various ways to extend JupyterLab, and even though my JavaScript/TypeScript skills are getting better, the Jupyter docs seem to get more complicated at a faster pace, so there's no way I catch up. In addition, I stepped down as maintainer of poliastro in December 2022 https://www.poliastro.space/blog/2022/12/21/juan-luis-steps-down/, so I'm not actively maintaining czml3 anymore.

Still, efforts are welcome to fix this issue "the right way", whatever that is.

Interesting links:

• https://jupyterlab.readthedocs.io/en/latest/extension/extension_migration.html#jupyterlab-3-x-to-4-x
• https://jupyterlab.readthedocs.io/en/latest/extension/extension_tutorial.html

[astrojuanlu]

Potentially helpful https://blog.jupyter.org/anywidget-jupyter-widgets-made-easy-164eb2eae102

[astrojuanlu]

Further inspiration jupyterlab/jupyterlab#14590

[vishwa2710]

Hello all, worth sharing that there is a fork of cesiumpy here that is starting to be maintained:
https://github.com/open-space-collective/cesiumpy
With that it's possible to create a viewer and render a CZML document, for example one that was created using CZML3. I'll provide some examples in the notebooks there soon. A useful stopgap till an official JupyterLab widget is available?

[astrojuanlu]

czml3 can also render CZML documents in HTML, but embedding them in JupyterLab is another story AFAIK

[astrojuanlu]

Glad to see open-space-collective is still alive and kicking!