Support In-Browser Execution? #72
Comments
|
@bollwyvl What's your current recommendation on this? I spent some time looking around in Jupyter Lite and, I must admit, I am not completely sure is this possible yet. If I would try this now, I start by cloning packages pyolite-kernel and pyolite-kernel-extensions and then try to figure out how to refactor robotkernel so that the service worker could patch robotkernel comms... but there could still be blockers that I am unaware. Has anyone tried something similar yet? |
Nah, and most of the maintainers are on holiday 🏄 I'm kinda holding my breath on the python side of the house until
Yeah, something like that. In addition, we probably want a I think doing it in-tree (off a fork of
|
|
oy, pyodide 0.18 is up: https://github.com/pyodide/pyodide/releases/tag/0.18.0 no idea on when i'll be able to hit it, but no external blockers is a good place to be... |
|
@bollwyvl Thanks for all the above! I am sure I am busy enough learning pyolite-kernel and -extension for some time :) |
|
Agreed that iteration is slow. My personal X220 seems no longer suitable for real work in typescript world... and there is no upgrade with matching keyboard available :(
But the approach above seems to work. I'm importing pyolite and using it publishing hooks. It seems that once I learn the quirks in pyolite display publishing API, I'll might have a POC. @bollwyvl What would be the right way to add marketplace/jupyterlab_robotmode into jupyterlite? |
|
Marvelous. Yeah, the cost of the webpack monster build, twice, is brutal,
even on more recent hardware. Once up and running, the watch mode is
_tolerable_, I guess.
My hope is we'll be able to use the same webpack tricks for everything, so
it will be 20, much smaller, webpack builds...
And make the system more composable.
As for robotmode: so we don't yet have any examples of shipping (even plain
old, not server) third-party extensions as part of core, but have a bunch
in the demo:
- add jupyterlab_robotmode to the binder environment.yml in the FEDERATED
EXTENSION area
- run "doit env" to update the examples jupyter-lite.json
- run "doit docs" to rebuild the example app
- serve it up with watch:docs
Had to cut some corners, so it's all conda, but it also works with wheels.
|
|
https://robolite.readthedocs.io/en/latest/try/lab/ I believe that robotmode is bundled, but does not yet activate. For now that fork includes prebuilt robotkernel wheel while I'm locally resolving issues with different message types. Update: Neither .robot -files are highlighted, nor is Robot Framework available as mime type manually for text files. Update: I'm also puzzled that "doit watch" keeps two of my cores 70-100% utilized even when there is no updates. Actual build on update is pretty fast though. |
|
@bollwyvl What could push robotmode into disabled plugins?
Update: So it is disabled, because it is not bundled after all. |
|
"regular" extensions should be provided via the https://github.com/jupyterlite/jupyterlite/blob/main/examples/jupyter_lite_config.json#L3 |
|
@bollwyvl Thanks. Locally I had issues with all possible download URL I could find, but by adding the Conda tar.bz2 package into repository worked. So, highlighting works now at https://robolite.readthedocs.io/en/latest/try/lab/ and now I focus on broken messages. |
|
Highly recommend getting https://github.com/jupyterlite/jupyterlite/blob/main/examples/jupyter_lite_config.json#L12 (ah, see you already do 😊 ) |
|
So, basic usage seems to work. Execution count is lost on successful executions but not on failures. Maybe I didn’t understand all the shortcuts I took. Keyword execution widgets don’t work yet. Anyway, it should be safe to say that this could be done once JupyterLite is “ready”. And I should continue earliest after Pyolite has been renamed 🤔 |
|
Sterling work, sir.
My rough order of getting ready for an 0.1.0a7 release:
I am not sure if this will constitute readiness, as I think we really need to have a way to add out-of-tree, user-specified kernels, and server extensions in general, without the WebPack build... preferably reusing the existing In the meantime, please don't hesitate to do a draft PR to lite itself, so other can be made aware of this effort, which is probably the first wrapper kernel anyone has been willing to share. |
|
Thanks again. I'll see if I can clean things up enough for a pull request. Don't hold your breath. I try to avoid writing new code, so my clones of pyolite-kernel and pyolite-kernel-extension have just the very minimal changes from their original versions, and then I've been refactoring robotkernel to be compatible with pyolite. Maybe at some point I could simply depend on pyolite-kernel and inherit its base classes to avoid maintaining copy & paste. Also, I finally tried this out on a desktop and that has enough juice to run the development environment in reasonable way. Keyword widgets mostly worked out of the box after all, with one issue: With custom output widget the output is not rendered: robotkernel/src/robotkernel/executors.py Line 128 in 827435d If I remove Update: Got trace of this. I need to fix kernel._parent_header. |
|
Widgets mostly work. Once readthedocs has updated. Widget execution clears the button, which is probably because the message helper from pyolite did not pass display id forward... Anyway. Added a smoketest notebook: https://robolite.readthedocs.io/en/latest/_static/retro/notebooks/?path=robolite%20-%20smoketest.ipynb |
Oooh, sounds like a real bug.
Yeah, we should enable this upstream to the extent possible. Presumably the sticky wicket is the huge Another thing i'd like to explore is normalizing the worker implementation with |
Well, that was too hesitated guess from me. The issue is still with robotkernel/src/robotkernel/executors.py Line 128 in 827435d For keyword execution widget execution, RobotKernel creates ipywidgets' output widget and relies on its context manager to make that the output default display output. Unfortunately, that does not work, but display commands are sent to the last active cell's default display output. (And sorry for the possibly inaccurate terms.) Even more unfortunate, I don't understand well enough how messages are targeted to understand, where things go wrong. I wonder, would you be able to understand Kenel spy output for the keyword example (what happens when button for the keyword cell is clicked) at "robolite - smoketest" at https://robolite.readthedocs.io/en/latest/_static/lab/ ? The expected behavior is to render keyword output or stderr below the button, before the log | report -links. What happens is the output replaces button and possible stderr is not displayed (Kernel spy does show the message, but it is not displayed.) That output widget context manager seems to work on just Pyolite, so this is a robolite issue. |
|
I've noticed some similar weirdness with Keeping the |
|
Yeah, i can't track much what's going on. A demonstration didn't get added with jupyterlite/jupyterlite#154 so it might not actually be fixed... |
|
@bollwyvl I'm able to reproduce this with pyolite Expected behavior is tsimilart that this does on normal Jupyter |
|
Awesome, that's a great reproducer, please issue! |
|
@bollwyvl I was about to start learning
Would you have an idea, how to make calling asynchronous Update: My best guess so far is that I might be able to use Atomics.wait to sleep in loop until fetch has been completed. Update: I was naive about Atomics.wait. It blocks also the worker executing fetch, so that approach would probably require creating a new worker from fly, executing fetch on that worker and then relying on Atomics.wait on the main worker to wait for the async fetch to complete. Update: Also worker started from the worker is only executed after the code that created it no longer blocks... 🙈 |
|
@bollwyvl After learning too much about web workers, I learned that XMLHttpRequest has synchronous mode 🙈 |
|
Yeah, we're generally going to run into a bunch of things around asynchrony. If you can get |
|
I had not tried yet, but I wonder, if the approach I tried would work better the other way around: if the main thread would provide postMessage based i/o -services for the worker thread (exposed through js module for Python) and the worker would call the main thread and then do Atomics.wait until the main thread has the answer. Although, I am not completely sure can SharedArrayBuffers really be passed with postMessage for this kind of signaling 🤔 |
|
Hm...
|
|
Doing a copy-paste fork of |
Right, so that's the first part, but there's a bit of care-and-feeding around making that typescript into a more-or-less self-contained "real" PyPI package. So one play could be:
|
|
@bollwyvl Minor update. Last weekend was not like I expected, but I am back on this starting from today. I have still things to clean up, but I expect to the the first version out (much) later today. I seem to be super slow nowadays :) |
when are they ever!
obviously rope me in if you hit any snags... meanwhile, stuff marches on over at lite: the next version of pyodide will be based on python 3.10, and does everything with wheels (no more |
|
@bollwyvl Almost there: https://robotkernel.readthedocs.io/en/latest/_/lab/index.html
I tried to stole everything I could from pidgylite. And probably a lot of things to polish. This repository is getting a bit messy, while I am dancing between Nix, Conda and Pip. Continuing tomorrow... |
Ah, that's from (failing to) use the Getting the pyolite wheels right is still error-prone... but it's hard to make everybody happy. for example, if you try to For
Excellent.
yeah, well we do what we can. at least no docker or windows... unfortunately, i'm unlikely to be of much help if nix is required to get going on it, as i still haven't had the time to dig into it. Anyhow: when you want to take a breath, it's worth trying to scope down the fork into an actual pip-installable package. I can slo |
|
@bollwyvl It works now https://robotkernel.readthedocs.io/en/latest/_/lab/ I added extra steps to cleanup the conflicting wheels. Next I will reorganize docs and link lab with collaboration features, set those pull request versions and then merge. A question: Pidgy and now also RobotKernel uses dynamic doit-tasks to build JupyterLite with the latest development version. But what is the official end-user way? Have Nix is just my personal passion, headache and mostly aesthethic issue because of the current bloat in this repo. I know Conda is more practical and works with all the services like RTD and even Windows. In my projects, I have preferred to have Nix setup for myself and then "graceful degradation" to pip for the others. In this case, I should forget bare pip and only support Nix and Conda 😄 |
|
But what is the official end-user way?
"jupyter lite build" is the way. It is pure python, so no web pack, and
it's tested on Linux, MacOS and windows.
Have jupyter_lite_config.json with absolute URLs
Wheels can also be relative paths, or checked in under ./pypi/ (next to
jupyter_lite_config.json).
for all the required extensions and PYPI packages?
We don't have a first-party, general tool for dependency resolution, much
less with packages-under-development in the mix.
So far every site has had special needs, and bespoke kernels that are built
into labextensions doubly so: it's important to have a real solver (e.g.
pip, poetry, conda) do the heavy lifting, and we really don't want that job.
More broadly, labextensions, by default, will be pulled from the
environment in which lite is installed... This also ensures a compatible
version of many tools we use during the build (e.g jupyter-server), and
should be package manager agnostic. So having a fully-described build
environment will take care of that, as well as in-development labextensions.
Wheels are... annoying, as all the tools (pip, poetry, Thoth, etc) use
different mechanisms to describe them, resolve them, put them on disk,
etc... and solving for the browser is not something the upstream tools are
going to support any time soon. Further, almost every site has needed some
careful stub building.
We might get some conda/mamba/boa tooling soon, which will drastically
alter the situation, and not just for python.... but PyPI wheels will still
be a concern.
Anyhow, lite is pretty extensible, so I see the most likely near term thing
being a jupyterlite-pip-tools, jupyterlite-poetry or something... But not
all of them at once, with unmanaged versions, so therefore not in core...
And I'm not going to commit to supporting them.
Is there way to generate example jupytger_lite_config.json?
No there's no particular way to do that yet. The zero-config,
well-known-paths, usually shadowing where they would be served, should work
but still have some rough edges. We've been slowly moving approaches that
are a good idea from the lite dodo.py into the shipped package. Our setup
seems to give a stable site at runtime, but some of it's still pretty hairy
and fragile at build time, hence checking in so much stuff.
forget bare pip
Nah, bare pip/pypi is still quite important, as no matter how much fancy
stuff is built on top, it is the official, supported tool. If it doesn't
work there, in GHA/GLP and RTD, then it doesn't work for most people.
and only support Nix and Conda 😄
Of course I'd like to support robolite (or whatever we call it) on
conda-forge, but the best path for that is a robust tool chain to pypi
sdist that doesn't need submodules/npm to install.
… |
|
@bollwyvl Thanks again. So, following https://pypi.org/project/jupyterlite-robotkernel/ Thanks to robotkernel wheels being fetched by overridden entrypoint: This seems to work, but would you prefer to do it otherwise? Another question: Should jupyter-videochat already work in lite? Eg. at https://pidgy.readthedocs.io/en/latest/_/lab/index.html?path=tangle_examples.ipynb&room= |
|
Figured out that jupyter-videochat is just missing a release (and the supported version is at develop branch). |
🤯
that's awesome that it works, but might have unwanted longer-term side-effects. in this case, the
obviously this should work better.
yeah... i merged it, and tried it out for a demo: i introduced some nasty react focus bugs from something i did in the last refactor, and i started getting fed up with how i'd been adding features to react (just use more props!) instead of proper components, and got somewhat frustrated, and then moved across the country, and never came back. if we don't mind some react hiccups, we could check in the the CI-built wheel in the demo repo, and configure rooms with settings to use the current real release, could make a short lifespan lite "server" extension that implemented the three API endpoints... i'll see if i can make that real. |
|
@bollwyvl Thanks for explaining all the steps in pidgy for bundled Python packages step by step. I kind of followed it through earlier, to but could not believe that I should bundle robotkernel and all its dependencies into jupyterlite-robotkernel. That would make a 23M wheel. Sure it would be robust (without side-effects and breakages due to updating packages) and also offline-runnable then. So, maybe I'll do that anyway after all. |
you'd only need to bundled hacked dependencies in the labextension that would be bad or impossible to put on PyPI (because you don't own them, or they only work in the browser). pidgy is still super experiemental, but robotkernel has been through a bunch of a releases. If the off-the-shelf kernel and all dependencies work as-downloaded, then no bundled wheels are required... but does put the onus of specific wheel version management on the site owner... but that's probably just fine as they are going to want to configure a bunch of stuff. if anything, a new And indeed, that's what we do upstream on pyolite:
right, that's a huge wheel, and feels bad to push to npm and or pypi if all the stuff already exists on pypi....
...but for an actual site to be deployed that could stay running for years without any maintainence, it would be totally worth it to either check those 23M all in, or at least the whole "big list of wheel URLs", and have a plan (outside of dumb-ol-jupyterlite) to get those wheels to the right place. |
no dice: it doesn't use the right |
It seems that there might be still some discovery issues when labextension has been installed into virtualenv. I think, the discovery is only triggered for piplite_urls from jupyter_lite_config.json. Anyway, I seem to have three options:
I think, I start with 1), but with generating I did try 3) and was able to resolve wheels ( |
the labextension stuff is older, and.... differently complicated than the piplite stuff. But yeah, it's not super clever, but has been working well enough on e.g. RTD. Unless to to
Maybe a fourth, of actually adding (instead of overridng) another
yeah, this is a tough one, as a balance between upgradability and stability. perhaps a middle ground of shipping a known set of pypi urls to be cached and copied to the I guess let me know how you'd like to proceed, and i can try to PR something that moves towards it, wherever it should go... we're doing some fairly novel stuff here. |
Well, I could think if "hook entrypoints", which could have multiple registration and the default implementation would call all of them in some random registration order. For example, right now I would need something like: But those would be quite specific indeed. Therefore, because the current approach already works (until someone wants to compose |
|
oy, hadn't actually looked at the code, but... a lot of that isn't going to work cross-platform. So anyhow:
In the case of wheels, they just need to get into import json
from jupyterlite.addons.base import BaseAddon
import traitlets
# or importlib/pkginfo or wathever
DEFAULT_WHEELS = json.loads(
(Path(__file__).parent / "wheels.json").read_text(encoding="utf-8")
)
class RobotKernelAddon(BaseAddon):
"""Ensures the unique dependencies of robotkernel are available."""
__all__ = ["pre_build"]
wheel_urls = traitlets.List(DEFAULT_WHEELS).tag(config=True)
def pre_build(self, manager):
"""Downloads wheels.""""
for wheel in self.wheel_urls():
dest = self.output_dir / "pypi" / Path(wheel).name
if dest.exists():
continue
yield dict(
actions=[(self.fetch_one, [wheel, dest])],
targets=[dest]
)and then overridden in {
"LiteBuildConfig": {},
"RobotKernelAddon": {
"wheel_urls": [
"..."
]
}
} |
|
Thanks again!
Something else that How is that |
|
Thanks again!
oy, hadn't actually looked at the code, but... a lot of that isn't going
to work cross-platform.
Something else that rm calls, which could be replaced with stdlib calls?
Yeah, that's the first thing that jumped out... Didn't look too closely.
How is that RobotKernelAddon discovered? Is the discovery part of
traitlets (and just because it inherits `BaseAddon``)
Good point, that was missing from the above: they are added by
entry_points, e.g.
https://github.com/jupyterlite/jupyterlite/blob/main/py/jupyterlite/pyproject.toml#L93
If we get some useful pattern, we could work it up as a docs PR... e.g.
"writing a wrapper kernel with pyolite"
|
Oh, so I can just invent my own, like that was the thing, I somehow missed from the beginning. I though, I could only override some of the predefined entry-points. Now all the above makes much more sense! Thanks for your patience! Iterations, iterations... RoboCon is expecting recordings around Mid-April, so there is still some time. I suppose, I can finally make this jupyterlite-robotkernel proper and cross-platform. Next I need to update my Camunda-integration with everything I have learned to have a better demo. Hopefully be able to start outlining around weekend... |
|
Could not get the overriding part to work {
"LiteBuildConfig": {},
"RobotKernelAddon": {
"wheel_urls": [
"..."
]
}
}Any tips to debug, what could be wrong? |
|
Can you point me at a PR where you are working? It's hard out of context...
…On Tue, Mar 29, 2022, 14:33 Asko Soukka ***@***.***> wrote:
Could not get the overriding part to work
{
"LiteBuildConfig": {},
"RobotKernelAddon": {
"wheel_urls": [
"..."
]
}
}
Any tips to debug, what could be wrong?
—
Reply to this email directly, view it on GitHub
<#72 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAALCRFBU3536DXHVN3UO3DVCNLHJANCNFSM472YABVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Actually did some looking: addons are not getting initialized with the parent configuration... PR incoming... |
Does now (just dropped 0.6.0). The room argument is different |
|
@bollwyvl Would you be available to complete our RoboCon recording next week? I'll be working on plan and slides little every day and then the next weekend, and would be available from Mon to Thu (18–21th). Unfortunately, I am not available during the weekend after that, but then again 25th–. Feel free to contact me by email. I'll try to adapt to your available slots. |
Elevator Pitch
Support in-browser, interactive execution of plain-language Task and Test Cases, without a (necessarily) python runtime running on the server.
Motivation
For a number of use cases, like documentation/playground websites, the weight of standing up a full binder is pretty heavy. If we could run all of robotkernel and some of its more fun extensions entirely within the browser, it would be just that much faster to get started.
Key Challenges
Design ideas
JupyterLite is an in-browser implementation of a subset of the Jupyter Server and Kernel APIs hosted together with a re-configured JupyterLab, and any number of supported federated extensions.
At present, the "pyolite" (name may change) kernel built on
pyodideimplements much of theIPythonandipykernelnamespaces. At any rate, it's enough to load uprobotframework:It's likely just a few shims we need to figure out properly to get
await micropip.install("robotkernel")to just work and start experimenting, and be able to put browser/server kernels through the same tests.The text was updated successfully, but these errors were encountered: