[squash] Jupyterlab3 CI updates #454

bollwyvl · 2021-01-03T22:25:28Z

References

hopefully can be cherry-picked (with squash) into JupyterLab 3 support v2 #452
- just this diff: krassowski/jupyterlab-lsp@jupyterlab3...bollwyvl:jupyterlab3-ci

Code changes

Packaging/metdata

moves more files in-tree in packages
- was getting some weird stuff when installed as-packaged
updates install.json and jupyterlab-lsp/package.json to point at the new package name

CI

adds a build step to do the all the distribution builds, uploads for downstream jobs
- nice side-effect: two commands can upload everything from CI-built assets 🎉
- +~4min before any atest jobs start
uses --find-links and --no-index to install whls without knowing the version
- removes this check from integrity
skips js builds altogether on atest
- windows: -~3min
moves js unit tests to lint (since it already has to do a lot with js stuff)
uses mambaforge
- windows: -~2min
tries to cache node_modules, falls back to .yarn-packages
- windows: -~4min
removes the jedi and tectonic cache
- these weren't doing anything positive because we overload $HOME
  - bringing them back would require more research into how to specify these locations with env vars or config, but would very probably reduce some robot timeouts
updates binder postBuild to (probably) work
- haven't tested

timings above are:

vs the last run of JupyterLab 3 support v2 #452

with a hot cache

User-facing changes

n/a

Backwards-incompatible changes

n/a

Chores

linted
tested
documented
changelog entry

krassowski · 2021-01-04T00:58:48Z

I was thinking about making release workflow as separate from test, with some fancy trigger.

bollwyvl · 2021-01-04T01:11:28Z

I was thinking about making release workflow as separate from test,

I think the actual uploading is a separate concern. With this approach, I'm just ensuring we are testing exactly what we would be shipping. The release step could pull from this artifact, though I still am not super-excited about putting my pypi/npm creds into CI, no matter the encryption, etc. Another approach is to create github "draft" releases from tags, and then

with some fancy trigger.

If you do want to put creds in, a nice approach is posting a "draft" github release, which you can then Push The Button to promote it, and have that make the magic happen.

krassowski · 2021-01-04T01:19:20Z

Right. Though it does not that bad - no need for full credentials, just API token which may be limited in scope to only one project, as per https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

Edit: also missed one lint issue earlier - sorry!

bollwyvl · 2021-01-04T01:22:21Z

Right, but I like those creds to only be available to the release job, and not to every random npm post-install script and github action.

bollwyvl · 2021-01-04T06:01:24Z

Calling it for the day. I updated the pr description with all the stuff I've been fiddling with. This is probably ready to be merged into #452, if there's nothing overly-offensive happening, but haven't retargeted as:

i'll happily fix anything you don't like
I don't know if re-targeting would kill the currently-running jobs which would have the most updated timings

Still no update on conda-forge/jupyter_server-feedstock#34 for py36 builds, maybe the PR fairy will come over night 🧚 .

bollwyvl · 2021-01-04T13:42:34Z

Timings look good... getting close to 10 minutes saved on the slowest windows excursion, and inching us back from the dreaded hour.

Future versions of setup-miniconda will obviate a bunch of the hackery this introduces, as well.

Without doing more entropy plumbing with make/doit (e.g. pre-solved lock files), I don't think I can get it down much more... I guess the build could use setup-node and setup-python, which could shave another minute. Robot waiting Until Keyword Suceeds is definitely the long pole: I've found pabot to be awesome on linux/osx (up to about 4 processes on CI, basically linear speedup, though the server gets stood up more) but windows can't seem to handle it, so wouldn't move the needle much.

Will re-target, and will be off open source stuff (but occasionally checking email) for the next $DAY_JOB worth of hours... I've no idea what started on fire over the holidays!

krassowski · 2021-01-04T19:45:23Z

python_packages/jupyterlab_lsp/jupyterlab_lsp/install.json

-  "packageName": "jupyter_lsp",
-  "uninstallInstructions": "Use your Python package manager (pip, conda, etc.) to uninstall the package jupyter_lsp"
+  "packageName": "jupyterlab_lsp",
+  "uninstallInstructions": "Use your Python package manager (pip, conda, etc.) to uninstall the package jupyterlab_lsp"


krassowski · 2021-01-04T19:47:10Z

packages/jupyterlab-lsp/package.json

@@ -130,7 +129,7 @@
    "discovery": {
      "server": {
        "base": {
-          "name": "jupyter-lsp"
+          "name": "jupyterlab_lsp"


Well, if I publish another npm release and someone installs from npm, jupyter-lsp would suffice. Otherwise they would not see this anyway.

yeah, who knows. just thought of it while fixing the other one...

krassowski · 2021-01-04T19:57:42Z

.github/workflows/job.test.yml

  acceptance:
    runs-on: ${{ matrix.os }}-latest
-    name: ${{ matrix.os }} py${{ matrix.python }} node${{ matrix.nodejs }}
+    name: atest ${{ matrix.os }} py${{ matrix.python }}


Not a fan of atest abbreviation, but we already use it so it can stay I guess.

we can call it whatever you like! but shorter is better so we can get some info off the UI. robot, browser, etc.

if they could be emoji, i'd be all for 🤖

krassowski · 2021-01-04T20:03:10Z

Great work, I like the idea with building once! I restarted the CI to make sure it works with 3.6.

bollwyvl · 2021-01-04T20:57:46Z

omg so green 🤯

bollwyvl force-pushed the jupyterlab3-ci branch from c8f924f to 304edb7 Compare January 4, 2021 00:53

bollwyvl force-pushed the jupyterlab3-ci branch from b44e168 to 6ac92ef Compare January 4, 2021 01:03

bollwyvl mentioned this pull request Jan 4, 2021

JupyterLab 3 support v2 #452

Merged

10 tasks

bollwyvl added 7 commits January 3, 2021 20:55

add build step to test workflow

8bef655

get funky characters out of cache name

7110bb8

remove integrity check vs ci versions

3246ce1

fix mambaforge url again

8832e1a

fix test mamab urls

1454a71

some more mamba invocation work

b60659d

combine all CI jobs, slim down node build steps

6d49e83

bollwyvl force-pushed the jupyterlab3-ci branch from 4f9c475 to 6d49e83 Compare January 4, 2021 01:56

bollwyvl added 15 commits January 3, 2021 21:01

apparently offline not really supported

0b4587c

more shell stuff for windows

73e7645

add pip to gha.yml

106455c

update postBuild

fe69aa6

move more files in-tree

8acee1a

some more work on (re)building packages

0c4995a

remove --no-banner

ad66f6b

re-build for frontend unit tests, only on linux

62e7fa9

more shell work

ad02e5e

only run frontend unit on _linux_

5f54d5f

clean up prettierignore

24455ca

try conservative caching of node_modules

4c123f6

fix jlpm var

2fae4fa

move js utest to lint

b57e248

job names

0b70041

bollwyvl added 4 commits January 4, 2021 00:21

remove LINKED_EXTENSIONS

cc42c1f

remove lab:link commands

843bc7a

update module name in install.json

dc3cca7

update discovery data

750c757

bollwyvl changed the base branch from master to jupyterlab3 January 4, 2021 13:42

bollwyvl closed this Jan 4, 2021

bollwyvl reopened this Jan 4, 2021

bollwyvl changed the title ~~[wip] Jupyterlab3 CI updates~~ [squash] Jupyterlab3 CI updates Jan 4, 2021

krassowski reviewed Jan 4, 2021

View reviewed changes

krassowski merged commit f59f598 into jupyter-lsp:jupyterlab3 Jan 4, 2021

krassowski mentioned this pull request Mar 26, 2021

Instruction to download jupyterlab-lsp on jupyterlab GUI is wrong #568

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[squash] Jupyterlab3 CI updates #454

[squash] Jupyterlab3 CI updates #454

bollwyvl commented Jan 3, 2021 •

edited

Loading

krassowski commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

krassowski commented Jan 4, 2021 •

edited

Loading

bollwyvl commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

krassowski Jan 4, 2021

krassowski Jan 4, 2021

bollwyvl Jan 4, 2021

krassowski Jan 4, 2021

bollwyvl Jan 4, 2021

bollwyvl Jan 4, 2021

krassowski commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

[squash] Jupyterlab3 CI updates #454

[squash] Jupyterlab3 CI updates #454

Conversation

bollwyvl commented Jan 3, 2021 • edited Loading

References

Code changes

Packaging/metdata

CI

User-facing changes

Backwards-incompatible changes

Chores

krassowski commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

krassowski commented Jan 4, 2021 • edited Loading

bollwyvl commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

krassowski Jan 4, 2021

Choose a reason for hiding this comment

krassowski Jan 4, 2021

Choose a reason for hiding this comment

bollwyvl Jan 4, 2021

Choose a reason for hiding this comment

krassowski Jan 4, 2021

Choose a reason for hiding this comment

bollwyvl Jan 4, 2021

Choose a reason for hiding this comment

bollwyvl Jan 4, 2021

Choose a reason for hiding this comment

krassowski commented Jan 4, 2021

bollwyvl commented Jan 4, 2021

bollwyvl commented Jan 3, 2021 •

edited

Loading

krassowski commented Jan 4, 2021 •

edited

Loading