Can you help pairing ipynb and R format using spin ?

[statquant]

Hello @mwouts and thank you for jupytext it really looks amazing.
From the docs I think I understand that not only can I sync ipynb and Rmd formats (I succeeded in doing that out of the box) but I should even be able to do ipynb and R given that running knitr::spin on a R file generates a Rmd I kinda can see the magic here.

Is my understanding correct ?
Would you be kind enough to show me a minimal reproducible example for this ?
Many thanks

[statquant]

Actually I looked into other issues with spin and figured out how to do it.
I'll leave a trace here for a next wonderer

bash) mkdir /tmp/workfow; cd !$
bash) wget https://raw.githubusercontent.com/daattali/UBC-STAT545/master/hw/hw12_web-scraping-api/hw12_web-scraping-api.R
bash) jupytext --set-formats R:spin,ipynb,Rmd --sync hw12_web-scraping-api.R
# Here you are everything works out of the box, that's purely amazing
bash) jupyter lab

I can see the following

This is truly amazing

[mwouts]

Thank you @statquant for your feeback. I'm glad you like the program!

Yes, that's it, you can put as many formats as you want in --set-formats, or in the jupyter.jupytext.formats metadata in the YAML header of the document.

If you want to use two different formats with the same extension, you can use a extension prefix. This is how we pair the world population example notebook to seven different formats:

jupytext/demo/World population.Rmd

Line 5 in 55e3e80

formats: ipynb,.pct.py:percent,.lgt.py:light,.spx.py:sphinx,md,Rmd,.pandoc.md:pandoc

Also, the R:spin format is specific to R, and it may lack the support for cell metadata. If that turns out to be an issue, you can give a try to the other formats (light or percent).

[statquant]

Hello @mwouts I actually want to use R:spin.
Being able to use native R files solves what I think to be the major problem of notebooks: the fact that they break the workflow (necessity to add cells, no multipage for coding, no proper console)
As it is I can stay totally in R (I use vim and nvimR), and I get to keep all the greatness of rmarkdown::render.

The only thing I do not know how to do is how to add other yaml instructions that the ones that jupytext adds.
For instance if I modify the yaml to

  #' ---
  #' title: "An example Knitr/R Markdown document"
  #' author: "Karl Broman"
  #' date: "3 Feb 2015"
  #' jupyter:
  #'   jupytext:
  #'     cell_metadata_filter: fig.height,fig.width,tags,cache,name,-all
  #'     formats: R:spin,ipynb,Rmd
  #'     text_representation:
  #'       extension: .R
  #'       format_name: spin
  #'       format_version: '1.0'
  #'       jupytext_version: 1.3.2
  #' ---

I get the spurious following on jupyterlab

So I need to either find a way to filter this with jupytext or find a way to specify those options in rmarkdown::render (or knitr::spin)

[statquant]

Actually maybe you can help me on this tiny detail, it seems to be something that should be handle in jupytext, I opened a SO question but it feels really wrong

[mwouts]

Well representing the non-jupyter part of the YAML header in a raw cell allows makes it easy to edit in the notebook... But I understand that you may not want to see that in the notebook.

Maybe we could offer a boolean option root_metadata_under_jupytext, that would have the effect to store the non-jupyter part of the YAML header under jupytext.root_metadata when the file is converted to a notebook. Let me ask @grst what he thinks about this.

[statquant]

Hello @mwouts

Well representing the non-jupyter part of the YAML header in a raw cell allows makes it easy to edit in the notebook.

Do you mean that people would like to be able to modify the non jupyter yaml part in a jupyter notebook even if it is likely that they won’t be able to act upon it in jupyter ?
I did not think about this but now that you mention it I can see that if you consider jupyter to be an editor it makes sense.
But then shouldn’t we also be able to see the jupyter part of the yaml ?

Many thanks

[grst]

@statquant, I'm not sure if I understand what you are trying to achieve... What's your rationale for not having a yaml header? I suppose it's an integral part of the Rmd/spin formats.

@mwouts, about the yaml headers:

• I agree that is convenient to edit the metadata in the raw cell.
• We discussed earlier (Getting the title from jupyter notebooks (ipynb) jgm/pandoc#5905 (comment)) that it might be useful to have the contents of yaml in the notebook metadata, too, so that e.g. pandoc can read it.
• I'm not a big fan of adding too many configuration options that change the way the metadata is structured. It makes everything more complicated.

[statquant]

Hello @grst my view on this is that that either we should have access (hence display) to all the yaml preambules (in my example the jupyter: is hidden in jupyter but the author: date: output: are visible) or none at all.
Because (please correct me if I am wrong) in jupyter the "Rmd" yaml information will not be acted upon I see little case for displaying it.
Functionality aside I think it would look much cleaner without
Regards

[statquant]

Closing this

[mwouts]

Hello @statquant , sorry I did not follow up on your previous comment. I think your question is completely legitimate, and that not displaying the (non-jupyter) metadata in the .ipynb notebook may be more natural than mapping them to the raw cell. Would you mind if I reopen this, and give it a try when time permits?

[statquant]

Hello @mwouts sorry for my late answer. Thank you very much for your message, feel free to reopen. That's much appreciated

[mwouts]

@statquant , with the upcoming release you will be able to not get the raw cell by adding root_level_metadata_as_raw_cell = false to your jupytext.toml file.

If you want you can give a try to the development version, use

pip install jupytext==1.6.0rc0

[statquant]

Hello, I took a test at this but can't make it work.
I added a jupytext.toml then a .jupytext file containing root_level_metadata_as_raw_cell = False but for some reason it seems to be breaking something as starting jupyter-lab --notebook-dir ${PWD} leads to an "empty" jupyter window.

[mwouts]

Hi @statquant , thanks for reporting this. My bad...I said False above but it should have been false, because this is TOML 😄
Anyway I see that the error in the config file is very hard to spot, I will see how to improve this, see #613 .

[statquant]

Hello @mwouts indeed it now works perfectly. Many thanks for this that's much appreciated.