Missing "formats" attribute for autogenerated metadata

[bxsx]

Hi,

I have setup the following lines in the Jupyter config file:

c.NotebookApp.contents_manager_class = 'jupytext.TextFileContentsManager'
c.LabApp.contents_manager_class = 'jupytext.TextFileContentsManager'
c.ContentsManager.default_jupytext_formats = "ipynb,py"
c.ContentsManager.preferred_jupytext_formats_save = "py:percent"

While creating a new ipynb file via Jupyter Lab, the Python file is also created and everything seems to work (updating content between files).

However, I noticed that the header of the files is missing a formats information.

What I get in my files?

 "metadata": {
  "jupytext": {
   "text_representation": {
    "extension": ".py",
    "format_name": "percent",
    "format_version": "1.3",
    "jupytext_version": "1.6.0"
   }
  },

# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.6.0

What I was expecting?

"metadata": {
  "jupytext": {
   "formats": "ipynb,py:percent"
   "text_representation": {
    "extension": ".py",
    "format_name": "percent",
    "format_version": "1.3",
    "jupytext_version": "1.6.0"
   }
  },

# jupyter:
#   jupytext:
#     formats: ipynb,py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.6.0

There is a missing formats attribute. Maybe it's not required anymore. However, I noticed that in Commands view (can't say for menu as it's not working for me: #630) the "Unpair Notebook" command is disabled.

When I use jupytext --set-formats ipynb,py:percent Foo.ipynb then the generated metadata contains formats attribute and "Unpair Nootebook" command is available.

[mwouts]

Hi @bartskowron , well you are pointing out at two different facts:

1. The default_jupytext_formats value is not set on .ipynb files. Well that is a choice we made a long time ago, and many users prefer it to be this way: you can pair an ipynb file to a text file without impacting at all the .ipynb file. I really mean at all - the jupytext metadata that you show above is filtered when the notebook is saved on disk.
2. The Jupytext Lab extension (and the Jupytext Menu) are not aware of the default pairing. This is a known (and old) issue, see Jupytext extensions should be aware of the default Jupytext configuration #177. I have not fixed it yet because 1. I am not so fluent in Javascript & Typescript and 2. it was not clear how to access the default pairing information. But of course we can think about it!

[bxsx]

1. The default_jupytext_formats value is not set on .ipynb files. Well that is a choice we made a long time ago, and many users prefer it to be this way: you can pair an ipynb file to a text file without impacting at all the .ipynb file. I really mean at all - the jupytext metadata that you show above is filtered when the notebook is saved on disk.

Thanks for the clarification. However the default_jupytext_formats is not also set on .py files. So where is it set then?
Do you also mean that using jupytext --set-formats ipynb,py:percent Foo.ipynb is not the best way as this interact with the .ipynb files which is not necessary?

When I changed in the config file to c.ContentsManager.default_jupytext_formats = "py" then no .py file was created when I was creating a new Notebook file.
`
I'm basically looking for solutions for the following scenarios:

• starting from a Python source code that I want to convert and pair with Notebook file
• starting from a Notebook that I want to convert and pair with Python source

I have a feeling that I'm doing something wrong even if based on the documentation ;) I mean - it's working - but as I'm understanding there are some redundancy.

2. The Jupytext Lab extension (and the Jupytext Menu) are not aware of the default pairing. This is a known (and old) issue, see Jupytext extensions should be aware of the default Jupytext configuration #177. I have not fixed it yet because 1. I am not so fluent in Javascript & Typescript and 2. it was not clear how to access the default pairing information. But of course we can think about it!

Pity but promising ;-)

[mwouts]

I'm basically looking for solutions for the following scenarios (...)

Well you have the choice between

1. Using the Jupyter Lab extension (or the menu) to pair the notebook (or the script opened as a notebook)
2. Using jupytext --set-formats ... on the .py or .ipynb files
3. Using a configuration file. Rather than using jupyter_notebook_config.py (the only option until version 1.5.0), I'd advice to use a jupytext.toml file in the notebook folder, or in a parent folder. See https://jupytext.readthedocs.io/en/latest/config.html#jupytext-configuration-file

Note that 1. and 2. have the same effect - they create a jupytext.formats metadata on both the .py and .ipynb files. 3. is a bit different since it does not create that metadata - the pairing is for all the notebook in the folder, unless they have another value for jupytext.formats.

[bxsx]

Oh, I missed this part of the manual! It's completely clear now.

Rather than using jupyter_notebook_config.py (the only option until version 1.5.0), I'd advice to use a jupytext.toml file in the notebook folder, or in a parent folder.

However, these 2 lines have to be in Jupyter config file?

c.NotebookApp.contents_manager_class = 'jupytext.TextFileContentsManager'
c.LabApp.contents_manager_class = 'jupytext.TextFileContentsManager'

If so, I would rather keep the Jupytext all settings together under the Jupyter config file.

[mwouts]

However, these 2 lines have to be in Jupyter config file? (...)

They don't... at least when the jupytext server extension is active, which seems to be ok for you according to #630

[bxsx]

You are right. Thanks for all the informations. As everything said here is also documented in the documentation: https://jupytext.readthedocs.io/en/latest/config.html
I think this issue can be closed 👍
Thanks for the help!