Support pyproject.toml and poetry.lock for Python packages

[pawamoy]

A bit surprised not to see this feature request already opened 😄

More and more projects are using Poetry to manage their dependencies. It could be great if Libraries.io could parse these dependencies in the pyproject.toml and poetry.lock files.

Example pyproject.toml:

[tool.poetry.dependencies]
python = "^3.6"
beautifulsoup4 = "^4.8.2"
mkdocs = "^1.1"
pymdown-extensions = ">=6.3, <8.0"
pytkdocs = ">=0.2.0, <0.4.0"

[tool.poetry.dev-dependencies]
bandit = "^1.5"
black = "^19.10b0"
coverage = "^5.0.4"
failprint = "^0.1.1"
flake8 = "^3.6"
flake8-bandit = "^2.1.2"
flake8-black = "^0.1.1"
flake8-builtins = "^1.5.2"
flake8-comprehensions = "^3.2.2"
flake8-docstrings = "^1.5.0"
flake8-isort = "^3.0.0"
flake8-tidy-imports = "^4.1.0"
flake8-variables-names = "^0.0.3"
git-changelog = "^0.2.0"
ipython = "^7.2"
isort = { version = "^4.3", extras = ["pyproject"] }
jinja2-cli = "^0.7.0"
mkdocs-material = ">=4.5, <6.0"
mypy = "^0.770"
pytest = "^4.3"
pytest-cov = "^2.8"
pytest-sugar = "^0.9.2"
pytest-xdist = "^1.26"
requests = "^2.23.0"
toml = "^0.10.0"

Example poetry.lock

[[package]]
category = "dev"
description = "Produce colored terminal text with an xml-like markup"
name = "ansimarkup"
optional = false
python-versions = "*"
version = "1.4.0"

[package.dependencies]
colorama = "*"

[package.extras]
devel = ["bumpversion (>=0.5.2)", "check-manifest (>=0.35)", "readme-renderer (>=16.0)", "flake8", "pep8-naming"]
tests = ["tox (>=2.6.0)", "pytest (>=3.0.3)", "pytest-cov (>=2.3.1)"]

[[package]]
category = "dev"
description = "apipkg: namespace control and lazy-import mechanism"
name = "apipkg"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.5"

[[package]]
category = "dev"
description = "A small Python module for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
name = "appdirs"
optional = false
python-versions = "*"
version = "1.4.3"

[[package]]
category = "dev"
description = "Disable App Nap on OS X 10.9"
marker = "sys_platform == \"darwin\""
name = "appnope"
optional = false
python-versions = "*"
version = "0.1.0"

[[package]]
category = "dev"
description = "Atomic file writes."
name = "atomicwrites"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.3.0"

[pawamoy]

Actually, this feature request could also include support for pdm and pyflow. They are both very similar to Poetry, but bring support for PEP 582: a __pypackages__ directory, similar to node_modules and others. PEP 582 allows to get rid of virtualenvs.

The three tools (poetry, pdm, pyflow) use the TOML format for their lock file and pyproject.toml with the same data structures (I believe both pdm and pyflow got inspiration from poetry for the lock file format and dependencies specification in pyproject.toml, and pdm seems to be fork of poetry).

[brainwane]

pyproject.toml is getting increasingly important to support as there's now a PEP draft coauthored by many key people: "PEP 621: Storing project metadata in pyproject.toml".

[brainwane]

Although people are still working out that syntax, as you can see in https://www.python.org/dev/peps/pep-0621/#how-to-specify-dependencies .

[pawamoy]

Something like this?

def self.parse_pyproject(file_contents)
  manifest = TomlRB.parse(file_contents)
  map_dependencies(manifest['dependencies'], 'runtime') + map_dependencies(manifest['optional-dependencies'], 'develop')
end

def self.parse_pyproject_poetry(file_contents)
  manifest = TomlRB.parse(file_contents)
  map_dependencies(manifest['tool']['poetry']['dependencies'], 'runtime') + map_dependencies(manifest['tool']['poetry']['dev-dependencies'], 'develop')
end

def self.parse_pyproject_pdm(file_contents)
  manifest = TomlRB.parse(file_contents)
  map_dependencies(manifest['tool']['pdm']['dependencies'], 'runtime') + map_dependencies(manifest['tool']['pdm']['dev-dependencies'], 'develop')
end

def self.parse_pyproject_pyflow(file_contents)
  manifest = TomlRB.parse(file_contents)
  map_dependencies(manifest['tool']['pyflow']['dependencies'], 'runtime') + map_dependencies(manifest['tool']['pyflow']['dev-dependencies'], 'develop')
end

def self.parse_poetry_lock(file_contents)
  manifest = TomlRB.parse(file_contents)
  deps = []
  manifest['package'].each do |info|
    group = 'runtime' if info['category'] == 'main' else 'develop'
    deps << {
      name: info['name'],
      requirement: map_requirements(info),
      type: group
    }
  end
  deps
end

And add these to mappings:

match_filename("poetry.lock") => {
  kind: 'lockfile',
  parser: :parse_poetry_lock
},
match_filename("pdm.lock") => {
  kind: 'lockfile',
  parser: :parse_poetry_lock
},
match_filename("pyflow.lock") => {
  kind: 'lockfile',
  parser: :parse_poetry_lock
},
match_filename("pyproject.toml") => {
  kind: 'manifest',
  parser: :parse_pyproject_poetry
}

Not sure how to parse both Poetry/PDM/PyFlow and native dependencies in pyproject.toml.
Also, total guesses on the Ruby syntax 😄

[katzj]

@tyrelsouza implemented in #477

[brainwane]

Awesome! Thank you!

[pawamoy]

Any idea on when this will be available on libraries.io 🙂 ? Or if this is already deployed, when/how the data will start being updated?