docstrings for compiled builtin modules

[DetachHead]

fixes #160

a few things we need to address before merging tho:

• this currently only generates stubs for the python version & platform build script is running on. need to set up github actions to run them on every platform and python version, then combine them at the end
• currently this PR drops support for python 3.8 because docify doesn't support it. we should probably wait until 3.8 goes eol before merging this, or do you think it's feasible to add 3.8 support @AThePeanut4?
  • edit: turned out this was as easy as downgrading libcst to 1.1.0
• doesn't work on the playground because it uses the npm package, and docify is currently only run in the pypi package
• doesn't work when using the version of basedpyright bundled with the vscode extension for the same reason. (though not an issue if you have the pypi package installed and are using the "fromEnvironment" importStrategy)
• document it, and also mention the docify repo if users need to generate docs for their own stubs
• add tests

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

[AThePeanut4]

@DetachHead python 3.8 has a few other minor issues besides just the libcst version, I've pushed a fix to support 3.8 fully. I think you missed 3.8 in based_build/generateAllDocstubs.sh.

I've also added a run function that accepts kwargs rather than a list of string arguments, for easier usage when running programmatically.

[DetachHead]

oh oops, nice catch. thanks!

fixed in #423

[jfcherng]

I think docstrings can be added to the whole stdlib. Current it seems to be only done with builtin.pyi?

[DetachHead]

yeah i did that on purpose. i only made it run on compiled builtins because i figured running it on the entire stdlib would unnecessarily make it slower to build and increase the risk of something going wrong. since pyright can already see docstrings from all the other stdlib modules anyway i didn't see the point.

feel free to raise an issue if you disagree. i'm open to changing this behavior in the future if that's what people want

[jfcherng]

Ah, got it. We only need it for compiled modules. NVM.

[jfcherng]

Ah, got it. We only need it for compiled modules. NVM.

Hmm... doesn't seem to be this case.

import re

if m := re.search('foo', 'foo'):
    aa = m.group()
    #      ^^^^^ no docstring for `group`

>>> import inspect, re; m = re.search('foo', 'foo'); inspect.getdoc(m.group)
'group([group1, ...]) -> str or tuple.\nReturn subgroup(s) of the match by indices or names.\nFor 0 returns the entire match.'

There can a lot of built-in things still implemented in C. So I would still consider that dosctrings should be generated for the whole stdlib typeshed-fallback.

[DetachHead]

oh man that's annoying, i didn't know there were modules that are only partially compiled. also it looks like pylance has the same issue

i opened #428 to track this