Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug with loading dependencies when running mypy #18519

Open
ryanking opened this issue Mar 16, 2023 · 18 comments
Open

bug with loading dependencies when running mypy #18519

ryanking opened this issue Mar 16, 2023 · 18 comments
Labels
backend: Python Python backend-related issues bug

Comments

@ryanking
Copy link
Contributor

Pants version
2.16, but also was happening on 2.15

OS
Mac

I have been trying to hunt down a bug in pants for a few weeks and finally have a reproduction, but don't know what do do with it.

From time to time, a mypy run will fail due to missing dependencies. Dependencies which are definitely configured and findable. Often a re-run will succeed, which has made this very hard to reproduce.

I now found a case where running a single file will fail, but running with :: will succeed. I have both sandboxes and when I edit __files.txt in the latter it passes. However I don't know what to look for in these sandboxes to figure out what's going on.

Failure from the "bad" sandbox:

.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: error: Library stubs not installed for "six.moves.http_client"  [import]
    from six.moves import http_client
    ^
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: Hint: "python3 -m pip install types-six"
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: (or run "mypy --install-types" to install all missing stub packages)
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: error: Library stubs not installed for "six.moves"  [import]
    from six.moves import http_client
    ^
src/django_apps/counseling/google_calendar.py:74:14: error: Module has no attribute "parser"  [attr-defined]
                (dateutil.parser.parse(e['start']), dateutil.parser.parse(e['end']))
                 ^~~~~~~~~~~~~~~
Found 3 errors in 2 files (checked 1 source file)

Success from the good sandbox, edited to run only the file in question:

Success: no issues found in 1 source file

A fun complication is that the mypy cache appears to get poisoned. If I have a failed run on the "bad" sandbox, the good one will then fail. If I reset the cache it will work.

@ryanking ryanking added the bug label Mar 16, 2023
@kaos
Copy link
Member

kaos commented Mar 17, 2023

Huh, interesting observation regarding the cache. I too have seen this issue on and off for months without being able to pin it down.

@tdyas tdyas added the backend: Python Python backend-related issues label Mar 17, 2023
@ryanking
Copy link
Contributor Author

Yeah I think there are potentially two problems here:

  1. I somehow, sometimes get a sandbox that is missing some dependencies
  2. incorrect results sometimes get in the mypy cache. this may not be a pants specific problem but might be obscuring problems in pants

@ryanking
Copy link
Contributor Author

Spent some time debugging this with @stuhood in slack DM. A few things I have found–

  1. It does not appear to be related to caching. I get the same behavior with pants --no-local-cache --no-remote-cache-read --no-remote-cache-write --no-pantsd check and incremental = False in mypy.ini
  2. This might be very specific to dateutil, as changing from import dateutil to from dateutil import parser seems to have solved the problem :tableflip:

@achimnol
Copy link

@achimnol
Copy link

This may be related with the "--changed" subsystem.
image
image

@achimnol
Copy link

achimnol commented Jun 5, 2023

Patching the PR like

diff --git a/BUILD b/BUILD
index c9bda9306..8d74b8143 100644
--- a/BUILD
+++ b/BUILD
@@ -23,7 +23,7 @@ python_requirements(
         "types-PyYAML": ["yaml"],
         "types-python-dateutil": ["dateutil", "dateutil.parser", "dateutil.tz"],
         "types-redis": ["redis"],
-        "types-six": ["six"],
+        "types-six": ["six", "graphql", "promise"],
         "types-tabulate": ["tabulate"],
     },
 )

resolves the issue.

So this seems to be an issue of mismatch in transitive dependency resolution/requirement in mypy and pants.

@achimnol
Copy link

achimnol commented Jun 7, 2023

Still happens, especially with external package (graphql, promise, _pytest) imports where those have py.typed.

@achimnol
Copy link

achimnol commented Jun 7, 2023

I had to do it like lablup/backend.ai@c7bb82e

diff --git a/BUILD b/BUILD
index f85cfa7b..d9b923ba 100644
--- a/BUILD
+++ b/BUILD
@@ -23,8 +23,13 @@ python_requirements(
         "types-PyYAML": ["yaml"],
         "types-python-dateutil": ["dateutil", "dateutil.parser", "dateutil.tz"],
         "types-redis": ["redis"],
-        "types-setuptools": ["setuptools", "pkg_resources", "_pytest"],
-        "types-six": ["six", "graphql", "promise", "_pytest", "ai.backend"],
+        "types-setuptools": [
+            "setuptools",
+            "pkg_resources",
+            "ai.backend.testutils",
+            "ai.backend.test",
+        ],
+        "types-six": ["six", "graphql", "promise", "ai.backend.manager.models"],
         "types-tabulate": ["tabulate"],
     },
 )

@kuza55
Copy link

kuza55 commented Mar 19, 2024

I'm running into some flakiness with mypy too. Blowing away the cache is unnecessary, merely re-running pants check is sufficient. I am running into this both in pre-commit as well as just running pants check from the command line.

I am getting this error with other modules than mentioned here:

.cache/pex_root/venvs/d17349559f692efab258d5f40c73f81d7382b44c/2540ac6a914a2a78bfd89ca0676ffdc7f0dfe44f/lib/python3.11/site-packages/pandas-stubs/io/formats/style_render.pyi:11: error: Cannot find implementation or library stub for module named "jinja2"  [import-not-found]
.cache/pex_root/venvs/d17349559f692efab258d5f40c73f81d7382b44c/2540ac6a914a2a78bfd89ca0676ffdc7f0dfe44f/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py:57: error: Library stubs not installed for "requests.models"  [import-untyped]

I have a separate lockfile for mypy due to some issue with the version of mypy that is bundled with pants, no idea if related.

The error seems relatively new, I don't recall seeing anything like this pre 2.19.0

@isra17
Copy link
Contributor

isra17 commented Mar 19, 2024

Having a similar bug with opensearch-py and mypy complaining about types-simplejson or type-six. Happens only once, and then it succeed, even with --no-pantsd and --no-local-cache. I am unfortunately unable to reproduce reliably or find anything specific in the failure case.

@isra17
Copy link
Contributor

isra17 commented Mar 19, 2024

Having a similar bug with opensearch-py and mypy complaining about types-simplejson or type-six. Happens only once, and then it succeed, even with --no-pantsd and --no-local-cache. I am unfortunately unable to reproduce reliably or find anything specific in the failure case.

Not sure if related, but looking at Pants log, it seems like the the requirements.pex get cancelled:

20:28:53.99 [INFO] Starting: Building 3 requirements for mypy.pex from the pants/3rdparty/python/mypy.lock resolve: mypy==1.7.0, pydantic==1.10.14, sqlalchemy[mypy]==1.4.51
20:28:55.40 [INFO] Starting: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)
20:28:55.62 [INFO] Canceled: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)
20:28:55.62 [INFO] Starting: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)
20:28:59.10 [INFO] Completed: Building 3 requirements for mypy.pex from the pants/3rdparty/python/mypy.lock resolve: mypy==1.7.0, pydantic==1.10.14, sqlalchemy[mypy]==1.4.51
20:29:10.23 [INFO] Completed: Building 66 requirements for requirements.pex from the pants/3rdparty/python/default.lock resolve: authlib==0.15.4; python_full_version >= "3.11.3" and python_full_version < "3.12.0", bcrypt==4.1.2; p... (6017 characters truncated)

Now, even successful run has the Cancelled: Building ..., but the one run that failed has a [DEBUG] Canceled: setup_sandbox right before. Am I getting warmer?

@isra17
Copy link
Contributor

isra17 commented Mar 19, 2024

I'm actually thinking again, and doesn't it look like mypy is parsing these third-parties file directly?

Playing in a venv with a similar configuration, I can run mypy without any of my dependency's dependencies and mypy will set them as Any. I don't think there should be error from missing import in third-party libraries.

This might be related to python/mypy#12162

@isra17
Copy link
Contributor

isra17 commented Mar 19, 2024

Now, even successful run has the Cancelled: Building ..., but the one run that failed has a [DEBUG] Canceled: setup_sandbox right before. Am I getting warmer?

Disregard this, I was able to get another failure and it didn't have this log.

@isra17
Copy link
Contributor

isra17 commented Mar 19, 2024

I found it. This is an issue with mypy cache. If it runs again without some 4th-party dependencies, mypy will raise an error instead of settings these import as Any as usual. I filed a bug and easy repro here: python/mypy#17049

From pants side of things, I don't know if there could be a way to make the mypy cache specific to the venv and avoid reusing the same cache with different dependencies.

@isra17
Copy link
Contributor

isra17 commented Mar 20, 2024

FYI,

# pants.toml
[mypy]
args = ["--no-incremental"]

Should fix it until this bug get fixed in mypy.

@hauntsaninja
Copy link

python/mypy#17420 should fix a thing here

@cburroughs
Copy link
Contributor

Thank you for the work on python/mypy#17420 !

With both 1.10.1 and 1.11.0 I'm still unable to enable the incremental cache on our repo due to errors like:

foon.py:1:1: error: Cannot find implementation or library stub for module named "cvxpy.atoms.sum_squares"  [import-not-found]
    from dataclasses import dataclass
    ^
Found 126 errors in 41 files (checked 12 source files)

I have not been able to find a reproduction case yet. Note that here cvxpy.atoms.sum_squares is obviously not dataclasses so this seems like a different error. Like the original case, a re-run fixes it. All of the examples so far are with the same package (cvxpy) which I find puzzling.

I'm interested if others have had more luck.

@isra17
Copy link
Contributor

isra17 commented Aug 2, 2024

Yes, sorry didn't have much time to come back, but I had similar issues...

Unfortunately, unlike the previous instance of the bug, I'm unable to make a minimal repro. Like before, the cache end up in a state that will yield an error, but succeed on the next run. If I take a copy and reuse it I can reproduce the failure. (I can't share it since it's from our internal project though).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend: Python Python backend-related issues bug
Projects
None yet
Development

No branches or pull requests

8 participants