Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support very long path names for compiled artifacts on Windows #2558

Closed
drewbanin opened this issue Jun 17, 2020 · 4 comments · Fixed by #2566
Closed

Support very long path names for compiled artifacts on Windows #2558

drewbanin opened this issue Jun 17, 2020 · 4 comments · Fixed by #2566
Labels
bug Something isn't working windows Everyone's favorite OS that's sometimes a little weird
Milestone

Comments

@drewbanin
Copy link
Contributor

drewbanin commented Jun 17, 2020

Describe the bug

A user is reporting an issue when running dbt compile. The error is for a schema test .sql artifact, and it reads:

No such file or directory: "target\\compiled\\<230 more characters>.sql"

My guess is that the full path to this file is too long, and Windows either truncates the file path, or does something else unexpected, when dbt tries to write out this sql file.

Instead, let's catch this error, log a warning, and move on. Alternatively if there's a way to increase the file path length on windows to beyond ~260 chars, we should do that.

This error may start happening for folks on 0.17.0 because we are now including more information (like the schema.yml file name) in the path for the .sql artifact.

Steps To Reproduce

  • Run dbt compile on a project that would try to write a filepath > 260 chars long

Expected behavior

  • dbt should not fail here, but it's ok for dbt to omit writing these artifacts if need be

The output of dbt --version:

0.17.0

The operating system you're using: Windows 10 (Windows Server 2019)

The output of python --version: TBD

@drewbanin drewbanin added bug Something isn't working windows Everyone's favorite OS that's sometimes a little weird labels Jun 17, 2020
@beckjake
Copy link
Contributor

I believe this is pretty fixable! We need to prefix all our paths with r'\\?\' on windows, and make sure to normalize any paths we find because that syntax doesn't accept any / in paths. This is mostly hard because it's tedious to get every spot that writes files, not because it's actually a hard problem.

@drewbanin
Copy link
Contributor Author

Right on, thanks @beckjake! This one isn't really an 0.17.0 regression -- I'd be happy to do a cheapo version for 0.17.1 (that just catches the error and logs a warning) in the near-term, then do a more fundamental overhaul of windows pathing for our next scheduled feature release. What do you think?

@beckjake
Copy link
Contributor

I think if we're catching this error, we might as well fix the path and try again, rather than log a warning.

@beckjake beckjake mentioned this issue Jun 17, 2020
4 tasks
@drewbanin drewbanin added this to the 0.17.1 milestone Jun 18, 2020
@drewbanin
Copy link
Contributor Author

closed by #2566

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working windows Everyone's favorite OS that's sometimes a little weird
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants