-
Notifications
You must be signed in to change notification settings - Fork 570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error on windows when trying to deploy - [WinError 32] #494
Comments
Hmm
Can you check that you aren't running multiple Scrapyd processes? |
Yeah, i wasn't. I even rebooted my pc and retried everything with a clean environment |
Does Windows have a utility to determine which processes are using a given file? On Linux, I don't think Scrapyd is causing the issue, as only one process would be trying to access the egg. |
Same issue in win11, Inserting |
I've committed a fix to HEAD. Can you test with the version of Scrapyd from GitHub? |
It seems to have no relation to import os
import shutil
import sys
import tempfile
# from contextlib import contextmanager
from scrapy.utils.misc import load_object
from scrapyd import Config
from scrapyd.eggutils import activate_egg
def project_environment(project):
eggversion = os.environ.get('SCRAPYD_EGG_VERSION', None)
config = Config()
eggstorage_path = config.get(
'eggstorage', 'scrapyd.eggstorage.FilesystemEggStorage'
)
eggstorage_cls = load_object(eggstorage_path)
eggstorage = eggstorage_cls(config)
version, eggfile = eggstorage.get(project, eggversion)
if eggfile:
prefix = '%s-%s-' % (project, version)
f = tempfile.NamedTemporaryFile(suffix='.egg', prefix=prefix, delete=False)
shutil.copyfileobj(eggfile, f)
f.close()
activate_egg(f.name)
else:
f = None
return f
# try:
# assert 'scrapy.conf' not in sys.modules, "Scrapy settings already loaded"
# yield
# finally:
# if f:
# os.remove(f.name)
def main_finally():
project = os.environ['SCRAPY_PROJECT']
f = None
try:
f = project_environment(project)
from scrapy.cmdline import execute
execute()
finally:
if f:
os.remove(f.name)
def main():
project = os.environ['SCRAPY_PROJECT']
f = None
f = project_environment(project)
from scrapy.cmdline import execute
execute()
if f:
os.remove(f.name)
if __name__ == '__main__':
main() # work fine
# main_finally() # rasie |
They are connected - if the tempfile is never created, then it can never be removed. Anyway, can you add eggfile.close() after f.close() to see what happens? |
The error doesn’t occur when you remove exception handling, because Scrapy raises SystemExit, which causes the process to end - but we’re trying to capture that |
import os
import sys
import tempfile
import shutil
import operator
import functools
import pkg_resources
import itertools
from importlib.metadata._itertools import unique_everseen
from importlib.metadata import distributions
def activate_egg(eggpath):
"""Activate a Scrapy egg file. This is meant to be used from egg runners
to activate a Scrapy egg file. Don't use it from other code as it may
leave unwanted side effects.
"""
try:
d = next(pkg_resources.find_distributions(eggpath))
except StopIteration:
raise ValueError("Unknown or corrupt egg")
d.activate()
settings_module = d.get_entry_info('scrapy', 'settings').module_name
os.environ.setdefault('SCRAPY_SETTINGS_MODULE', settings_module)
def main():
eggfile = open('./0_1_0.egg', 'rb')
f = None
try:
f = tempfile.NamedTemporaryFile(suffix='.egg', delete=False)
shutil.copyfileobj(eggfile, f)
activate_egg(f.name)
f.close()
# from scrapy.cmdline import execute
# execute(['C:\\Users\\Sanze\\AppData\\Local\\pdm\\pdm\\Cache\\packages\\scrapyd-1.4.2-py2.py3-none-any\\lib\\scrapyd\\runner.py', 'list', '-s', 'LOG_STDOUT=0'])
# traceback
#
# scrapy.cmdline.execute
# |
# scrapy.cmdline._get_commands_dict
# |
# scrapy.cmdline._get_commands_from_entry_points
# |
# importlib.metadata.entry_points
norm_name = operator.attrgetter('_normalized_name')
unique = functools.partial(unique_everseen, key=norm_name)
list(dist.entry_points for dist in unique(distributions()))
#### cause PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
finally:
if f:
os.remove(f.name)
if __name__ == '__main__':
main() |
As I requested, please test by add |
Please test HEAD again – I added that line myself, and also got rid of the temporary file. |
I had already tired it, it still threw that error. The key is |
That line must be opening the file a second time. But the error must be that it’s opened a first time somewhere else. I doubt the Python standard library (importlib) has a Windows error that opens files twice on its own. Can you try the new HEAD from GitHub? |
I checked the HEAD, having no changes. Did you have any committed code. |
Ah, sorry, I forgot to push: you can try now. |
I have tried it, it works. |
Thank you for confirming! |
I have successfully installed and run scrapyd on Windows. However, when i try to deploy to scrapyd I have the following error:
I have tried with simple scrapyd API (curl http://localhost:6800/addversion.json -F project=myproject -F version=r23 -F egg=@myproject.egg) and with scrapy-deploy from scrapy-client.
I am using Windows 11 with python 3.11.4
Any help will be appreciated!
The text was updated successfully, but these errors were encountered: