__del__ is insufficient - need a close() #596
Comments
|
I second that. I'd love to have a context manager that takes care of the cleanup. |
|
Thanks very much for the insight @Julian-O. Let's maybe wait for one or two other opinions, but if it was all good, would you be able to clean the code and submit a PR ? |
|
I agree.. I'd like to see if his suggestions will fix the issue. I don't see any harm in any of the things he is proposing. |
|
@gyglim: Yes, I considered explicitly calling |
|
I am underway with this change. Here is my current understanding of how it will work.
Do you think we will be able to clearly explain to users which instances they need to call close() on, and when calling close() will also make the copies fail? |
…led. The work should be done in close(). Deleting can be left for the garbage collector.
Again, avoid depending on __del__. Add a context manager interface. Use it lower down.
Was concerned that lambda might include a reference to reader that wasn't cleaned up by close, so changed it over to an equivalent self.reader. Probably has no effect, but feels safer.
Note: It does NOT close all the subclips, because they may be used again (by the caller). It is the caller's job to clean them up. But clips created by this instance are closed by this instance.
test_resourcereleasedemo exercises the path where close is not called and demonstrates that there is a consistent problem on Windows. Even after this fix, it remains a problem that if you don't call close, moviepg will leak locked files and subprocesses. [Because the problem remains until the process ends, this is included in a separate test file.] test_resourcerelease demonstrates that when close() is called, the problem goes away.
* Without tests changes, many of these existing tests do not pass on Windows.
|
I still haven't touched the documentation. I don't want to pollute simple introductory examples with complexity they don't need, but I also want to demonstrate good, clean code that uses close() correctly. |
|
Please tag a look at these changes, and tell me how you'd like to proceed. They have been swallowed up in a separate PR. |
* Exceptions do not have a .message attribute. * Help tests run on Windows - don't assume temp dir or fonts. * Python already has a feature for finding the temp dir. Changed test helper to take advantage of it. * Still outstanding: Several hard-coded references to /tmp appear in the tests. * Liberation-Mono is not commonly installed on Windows, and even when it is, the font has a different name. Provide a fall-back for Windows fonts. (Considered the use of a 3rd party tool to help select, but seemed overkill.) * Help tests run on Windows - allow some flexibility in versions. Building/finding binaries on Windows is non-trivial. Aallow some flexibility in the path levels. (I don't want to force existing users to upgrade, but new users should be allowed the later patches.) * Issue 596: Add initial support for closing clips. Doesn't do anything yet. The work is done in the subclasses that need it. Also supports context manager, to allow close to be implicitly performed without being forgotten even if an exception occurs during processes. * Issue 596: Update doctest examples to call close. Demonstrate good practice in the examples. * More exception details for easier debugging of ImageMagick issues. Especially for Windows. * Issue #596: Move away from expecting/requiring __del__ to be called. The work should be done in close(). Deleting can be left for the garbage collector. * Issue #596: Move ffmpeg_writer to using close. Again, avoid depending on __del__. Add a context manager interface. Use it lower down. * Issue #596: Update ffmpeg_audiowriter to support close/context manager. * Issue #596: Move AudioFileClip to use close(), away from __del__. Was concerned that lambda might include a reference to reader that wasn't cleaned up by close, so changed it over to an equivalent self.reader. Probably has no effect, but feels safer. * Issue #596: Support close() on CompositeVideoClip. Note: It does NOT close all the subclips, because they may be used again (by the caller). It is the caller's job to clean them up. But clips created by this instance are closed by this instance. * Issue #596: Add tests to see if this issue has been repaired. test_resourcereleasedemo exercises the path where close is not called and demonstrates that there is a consistent problem on Windows. Even after this fix, it remains a problem that if you don't call close, moviepg will leak locked files and subprocesses. [Because the problem remains until the process ends, this is included in a separate test file.] test_resourcerelease demonstrates that when close() is called, the problem goes away. * Issue #596: Update tests to use close(). * Without tests changes, many of these existing tests do not pass on Windows. * Further to PR #597: Change to Arial Helvetica wasn't recognised by ImageMagick. Changing to another arbitrary font that should be available on all Windows machines. * Issue #596 and #598: Updated test to support close(). Also changed test to meet Issue #598, but that is also being done in PR#585, so will require a merge. * Revert "More exception details for easier debugging of ImageMagick issues." This reverts commit dc4a16a. I bundled too much into one commit. Reverting and reapplying as two separate commits for better history. * Issue #599: test_6 doesn't test anything. Removed as it was crashing on Windows, achieving nothing on Linux. * Issue #596: Move comment to avoid incorporate into documents. * Issue #596: Add usages tips to documentation. * Clip class missing from reference documents. Due to failing import. * Copy-edit: Clumsy sentence in documentation. * Fix failing doctest. * Issue 596: Add initial support for closing clips. * Add key support for close() * FFMPEG_VideoWriter and FFMPEG_AudioWriter: Support close() and context managers. * Clip: support close() and context manager. Doesn't do anything itself. The work is done in the subclasses that need it. * Clip subclasses: Overrride close. * Move away from depending on clients calling__del__(). Deleting can be left to Garbage Collector. * CompositeVideoClip: Note: Don't close anything that wasn't constructed here. The client needs to be able to control the component clips. * AudioFileClip: Was concerned that lambda might include a reference to reader that wasn't cleaned up by close, so changed it over to an equivalent self.reader. Probably has no effect, but feels safer. * Update tests to use close(). * Note: While many tests pass on Linux either way, a large proportion of the existing unit tests fail on Windows without these changes. * Include changes to many doctest examples - Demonstrate good practice in the examples. * Also, migrate tests to use TEMPDIR where they were not using it. * test_duration(): also corrected a bug in the test (described in #598). This bug is also been addressed in #585, so a merge will be required. * Add two new test files: * test_resourcereleasedemo exercises the path where close is not called and demonstrates that there is a consistent problem on Windows. Even after this fix, it remains a problem that if you don't call close, moviepg will leak locked files and subprocesses. Because the problem remains until the process ends, this is included in a separate test file.] * test_resourcerelease demonstrates that when close() is called, the problem goes away. * Update documentation to include usage tips for close() Not included: * Example code has not been updated to use close(). * Merge branch 'WindowsSupport' of C:\Users\xboxl\OneDrive\Documents\MyApps\moviepy with conflicts. * Neaten up output and PEP8 compliance. Also, make runnable directly (to help debugging) * Remove references to /tmp to allow to run on Windows. * Reference to PermissionError failing on Python 2.7. * Migrate to use requests to avoid certificate problems. Old versions of urlretrieve have old certificates which means one of the video downloads was failing. Also requires changes to setup.py, to come. * Clean up of dependencies. Including adding ranges, removing unnecessary entries, adding missing entries, adding environment markers, changing versions, and updating pytest parameter handling. * Simplification of Travis file - letting te setup.py do the heavy lifting Remove conditional installations repeating the rules in setup.py Remove some installation of test needs repeating the rules in setup.py Add testing of installation options. * Add Appveyor support. * Solve Issue 629.
|
@Julian-O @Earney Sorry for the late reaction, but is this issue solved at least on Gtihub now ? Looking at this thread I understand that there is an old PR pending. As there were two issues related to ffmpeg closing recently, what's the general opinion on merging @Julian-O 's solution ? (I haven't had time to review it all ad think of side effects but looks good at first sight) |
|
@Zulko: I've been using my fork with the patches (and other patches documented in other pull requests) included, and it is passing tests and working fine for me. So, yes, it has been solved somewhere on GitHub, but not in the main branch yet. |
|
sorry guys, which is the PR mentioned? close was implemented in 02fc129, and that means this could be closed, right? |
|
Linking this as I didn't see it linked upthread #608 |
|
@Julian-O Hmm, I see that one of the checks failed and that there was no follow-up on the code review (e.g. in form of a new commit), maybe that's why no-one continued looking into this |
I am working on refactoring some code where the original author uses moviepy. So, I am not too familiar with the code usage, sorry.
I am having some problems that echo the problems described in #101, #110, #164, and #518. Examining the code, I believe they all have the same root cause.
Various classes (e.g. VideoFileClip) follow the same pattern:
I believe there are two misunderstandings here.
As a result, you are seeing Windows users complain that temporary files can't be deleted, and other users complaining that the number of "zombie" subprocesses is climbing as they continue processes.
The solution is:
The symptoms I am seeing, broadly:
Running Windows 10, Python 3.6.0. moviepy==0.2.3.2
The text was updated successfully, but these errors were encountered: