FFmpeg validation#4843
Closed
bastimeyer wants to merge 2 commits into
Closed
Conversation
New utility class for executing a subprocess and asynchronously reading its stdout and stderr streams line by line while the process is running, with an optional timeout which kills the process. The output streams and exit code get passed to callback methods which can control the process execution and final output result. This is useful for validating unknown executables, e.g. when querying a version string output, as it avoids reading the entire stdout/stderr at once and also avoids waiting for the process to terminate on its own.
and log FFmpeg version output on the debug logging level WIP: update and fix tests
Member
Author
|
I'll open a new PR in a couple of days once I've finished updating the tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We've recently added warning log messages to the
FFMPEGMuxer's resolver when FFmpeg could not be found, so that the user can be informed that muxing is not supported. However, there are still a couple of issues left which I think should be addressed:This is a problem when the user has set an invalid FFmpeg executable, or when it is broken and doesn't work. In case of invalid executables, the output and stream muxing are pretty much undefined, and broken executables lead to empty stream outputs, and unless verbose logging is enabled, no error messages get printed, making the user believe there's an issue with the input stream.
Logging the FFmpeg version makes debugging issues easier. The recent issue which comes to mind is:
stream.ffmpegmux: stream with independent segments (audio+video) cannot be remuxed #4825
This PR doesn't add any version checks, but this could be added in the future if it's needed, based on this work.
hls-multi from arte.tv has seconds of hanging video #4520 (comment)
Opening the PR as a draft, as it's unfinished work.
The first commit implements a generic
ProcessOutpututility class for asynchronously spawning a subprocess and reading its stdout/stderr streams.This is necessary, because using APIs like
subprocess.run(capture_output=True)orasyncio.create_subprocess_exec().communicate()read the entire stdout/stderr streams into memory until the process terminates on its own. This is bad when executing unknown binaries. And reading streams of a spawned subprocess in separate threads as well as dealing with timeouts is unnecessarily difficult to implement in regards to making everything thread-safe (especially when writing tests, just see the messy HLS stream tests for example).Asynchronously iterating output streams and using
asyncio.sleep()makes this fairly easy and straightforward to implement, but testing stuff unfortunately requires adding two more dev-requirements:pytest-asyncio- already installed as a transitive dependency, so no changes heremock- compatibility backport of theunittest.mockstdlib implementation for py37 (AsyncMockis missing)mockwas already removed from the dev-requirements in d04767f, and the only gotchas were theargs/kwargsproperties inMock.call_argswhich are missing on py37, but that doesn't justify adding compatibility imports everywhereAnother issue I ran into while writing tests was
freezegun, which currently affects the internals ofasyncioin its most recent version, and there's an open PR on their repo which will change that. I managed to get the tests working (after lots of issues withasyncio.sleep()), but I'm not sure how the nextfreezegunrelease(s) will affect the tests. We'll see...The second commit adds FFmpeg executable validation and version logging. The validation can be turned off by setting
--ffmpeg-no-validation.Still need to update and fix tests.
I've included some links in the code to the FFmpeg version string definitions, which have been stable for 12 years now.