File download causes crash with MemoryError #3762
Description
Expected behaviour:
File is downloaded
Actual behaviour:
I'm stuck with 3.2.7 since it's the last one that works fine.
I updated to 3.6.7 using my custom docker image maxcanna/flexget but it crashes with OverflowError
I added arm support to official image to be able to test it on my Raspberry Pi but it crashes too with MemoryError
I tracked the problem down to https://github.com/psf/requests but I'm not able to help more than that.
Steps to reproduce:
- Step 1: Create an entry with an url like
https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4
Config:
configure_series:
from:
filesystem:
- /media/
download: /media/{{ series_name }}/
inputs:
- rss: https://[redacted]/rss/0.xml
series:
- Vera: {}
set:
filename: '{{ description }}'Log:
(click to expand)
2023-05-08 17:47:20 VERBOSE task rai ACCEPTED: `Vera - Un'infermiera in campagna 2023.05.07` by series plugin because choosing first acceptable match
2023-05-08 17:47:21 DEBUG series rai processing series took 0.0376355589999946
2023-05-08 17:47:21 DEBUG urlrewriter rai Checking 1 entries
2023-05-08 17:47:21 INFO download rai Downloading: Vera - Un'infermiera in campagna 2023.05.07
2023-05-08 17:47:21 DEBUG download rai Downloading url 'https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4'
2023-05-08 17:47:21 DEBUG utils.requests rai GETing URL https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4 with args () and kwargs {'auth': None, 'headers': {'User-Agent': 'FlexGet/3.6.7 (www.flexget.com)', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '*/*', 'Connection': 'keep-alive'}, 'allow_redirects': True, 'timeout': 30}
2023-05-08 17:48:36 CRITICAL task rai BUG: Unhandled error in plugin download:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap
self._bootstrap_inner()
│ └ <function Thread._bootstrap_inner at 0xb692cb18>
└ <Thread(task_queue, started daemon 2962677100)>
File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
self.run()
│ └ <function Thread.run at 0xb692c988>
└ <Thread(task_queue, started daemon 2962677100)>
File "/usr/local/lib/python3.11/threading.py", line 975, in run
self._target(*self._args, **self._kwargs)
│ │ │ │ └ {}
│ │ │ │ └ <Thread(task_queue, started daemon 2962677100)>
│ │ │ └ ()
│ │ └ <Thread(task_queue, started daemon 2962677100)>
│ └ <bound method TaskQueue.run of <flexget.task_queue.TaskQueue object at 0xb18f5d70>>
└ <Thread(task_queue, started daemon 2962677100)>
File "/usr/local/lib/python3.11/site-packages/flexget/task_queue.py", line 46, in run
self.current_task.execute()
│ │ └ <function Task.execute at 0xb3f3b758>
│ └ <flexget.task.Task object at 0xb17d0a70>
└ <flexget.task_queue.TaskQueue object at 0xb18f5d70>
File "/usr/local/lib/python3.11/site-packages/flexget/task.py", line 87, in wrapper
return func(self, *args, **kw)
│ │ │ └ {}
│ │ └ ()
│ └ <flexget.task.Task object at 0xb17d0a70>
└ <function Task.execute at 0xb3f3b708>
File "/usr/local/lib/python3.11/site-packages/flexget/task.py", line 725, in execute
self._execute()
│ └ <function Task._execute at 0xb3f3b6b8>
└ <flexget.task.Task object at 0xb17d0a70>
File "/usr/local/lib/python3.11/site-packages/flexget/task.py", line 694, in _execute
self.__run_task_phase(phase)
│ └ 'download'
└ <flexget.task.Task object at 0xb17d0a70>
File "/usr/local/lib/python3.11/site-packages/flexget/task.py", line 514, in __run_task_phase
response = self.__run_plugin(plugin, phase, args)
│ │ │ └ (<flexget.task.Task object at 0xb17d0a70>, '/media/{{ series_name }}/')
│ │ └ 'download'
│ └ <PluginInfo(name=download)>
└ <flexget.task.Task object at 0xb17d0a70>
> File "/usr/local/lib/python3.11/site-packages/flexget/task.py", line 547, in __run_plugin
result = method(*args, **kwargs)
│ │ └ {}
│ └ (<flexget.task.Task object at 0xb17d0a70>, '/media/{{ series_name }}/')
└ <Event(name=plugin.download.download,func=on_task_download,priority=128)>
File "/usr/local/lib/python3.11/site-packages/flexget/event.py", line 20, in __call__
return self.func(*args, **kwargs)
│ │ │ └ {}
│ │ └ (<flexget.task.Task object at 0xb17d0a70>, '/media/{{ series_name }}/')
│ └ <bound method PluginDownload.on_task_download of <flexget.plugins.output.download.PluginDownload object at 0xb1c6eed0>>
└ <Event(name=plugin.download.download,func=on_task_download,priority=128)>
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 88, in on_task_download
self.get_temp_files(
│ └ <function PluginDownload.get_temp_files at 0xb3b47708>
└ <flexget.plugins.output.download.PluginDownload object at 0xb1c6eed0>
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 196, in get_temp_files
self.get_temp_file(task, entry, require_path, handle_magnets, fail_html, tmp_path)
│ │ │ │ │ │ │ └ '/config/temp'
│ │ │ │ │ │ └ True
│ │ │ │ │ └ False
│ │ │ │ └ False
│ │ │ └ <Entry(title=Vera - Un'infermiera in campagna 2023.05.07,state=accepted)>
│ │ └ <flexget.task.Task object at 0xb17d0a70>
│ └ <function PluginDownload.get_temp_file at 0xb3b47668>
└ <flexget.plugins.output.download.PluginDownload object at 0xb1c6eed0>
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 138, in get_temp_file
error = self.process_entry(task, entry, url, tmp_path)
│ │ │ │ │ └ '/config/temp'
│ │ │ │ └ 'https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4'
│ │ │ └ <Entry(title=Vera - Un'infermiera in campagna 2023.05.07,state=accepted)>
│ │ └ <flexget.task.Task object at 0xb17d0a70>
│ └ <function PluginDownload.process_entry at 0xb3b47758>
└ <flexget.plugins.output.download.PluginDownload object at 0xb1c6eed0>
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 216, in process_entry
self.download_entry(task, entry, url, tmp_path)
│ │ │ │ │ └ '/config/temp'
│ │ │ │ └ 'https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4'
│ │ │ └ <Entry(title=Vera - Un'infermiera in campagna 2023.05.07,state=accepted)>
│ │ └ <flexget.task.Task object at 0xb17d0a70>
│ └ <function PluginDownload.download_entry at 0xb3b477a8>
└ <flexget.plugins.output.download.PluginDownload object at 0xb1c6eed0>
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 264, in download_entry
response = task.requests.get(url, auth=auth, raise_status=False, headers=headers)
│ │ │ │ │ └ {'User-Agent': 'FlexGet/3.6.7 (www.flexget.com)', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '*/*', 'Connection': 'kee...
│ │ │ │ └ None
│ │ │ └ 'https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4'
│ │ └ <function Session.get at 0xb45bcbb8>
│ └ <flexget.utils.requests.Session object at 0xaf956cb0>
└ <flexget.task.Task object at 0xb17d0a70>
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
│ │ │ └ {'auth': None, 'raise_status': False, 'headers': {'User-Agent': 'FlexGet/3.6.7 (www.flexget.com)', 'Accept-Encoding': 'gzip, ...
│ │ └ 'https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4'
│ └ <function Session.request at 0xb3f98f28>
└ <flexget.utils.requests.Session object at 0xaf956cb0>
File "/usr/local/lib/python3.11/site-packages/flexget/utils/requests.py", line 267, in request
result = super().request(method, url, *args, **kwargs)
│ │ │ └ {'auth': None, 'headers': {'User-Agent': 'FlexGet/3.6.7 (www.flexget.com)', 'Accept-Encoding': 'gzip, deflate, br', 'Accept':...
│ │ └ ()
│ └ 'https://creativemedia2-rai-it.akamaized.net/podcastmhp_world/replaytv_world/raiuno_world/sera_world/19925698_2400.mp4'
└ 'GET'
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
│ │ │ └ {'timeout': 30, 'allow_redirects': True, 'proxies': OrderedDict(), 'stream': False, 'verify': True, 'cert': None}
│ │ └ <PreparedRequest [GET]>
│ └ <function Session.send at 0xb45bcde8>
└ <flexget.utils.requests.Session object at 0xaf956cb0>
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 745, in send
r.content
│ └ <property object at 0xb45b9910>
└ <Response [200]>
File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
│ │ │ │ └ 10240
│ │ │ └ <function Response.iter_content at 0xb45bc078>
│ │ └ <Response [200]>
│ └ False
└ <Response [200]>
MemoryError
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/flexget/task.py", line 547, in __run_plugin
result = method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flexget/event.py", line 20, in __call__
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 88, in on_task_download
self.get_temp_files(
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 196, in get_temp_files
self.get_temp_file(task, entry, require_path, handle_magnets, fail_html, tmp_path)
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 138, in get_temp_file
error = self.process_entry(task, entry, url, tmp_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 216, in process_entry
self.download_entry(task, entry, url, tmp_path)
File "/usr/local/lib/python3.11/site-packages/flexget/plugins/output/download.py", line 264, in download_entry
response = task.requests.get(url, auth=auth, raise_status=False, headers=headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flexget/utils/requests.py", line 267, in request
result = super().request(method, url, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 745, in send
r.content
File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 899, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError
Additional information:
- FlexGet version: 3.6.7
- Python version: 3.11.3
- Installation method: docker container
- Using daemon (yes/no): yes
- OS and version: Raspbian GNU/Linux 10 (buster)
- Link to crash log: