Improve Exception Handling in `File.download_*` #4542

Bibo-Joshi · 2024-10-31T15:03:10Z

Response to the test failures on master https://github.com/python-telegram-bot/python-telegram-bot/actions/runs/11607799025/job/32340491289

These are caused by Telegram failing to return a file_path in the get_file response. I'm not sure why they stopped doing that. Further investigation might be necessary. I realized (maybe for the first time) that file_path is actually documented to be optional.

What this PR does is mostly soften the blow, i.e. provide understandable error handling.
This will not immediately fix the test suite. I'd like to wait a moment and see if the issue solves itself. If it does not, we'll have to adjust the other tests/mark them as xfail/report to telegram.

PS: see also tdlib/telegram-bot-api#658

codecov · 2024-10-31T15:14:22Z

❌ 6 Tests Failed:

Tests completed	Failed	Passed	Skipped
5997	6	5991	265

View the top 3 failed tests by shortest run time

tests._files.test_file.TestFileWithoutRequest test_download_bytearray

Stack Traces | 0.002s run time

self = &lt;tests._files.test_file.TestFileWithoutRequest object at 0x000001BA36564D70&gt;
monkeypatch = &lt;_pytest.monkeypatch.MonkeyPatch object at 0x000001BA3AAB63C0&gt;
file = File(file_id='NOTVALIDDOESNOTMATTER', file_size=28232, file_unique_id='adc3145fd2e84d95b64d68eaa22aa33e')

    async def test_download_bytearray(self, monkeypatch, file):
        async def test(*args, **kwargs):
            return self.file_content
    
        monkeypatch.setattr(file.get_bot().request, "retrieve", test)
    
        # Check that a download to a newly allocated bytearray works.
&gt;       buf = await file.download_as_bytearray()

tests\_files\test_file.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = File(file_id='NOTVALIDDOESNOTMATTER', file_size=28232, file_unique_id='adc3145fd2e84d95b64d68eaa22aa33e')
buf = None

    async def download_as_bytearray(
        self,
        buf: Optional[bytearray] = None,
        *,
        read_timeout: ODVInput[float] = DEFAULT_NONE,
        write_timeout: ODVInput[float] = DEFAULT_NONE,
        connect_timeout: ODVInput[float] = DEFAULT_NONE,
        pool_timeout: ODVInput[float] = DEFAULT_NONE,
    ) -&gt; bytearray:
        """Download this file and return it as a bytearray.
    
        .. versionchanged:: NEXT.VERSION
            Raises :exc:`RuntimeError` if :attr:`file_path` is not set.
    
        Args:
            buf (:obj:`bytearray`, optional): Extend the given bytearray with the downloaded data.
    
        Keyword Args:
            read_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.read_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
    
                .. versionadded:: 20.0
            write_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.write_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
    
                .. versionadded:: 20.0
            connect_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.connect_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
    
                .. versionadded:: 20.0
            pool_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.pool_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
    
                .. versionadded:: 20.0
    
        Returns:
            :obj:`bytearray`: The same object as :paramref:`buf` if it was specified. Otherwise a
            newly allocated :obj:`bytearray`.
    
        Raises:
            RuntimeError: If :attr:`file_path` is not set.
    
        """
        if not self.file_path:
&gt;           raise RuntimeError("No `file_path` available for this file. Can not download.")
E           RuntimeError: No `file_path` available for this file. Can not download.

telegram\_files\file.py:341: RuntimeError

tests._files.test_file.TestFileWithoutRequest test_download_no_filename

Stack Traces | 0.002s run time

self = &lt;tests._files.test_file.TestFileWithoutRequest object at 0x000001BA363CB450&gt;
monkeypatch = &lt;_pytest.monkeypatch.MonkeyPatch object at 0x000001BA3AAB7A80&gt;
file = File(file_id='NOTVALIDDOESNOTMATTER', file_size=28232, file_unique_id='adc3145fd2e84d95b64d68eaa22aa33e')

    async def test_download_no_filename(self, monkeypatch, file):
        async def test(*args, **kwargs):
            return self.file_content
    
        file.file_path = None
    
        monkeypatch.setattr(file.get_bot().request, "retrieve", test)
&gt;       out_file = await file.download_to_drive()

tests\_files\test_file.py:192: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = File(file_id='NOTVALIDDOESNOTMATTER', file_size=28232, file_unique_id='adc3145fd2e84d95b64d68eaa22aa33e')
custom_path = None

    async def download_to_drive(
        self,
        custom_path: Optional[FilePathInput] = None,
        *,
        read_timeout: ODVInput[float] = DEFAULT_NONE,
        write_timeout: ODVInput[float] = DEFAULT_NONE,
        connect_timeout: ODVInput[float] = DEFAULT_NONE,
        pool_timeout: ODVInput[float] = DEFAULT_NONE,
    ) -&gt; Path:
        """
        Download this file. By default, the file is saved in the current working directory with
        :attr:`file_path` as file name. If the file has no filename, the file ID will be used as
        filename. If :paramref:`custom_path` is supplied as a :obj:`str` or :obj:`pathlib.Path`,
        it will be saved to that path.
    
        Note:
            If :paramref:`custom_path` isn't provided and :attr:`file_path` is the path of a
            local file (which is the case when a Bot API Server is running in local mode), this
            method will just return the path.
    
            The only exception to this are encrypted files (e.g. a passport file). For these, a
            file with the prefix `decrypted_` will be created in the same directory as the
            original file in order to decrypt the file without changing the existing one
            in-place.
    
        .. seealso:: :wiki:`Working with Files and Media &lt;Working-with-Files-and-Media&gt;`
    
        .. versionchanged:: 20.0
    
            * :paramref:`custom_path` parameter now also accepts :class:`pathlib.Path` as argument.
            * Returns :class:`pathlib.Path` object in cases where previously a :obj:`str` was
              returned.
            * This method was previously called ``download``. It was split into
              :meth:`download_to_drive` and :meth:`download_to_memory`.
    
        .. versionchanged:: NEXT.VERSION
            Raises :exc:`RuntimeError` if :attr:`file_path` is not set.
    
        Args:
            custom_path (:class:`pathlib.Path` | :obj:`str` , optional): The path where the file
                will be saved to. If not specified, will be saved in the current working directory
                with :attr:`file_path` as file name or the :attr:`file_id` if :attr:`file_path`
                is not set.
    
        Keyword Args:
            read_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.read_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
            write_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.write_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
            connect_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.connect_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
            pool_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.pool_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
    
        Returns:
            :class:`pathlib.Path`: Returns the Path object the file was downloaded to.
    
        Raises:
            RuntimeError: If :attr:`file_path` is not set.
    
        """
        if not self.file_path:
&gt;           raise RuntimeError("No `file_path` available for this file. Can not download.")
E           RuntimeError: No `file_path` available for this file. Can not download.

telegram\_files\file.py:186: RuntimeError

tests._files.test_file.TestFileWithoutRequest test_download_file_obj

Stack Traces | 0.003s run time

self = &lt;tests._files.test_file.TestFileWithoutRequest object at 0x000001BA363CB550&gt;
monkeypatch = &lt;_pytest.monkeypatch.MonkeyPatch object at 0x000001BA3AAB5FD0&gt;
file = File(file_id='NOTVALIDDOESNOTMATTER', file_size=28232, file_unique_id='adc3145fd2e84d95b64d68eaa22aa33e')

    async def test_download_file_obj(self, monkeypatch, file):
        async def test(*args, **kwargs):
            return self.file_content
    
        monkeypatch.setattr(file.get_bot().request, "retrieve", test)
        with TemporaryFile() as custom_fobj:
&gt;           await file.download_to_memory(out=custom_fobj)

tests\_files\test_file.py:206: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = File(file_id='NOTVALIDDOESNOTMATTER', file_size=28232, file_unique_id='adc3145fd2e84d95b64d68eaa22aa33e')
out = &lt;tempfile._TemporaryFileWrapper object at 0x000001BA392DF610&gt;

    async def download_to_memory(
        self,
        out: BinaryIO,
        *,
        read_timeout: ODVInput[float] = DEFAULT_NONE,
        write_timeout: ODVInput[float] = DEFAULT_NONE,
        connect_timeout: ODVInput[float] = DEFAULT_NONE,
        pool_timeout: ODVInput[float] = DEFAULT_NONE,
    ) -&gt; None:
        """
        Download this file into memory. :paramref:`out` needs to be supplied with a
        :obj:`io.BufferedIOBase`, the file contents will be saved to that object using the
        :obj:`out.write&lt;io.BufferedIOBase.write&gt;` method.
    
        .. seealso:: :wiki:`Working with Files and Media &lt;Working-with-Files-and-Media&gt;`
    
        Hint:
            If you want to immediately read the data from ``out`` after calling this method, you
            should call ``out.seek(0)`` first. See also :meth:`io.IOBase.seek`.
    
        .. versionadded:: 20.0
    
        .. versionchanged:: NEXT.VERSION
            Raises :exc:`RuntimeError` if :attr:`file_path` is not set.
    
        Args:
            out (:obj:`io.BufferedIOBase`): A file-like object. Must be opened for writing in
                binary mode.
    
        Keyword Args:
            read_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.read_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
            write_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.write_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
            connect_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.connect_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
            pool_timeout (:obj:`float` | :obj:`None`, optional): Value to pass to
                :paramref:`telegram.request.BaseRequest.post.pool_timeout`. Defaults to
                :attr:`~telegram.request.BaseRequest.DEFAULT_NONE`.
    
        Raises:
            RuntimeError: If :attr:`file_path` is not set.
        """
        if not self.file_path:
&gt;           raise RuntimeError("No `file_path` available for this file. Can not download.")
E           RuntimeError: No `file_path` available for this file. Can not download.

telegram\_files\file.py:274: RuntimeError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

…` to tests as workaround for tdlib/telegram-bot-api#658

Bibo-Joshi · 2024-11-02T17:58:01Z

Lastest commit

adds retries of get_file to tests suite as workaround for getFile stopped returning file_path tdlib/telegram-bot-api#658, where it was observed that successive calls to that method do return the file_path
Removes the special casing from File.download_to_drive designed to handle cases where file_path is not present but file_id is. Without file_path, it's not possible to download a file, as you need the URL for that. I tracked that if-clause back to [BUG] Cannot Download a File if it doesnt have a file_path or if custom_path is not provided to download() #1590 and fix download without path arguments #1591 and TBH I can't see why we merged that. The fact that the test case worked only by mocking the http request already shows that it's not an actual use case IMO

Poolitzer · 2024-11-03T12:15:28Z

Well hindsight is is 20/20.

Poolitzer

LGTM

This reverts commit 1e3b147.

Bibo-Joshi · 2024-11-03T13:52:50Z

New commits revert the workaround for get_file again as tdlib/telegram-bot-api#658 has been resolved by now 😅

Poolitzer

LGTM

Improve Exception Handling in File.download_*

1878a4b

Bibo-Joshi added the enhancement label Oct 31, 2024

Bibo-Joshi requested review from Poolitzer and harshil21 October 31, 2024 15:03

Bibo-Joshi added 2 commits October 31, 2024 16:03

Merge branch 'master' into stabilize-file-download

6230870

A bit more documentation

77297da

Bibo-Joshi added 2 commits November 2, 2024 18:53

Remove file_id case in download_to_drive and add `stable_get_file…

bf883a5

…` to tests as workaround for tdlib/telegram-bot-api#658

Merge branch 'refs/heads/master' into stabilize-file-download

2c31492

Use Union instead of |

1e3b147

Poolitzer approved these changes Nov 3, 2024

View reviewed changes

Bibo-Joshi added 2 commits November 3, 2024 14:45

Revert "Use Union instead of |"

d36f12c

This reverts commit 1e3b147.

Revert addition of stable_get_file

1ac480c

Coverage problem

fe58617

Bibo-Joshi requested a review from Poolitzer November 3, 2024 14:16

Poolitzer approved these changes Nov 3, 2024

View reviewed changes

Bibo-Joshi merged commit 507d6bc into master Nov 3, 2024
22 of 23 checks passed

Bibo-Joshi deleted the stabilize-file-download branch November 3, 2024 15:35

Bibo-Joshi added 🔌 enhancement pr description: enhancement and removed enhancement labels Nov 3, 2024

github-actions bot locked and limited conversation to collaborators Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Exception Handling in `File.download_*` #4542

Improve Exception Handling in `File.download_*` #4542

Bibo-Joshi commented Oct 31, 2024 •

edited

Loading

codecov bot commented Oct 31, 2024 •

edited

Loading

Bibo-Joshi commented Nov 2, 2024

Poolitzer commented Nov 3, 2024

Poolitzer left a comment

Bibo-Joshi commented Nov 3, 2024

Poolitzer left a comment

Improve Exception Handling in File.download_* #4542

Improve Exception Handling in File.download_* #4542

Conversation

Bibo-Joshi commented Oct 31, 2024 • edited Loading

codecov bot commented Oct 31, 2024 • edited Loading

❌ 6 Tests Failed:

Bibo-Joshi commented Nov 2, 2024

Poolitzer commented Nov 3, 2024

Poolitzer left a comment

Choose a reason for hiding this comment

Bibo-Joshi commented Nov 3, 2024

Poolitzer left a comment

Choose a reason for hiding this comment

Improve Exception Handling in `File.download_*` #4542

Improve Exception Handling in `File.download_*` #4542

Bibo-Joshi commented Oct 31, 2024 •

edited

Loading

codecov bot commented Oct 31, 2024 •

edited

Loading