Both of these methods falsely advertise that their return type is a string when they multiple instances boolean returns in their method body.

The DocBlock comments do indeed need to be improved. Unfortunately, that's not really my highest priority. Right now I'm working on the documentation for v3.0. If someone wants to do a PR for the return types they're very welcome to do so.

As I noted in your PR you're proposed fix is problematic because throwing exceptions in the 2.0 branch would be a PR break. So here's what I'll do.

In the 1.0 / 2.0 branches I'll add a line to return false if $payload is false, right above the line you included in your post.

In the 3.0 / master branches I'll look for more return false's that can be replaced with exceptions. Like I said in your PR the 3.0 / master branches already do throw exceptions so throwing a few more ain't no thing.

Looking at the code (1.0) it is not clear to me how _filter() could get anything other than a string for $payload. As you note, _filter() is only called from _get_binary_packet().

$payload is set thusly:

$payload = $this->_string_shift($raw, $packet_length - $padding_length - 1);

Here's the definition of _string_shift:

    /**
     * String Shift
     *
     * Inspired by array_shift
     *
     * @param string $string
     * @param int $index
     * @return string
     * @access private
     */
    function _string_shift(&$string, $index = 1)
    {
        $substr = substr($string, 0, $index);
        $string = substr($string, $index);
        return $substr;
    }

So apparently substr can return false, which I didn't realize. eg. substr('', 2) returns false. substr(false, 0, 5) returns false on PHP 5.6 and earlier but not on PHP 7.0+.

Tracing through the code I don't see substr('', 2) as really being applicable. Before any _string_shift() calls on $raw there's this:

        if (strlen($raw) < 5) {
            return false;
        }

The next operation on $raw is this:

        if (strlen($buffer)) {
            $raw.= $this->decrypt !== false ? $this->decrypt->decrypt($buffer) : $buffer;
        }

If $buffer isn't a multiple of the block length it is null padded to a multiple of the block length. Because message authentication isn't performed here a string is always going to be returned and because of the way the the loop that populates $buffer is built I don't see how $buffer could ever be anything other than a string of the desired length:

        $buffer = '';
        while ($remaining_length > 0) {
            $temp = fread($this->fsock, $remaining_length);
            if ($temp === false || feof($this->fsock)) {
                $this->bitmap = 0;
                user_error('Error reading from socket');
                return false;
            }
            $buffer.= $temp;
            $remaining_length-= strlen($temp);
        }

The next operation on $raw after that is the line where $payload is set:

$payload = $this->_string_shift($raw, $packet_length - $padding_length - 1);

So we already know it's of the appropriate length. But even if it wasn't, I still don't see how substr($string, 0, $index) could return false.

I'm all for "defensive programming" (indeed, that's why I do some of the length checks that I do) but there is such a thing as paranoid redundancy, as well. eg.

$a = (int) $_GET['a'];
if (!is_numeric($a)) {
    throw new \Exception('$a must be an integer');
}

In that example that !is_numeric call is pointless.

I'll look into adding more exceptions for the 3.0 / master branch but at this point I'm thinking I won't make _filter return false, at the beginning, if $payload is false. Not until I better understand how that's even possible...

What version of PHP are you running, out of curiosity? The latest 2.0 version of phpseclib?

I am running the latest 2.0 version of phpseclib with PHP 7.4.
I can pretty easily reproduce this error, but not with a public server, so I can not give you any details.

I will try to get to this asap, but it could take some time. Sorry.

Both of these methods falsely advertise that their return type is a string when they multiple instances boolean returns in their method body.

#1473 addresses this.