This feature would be extremely valuable, thank you a lot for working on this!

I have one question: What if the timeout occurs between two bytes of a multi-octet UTF-8 sequence? For instance, if the file or stream content (eventually) is "\xf0\\x9f\\x92\\x9c\", which is the UTF-8 encoding of a single character, is the read always "atomic" in the sense that it reliably reads a full character, if any, also if a timeout occurs?

If possible, could you please consider adding a test for this? Thank you a lot!

Regarding the message format("Got ~d bytes~n", [N]).: It should be chars, shouldn't it?

This feature would be extremely valuable, thank you a lot for working on this!

I have one question: What if the timeout occurs between two bytes of a multi-octet UTF-8 sequence? For instance, if the file or stream content (eventually) is "\xf0\\x9f\\x92\\x9c\", which is the UTF-8 encoding of a single character, is the read always "atomic" in the sense that it reliably reads a full character, if any, also if a timeout occurs?

If possible, could you please consider adding a test for this? Thank you a lot!

Great question. I have a thought and a question. My thought is that my usecase for this was to prevent blocking on idle connections, not active ones. So, it would be easy enough to make the default behavior continue to consume while there is data available.

Of course, that could interfere with cooperative concurrency, so if we were to enforce a cutoff, what should the behavior be?

should I push the half char back onto the stream?

Personally, if I set a timeout, I would expect the system to respect it in all cases, and to read as much as possible within the timeout.

In addition, I would expect the API to be implemented in such a way that it reliably yields correct results. In the case of get_n_chars/4, this means that I expect to get what the sender transmitted.

For instance, if the sender sent the character '💜', then I would not expect to receive ['\xF0\'|Rs], because '\xf0\' is a different character (even though its byte value is part of the prefix of the UTF-8 encoding of '💜').

Personally, if I set a timeout, I would expect the system to respect it in all cases, and to read as much as possible within the timeout.

In addition, I would expect the API to be implemented in such a way that it reliably yields correct results. In the case of get_n_chars/4, this means that I expect to get what the sender transmitted.

For instance, if the sender sent the character '💜', then I would not expect to receive ['\xF0\'|Rs], because '\xf0\' is a different character (even though its byte value is part of the prefix of the UTF-8 encoding of '💜').

I don't think this is even theoretically possible though, is it? I need to think this through a bit. It seems to me to be entirely possible that there are a number of conditions which means that a half char would be received when a full char was sent.

So I think maybe the question resolves to, "how do we know what the senders intent was?"

I suppose there are two situations that come to mind, but I'm sure it's nonexhaustive.

1. A transmission is "stalled" on a half char and THEN the timeout occurs
2. The timeout occurs on a half char

In situation 1, it is unclear if waiting longer would result in additional information or not.
In situation 2, it seems that truncating the half char would correct but possibly incomplete?

I don't think this is even theoretically possible though, is it?

On the byte-level, it is definitely possible to detect whether the prefix of a UTF-8 encoded character is encountered! In the case of get_n_chars/4, this detection cannot happen on the Prolog level, because this predicate yields chars, i.e., it presumes completely received characters.

Ah, of course. This is in charsio, we are only interested in full characters. So it would never make sense to receive a half character.

Also, only situation 2 matters: The timeout occurs on a half char.

In that case, the implementation must take care to ensure that I can (later) receive the full character (of course presuming it ever arrives).

What must never happen is that the predicate tells us: "This is a character that was received", when that character was not sent.

This is going to be somewhat of an expanded footprint (esp w/regard to tests) in order to main sequential serialization of reads and not having any existing predicates break when they try to read from a half char stuck in a stream.

If this cannot be supported (yet) for text streams, then one option may be to make timeouts only work for binary streams? For binary streams, bytes can be read individually, and there is no risk of breaking up a byte incorrectly.

The predicate could throw an error if it is given a stream for which it cannot reliably perform the operation.

Maybe a permisson_error on the grounds that it is not allowed to read input from a text stream with a timeout?

  e) There shall be a Permission Error when it is not
  permitted to perform a specific operation. It has the form
  permission_error(Operation, PermissionType,
  Culprit) where

     Operation E {
       access,
       create,
       input,
       modify,
       open,
       output,
       reposition
     },

I see you now already implemented it also for text streams! Incredibly contribution, thank you a lot!

If this works correctly, then it will tremendously increase Scryer Prolog's application opportunities, notably for hosting sophisticated web services.

I see you now already implemented it also for text streams! Incredibly contribution, thank you a lot!

If this works correctly, then it will tremendously increase Scryer Prolog's application opportunities, notably for hosting sophisticated web services.

🤞 🤞 🤞

and maybe even web actors 🤫 🤫 🤫

It looks very good!

The only remaining small issue I noticed is in the test cases, and can be addressed at any time (also later): setup_call_cleanup/3 may be useful to reliably close started processes:

setup_call_cleanup(process_create(Echo, [Content], [stdout(pipe(Out))]), ..., close(Out))

It doesn't matter much for the tests themselves, but it may be valuable to use and see this pattern more.

It looks very good!

The only remaining small issue I noticed is in the test cases, and can be addressed at any time (also later): setup_call_cleanup/3 may be useful to reliably close started processes:

setup_call_cleanup(process_create(Echo, [Content], [stdout(pipe(Out))]), ..., close(Out))
It doesn't matter much for the tests themselves, but it may be valuable to use and see this pattern more.

In the test cases or the implementation?

It looks very good!

The only remaining small issue I noticed is in the test cases, and can be addressed at any time (also later): setup_call_cleanup/3 may be useful to reliably close started processes:

setup_call_cleanup(process_create(Echo, [Content], [stdout(pipe(Out))]), ..., close(Out))
It doesn't matter much for the tests themselves, but it may be valuable to use and see this pattern more.

@triska c16dec0

Thank you! The explicit module qualifications should not be necessary!

I agree they should not be, but unfortunately they are. Perhaps not for the external calls to iso_ext:setup_call_clean/3, but for the arguments to it, those must be module qualified otherwise they will error as undefined.

Once we have dyadic quads, the current approach along with explicit qualification will no longer be necessary.

I now see the terminology "partial data" is still present: In the context of Prolog, "partial" is associated with "partially instantiated", from the documentation I would have thought the call may now yield partially instantiated lists. But apparently that is not the case? It only may yield fewer chars than asked for, if a timeout occurs, is that correct?

And more importantly:

returns partial data (distinguishable from EOF).

On EOF, we get [], and on timeout, we may also get [], so how are the cases distinguishable? From the documentation, I thought that the timeout case is distinguished from EOF by a partially instantiated list, or (in case nothing at all arrived within the timeout) with a variable.

For completeness: We can always manually use at_end_of_stream/1, so read_n_chars/4 itself need not even provide special facilities to distinguish this condition!

And more importantly:

returns partial data (distinguishable from EOF).

On EOF, we get [], and on timeout, we may also get [], so how are the cases distinguishable? From the documentation, I thought that the timeout case is distinguished from EOF by a partially instantiated list, or (in case nothing at all arrived within the timeout) with a variable.

I was using the informal sense of the world "partial" here. Thanks for pointing this out, I will fix the terminology to be more precise.

I am open to suggestions on how to differentiate EOF vs timeout.

We could also put an option in the 3rd argument for termination characterization.

suggestions on how to differentiate EOF vs timeout.

Regarding the general API, I have now compared this with the Elisp function accept-process-output, which says:

(accept-process-output &optional PROCESS SECONDS MILLISEC JUST-THIS-ONE)

Allow any pending output from subprocesses to be read by Emacs.
It is given to their filter functions.
...

Optional second argument SECONDS and third argument MILLISEC
specify a timeout; return after that much time even if there is
no subprocess output. 

I think this is the direction you initially envisaged: That the timeout only plays a role if there is nothing to read. On the other hand, if there is data available, then why not read and yield it?

In our case, both semantics do not mesh well with the N argument of get_n_chars/4. So maybe a slightly different API would be useful for reading with timeouts, analogous to Elisp's accept-process-output?

Maybe: get_chars_timeout(+Stream, -Chars, +Timeout): Yield as many Chars as can be read from Stream within Timeout?

This is true, and that was a thought I had. However the immediate reason that comes to mind is a poorly behaved process could flood the server.

What about a get_n_chars/5 where the final argument is EOF or timeout?

However the immediate reason that comes to mind is a poorly behaved process could flood the server.

Good point! What I was thinking about initially is an application with real-time requirements, such as audio processing, where another thing must be handled after a number of milliseconds.

Therefore, yes, good point, for these reasons I think it makes sense to honor the timeout in all situations.

As mentioned, I think there is no need to indicate EOF within this predicate. When needed, that can be tested completely separately, with the standard predicate at_end_of_stream/1. Note that the existing predicate get_n_chars/3 also does not indicate EOF.

I think it only remains to document the arguments correctly.

Ok I think I unbuggered everything after a failed rebase, could use another set of eyes