-
Notifications
You must be signed in to change notification settings - Fork 950
machine: [rp2] discount scheduling delays in I2C timeouts #4876
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
These changes may have a negative impact in cases where the timeout is real. Have you tried "refreshing" the deadline between the write and read section of I2C.tx and seeing if that solves the issue as well? (right above rxStart declaration)
|
eed26f2
to
456e38f
Compare
Good idea! Please take another look. I also expanded the commit message. |
Hol up. I believe the deadline can never trigger now. You are declaring the deadline in-loop, so it is refreshed every time a byte is written over I2C. I suggest going back to the deadline declaration at the start of the function and refreshing it after the write section (right above rxStart declaration). This is because I suspect your program is timing out during the read section due to how the conditions are ordered. If this is not the case let me know |
What is the deadline for, other than buggy i2c hardware or faulty data lines? The calling code says // timeout in microseconds.
const timeout = 40 * 1000 // 40ms is a reasonable time for a real-time system.
return i2c.tx(uint8(addr), w, r, timeout) but TinyGo is no way near "a real-time system" unless there's only one goroutine running. With your proposed change (deadline extension just before starting read), there's still a risk of unfortunate goroutine scheduling triggering a timeout. Simplified case, for the transmission loop: deadline := now() + 40ms
for _, b := range txBuf { // for each byte to transmit
i2cTransmit(b)
for i2cTxBufferFull() {
if now() > deadline { return ... }
gosched() // Takes, say, 100ms of time.
}
} As far as I can tell, the second byte will immediately timeout if for _, b := range txBuf {
i2cTransmit(b)
deadline := now() + 40ms
for i2cTxBufferFull() {
if now() > deadline { return ... }
gosched()
}
} which ensure that no matter how long a |
OK, I've just realized I had a miconception about the looping logic when I asked you to perform both changes. Can we try your original change and doing some basic math to calculate ideal timeout? Lets look at a scenario: We have our RP2040 running a slow 10kHz I2C baud. The time it takes for transaction to complete is at least (len(w)+len(r))*bitSendTime where bitSendTime is 1./10000 seconds = 0.1ms = 100us. We can thus set the timeout to be (len(w)+len(r))*100, and we can calculate it correctly. With your original change then we can then discard the time spent between byte transmits (which I had not realized the I2C algorithm could block between every single byte. |
I'd also add in a security factor, like *2 or something of the sort |
456e38f
to
2a25227
Compare
Done. I didn't add a security factor, because that's already builtin to the pessimistic bus frequency (10kHz). |
01f79a9
to
f3c58b1
Compare
I've pushed an update that replaces the bitrate computation with a simple "long" timeout. I also replaced another tight deadline in Longer deadlines are not great, but my rationale is that the deadlines are only there for faulty hardware or programming errors. And they're not that long (40ms). EDIT: finally, the deadlines can be tightened in a future DMA implementation. |
src/machine/machine_rp2_i2c.go
Outdated
@@ -218,8 +215,8 @@ func (i2c *I2C) enable() { | |||
// | |||
//go:inline | |||
func (i2c *I2C) disable() error { | |||
const MAX_T_POLL_COUNT = 64 // 64 us timeout corresponds to around 1000kb/s i2c transfer rate. | |||
deadline := ticks() + MAX_T_POLL_COUNT | |||
const timeout_us = 40000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you change this line? This loop does not call gosched
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did it have a noticeable effect? What baud are you running?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I run i2c at 400kHz.
I had to bump the disable
deadline otherwise I saw timeouts from it (presumably because of a long interrupt handler in my program). Unlike gosched
, interrupt handlers can reasonably be assumed to complete in short time (<40ms), but not in (in this case) 64μs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have some reservations jumping the disable timeout by factor ~1000 than what the reference implementation suggests. If a interrup handler runs in over a few ms i believe that to be a real issue for tinygo which usually encourages really short interrupts to not negatively interfere with the runtime
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you try 400us and 4ms and see if any of those work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the timeouts to 4ms; my interrupt handler completes in ~1.5ms. Long interrupts are not great, but what else can be done about real-time requirements (e.g. interrupt "bottom halves")? TinyGo doesn't offer goroutine priorities.
Just for my curiosity, where do you see the disable deadline in the pico-sdk reference? I couldn't see it in https://github.com/raspberrypi/pico-sdk/blob/master/src/rp2_common/hardware_i2c/i2c.c.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the reference.
Increment POLL_COUNT by one. If POLL_COUNT >= MAX_T_POLL_COUNT, exit with [...] error [...]
reads to me that MAX_T_POLL_COUNT is a loop counter, not a deadline. And
Define a maximum time-out paramater, MAX_T_POLL_COUNT
reads to me as the value being implementation defined. Only the loop iteration delay, t_i2c_poll
, is defined in terms of bus speed.
Am I reading the datasheet wrong here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are correct! Looks like I slipped on the naming
The `gosched` call introduce arbitrary long delays in general, and in TinyGo particular because the goroutine scheduler is cooperative and doesn't preempt busy (e.g. compute-heavy) goroutines. Before this change, the timeout logic would read, simplified: deadline := now() + timeout startTX() for !txDone() { if now() > deadline { return timeoutError } gosched() // (1) } startRx() // (2) for !rxDone() { // (3) if now() > deadline { return timeoutError } gosched() } What could happen in a busy system is: - The gosched marked (1) would push now() to be > than deadline. - startRx is called (2), but the call to rxDone immediately after would report it not yet done. - The check marked (3) would fail, even though only a miniscule amount of time has passed between startRx and the check. This change ensures that the timeout clock discounts time spent in `gosched`. The logic now reads, around every call to `gosched`: deadline := now() + timeout startTX() for !txDone() { if now() > deadline { return timeoutError } before := now() gosched() deadline += now() - before } I tested this change by simulating a busy goroutine: go func() { for { // Busy. before := time.Now() for time.Since(before) < 100*time.Millisecond { } // Sleep. time.Sleep(100 * time.Millisecond) } }() and testing that I2C transfers would no longer time out.
f3c58b1
to
adc7c79
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
The
gosched
call introduces arbitrary long delays in general, and in TinyGo particular because the goroutine scheduler is cooperative and doesn't preempt busy (e.g. compute-heavy) goroutines. This change discounts such delays from the rp2xxx I2C timeout logic.I tested this change by simulating a busy goroutine:
and testing that I2C transfers would no longer time out.