Thanks to visit codestin.com
Credit goes to github.com

Skip to content

machine: [rp2] discount scheduling delays in I2C timeouts #4876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 7, 2025

Conversation

eliasnaur
Copy link
Contributor

@eliasnaur eliasnaur commented Apr 29, 2025

The gosched call introduces arbitrary long delays in general, and in TinyGo particular because the goroutine scheduler is cooperative and doesn't preempt busy (e.g. compute-heavy) goroutines. This change discounts such delays from the rp2xxx I2C timeout logic.

I tested this change by simulating a busy goroutine:

go func() {
	for {
		// Busy.
		before := time.Now()
		for time.Since(before) < 100*time.Millisecond {
		}
		// Sleep.
		time.Sleep(100 * time.Millisecond)
	}
}()

and testing that I2C transfers would no longer time out.

@deadprogram deadprogram requested a review from soypat April 30, 2025 17:43
@soypat
Copy link
Contributor

soypat commented Apr 30, 2025

These changes may have a negative impact in cases where the timeout is real. Have you tried "refreshing" the deadline between the write and read section of I2C.tx and seeing if that solves the issue as well? (right above rxStart declaration)

deadline += timeout_us

@eliasnaur
Copy link
Contributor Author

Good idea! Please take another look. I also expanded the commit message.

@soypat
Copy link
Contributor

soypat commented Apr 30, 2025

Hol up. I believe the deadline can never trigger now. You are declaring the deadline in-loop, so it is refreshed every time a byte is written over I2C. I suggest going back to the deadline declaration at the start of the function and refreshing it after the write section (right above rxStart declaration). This is because I suspect your program is timing out during the read section due to how the conditions are ordered. If this is not the case let me know

@eliasnaur
Copy link
Contributor Author

What is the deadline for, other than buggy i2c hardware or faulty data lines?

The calling code says

	// timeout in microseconds.
	const timeout = 40 * 1000 // 40ms is a reasonable time for a real-time system.
	return i2c.tx(uint8(addr), w, r, timeout)

but TinyGo is no way near "a real-time system" unless there's only one goroutine running.

With your proposed change (deadline extension just before starting read), there's still a risk of unfortunate goroutine scheduling triggering a timeout.

Simplified case, for the transmission loop:

deadline := now() + 40ms
for _, b := range txBuf { // for each byte to transmit
   i2cTransmit(b)
   for i2cTxBufferFull() {
      if now() > deadline { return ... }
      gosched() // Takes, say, 100ms of time.
   }
}

As far as I can tell, the second byte will immediately timeout if i2cTxBufferFull returns true just once. My proposed change does

for _, b := range txBuf {
   i2cTransmit(b)
   deadline := now() + 40ms
   for i2cTxBufferFull() {
      if now() > deadline { return ... }
      gosched()
   }
}

which ensure that no matter how long a gosched takes, the timeout will only happen if at least 40ms passed with a full buffer.

@soypat
Copy link
Contributor

soypat commented May 2, 2025

OK, I've just realized I had a miconception about the looping logic when I asked you to perform both changes. Can we try your original change and doing some basic math to calculate ideal timeout?

Lets look at a scenario: We have our RP2040 running a slow 10kHz I2C baud. The time it takes for transaction to complete is at least (len(w)+len(r))*bitSendTime where bitSendTime is 1./10000 seconds = 0.1ms = 100us. We can thus set the timeout to be (len(w)+len(r))*100, and we can calculate it correctly. With your original change then we can then discard the time spent between byte transmits (which I had not realized the I2C algorithm could block between every single byte.

@soypat
Copy link
Contributor

soypat commented May 2, 2025

I'd also add in a security factor, like *2 or something of the sort

@eliasnaur eliasnaur force-pushed the push-vstovnronupo branch from 456e38f to 2a25227 Compare May 2, 2025 08:22
@eliasnaur
Copy link
Contributor Author

Done. I didn't add a security factor, because that's already builtin to the pessimistic bus frequency (10kHz).

@eliasnaur eliasnaur force-pushed the push-vstovnronupo branch 2 times, most recently from 01f79a9 to f3c58b1 Compare May 3, 2025 09:36
@eliasnaur
Copy link
Contributor Author

eliasnaur commented May 3, 2025

I've pushed an update that replaces the bitrate computation with a simple "long" timeout. I also replaced another tight deadline in I2C.disable. I was still seeing timeouts, presumably because of an interrupt handler that can run for a while because of real-time constraints.

Longer deadlines are not great, but my rationale is that the deadlines are only there for faulty hardware or programming errors. And they're not that long (40ms). EDIT: finally, the deadlines can be tightened in a future DMA implementation.

@@ -218,8 +215,8 @@ func (i2c *I2C) enable() {
//
//go:inline
func (i2c *I2C) disable() error {
const MAX_T_POLL_COUNT = 64 // 64 us timeout corresponds to around 1000kb/s i2c transfer rate.
deadline := ticks() + MAX_T_POLL_COUNT
const timeout_us = 40000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you change this line? This loop does not call gosched

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did it have a noticeable effect? What baud are you running?

Copy link
Contributor Author

@eliasnaur eliasnaur May 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I run i2c at 400kHz.

I had to bump the disable deadline otherwise I saw timeouts from it (presumably because of a long interrupt handler in my program). Unlike gosched, interrupt handlers can reasonably be assumed to complete in short time (<40ms), but not in (in this case) 64μs.

Copy link
Contributor

@soypat soypat May 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some reservations jumping the disable timeout by factor ~1000 than what the reference implementation suggests. If a interrup handler runs in over a few ms i believe that to be a real issue for tinygo which usually encourages really short interrupts to not negatively interfere with the runtime

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try 400us and 4ms and see if any of those work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the timeouts to 4ms; my interrupt handler completes in ~1.5ms. Long interrupts are not great, but what else can be done about real-time requirements (e.g. interrupt "bottom halves")? TinyGo doesn't offer goroutine priorities.

Just for my curiosity, where do you see the disable deadline in the pico-sdk reference? I couldn't see it in https://github.com/raspberrypi/pico-sdk/blob/master/src/rp2_common/hardware_i2c/i2c.c.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
Currently on vacations so cant spend too much time right now. Looks like I diverted from the Pico SDK and took the datasheet at its word here. Prolly get back to this by mid week

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the reference.

Increment POLL_COUNT by one. If POLL_COUNT >= MAX_T_POLL_COUNT, exit with [...] error [...]

reads to me that MAX_T_POLL_COUNT is a loop counter, not a deadline. And

Define a maximum time-out paramater, MAX_T_POLL_COUNT

reads to me as the value being implementation defined. Only the loop iteration delay, t_i2c_poll, is defined in terms of bus speed.

Am I reading the datasheet wrong here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct! Looks like I slipped on the naming

The `gosched` call introduce arbitrary long delays in general, and in
TinyGo particular because the goroutine scheduler is cooperative and
doesn't preempt busy (e.g. compute-heavy) goroutines.

Before this change, the timeout logic would read, simplified:

  deadline := now() + timeout
  startTX()
	for !txDone() {
	   if now() > deadline { return timeoutError }
		 gosched() // (1)
	}
  startRx() // (2)
	for !rxDone() {
	   // (3)
	   if now() > deadline { return timeoutError }
		 gosched()
	}

What could happen in a busy system is:

- The gosched marked (1) would push now() to be > than deadline.
- startRx is called (2), but the call to rxDone immediately after would
report it not yet done.
- The check marked (3) would fail, even though only a miniscule amount
of time has passed between startRx and the check.

This change ensures that the timeout clock discounts time spent in
`gosched`. The logic now reads, around every call to `gosched`:

  deadline := now() + timeout
  startTX()
	for !txDone() {
	   if now() > deadline { return timeoutError }
		 before := now()
		 gosched()
		 deadline += now() - before
	}

I tested this change by simulating a busy goroutine:

	go func() {
		for {
      // Busy.
			before := time.Now()
			for time.Since(before) < 100*time.Millisecond {
			}
      // Sleep.
			time.Sleep(100 * time.Millisecond)
		}
	}()

and testing that I2C transfers would no longer time out.
@eliasnaur eliasnaur force-pushed the push-vstovnronupo branch from f3c58b1 to adc7c79 Compare May 3, 2025 14:43
Copy link
Contributor

@soypat soypat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@soypat soypat merged commit d840971 into tinygo-org:dev May 7, 2025
27 of 29 checks passed
@eliasnaur eliasnaur deleted the push-vstovnronupo branch May 7, 2025 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants