We spend a big chunk of last weekend trying to track down an issue with our motor driving logic. The problem was that sometimes a fast change of direction would cause the i2c bus to die; writes from the Pi would fail and the bot would go crazy as result.
We knew it was likely to be one of a couple of factors:
- High current draw from the quick change in direction causing a brownout.
- Motor switching causing interference/voltage spikes.
Unfortunately, not owning an oscilloscope, it was hard to pinpoint the exact cause so we resorted to magic capacitive pixie dust and software changes:
- We added large reservoir capacitors to the power inputs of the various boards to provide a store of charge in case the power rail momentarily dropped.
- We added small decoupling capacitors too to help filter any noise.
Those changes did seem to help but they didn’t eliminate the problem completely. An extra layer of software changes seems to have done the trick:
- We changed the i2c driver code to close and reopen the device file after a failure. The hope is that that resets the bus more thoroughly than simply retrying the write.
- After John mentioned that he’d seen issues with it in the past, we took control of the GPIO pin that is attached to the propeller’s reset pin and started actively driving it rather than letting it be weakly pulled up with a resistor.
- We beefed up our retry loop, with escalation. If it fails enough times, it resets the propeller and reflashes it. That takes about a second but it might just save us on the day!
- We implemented a maximum ramp rate for the motors so that we change their speed a little slower.
- We put the motor PWMs out-of-phase so that they don’t all start and stop on the same clock cycle.
With all those changes in place, we’ve seen a few retries but it’s hasn’t escalated to a reset yet so, fingers crossed, the problem is fixed enough.