2/13/2014— Eric Kolker
Hey Tesselators, Eric here.
As many of you have heard by now, we found a problem with the main Tessel board about a week ago related to how quickly different parts of the board turn on (we’re talking microsecond timing here). We’re hard at work gathering data about the existing system and testing out possible fixes.
The “reset issue”
When I started this post on Friday (after one or two days of data collection but before parts had arrived from DigiKey), we were pretty sure that the SPIFI flash chip’s boot time was to blame for the behavior we were seeing. The theory was that the LPC1830 (our main processor) was trying to communicate with the flash chip, which contains all user code, before the flash was ready. Because the flash was unresponsive, the code entered a 60-second countdown after which the LPC reset itself and tried again. Since the flash had stabilized by this point, the Tessel would boot normally and detect as a USB device in “application mode.” Translation: Tessel would work after it sat untouched for a minute, but not immediately after being plugged in.
The easiest way to cut through the ridiculous 60-second timer was simply to hit the reset button on the board, which resets the LPC, thereby restarting the boot sequence. Assuming you hit the button more than about 300 microseconds after the board is plugged in (the time it takes the flash to init), the Tessel would boot properly and become usable. We decided, though, that the need for user intervention was unacceptable (read: not shippable), so we’ve been looking into other ways to make the Tessel reset immediately after being plugged in.
Fast forward a few days and we’ve found that the flash is definitely to blame. For a variety of reasons (hysteresis, indeterminate state inside an IC during power up, etc.), we believe that the best way to fix this issue is to add a rest control chip to the board. All that this chip does is keep the LPC’s reset line asserted for a few milliseconds after the board is powered, thereby eliminating the need for the user to hit the reset button (although this may sound like a hack, trust me that this technique is very common).
However, one additional change is also required in order to guarantee that the reset chip does the trick. Some of the lines the LPC uses to talk to the RAM (not the flash) can serve the dual purpose of controlling how the LPC boots. More specifically, when the LPC is in its default boot-mode configuration, these lines control what software runs when the LPC boots, a feature which we’ve been exploiting to make it easy to update the firmware on Tessel. This also makes the Tessel unbrickable, which is a really nice feature.
In order to make use of the versatility of the default configuration, we have resistors on some of the boot lines to configure them to boot from flash and actually break out others to through hole test points. This allows us to switch between “normal” boot and DFU mode (“device firmware update” — a class of USB device that lets us do the firmware update) without permanently configuring the LPC one way or another. That is to say, this keeps it flexible and allows us to physically select the boot mode.
However, the LPC also has internal resistors on these lines which are controlled programmatically. The trouble is, these resistors are not configured properly as the device is booting up (their control circuitry is turning on at the same time). Pair this with the fact that the internal resistors typically fight (pull in the opposite direction from) the external ones, and the device may inadvertently boot into an undesirable state from which it may or may not be able to recover without an external reset event.
We have gotten good results with a workaround which involves removing the external resistors (which has the added bonus of allowing us to increase the clock rate for the entire board), but this requires that we permanently configure the LPC to boot into flash, as opposed to keeping the boot options open/configurable in hardware.
The last piece of the puzzle is a bootloader that will live in a quasi-reserved section of the flash to ensure that the Tessel remains unbrickable (unless you try to load your own firmware…without using our tools, which will refuse to write to the restricted zones). I should also clarify that the Tessel can always be recovered using a JTAG programmer (so in this case “bricked” really means “you can’t get Tessel working again without a hardware debugger”), but that we don’t ever want to force you to use anything outside of USB and WiFi to program Tessel.
What else is new
We’d like to take this opportunity to tweak our power input stage. We’re looking to move from a low dropout regulator (a subset of linear regulators) to a switching regulator because it will give us more current available on the 3.3 V rail. A part swap may also allow us to increase our maximum input voltage (for powering the Tessel from an external battery/supply).
A switching regulator would allow us to draw more current from the 3.3 V rail. The two families of voltage regulators (linear and switching) differ in how they deal with the power available to them. Linear regulators, for a given output voltage and load current (P = IV), burn any excess power by dissipating it as heat. In other words, for a given regulated voltage Vout:
As you can see, these regulators dissipate the power that comes from “extra” voltage as heat, and are poor choices when there’s a large difference between the input and output voltages.
Switching regulators, however, conserve power from input to output and can therefore provide a higher output current at a lower output voltage (or a lower current at a higher voltage):
Here’s to physics! (NOTE: no process is 100% efficient, so we’ll always get less than the theoretical maximum.)
In our case, dropping from 5 V USB 2.0 down to 3.3 V allows us to get ~750 mA out (in theory). The absolute maximum is, at that point, governed by the rating of the regulator, so as much as ~1.8 A might be possible with 12 V in.
While we’re on the subject of input voltage, right now the Tessel can be powered from a maximum of 5.5 V, which is fine for USB and single cell LiPos. We’d be looking to move to a system that gets us up to 12 V in, which would let the Tessel run off of two-cell LiPos, 6/12 V sealed lead acid (SLA) batteries, and even everyday 9 V batteries. We’re in the midst of testing parts and propagating the necessary changes to the rest of Tessel, and should have a new design by the end of next week.
Changes wrap-up/most recent state
We’ve found a switching regulator we like, a reset controller, and have parts in the mail for a power multiplexer (one of those changes that needed propagating). We need to test everything together, update the design files, and then place an order. We continue to live in interesting times…
The original post - or - “Eric, your handwriting is terrible”
Finally, I leave you with the contents of my lab notebook, complete with the scope data I took as a way of showing you what the hardware debugging process can look like sometimes. Fair warning, what follows was copied verbatim from my notebook, so it’s dense, cryptic, and a lot of the supporting information elsewhere in my notes. The general pattern it follows is:
- graph description/experimental setup
- notes/comments/musings on the most recent graph
If you’d like to try your hand at interpreting the graphs, keep the following things in mind:
- I only had a 2-channel scope; if I had had a larger one, many of these graphs would have been combined
- Timing is important. Be sure to look at the values on the x axis.
- Conditions inside an IC (voltages, memory state, etc.) are highly unpredictable during boot-up, so we can’t know what’s going on inside our chips until they tell us that they’re ready. This is why it’s important to turn things on in the right order.
- On a related note, hysteresis exists in many of these chips
Finally, as a refresher, we’re looking into an issue which we believe has its roots in when different chips on the Tessel turn on after the Tessel is plugged in. I’m measuring voltages.
Enjoy, noodle, and hit me with questions.
From my notebook
Want to diagnose boot behavior…
ch1 - 3.3V
ch2 - reset
w/ reset button
1 - 3.3V 2- reset
ch1 - DFU
ch2 - reset
1 - DFU
2 - reset
chip did not come alive
reset chip to blame?
chip power disconnected
run it again!
board instantly came alive on 3.3V wire cut to reset controller. …what?
05 DFU/reset 1V, 200us
06 3.3V/reset 1V/200us
60 second timeout is definitely not working
hypothesis: it is hysteresis. Cap time
added 1uF to reset
still no 60s boot…
but the button works…
button on power plug does not work.
repeated plugging in rapid succession also doesn’t work…
~3s delay before button push works edit: no, it doesn’t
long wait be
plug + reset + 3s delay to lift works
with or without the cap, plugging in + arbitrary delay + reset button works
07 DFU/reset 1V, 100ms
[ NOTE: data file unreadable and data omitted for CH1 (DFU) ]
1uF cap button push after plug
08 no cap, same as above
…for using Excel. My D3-fu is weak but, frankly, I really just miss MATLAB right about now.