It’s been snowing like crazy here in Boston, but we’re warm, indoors, and hacking away on Tessel and its modules.
Last Friday we had one of the roughest days of our company’s relatively short existence. We discovered a bug in our hardware design just days before we were going to give our assembly house the go-ahead to start assembling all 1450 Tessel PCBs that were manufactured. The bug caused Tessel to hang each time it was plugged into USB, necessitating a push of the reset button to be able to use it.
A Bug on the Hardware
The tough part for us as engineers, was that we had known about this bug since July, but had diagnosed it as a minor “software issue” during several independent investigations. The root cause is a race condition between the bootup of the NXP chip (the main microcontroller) and the external flash chip. Our hardware debuggers attach to the reset pin of the Tessel, so we were changing the behavior during debugging.
The microcontroller powers up around 100 to 200 microseconds before the flash chip, which contains the firmware. The NXP boot code (in read-only memory) checks for the firmware in flash, doesn’t find it, then only checks again 60 seconds later. We’re not the only crowdfunding team to hit NXP’s bug, and it’s not something we’re comfortable shipping out to our users.
There is not an insurmountable problem, it’s just a matter of reconciling the consequences. Here are our options:
Solutions and Consequences
NXP has documented that they fixed the “60 second bug” on their recently manufactured Cortex M4 series chips. We’re currently trying to get in touch with someone at NXP who knows if it was fixed for the M3 series as well. If this turns out to be true, we’ll just need to make sure we have those newer chips, then we can still use the same PCB. This is the most ideal situation as our shipping schedule will likely stay the same. That being said, the current chips we have were all manufactured in the last weeks of 2013, so it is unlikely there would be newer chips with this bug fixed.
We could add a reset controller to our main board, which will allow us to delay the propagation of the reset signal to the microcontroller, thereby eliminating the race condition. This will require another board revision, but allows for minimal changes to Tessel’s functionality and has the silver lining of allowing us to make a handful of incremental improvements to Tessel including making USB more robust and picking a better voltage regulator. This our most likely solution and will push our shipping schedule back to late March or early April.
We’re still learning how to create a sustainable hardware company, and this issue was a real eye opener in terms of much more time and energy we need to dedicate towards testing prior to getting products ready for shipping. We have a lot of pressure on us to ship to you, our backers, as close to the date we expected as possible, but quality is more important to us than schedule.
While we honestly can’t think of anything we could have done differently to avoid this specific bug (besides fix every existing software and hardware issue), we’re taking several steps to ensure the quality of our hardware and software in the future. Specifically, we’ve set up a continuous integration rig to automatically run any new firmware on every version of Tessel prior to consolidating into the rest of our code base. We’ll soon be setting up continuous integration rigs for all of our modules as well.
But we do also have good news! We’ve been making progress on several items recently:
- We’re able to write JS programs to flash so that they will run as soon as Tessel is powered up.
- We’ve fixed over 50 firmware bugs reported by our beta backers.
- We’ve been interviewing folks for various positions within the company (want to be an intern or a full-time employee? We’re looking for engineers and designers.).
- We’ve finished the Infrared Module hardware, driver, firmware, and testbench.
- We’ve done a major overhaul of the Audio Module. Here’s the latest prototype:
We now support headphones and line-out and, more importantly, the module can now record audio through an onboard microphone or a line-in jack.
We’ll keep you updated over email as things progress.
Kelsey, Jia, Eric, Tim, Kevin, and Jon