• High-Level Automated Hardware Debugging

    Wednesday, April 23, 2014

    4/23/2014— Tim Ryan

    The CC3000 is the on-board component that enables the Tessel to connect to WiFi. This $12 chip communicates over SPI, has a built-in TCP/IP stack, and comes with a host library that enables using it indistinguishably with the BSD sockets API.

    For example: Tessel, while not running Linux, can easily drop in a library like c-ares (used to look up the IP address correlated with a domain name) because we’re using the same API used to make TCP/UDP queries on the desktop.

    A chip this capable is naturally complicated, and so is Tessel’s firmware. Debugging can occasionally be much more difficult than throwing open a GDB console and stepping through code. Is the state of the CC3000 chip known by the code? Was an incorrect command sent most recently, or a thousand commands ago? Is this TCP command originating through the JavaScript VM or directly from third-party code?

    Instead of debugging all of these subsystems simultaneously, it’s more straightforward to just analyze a) what commands the Tessel sends and b) what commands the CC3000 sends back. Using these exposed pins on the Tessel board and a logic analyzer, we can record the SPI communication emitted by any Tessel code and dissect exactly what went on.

    Tessel exposes breakout SPI pins for the CC3000.

    We have several of the Saleae Logic line of logic analyzers around our office. The Logic and its larger brother the Logic 16 can record high-speed signals, for example the 4MHz SPI communication to the CC3000, as well as I2C, UART, or USB signals, then display them in a zoomable graph.

    The startup SPI sequence of the CC3000.

    The interface for the Logic analyzer is nice for being able to visualize erroneous communications, but we aren’t much closer to understanding what is happening on the signal level. In particular:

    1. The Logic analyzer has helpful visual tools to see what bytes are actually being sent over the protocol, but not what groups of bytes mean.
    2. We have to specify an upper bound for the number of samples we want to take, rather than continuously logging.
    3. The options to export data have to be configured each time.
    4. We can’t use the command line.

    Luckily, the open source world of reverse engineering is alive and well!

    Sigrok

    Sigrok is an open-source library with a collection of drivers for various open-source projects. In our case, Sigrok supports a family of logic analyzers using the FX2 chip, including the Logic 8. Using brew install sigrok or any version 0.4.0 or newer from your repository, we can communicate with our analyzer directly from the command line.

    sigrok-cli \
      --driver fx2lafw \
      --continuous \
      --config samplerate=8M \
      --output-format vcd \
      --output-file /tmp/sigrok.vcd \
      --probes 0=cs,1=miso,2=irq,3=mosi,4=sck,5=sw_en
    

    We’re using the fx2 driver, with a sample rate of 8MHz (twice the speed our CC3000 runs at), outputting in vcd format to /tmp/sigrok.vcd, with probes 0-5 (named according to how they are named in the first diagram).

    The command will begin logging data from the Logic analyzer in perpetuity using the --continuous option, as long as the USB drivers can keep up*.

    Now that we’re logging to /tmp/sigrok.vcd, we can throw together a few Node modules to utilize it:

    • tail-stream creates a stream of a file that is actively being written, outputting more data whenever the file size grows.
    • vcd parses a textual value-change dump file, turning it into a series of “sample” events.
    • cc3000-log is our own script for interpreting the signals of the CC3000, interpreting them as SPI and generating a human-readable list of communication:

    The interpreted output of the CC3000 communication. Support for interpreting the payload bytes on the right is partially implemented.

    Instead of adding extensive debug facilities to many locations in our code (slowing it down and impacting the operation of the CPU), we can now be a silent observer: tapping the communication of the Tessel and the CC3000 and viewing in real-time how the IC is being controlled. This technique doesn’t just apply to the CC3000, but for any module or IC that the Tessel communicates with!

    TLDR: Sigrok is an open source library for communicating with commercial logic analyzers. Using it, you can make powerful, customized tools to debug ICs and signals from your command line, a welcome addition to automated testing.

    We highly recommend and use the Saleae Logic. But if you’re looking for something much cheaper (in price and quality) or throwaway devices to augment automated hardware testing, there are a variety of clones from foreign markets also.


    * 8Mhz samples are a lot; originally I was trying to set --output-format as binary or hex, which would inevitably cause the device to stop responding. It wasn’t bounded by my USB driver, it was bounded by the I/O speed of piping that many bytes to a file! The VCD format only outputs when values have changed, creating a drastically smaller log. In the future, we would bind directly to libsigrok for even faster throughput, but first hack first.

    #tim ryan #wifi #cc3000 #hardware #reverse engineering #open source #tessel #technical machine #logic #saleae logic #saleae #SPI #tcp #ip #sigrok #debugging #high level #automated debugging

  • Tessels are in production!

    Monday, April 21, 2014

    4/21/2014— Updates

    Tessel PCBs are in manufacturing as of about two weeks ago.

    Meanwhile, our modules have been coming in:

    That’s about 1700 modules in bags on our desk, and more are rolling in all the time.

    On the software side, we’ve begun continuous integration for all Desktop tools on Windows, Linux, and OS X, thanks to Travis CI and Appveyor. We’re also running continuous integration for our firmware builds on ARM. Look for details in an upcoming blog post.

    Additional improvements:

    • We’ve completed major firmware infrastructure improvements & bug fixes
    • Eric updated hardware design documentation
    • Jon has the camera module taking pictures:

    And here’s what we’re working on:

    • Building out community site
    • Improving Wireless communications
    • Finishing module drivers
    • Improving power consumption and event loop
    • Moving to our new office in Berkeley, CA!

    #update #manufacturing #tessel #technical machine #pcbs

  • How to Get Stuff Done

    Monday, April 14, 2014

    4/15/2014— Kelsey Breseman

    As a company with six people and a lot of work to do, it’s a challenge to keep track of tasks and priorities. Six people is just big enough that you might not know what everyone is doing, and small enough that there are always more immediately important tasks to do than people to do them.

    We’ve tried a few different ways to make sure things get done efficiently, and usually end up using some combination of all of these:

    Post-it graphs

    Post-its are visual and moveable. If you put them on a nearby wall, you’ll notice them. They’re some of our best productivity tools.

    Jon came up with a really good graph:

    Every yellow post-it represents a task to be done before we ship. The X-axis goes from low to high priority, and the Y-axis is completeness. A fully complete task can be moved to the top. The pink marks on some tasks are ripped pink Post-it denoting that the task is blocked until another task is completed.

    This is much better than a binary list, because there are a number of almost-but-not-quite finished tasks, and so it more accurately represents the state of the project, and we move our tasks up the wall at the end of the day. Additionally, since they’re ordered by priority, it’s really easy to look at the wall and figure out what you should be working on. Ideally, your graph is y=x at any given time as you try to push up the lower right quadrant.

    The blue and purple post-its are larger-scale categories of tasks, also ordered by priority.

    We also tried post-its for public shaming:

    The blog is important for all manner of reasons, but we still regularly neglect to write. We’ve tried various tricks (setting exact due dates, me personally reminding each person, etc.). This one is simple: everyone’s name is on a Post-it. The masking tape divides people who owe a blog post from people who have written recently. The objective is to get across to the safe side of the line.

    We took this one down for fear of the tape bonding to the wall paint, but it’s intact in principle.

    Meetings

    Bread o’clock is around 4pm, when we’re all kind of hungry. We stand up, look at and adjust the post-it wall, and informally discuss what we’re working on. This facilitates a “let me help you with your problem” exchange between team members so tasks get accomplished with a minimum of frustration.

    Wednesday is “food day”: the company buys lunch and we talk about the bigger picture (where the company is going) and things that just take a while to talk about. This makes sure we continue to think about where we’re going as a company, and not just the day-to-day.

    We eat lunch together anyway, but on Wednesdays we intentionally talk about work.

    Idonethis/Igonnadothis

    At the beginning of each day, in our company chat (Zulip), we each write what we’re planning to work on. At the end, we write what we’ve accomplished. This way we know what everyone is working on, and have a record of our own progress.

    Asana

    For the zillions of little tasks we need to complete, we use Asana. It lets you organize things into projects as a team, assign tasks to team members, and give due dates. Handy features include recurring tasks (I have one that reminds me to do accounting once a month) and emails/push notifications reminding you that your tasks are overdue.

    The difficulty is, it’s a task in itself to keep Asana tasks up to date, and if you forget to check Asana or filter out the emails, it quickly piles up into a mess of outdated assignments.

    We use Asana on-again, off-again. Mostly, I use it as my personal to-do list for company tasks.

    Calendars

    It helps to have a deadline for any given task, and calendars are great visual tools to remind yourself exactly how long you have left.

    We’ve found a giant calendar to be very helpful with long-term team milestones. Particularly last summer, when the founding team was still deciding whether they would continue when fall semester started back up, there were exact dates we needed to hit corresponding to board revisions– and if we didn’t hit them, the product wouldn’t launch.

    This motivated us into more than a couple of all-nighters, because you can’t argue with a calendar.

    All right, time to check the wall and see what’s next.

    Kelsey
    @selkeymoonbeam

    #kelsey breseman #organization #project management #pm #technical machine #tessel #asana #post-it #tasks #task management #company culture

  • Tessel Progression

    Wednesday, April 9, 2014

    4/9/2014— Jia Huang

    About 2 days ago we put Tessel in production. We just got back the time estimates from our manufacturer– it’ll take about 1.5 months for production to finish, and if we’re lucky, 2 months till ship.

    One year ago Jon, Tim, and I set out to build what would become Tessel. Now that we’ve put Tessel in production (for real this time), I think it’s a good time to go through all the different revisions we made along the way.

    Version 1, April 2013

    Jon mainly designed this board for a class on electronics prototyping. I believe it was supposed to stream music. The main MCU was a Freescale chip and there’s a breakout for Arduino shields. There’s a population option for a second chip (probably as a programmer of some sort). The Wifi chip is already present.

    Jon pretended that the LEDs could blink.

    Version 2, May 2013

    I started working on the PCB and electrical design at this point. The board is completely redesigned with the chips we are still using (although in different footprints). For this revision we were just trying to get the main LPC chip working on our own PCB. I messed up on the JTAG breakout that’s used for programming though, so I ended up having to splice on some wires in order to get the right pin connections.

    We got the LEDs actually blinking.

    Version 3, June 2013

    This is basically the same as the previous version except we fixed all the previous mistakes. All the main components are populated at this point though I think we still had some problems with WiFi.

    It was taking me about a day to assemble one of these things even with a stencil. Jon and I would sit side by side and make two of them, of which one would actually work. I think we only ever had two or three of these things working.

    We were able to run “javascript” on this… to blink some lights.

    Around this time I started getting really scared that WiFi would never work so I made all these WiFi breakouts. They look pretty similar but I was playing around with trace spacing and top and bottom copper.

    Version 4, July 2013

    We had Eric contracting for us at this point so I got lucky and didn’t have to lay out PCBs anymore. Obviously Eric is way better at it than me because this revision had us going from chips with visible pins (and solderable by hand) to Ball Grid Arrays. Worthington Assembly helped us make this batch.

    You can see the beginnings of our 10-pin header modules start to form along with the final formfactor.

    Wifi actually worked at this point. We could probably run the current Tessel firmware on these boards if we really wanted to.

    Version 5, August 2013

    Eric got fancy and rotated the chip which gave us room to add the GPIO bank at the back. We lost some room for the mounting holes though and ended up only having 2 in the middle. There’s also a WiFi debug port so we could debug that easier.

    We had some USB issues with this revision, but were able to run it fully. This is the model shown in our crowdfunding videos.

    Version 6, October 2013

    We added some additional power regulation features on this board so that it could be powered externally. There’s also a breakout for the WiFi antenna so that a bigger external antenna can be attached if the chip antenna does not provide a good signal.

    The boot selector pins (pins that determine if Tessel boots from USB or from internal Flash) are exposed so that we could revert bad firmware versions easily.

    We still ran into USB issues with this version though, so we had to do another prototyping round.

    Version 7, November 2013

    We finally fixed the USB issue with this revision. We thought this was the version we were going to use for production, until we hit that reset issue.

    We ended up ordering a bunch of PCBs for this revision and then scrapping them when we decided to push back production.

    Version 8, March 2014

    In order to fix the reset issue we added a chip that holds our MCU in reset until after our external Flash boots up. Since we had to do this revision anyway, we thought that we might as well add on another feature (bad idea). The extra feature involved changing around the power circuitry (all the chips on the bottom left) so that Tessel can be powered off of more than 5.5V externally.

    We also removed the external boot selection pins because they were on our high speed RAM lines and messing with them would sometimes crash Tessel or prevent it from booting correctly. In its place we wrote a custom bootloader to recover from bad firmware.

    We ended up having problems with the power chip though, so we did another revision to fix it…

    Version 9, April 2014

    Here we’re approaching maximum utilization of all available space. The power circuitry (bottom left) gets even more crowded.

    This is the version we actually put into production about 2 days ago.

    Originally Tim, Jon, and I thought that we would have this thing completed in 3 months over the summer, so we tripled the time (like everyone said we should) and gave ourselves an additional 6 months for the crowdfunding campaign to ship.

    Well, it turns out it was harder than we thought, so now 3 months after that deadline we’re almost there. We’re still patching up our firmware and related tools, along with trying to provide everyone with a superb first run experience.

    Jia

    #jia huang #update #updates #tessel #technical machine #revisions #iterative design #electrical engineering #technology #pcb #pcbs #history #company culture

  • Tessel's First Run

    Monday, April 7, 2014

    4/7/2014— Kelsey Breseman

    I’ve been on-and-off designing the first run experience for Tessel since January, and wanted to share a bit of the process and thinking behind our first run/installation sequence.

    One of the major selling points of Tessel is its ease of use: the code should be intuitive to a web developer. Adding physical modules should be just like adding code modules. You should be able to get from idea to simple prototype very quickly, and without any frustration related to using our tools.

    Therefore, in making the first run experience for Tessel, I want to show you how quickly you can make your code interact with physical hardware. I set a goal of two minutes maximum from unboxing to working code.

    I started here, thinking about all of the different backgrounds of our users and how we might serve each individually. There were two main components of this:

    1. Different modules for different people: everybody gets a different set of modules. I didn’t want to waste people’s time with modules they didn’t have, but wanted to make sure everyone got the instructions they needed.
    2. Different skill levels: most of our users are experienced web programmers. However, we do have a few who have never used the command line or even written a line of code. I wanted to make sure that our first run didn’t alienate anyone, even absolute beginners (this is something I’m still trying to resolve– suggestions welcome).

    This led me to make a first page with a lot of options:

    Original Page 1

    You could select any grouping of modules, and then click on one of the two big buttons. The “I’ve never programmed” option took you through a command line tutorial before proceeding on the main path of installation > tutorials > projects (filtered to show only projects involving your modules).

    On the page teaching use of each module, I only showed the modules the user had selected. You could switch between relevant tutorials by clicking on the icons shown at the top:

    Original modules tutorial page

    On the face of it, this was a nice “choose your own adventure” tutorial: you get just the information relevant to you by telling the tutorial your unique situation. But when I showed it to users (one advantage of a coworking space is how often people who have no idea what you’re working on wander by), they were paralyzed by choice.

    As I now recall, I hated “Choose Your Own Adventure” books as a kid. I always wanted to know what would happen if I made a different choice, and there were just too many choices to follow every forking plot line. Simultaneously, I felt boxed in by the choices I was allowed to make: I didn’t want to select from the multiple choice menu, I wanted to make my own creative decisions.

    Something similar was happening here: people had to make choices right away, with no information. Not only was module selection unclear, but they worried that if they skipped the beginner button, they’d be in over their heads and might not be able to turn back.

    So my next design made the choice a bit easier. The module icons were made more obvious with an increase in size, and I shrunk the text on the beginner button, adding explanatory text below on mouseover:

    Clip from Rev. 2 of the first page

    I implemented similar slight design changes throughout the first run experience in response to user feedback– and these changes did improve successive users’ flow through the process. However, there’s only so far you can go with a basically flawed design, and I realized that I’d become stuck in my original design paradigm. So I asked the team for a design review with a focus on improving the overall flow– somewhat inspired by this blog post.

    Design reviews are great. I highly recommend going back and forth between talking to users and getting feedback from your team, because they provide different kinds of feedback. Users can show you that they have a specific problem or reaction. Teammates in a design review are more likely to assess your bigger picture design.

    My team had some really good feedback as a group, and by the end of twenty minutes, I had some solid design paradigms and ideas for implementation, which have driven my next round of edits.

    Here’s the current flow (still in development, so anything could change):

    First page: install. (No, you can’t install tessel from npm yet. Soon.)

    First code: blink some lights.

    Modules page: an expandable list of each module, also navigable by the sidebar menu, which expands on this page to show each module.

    Modules page with the accelerometer tutorial expanded

    …and more to follow! I’m still messing with the tweet-from-a-Tessel tutorial and the page of projects at the end.

    Suggestions welcome!

    Is there anything you’d particularly like to see? Any particularly good first run experiences I should use as examples? Let me know.

    Kelsey Breseman
    kelsey@technical.io

    #first run #first run experience #installation #fre #kelsey breseman #technical machine #tessel #frontend #design #user testing #user oriented collaborative design #design reviews #design review

January 2018

July 2017

February 2017

November 2016

October 2016

September 2016

August 2016

July 2016

June 2016

April 2016

March 2016

February 2016

November 2015

September 2015

August 2015

July 2015

June 2015

May 2015

March 2015

February 2015

January 2015

December 2014

November 2014

October 2014

September 2014

August 2014

July 2014

June 2014

May 2014

April 2014

March 2014

February 2014

January 2014

December 2013

November 2013

October 2013

September 2013

August 2013

July 2013