1/17/2014— Jia Huang
It’s not uncommon to see “Beta Test” tiers on crowdfunded projects. Before we launched on Dragon, we tested Tessel with anyone we could find. We went back and forth about whether or not we should do a beta test tier. This was around mid August, and at this point Tessel wasn’t usable without a lot of hand-holding, so we were nervous about releasing it early. However, we knew that getting more beta testers would give us greater coverage across Tessel’s features.
Specifically, with the beta tier we wanted to test:
- Tessel usage across a longer period of time
- How easily others can get setup and use Tessel without us being there
- The way we handle issues
- Setup experience on other computers/setups
But we were also concerned about:
- How much more expensive it would be to manufacture early runs
- How difficult would it be for us to push breaking changes once Tessels were in the hands of users
- How much bandwidth we had to support testers. We wanted to be able to resolve issues as they come up, and as a small team, there are only so many issues we can handle at once.
This was in August and only about 3-4 modules were done at this point. We were on the 4th build of Tessel (we’re up to 14 modules and the 7th build of Tessel now). I was (and still am) afraid of people getting their hands on Tessel without it being fully “ready”.
We ended up setting up a beta tier that included 3 Tessels and all 14 modules for 1k. Testers would get their Tessels in November or December and modules as we finished building them. We hoped that the price of 1k would be enough to solve most of our concerns.
- 1k would cover the manufacturing costs of an early run
- 1k also limits the amount of backers at that level, which means we can handle issues individually.
Now that we’re close to shipping all of our beta tier rewards, I’d like to show where this 1k estimate got us:
|Item||Cost per tester|
|bad MicroSD round*||10.47|
|bad BLE round*||19.31|
|bad Ambient round*||12.0|
|bad IR round*||18.15|
*During module assembly, we caught a few design errors which resulted in us having to discard parts and PCBs.
The averaged shipping costs are a little misleading; shipping in the USA is around ~$5-$10, but about $70-$90 internationally.
I originally thought that a price tag of 1k would more than cover the costs, but it’s actually very close to the cost of running our beta program. And these prices are without paying for assembly, meaning that every time we ship to the testers, there are two of us who spend a day or two in a lab hand-assembling all the modules. The overall assembly time is probably around 70 hours in total.
Here’s what our shipping setup looks like:
So far we’ve had around 60 Github issues on Tessel, 20 of which were opened up by our testers, and we’ve closed about half of them. One beta tester even cleared some issues by themselves. Some of these issues were known before we shipped out Tessels to our testers, but having them opened up as an issue by someone other than the team helps us prioritize fixes.
My biggest worry before doing the beta tier was how much it would sidetrack us from our main development track. As it turns out, all the issues that testers run into are critical issues that we need to fix. Stability is a huge concern, and while we test all builds on our computers before pushing, there are always edge cases out there that we fail to catch. The “nice to have” features we thought we needed to work on are of a much lower priority.
Having live testers also forced us to put more effort on our debugging and test tools in general. While this takes up more of our time up front, it will allow us to iterate faster in the long run.
After we shipped out units to our first testers, we realized we didn’t really know how to run a beta program. We wanted feedback, but didn’t have many communication channels set up. Our code was on Github, and we just kind of assumed that it would be enough. Issues would just pop up and we’d resolve them on Github just like our normal workflow.
This ended up being a frustrating experience for both us and our testers, so on the next round that we shipped out we tried to be as communicative as possible. We ended up setting up an IRC channel and being a lot more upfront about how much we wanted to talk with our testers.
We’ve been getting more success with this approach, but there’s still a lot of room for improvement. If we run a beta program again, I would
- put more diagnostic tools on the device up front. A lot of communication is just sending back and forth error messages until we figure out what’s happening, and since we have testers in Europe, the time-zone lag takes up a lot of time.
- give testers an easier way to report issues.
- have more permanent communication tools setup. Right now it’s a mix of IRC, email, Github and Skype which makes a lot of the information very transient and hard to capture for others that might be running into the same problems.
- set up expectations of the beta program earlier. Tessel is still very much in beta, and as much as it pains me to say this, getting everything running smoothly is a an exception rather than the default.
Overall, having a beta program has been beneficial to us, and has helped us reprioritize issues that we otherwise wouldn’t have found out about. This will in turn make Tessel a better product for all of our backers.