• Costs & Benefits of Beta Testing

    Friday, January 17, 2014

    1/17/2014— Jia Huang

    It’s not uncommon to see “Beta Test” tiers on crowdfunded projects. Before we launched on Dragon, we tested Tessel with anyone we could find. We went back and forth about whether or not we should do a beta test tier. This was around mid August, and at this point Tessel wasn’t usable without a lot of hand-holding, so we were nervous about releasing it early. However, we knew that getting more beta testers would give us greater coverage across Tessel’s features.

    Specifically, with the beta tier we wanted to test:

    • Tessel usage across a longer period of time
    • How easily others can get setup and use Tessel without us being there
    • The way we handle issues
    • Setup experience on other computers/setups

    But we were also concerned about:

    • How much more expensive it would be to manufacture early runs
    • How difficult would it be for us to push breaking changes once Tessels were in the hands of users
    • How much bandwidth we had to support testers. We wanted to be able to resolve issues as they come up, and as a small team, there are only so many issues we can handle at once.

    This was in August and only about 3-4 modules were done at this point. We were on the 4th build of Tessel (we’re up to 14 modules and the 7th build of Tessel now). I was (and still am) afraid of people getting their hands on Tessel without it being fully “ready”.

    We ended up setting up a beta tier that included 3 Tessels and all 14 modules for 1k. Testers would get their Tessels in November or December and modules as we finished building them. We hoped that the price of 1k would be enough to solve most of our concerns.

    • 1k would cover the manufacturing costs of an early run
    • 1k also limits the amount of backers at that level, which means we can handle issues individually.

    Costs

    Now that we’re close to shipping all of our beta tier rewards, I’d like to show where this 1k estimate got us:

    Item Cost per tester
    Tessel v0 194.03
    Tessel v2 195.57
    Accelerometer 8.35
    Climate 13.93
    Servo 15.6
    Relay 14.22
    GPS 29.43
    BLE 29.98
    MicroSD v2 15.0
    Ambient 20.15
    Camera 23.33
    GPRS 25.1
    IR 19.0
    NRF24 9.96
    RFID 21.54
    Audio 13.56
    Shipping Nov 20
    Shipping Dec 40
    Shipping Jan 40
    bad MicroSD round* 10.47
    bad BLE round* 19.31
    bad Ambient round* 12.0
    bad IR round* 18.15
    total cost/tester 807.66

    *During module assembly, we caught a few design errors which resulted in us having to discard parts and PCBs.

    The averaged shipping costs are a little misleading; shipping in the USA is around ~$5-$10, but about $70-$90 internationally.

    I originally thought that a price tag of 1k would more than cover the costs, but it’s actually very close to the cost of running our beta program. And these prices are without paying for assembly, meaning that every time we ship to the testers, there are two of us who spend a day or two in a lab hand-assembling all the modules. The overall assembly time is probably around 70 hours in total.

    Here’s what our shipping setup looks like:

    Benefits

    So far we’ve had around 60 Github issues on Tessel, 20 of which were opened up by our testers, and we’ve closed about half of them. One beta tester even cleared some issues by themselves. Some of these issues were known before we shipped out Tessels to our testers, but having them opened up as an issue by someone other than the team helps us prioritize fixes.

    My biggest worry before doing the beta tier was how much it would sidetrack us from our main development track. As it turns out, all the issues that testers run into are critical issues that we need to fix. Stability is a huge concern, and while we test all builds on our computers before pushing, there are always edge cases out there that we fail to catch. The “nice to have” features we thought we needed to work on are of a much lower priority.

    Having live testers also forced us to put more effort on our debugging and test tools in general. While this takes up more of our time up front, it will allow us to iterate faster in the long run.

    Communication

    After we shipped out units to our first testers, we realized we didn’t really know how to run a beta program. We wanted feedback, but didn’t have many communication channels set up. Our code was on Github, and we just kind of assumed that it would be enough. Issues would just pop up and we’d resolve them on Github just like our normal workflow.

    This ended up being a frustrating experience for both us and our testers, so on the next round that we shipped out we tried to be as communicative as possible. We ended up setting up an IRC channel and being a lot more upfront about how much we wanted to talk with our testers.

    We’ve been getting more success with this approach, but there’s still a lot of room for improvement. If we run a beta program again, I would

    • put more diagnostic tools on the device up front. A lot of communication is just sending back and forth error messages until we figure out what’s happening, and since we have testers in Europe, the time-zone lag takes up a lot of time.
    • give testers an easier way to report issues.
    • have more permanent communication tools setup. Right now it’s a mix of IRC, email, Github and Skype which makes a lot of the information very transient and hard to capture for others that might be running into the same problems.
    • set up expectations of the beta program earlier. Tessel is still very much in beta, and as much as it pains me to say this, getting everything running smoothly is a an exception rather than the default.

    Overall, having a beta program has been beneficial to us, and has helped us reprioritize issues that we otherwise wouldn’t have found out about. This will in turn make Tessel a better product for all of our backers.

    @jia

    #jia huang #jialiya huang #beta testing #crowdfunding #tessel #technical machine #beta tier #beta program #beta #costs #pros and cons #kickstarter #dragon innovation

July 2017

February 2017

November 2016

October 2016

September 2016

August 2016

July 2016

June 2016

April 2016

March 2016

February 2016

November 2015

September 2015

August 2015

July 2015

June 2015

May 2015

March 2015

February 2015

January 2015

December 2014

November 2014

October 2014

September 2014

August 2014

July 2014

June 2014

May 2014

April 2014

March 2014

February 2014

January 2014

December 2013

November 2013

October 2013

September 2013

August 2013

July 2013