Why and How we Built a User Testing Lab
At Drop, we’re building solutions for the smart kitchen that help home cooks make the most of their appliances and save time cooking. The Drop Recipes app is the home for quick and easy guided recipes that are compatible with a range of connected and non-connected appliances.
Our mission is to help home cooks make food more often. We believe that the way to do this is to reduce both the complexity of cooking-related tasks and the time investment needed to produce a finished meal. For Drop, user testing is critical to achieving this end. We can only create simpler recipe instructions and connected solutions that get food to the table more quickly if we develop a sophisticated and detailed understanding of how home cooks work in their kitchens. This means observation, and lots of it.
Drop has always had a design-first approach to product development. This isn’t a surprise given that the company was founded by a group of product designers.
Putting our users right at the heart of our product design process has been instrumental to our implementation since day one, and building an in-house user testing lab was the next logical step.
We had run user testing in our office before, but these tended to be fairly disruptive to our normal working environment and required careful planning and set-up. In order to increase the frequency of our testing and reduce the overhead of running those tests, we needed to establish a permanent test lab. Cameras and audio equipment could be permanently installed, and tests could be run without interrupting the wider team’s work.
This permanent space enables a significant improvement in the frequency of tests. Instead of just running tests when we felt we had a concept or prototype important or “ready” enough, we can now test every two weeks, with whatever assets we have available. The regular and scheduled nature of the test acts as a forcing function to move us to lower-fidelity prototypes, and helps increase the cadence of our learning>develop>release loop.
And if we really don’t have any of our own projects to test, we test an interesting app from which we feel there is something we can learn. This was the key cultural change — testing as an organizational habit rather than a project-related milestone.
Planning the User Testing Lab
In planning, we outlined the key features we considered essential for success. In order to test recipes in a mobile environment, we needed our lab to cover the following:
1. Support mobility
The phone being tested should not be in a fixed position as it will be moved around while the participant is using it. It would be unnatural to restrict a person to only have the phone on the table or on a stand.
2. Capture how the user interacts with the kitchen environment
Drop Recipes’ place is intrinsically in the kitchen and, with or without smart appliances, our users will be cooking. Our testing lab needed to be a fully equipped kitchen that our testing participants can cook in. We also needed to able to record how our participants interacted with the app and appliances.
3. Record the session in a non-intrusive manner
User testing rooms can be intimidating for some test subjects. Whilst we cannot recreate the subjects’ home environment, we can try to make the kitchen feel more authentic and the participants more comfortable by making the recording equipment as unobtrusive as possible.
Creating The Set-Up
When setting up two rooms with a one-way mirror separating them wasn’t an option (goodbye, Mad Men dreams) we opted for building a sled. A sled lets you attach a camera to a phone pointing at the screen. In order to let user feedback be distributed to the entire company, we needed software solutions that would enable live streaming from the phone to a remote audience of observers.
We experimented with a series of software solutions, but each had its drawbacks. In the end, the best solution seemed to be the simplest one: strap a camera to a phone and place another in the user testing lab.
We took a Hue HD USB Camera, a black minimalistic phone case, a 10-meter USB extender and a bag of zip ties and got to work making a sled following this guide by Jon Crabb.
This sled setup allowed us to record what is on the users’ phone and what they tap, and also see if the user taps on something that doesn’t do anything. We could also track hesitation more effectively by observing when a user hovered their finger above something, which we would not be able to see using a screen mirroring app. These elements combined to give us the greatest insights into our users’ actions, as well as their comments.
Multi-Purpose User Testing
Thanks to the flexibility of our user testing lab, we are able to adapt the positioning of the tech to suit the type of testing we want to carry out.
We use simple prototype only tests to find issues early on in the development cycle. We also test recipes available in live builds of Drop Recipes to understand the user experience of the app as a whole. Testing a real cooking event is much closer to the experience users would have at home, and provides valuable insights into their behavior that we can use to continuously improve our UX.
When picking a recipe for the participants to cook, we work with our Content team to find recipes that suit the participants’ dietary requirements and show a range of different functionalities in the app.
These images show the difference between the set-up when we are (A) just testing a prototype and (B) testing a built app with a full cooking experience.
Testing a prototype
Testing a built app
Testing The Test
Before unleashing our user testing lab on the public, we ran six internal tests with Drop volunteers from a range of departments. This allowed members of the all aspects of the company — from marketing and engineering, to content and product — to understand user testing and usability and ensure user experience is at the focus of all our decisions.
It also ensured we were familiar with what steps users would be carrying out during the test, and that we had everything that would be required ready in the room.
Live-Streaming The User Testing
A key requirement for the test lab was that all Drop employees should be able to follow the tests in real time, no matter where in the world they were based. Using a combination of live streaming and Open Broadcaster Software (OBS), user testing can be observed in real-time to a private group.
Everyone in the company is sent a calendar invitation with the link for each test embedded, so they can tune in to each one when it is on or watch it back later after the test was complete.
Additionally, a core group of team members with a specific interest in the outcome of the test needs to be able to observe the tests together. Observer sheets are provided containing guidance for these team members, outlining what areas of the test to pay close attention to. This group then participates in a wrap-up meeting after the test sessions to discuss the results.
Recruiting The Right User Testing Participants
One challenge we faced was finding test subjects that fit our design persona. We set a baseline of having a minimum of three testing sessions on the same day, every two weeks — which meant at least 6 new test subjects each month.
We worked with a recruitment company, People for Research, to find participants, which we found to be the most effective way of finding our target personas.
Prior to this, we used Facebook ads, email outreach and a screening form using Typeform to find our target market locally. This approach was slow to yield results and required a lot of in-house work. Working with People for Research significantly simplified the process and allows us to concentrate on creating great prototypes and test plans.
Over the course of several months, we founded a permanent usability testing lab in our Dublin office, established a decent database of participants to pick from and created a system to recruit new participants. Conducting regular tests has had a large influence on the direction we are taking with the Drop Recipes app. It lets us tap into what our users’ needs and see any difficulties they encounter with the app.
”To design the best UX, pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior. Users do not know what they want.—Jakob Nielsen
The introduction of a user testing lab at Drop has been transformative. A frequent, formalized testing setup has helped us refine numerous experiments and product features. As well as bringing us closer to our users, the lab has helped democratize customer insight so that it is not just the preserve of a few members of the team, but is something accessible to everyone no matter what their role within the company. The insights gained help us improve several of our core objectives and gain more confidence in the overall direction of our product development.