Article — Design Research
Scaling Research at Dropbox
Responsible democratization: how the Design Research team at Dropbox empowers cross-functional partners to conduct competent research
Published:
March 2, 2021
Written by:
Artwork by:
Covid has thrown just about every life, plan, and connection off-kilter, sometimes in awful ways. Far down that list is the professional challenges it has presented to those of us fortunate enough to be working through it.
However, constraints can spark creativity. If you are looking for a little silver lining in a very dark cloud, we’ve found that a remote environment has not hurt the responsible democratization of research. In fact, it’s even improved our Real World Wednesday sessions in many ways.
Wait, what is Real World Wednesday, again? Why do I care?
Excellent questions.
What is it?
Real World Wednesday (RWW) is a rapid, speed-dating style research session. The RWW format is great for getting quick insights from users. It’s also lightweight enough that non-researchers can lead the sessions.
Prior to Covid, we had five different research teams set up their own tables in a large room, and invited five participants to come to our offices to chat about current Dropbox projects/prototypes. Throughout the 90-minute session, each participant started with one team and chatted for 15 minutes before rotating to the next table. By the end of the research session, the participant would have spoken to all five research groups.
If you want more details on the mechanics of how we set this up, you’re in luck. Dropbox published a fantastic blog post about this in 2019. The post gets incredibly granular about every stage of the process, from picking participants and prep work, to day-of steps and debriefs. It even included a diagram of how we set up the tables.
Why do you care?
This practice was developed to empower all disciplines to get user feedback through democratic research. Our RWW sessions were led by designers, PMs, engineers, UX writers, and anyone who had the need; those were the people talking directly to customers and getting immediate feedback.
Some fine print
As great as RWW is, its value and format are tied to certain types of research, specifically studies that do not require a high degree of confidence.
RWW was created to benefit:
- Moderated usability research
- Feature and flow comprehension (Do participants understand what X does?)
- Discoverability and flagging user concerns
- Language and copy feedback for information architecture purposes
- Onboarding flows
RWW sessions are NOT good for:
- Important product decisions
- Foundational, longitudinal research
- Research with large data sets
A prime example of how a program like RWW can be beneficial was Dropbox’s first-ever remote Hack Week in Summer 2020. Organizers asked us (the Research Operations team) to set up a RWW-esque program. Teams were able to share their brand new ideas with customers and integrate those learnings into their end-of-week share-outs. In less than five business days, teams designed and built products, got live customer feedback on them, and addressed that feedback when presenting to the rest of the company. That’s fast.
Also, as Dropbox moves toward being a Virtual First company, we’re finding iterative feedback to be more and more valuable in developing collaborative tools and ways of working; it is helping us respond to Covid by making changes to our environment.
For more info about using evaluative research (like RWW), check out our previous blog entry, Scaling research at Dropbox. It delves into these topics as well as the responsible democratization of research.
What we changed to make this work in a remote setting
As we all adjusted and settled into a new routine of working from home, our research team wanted to make sure we didn’t lose the program that empowers Dropbox product and design teams to be customer-centric and create customer empathy.
Adjusting the language
You may have noticed phrases we used above, such as “table set up diagrams,” “groups of people,” and “come to our offices.” You know, the things none of us do anymore.
Retrofitting our practices
We had to retrofit our practices to work in our new remote setting: Instead of our original five participants, we recruited seven participants to fill slots. We found it best to have two backups in case of no-shows.
Introducing new tools
We needed to quickly recreate the in-person round-robin experience on Zoom, including the backups. For our pilot, we decided on having seven different Zoom rooms, a Slack channel with all of the RWW facilitators, and a master-document, shared with all the participating teams, to track all of the moving parts for the session. With seven Zoom rooms things can get confusing pretty quickly, which is why we needed a document with:
- The participants’ first name
- The corresponding Zoom links
- The rotation schedule
This master-document was the primary source for the people who were leading research. It told them who they were talking to and when. They referred to it prior to and during the sessions. Any updates for the sessions were made here.
Ensuring everyone knows what to expect
In order for RWW to run smoothly, we needed to coach the people doing research on a few minor tech things:
- Teams had to adhere to a strict schedule (down to the minute!) of when to conduct their interviews so the session could start and stop on time. We did not want to risk anyone accidentally jumping onto their call (awkward!).
- Procedures for how to leave the Zoom room, without ending the call, or how to switch mouse control when sharing their screen.
- Most importantly, we established clear rules for recording and screen sharing with the people doing research prior to the sessions so that everyone was on the same page.
The RWW moderators (typically a Researcher and a Research Ops person) had to be on hand to admit all of the research participants into their sessions and troubleshoot any issues that came up via Slack.
This also meant letting the research participants know exactly what to do and expect during the day of the session. There are a lot of moving parts that they don’t need to see. We found it best to keep the details to a minimum:
- Click the Zoom link.
- Wait to be let into the chat room by a Dropbox team member (this may take a few minutes).
- Once you are in, you will stay in this same room for the entire 90-minute session.
- Every 15 minutes or so, a different group will enter the room to talk with you.
Examples from our master tracking document
Here are a few peeks of what the doc looks like: 🖥️ RWW Remote - AM Session - Date
What’s better?
When the RWWs were in-person, they worked great, but once we went virtual, they got better. We gained greater diversity among our participants. It was easier for participants to take part. It became simpler operationally.
Geographic/industry diversity
When reaching out for participants to take part in RWWs, we limited our search to those who were within a reasonable commuting distance of our NYC, SF, or Seattle offices. This limited who we talked to in a couple of ways.
Urban locations are natural filters for company types and sizes. Design and Healthcare industry professionals were always an option for our San Francisco office, which is located in the Design district and next to University of California, San Francisco (UCSF) hospitals. Along those lines, if an industry did not have a downtown office, it would have been tougher to get representation consistently.
Also, for our Seattle RWWs, we had more selective criteria for participants. Seattle is a less-populated metropolitan area that is more spread out, so we needed to be extra diligent to avoid repeat participants. There were only so many people who both met this criteria and considered it geographically convenient enough to attend.
Additionally, when we talked to people from specific geographies, we ended up talking to specific demographics, too. We were only talking to people who lived or worked in expensive urban areas. Of those people, only the ones who could block off 2.5-4 hours in a workday were able to travel to and from our offices to talk with us. This limited pool of participants was a larger hurdle to clear while searching for meaningful diversity in our research.
Now that we have gone virtual, teams in Seattle can talk to participants in New York, Arkansas, Illinois, and Arizona. Also, these participants can represent industries, job titles, and overall different backgrounds that don’t exist the same way in urban areas.
Lower commitment for participants
The in-person RWWs always felt like a big commitment for participants to take on. They happened during business hours. Each was a 90-minute session of activity, which is already long, plus arriving early, checking in at security, and getting escorted to and from the room—it added up.
Committing to a 90-minute Zoom call is less cumbersome than commuting to a location that may or may not be convenient for the participants.
Also, the role of backup is much more appealing. Instead of commuting for a chance to take part, participants are asked to standby near their computers at home for about 15 minutes.
Cancellations cause less chaos
Cancellations by participants or research teams, while still frustrating, are easier to smooth over. Finding a new video participant the day before a session may not be ideal, but it is much more feasible than arranging an in-person visit.
Internally, drumming up enough groups to participate is easier as well. Now, if our smaller Seattle office can’t fill their five slots, they can tuck in with the New York team, or vice versa. Previously, it meant waiting a week or more for projects and schedules to align.
Constraints = challenge. Challenges foster improvement.
We shifted our in-person RWW sessions from a conference room to multiple Zoom rooms. We created a detailed schedule, tracking documents and dedicated Zoom rooms to replace in-person communications and adjustments. We expanded our recruit from three cities to the entire nation, and made the research sessions more appealing to our customers.
In 12 RWW virtual sessions, we’ve talked to 105 different customers representing demographics, locations, and industries previously inaccessible in this format. We are also converting these participants with an improved recruitment rate. Any of our research teams across the country can take part when they are ready, and it is simpler to make last-minute operational adjustments.
This virtual format made it is easier for our researchers and non-researchers to talk to unique customers, and made it easier for those customers to talk to us.
Latest in Design Research
Article — Design Research
Do you like it? How to assess preference
Preference testing seems to be a go-to method for measuring aesthetic appeal amongst Design Researchers. But there are more rigorous ways to understand how users perceive visual design.