For the last month or so, the Write the Docs book club has been reading Don’t Make Me Think Revisited by Steve Krug. I recently finished the book and, for me, the best chapter by far was “Usability Testing on 10 Cents a Day” (chapter 9). In this chapter, Krug breaks down how to conduct usability tests on a regular basis. A usability test is when you observe a user trying to use something to complete a given task and you record where they succeed and fail.

In the case of documentation usability testing, you give someone the document you want to test and see if they can follow it to accomplish the task being documented.

Reading chapter 9 was yet another reminder of how awesome usability testing is. I’ve been aware of the incredible value of usability testing for many years, but every now and then I let it slip from my priorities—especially in the last month when I was focused on my company SaltStack’s acquisition by VMware.

In this blog, I want to get myself back on the wagon. I’m going to re-commit myself to usability testing by writing about why it’s valuable, examining what barriers are keeping me from doing it more regularly, and committing to a plan to do usability testing regularly.

I know the value of usability testing…

The best chapter by far was “Usability Testing on 10 Cents a Day” in chapter 9. In this chapter, Krug breaks down how to conduct usability tests on a regular basis.

Krug writes:

One of the most valuable things about doing usability testing is the effect it can have on the observers. For many people, it’s a transformative experience that dramatically changes the way they think about users: They suddenly “get it” that users aren’t all like them (123).

While reading this chapter, I was reminded of the first time I technical writing at UVU. I assigned my students to conduct a usability test on the documentation they had written for a previous assignment. (I had assigned them to document something important at their current job since everyone in the class worked full-time and took night school.)

On the day the usability test was due, several students came in wide-eyed and amazed. They talked about how conducting the user test had been incredibly eye-opening and valuable experience. Their perception of their documents had completely changed and they realized all of the areas where they needed to improve. But rather than being a humiliating experience, the students felt energized by the assignment: they now knew not only what they needed to do to fix their document, but also how to fix many major processes at work too. At the end of the semester, many students wrote about how this was the most important thing they had learned in the class and they intended to test everything (not just docs) for usability regularly. Wow!

Fast forward to this last year working at SaltStack. One of the members of our documentation team was passionate about conducting user testing and set a professional goal to start our usability testing program. She personally spearheaded initiatives to set up testing environments, recruit participants, and collate actionable data from the tests.

Her tests were invaluable. She identified several major problems in the doc sets we were testing that we were then able to immediately turn around and fix in our docs prior to release. One of those doc sets was our installation guide. A major component of that was usability testing to ensure accuracy in the docs. After we released our new installation guide, we saw an immediate improvement in average customer satisfaction scores to 3.7 out of 5, up from 2.8 for that doc set. I’ll say it again: wow! Usability testing clearly has tangible results.

So what keeps me from doing it?

Although I’m definitely aware of the benefits of usability testing, I think there are various logistical and mental barriers that are currently preventing me from doing more usability testing.

I mentioned in the previous section that we had a member of our team who was passionate usability testing and started all these initiatives by herself. I want to emphasize that I’m incredibly grateful for her work in this area. But an unfortunate consequence of her running the testing initiative by herself is that, when she left our company a month ago, the usability testing initiative left with her. Besides observing the occasional test, I was mostly an outsider to the testing initiative and didn’t feel any personal stakes or ownership in it other than enjoying the benefits of the tests (i.e. the insights for areas of improvement). As a result, the really cool structure for testing that she set up evaporated overnight when she left. One barrier that has kept me from testing is I now worry it would be a Herculean task to rebuild the testing environment, the testing participant pipeline, and the testing schedule.

Another barrier has to do with testing frequency. My coworker used to run usability tests on average 1-2 times every week. One advantage of testing this frequently was that major, high priority problems quickly began to stand out because multiple testers often identified the same problems. The flip side is that the quirky idiosyncratic problems that only seemed to show up for one user could be dismissed as being a lower priority issue.

The disadvantage of testing frequently was that it took a lot of resources (in terms of human resource hours running the tests) and they generated a lot of work. As that work piled up, sometimes it began to feel overwhelming—especially since my role on the team was usually to take the testing insights and translate them into action items. So I began to develop an attitude that testing, while valuable, was generating more work than I was able to get to.

What’s my plan to do usability tests going forward?

Given these structural and mental barriers to usability testing, my plan for now is to lower my expectations for myself a little bit. A common mantra I have to tell myself is: “Don’t let the perfect be the enemy of the good.” I’m going to have to accept that I maybe can’t do the Rolls Royce quality job that my coworker used to do and accept that my current Honda Civic quality work will have to suffice for now. It will still get me where I need to go.

So my current, Honda Civic-grade plan is to:

• Collaborate with other team members to get buy-in. Although it might be slower and less efficient, I plan to involve my coworkers in the developing the testing plan. Buy-in at the team level will distribute the work more evenly and ensure the usability testing structure has more resilience and won’t rely on one team member alone.
• Scale back testing to 1-2 users once a month. Cutting back on the tests will make the task seem less large and more manageable. It will also give me enough time to implement the actionable insights from the tests before the next round of testing begins.
• Follow Steve Krug’s advice. Krug’s chapter about usability testing has some excellent best practices that I plan to reference and use as I plan my tests. He also has a really handy script he uses to run the tests that I really loved.

Apparently, Krug also wrote a book called Rocket Surgery Made Easy that delves more deeply into usability testing best practices. I just so happen to have it sitting on my shelf, having bought it with good intentions to read it but never did. (Amazon tells me I bought it in 2016 and now I’m truly embarrassed.) Sounds like it’s time to pick it up and get started on docs usability testing!