IBM Watson Assistant:
Earning Trust in an AI That Teaches Itself
- Betaclosed May 2020, open Dec 2020
- 2core metrics improved
- 20+concepts tested down to 3
Disclaimer
The following case study is personal and does not necessarily represent IBM’s positions, strategies or opinions. I have omitted and obfuscated confidential information.

Summary
Overview
Watson Assistant is IBM’s platform for building AI chatbots. I joined in 2019 to help design Autolearning, a feature that learns from real conversations to improve an assistant automatically.
The problem
The algorithm was a black box, and people weren’t going to switch on something they didn’t understand. The team also had no shared definition of a “good” conversation beyond coverage and containment.
What I did
I led research and design strategy for Autolearning. I defined human-centered success metrics, like Customer Effort Score, and designed the controls that made the AI visible: an easy on/off switch and before-and-after comparisons.
Impact
Autolearning shipped in two beta releases and improved coverage and containment, the metrics customers already used to judge their assistants. Teams no longer had to tune them by hand.
Who I brought togetherThe IBM researchers behind the algorithm, a UX researcher and the team’s leads. We agreed on what success meant, and chose to hold back a metric that wasn’t ready instead of shipping it.

The Challenge
In 2019, IBM researchers were refining an algorithm that learns from end-user conversations and improves an assistant on its own. Until then, improving a chatbot was slow, manual work.
The algorithm was ready. The open questions were about people: how to earn their trust, and how to agree on what success looks like.
Research
Defining success
We asked users how they measured success. Some focused on coverage, meaning enough topics were trained. Others focused on containment, meaning no handoff to a live agent. Everyone agreed neither told them whether conversations were actually good for end users.
Acceptance rate
How often people accept the assistant’s answer.
Customer effort score
How many clicks it takes to get an answer.
Response quality score
Acceptance rate and effort combined into one score.
Earning trust
Users needed to see what the algorithm was doing. I sketched more than 20 concepts in a few days, and the team narrowed them to three for testing.


Concept Testing
I wrote the test plan with a UX researcher and made a short intro video so every participant heard the same explanation.
Effort matters most
Most testers preferred customer effort score. It measured what they cared about most: making things easy for end users.
Show the difference
Comparing performance with and without Autolearning made its value clear.
Keep an exit
An easy on/off switch eased anxiety. If people didn’t like the changes, they could switch back.
Iterating to an MVP
After several rounds of wireframes and testing, customer effort score wasn’t quite ready to ship. With the team’s leads, we held it back to refine it and launched with the sub-metrics that tested well. Customer effort score was still being refined when I left IBM.
We built the MVP in a few sprints.


The key decision
Launch with customer effort score, or only with the metrics that tested reliably?
The options
Testers loved customer effort score, but it wasn’t mature enough. We could launch with it anyway or hold it back.
What we chose and why
We held it back and launched with the sub-metrics that tested well. A trust feature that ships an unreliable number undermines itself.
The trade-off
The first release lacked the metric people were most excited about, in exchange for one they could rely on.
Impact
Autolearning replaced slow, manual tuning with an assistant that improves itself, and gave teams the controls they needed to trust it.
core metrics improved: coverage and containment
beta launches: closed in May 2020, open in December 2020
In the Wild
Learn more about Watson Assistant ↗ · Public coverage of this work.