Feedback Is Your Best Friend

Keeping track of your customer reviews is important, especially when it’s the only channel of communication you have with them. You often find interesting feature ideas and improvement suggestions, helping you increase retention and engagement.

This time I want to share an example journey that started with a sudden spike in bad user reviews of Categories Game, and solved with a bug report to the Amazon Alexa team.

We use Skilla, our review monitoring service, to ensure we don’t miss important feedback for our games. One day many of the reviews were turning negative:

So we started looking for signals about this “crashing” phenomena. Inspecting the logs did not help since there was no evidence of a crash. But one of the skill’s metrics proved that the symptom is there:

“Banana” (??)

Playing around with the skill a bit yielded an effective way to reproduce the issue: Saying banana makes Alexa play an earcon, which hints that it couldn’t understand. Our backend did not get any request and so the user was left frustrated, trying to make Alexa understand him to no avail.

Comparing the metrics for different regions showed that the issue was isolated to the US. So we were looking for a recent change in the US region’s model.

The Alexa platform does not offer a history view of your interaction model, you only get the Live version and the Development version. This made us store versions of our model in a git repository - a recommended practice. Comparing the differences between the current model and the pre-spike model highlighted some In Skill Purchasing (ISP) related intents that were added at that time.

The Smoking Gun

After some trial and error we narrowed it down to the Intent Confirmation feature that we turned on for our ISP Refund intent:

Turning it off fixed Alexa’s response and we were able to properly handle user interactions.

Sure enough, a few hours after the fixed model was deployed things were back to normal:

And this is how we made Categories Game great again™

Our takeaway from the incident is that monitoring user feedback is vital, and it’s a good idea to define other data-driven metrics that should be monitored and acted upon. How to define and monitor them? That’s a topic for another post.