Designing for skill discoverability issue in Alexa

Project Overview

There are more than 30,000 skills worldwide on the Alexa skill store. But finding “useful” skills still remains an area of friction for user’s engagement with Alexa. This project discusses some of the challenges users face while finding new skills and dives deeper into their definition of “useful” skills by identifying the scenarios in which people use Alexa skills.

Some of the key findings are — the use of Alexa is episodic. Users want to have the flexibility to search for skills by use case rather than strictly follow a specific category of skill. Currently, while recommending skills through voice output, Alexa plainly reads out the skills by their names and descriptions. In such cases, browsing the skills on the Alexa app is much faster and hence, users are discouraged to use voice for skill discovery.

Based on my user research findings and learnings from literature work, I propose design ideas to address these issues.

Skills: User Research, Literature Research, Affinity Diagramming, Dialogue flow modeling, Voice User Interface Design

Duration: 4 months

Independent project

Selected posts by Alexa users on Reddit regarding skill discovery issue

PROBLEM

Finding useful skills is one of the critical customer pain-points while using Alexa and using voice to find one becomes taxing because of the challenges that come along with VUIs

RESEARCH - PHASE 1

RESEARCH QUESTION

  • What skill discovery tools do people currently use to find new skills?
  • What work has already been done in the past to address the discoverability issue in VUIs?

To address my research question and come up with ideas to address this issue as a designer, I drafted a research plan.

CURRENT SKILL DISCOVERY TOOLS

  • Skill section in the Alexa app
  • Weekly emails by Amazon on ‘things to try’
  • Asking Alexa to recommend skills by using prompts like ‘Alexa! Recommend me skills.’ 
  • Using skills by Amazon like ‘Alexa Things to Try‘ —as part of Flash Briefing, Alexa provides a useful daily tip on new and existing Alexa features
  • Using Amazon’s ‘Skill Finder‘ skill (now removed from Alexa skill store)

REVIEW OF CURRENT SKILL DISCOVERY TOOLS

I reviewed the skill discovery tools mentioned above using my learnings from the literature research and the analysis of user responses on Reddit groups and Amazon’s website.

ALEXA APP

Skills are catalogued by categories

WHAT WORKS

  • Research has shown that users are more likely to resort to direct manipulation when communication success with VUI is less predictable. Here, direct manipulation means using other modalities like touch or type to directly control the system as opposed to using voice. Having a companion GUI app is helpful for users to fall back to in cases when they are unable to find new skills using voice.
  • Users have a wide variety of skills to choose from.

WHAT NEEDS IMPROVEMENT

  • Current skill categorization is overwhelming
  • Users need a way to filter out skills irrelevant to them
  • Users want curated lists of skills where they are categorized by use cases

WEEKLY EMAILS ON THINGS TO TRY

WHAT WORKS

  • Users get weekly updates of new features added to Alexa
  • Users get a quick view of new skills that they can try with Alexa

WHAT NEEDS IMPROVEMENT

  • Users don’t find emails from Amazon on ‘things to try’ of much use as they are long and are not personalized
  • Research has shown that users are more likely to feel motivated to discover system’s capabilities if it gives them recommendations based on their preferences and behaviors. Hence, user profiling matters.

SKILL DISCOVERY BY VOICE

Skills by Amazon to 'find new skills'

WHAT WORKS

Alexa Things to Try is popular among users (who use flash briefing feature) because users are already in the mindset of listening to “new updates” when they hear their flash briefing. Alexa giving users one new tip is directly applicable to their current goal of learning something new.

WHAT NEEDS IMPROVEMENT

  • Users want the ability to search for skills beyond just the categories they fall into.
  • They want to ask questions rather than just reply with a ‘Yes’ or ‘No’ answer.

Using prompts like "Alexa! Recommend me skills"

What the user can say is the biggest issue with VUIs. Research has shown that “guess and see approach” is a natural human inclination to voice interaction. A companion app can guide users initially by giving some prompt examples. But the examples should be such that they provide a structure to what more users can say.

Alexa app gives some prompts that users can use to discover new skills like:
“Alexa! Recommend some skills”
“Alexa! What are your top/latest/new/popular skills?”
“Alexa! What are your smart home/games skills?”
“Alexa! Let’s Barbecue”

  1. The first three prompts can only help users guess that they can ask for latest skills or request for skills by categories like games, smart home etc.
  2. Although the last prompt allows users to discover skills by a use case other than strictly category, it is not enough to help users guess what more they can say.
  3. Even if the user has successfully used the correct prompt, Alexa reads out the skills one by one by their names and descriptions. It is much faster to just use the app to browse the skills than listen to Alexa reading them out. So, Alexa is not making it easier to find new skills using voice discouraging users to come back and try again. Check out this audio:

KEY INSIGHTS

  • Users want a way to filter out skills which are irrelevant to them. 
  • They want to have the flexibility to search for skills by use case rather than strictly following a specific category of skill..
  • When it comes to voice interaction, users guess prompts most of the times. Interfaces should provide guidance to what users can guess.
  • While recommending skills through voice output, it is important to design better interactions rather than Alexa plainly reading out the skill name and description.

RESEARCH - PHASE 2

RESEARCH QUESTIONS

  1. What are the scenarios in which people use Alexa skills and why do they use them?
  2. How should voice interfaces make recommendations? What are the factors that govern whether or not users will accept the recommendations?

POPULAR USE CASES OF ALEXA SKILLS

The use of Alexa is episodic. Users enjoy interacting with her in a way that fits their routines, such as catching up with the news,  playing trivia games while having dinner with family or to unwind while doing some unpleasant chores.

Vasili Shynkarenka, CEO of Storyline summarized the popular use cases based on 5000 reviews of popular Alexa skills. Read the full article here.

USER STUDY

To address my second research question — “How should voice interfaces make recommendations? What are the factors that govern whether or not users will accept the recommendations?”, I conducted a user study with 12 Alexa users.
For the purpose of the study, I narrowed down to games and trivia skills.

STUDY PLAN

INSIGHTS

KEY INSIGHTS

  • User's mental model of Alexa matters. Two users rejected the recommendations saying that they don’t want to play with Alexa further as she is very annoying. Although it was the fault of the skill that it didn’t design for varied responses and better hints, users thought it was Alexa’s incapabilities.
  • While recommending a skill, it is important to understand the context in which that skill will be used and whether or not the user is in that state of mind.
  • Before recommending skills, know what’s that one new piece of information that you are giving to users and make that information explicit in the description of the skill.
  • Labeling the skills by their use cases helps users in getting a better picture of what to expect.

DESIGN GOALS

REDESIGN IDEAS

Adding a section in Alexa app with lists of skills curated by use cases that fit in people's routines

Reading out lists is boring, improving the experience of skill discovery by voice

Scenario 1: Users wants to find skills for a specific use case

Current Design: "Alexa! Help me Relax."

In the current design, when user asks Alexa to help her relax, Alexa reads out a list of skills that are similar in nature. She does not take into account the fact that the user might not be interested in say, sleep sounds or nature sounds. Current design doesn’t ask for user’s preferences

Note: Although currently Alexa accepts the prompt “help me relax”, just by looking at the app, there’s no way users could have guessed that they can say a prompt like this.

Redesign: "Alexa! Help me Relax."

  • Instead of listing the skills by their names and descriptions, the skills are labelled with the purpose they serve. For example, instead of suggesting skills like Sleep Sounds: Babbling Brook, Ambient Sounds: Ocean Sounds etc., user is asked if she is interested in relaxing sounds
  • User is given a choice upfront to choose between meditation skills and relaxing sounds skills
  • Filtering skills based on user preferences. For example, in this case, asking “Do you want to hear nature sounds or space sounds?”

Redesigned dialog flow

Scenario 2: Users wants to discover a new use case

Current Design: "Alexa! Recommend me skills."

  • Doesn’t take into account the context in which the user might be. It might be the case that the user is not in the mood of listening to games and trivia skills.
  • Reads out the descriptions of the skills that are redundant and are not giving user any new information to help them know what to expect out of the skills.
  • Doesn’t let users to specify their preferences

Redesign: "Alexa! Recommend me skills."

  • Understanding the user mood by asking if she is interested in listening to games and trivia skills
  • Not all skills need descriptions. Jeopardy is a famous game, asking users if they have heard about the game to avoid telling users something they already know
  • Filtering skills by asking users for more information like number of people that are playing
  • Describing the skill in a way that it gives a clear picture of what to expect out of the skill using Grice’s maxims of quality, quantity, relation and manner.
  • Making the conversation feel more social to keep the users engaged by adding a subtle humor like “Hmm.. You are difficult to please…but I will try” and letting users specify what they want by asking “How about you tell me a game that you played in the past…..”

Redesigned dialog flow

PROTOTYPING SOFTWARE

I used Storyline to prototype the skill demos.

FUTURE WORK

A longitudinal study with Alexa users is required to observe usage of Alexa in a private setting like homes or less private setting like corporate offices. The objective of the study should be to identify interaction habits of users with Alexa. Here are the research questions that this study should seek answers to the following questions:

  • What is the mental model of users of Alexa?
  • Is there a time of the day when users are most likely to engage with Alexa? How does the usage differ on weekdays and weekends?
  • Is there a pattern in the launching of the skills based on time or location?
  • Is there a pattern in which certain category of skills are launched?
  • Which skills are tended to be re-used by the users and why?
  • What are the different ways in which users figure out the skills useful for them? Can that behavior be extended to help them find more relevant skills?
  • Do users like to toggle between different skills fulfilling the same task? For example, toggling between two different types of skills for weather forecast.
  • When do users like recommendations and when do they not like them?
  • How can the success of a recommendation system be measured?
  • Are users open to getting pro-active recommendations? For example, after they finish playing a game, would they like it if another game is suggested?
  • If yes, how should these proactive recommendations be designed, so that they are enough to keep the interest of the user, and at the same time, that they are not excessive, or too out of context, to make Alexa annoying?

If patterns emerge, they can be used not only to curate the skills by their use cases, but also to personalize the emails sent by Amazon, improve auto skill suggestions in the Alexa app, curate the recommendations when users explicitly ask Alexa to recommend them skills and to proactively make recommendations.