top of page

Chatbot Benchmark

Project Backround

  • Chatbot was one of the key drivers in sales and a key indicator of company health/likelihood of renewing.

    • Different chatbot types: Job matching, Smart FAQs, Live Chat, Pre-Screening, Sourcing

    • Common components shared by all

  • Many users were scared/overwhelmed at building their own chatbot and instead relied on their customer service managers to build and implement their chatbots

chatbot image.jpeg

Study Goals

  • Gather an understanding of how well the chatbot builder works for users 

  • Identify areas of the chatbot that could be improved for both new and existing users

  • Gather performance metrics for success, time, and errors

Cross-functional team

  • Designer

  • Product Manager

  • CS team members

  • UX Researcher ( me)

Determining what to research

  • Kickoff 

    • Introduce project

    • Discuss current pain points with chatbot

      • Users lack confidence to build

      • Users are not sure where to start

      • Users feel it is easier to have the CS team build the chatbot

    • Identify what we wanted to test

      • Wanted to incorporate as many different question types as possible in a fluid experience

    • Create a research plan identifying 

      • Goals

      • Metrics collected

      • Tasks

      • Dependencies

flat-people-asking-questions-illustration_23-2148907176 copy.jpg

Methodology

  • Moderated research​

    • Technical constraints with the product and online testing tool did not allow unmoderated research. 

    • Ideally would have done a combination of moderated and unmoderated research​

Test script

  • Created a test script to guide users throughout the activity. Did not tell users exactly what they would need to do, but provided them a guide of what they wanted to accomplish with the pre-screening bot. 

    • Edit the introduction to use dynamic text

    • Split conditional options for answers

    • Add different items to the bot:

      • Pre-existing question

      • New question

      • Multi-select question

      • Single-select question

      • Calendar invitation

      • Notification to your team

​Test script example

  • Today you will be building a chatbot for candidates that have already submitted their application

  • You want to edit the greeting to include their name 

  • Ask them if they want to complete the chatbot now. 

    • If yes take them to the next question

    • If not, thank them and end the bot.

  • Find out where they are in their job search. 

    • If they aren’t looking, thank them and end the bot. 

Recruiting participants

  • 2 groups of users we wanted to talk with

    • Current clients - recruited in cooperation with the CS team

    • Non-customers recruited through Respondent

  • 2 recruits in Respondent​

    • Create a screener in Respondent

    • Start with a few generic questions to limit users guessing what we were looking for

    • Provide more specific questions with logic to identify what product users were using and eliminate users that did not qualify

    • First looked for people that had used a chatbot builder in past and did not get enough replies

    • Switched to users that had used any automation (Marketing, Communication, etc.)

Running research and communicating with the team

  • Run study via Zoom

  • During the study

    • Minimal helping of participants

    • Took top level notes by hand (success, issues, help)

  • Post session

    • Created a Figjam to capture issues and screen shots

    • Send slack messages to the cross functional team with high level issues and link to Figjam and recordings

  • After the first 3 some trends started emerging and added a common issues section to the figjam file

Figjam example.png

Analysis of session recordings

  • After sessions watched videos of each session again.

  • Created a spreadsheet to capture all information

    • Time 

    • Success

    • Errors

    • Help

    • Ease of use (5-point Likert scale)

    • Confidence they could build a chatbot (5-point Likert scale)

  • Calculated

    • Average

    • Max

    • Min

  • Groups

    • All

    • Customers

    • Non-customers 

  • Capture video clips to include in readout

Rating issue severity

  • After issues were identified and quantified, worked with the cross-functional team to understand how we would rate the severity of those issues

  • High Severity:

    • Issues that can cause severe task failure, delays, or lead to site exit. These issues should be addressed first and can provide the largest potential performance and user experience improvements.

  • Medium Severity:

    • Issues that can cause task delays or frustration. These issues should be addressed next and can provide smaller improvements. 

  • Low Severity:

    • Issues that have minimal impact on performance such as minor or cosmetic issues. These issues should be addressed last, after higher priority issues have been resolved.

Presenting findings to larger team

  • 1-hour meeting with the larger team to go over findings

    • Report included​

      • Purpose and goals of the study​

      • Methodology

      • Executive summary

      • Participant profiles

      • Task information and metrics​​

      • Videos showing issues and positive feedback

      • Tables with information

      • Issues and Recommendations

Post meeting actions

  • Issues were added to product backlog

  • After this research, team wanted to have additional information on how we could partner the data from the research and other data points like NPS, CSAT, and support ticket

  • Created a UX scorecard that shared all this info on a single figjam file.  

    • Automated data calculations from Chameleon 

    • Worked with the Support team to gather their metrics

Report example available upon request

bottom of page