A novel product solution ventured to new territory, but what lurked in the unknown?
Analyzing risk and usability in a gamified, hybrid product solution.
BEHAVIORAL / FITNESS / Gamification / hybrid product
4 MONTHS, 280 HOURS
Validation
Usability Testing
Strategy
Explanatory
THOMAS JEFFERSON UNIVERSITY, PHILADELPHIA, FEB - MAY 2024
How can we help couples interested in partnering in fitness?
Healthcare companies and professionals have been seeking new ways to nudge people into preventive self-care routines like healthy eating and exercise. Previous research I conducted through interviews and secondary research revealed a unique advantage for couples. Couples face a unique set of problems as they balance preferences and skill levels, all while trying to keep their fitness activities engaging and interactive. How can we help coupled users interested in partnering in fitness? How can we keep it fun?
Exploring the strengths and weaknesses of a designed solution, I set out to validate and test usability
For this project, I worked solo but relied on industry veterans and stakeholders for constant feedback and guidance. My responsibilities included identifying possible strategies for success, analyzing risky assumptions, building prototypes, and performing both validation and usability testing.
Research Process:
-
Defining Success - Leveraging data-based metrics to build for success ....
-
Hypothesis to Strategy - Building towards goals & planning for testing ......
-
Measuring Risk - Testing for uncertainty & exposure in design solutions ...
-
Aligning with Users- Finding usability problems and prioritizing changes ...
Index
KEI
Regularly feels engagement in partnered fitness activities
pre & post survey
+15%
KEI
REGULARLY FEELs a balance of PREFERENCE and variety in fitness activities
pre & post survey
+15%
KEI
REGULARLY FEELs a SENSE OF PARTNERSHIP in fitness activities
pre & post survey
+15%
Defining Success
Leveraging data-based metrics to build successful outcomes
To build a strategic roadmap, my first objectives where to understand:
-
Key Experience Indicators - What does success look like for fitness partnering couples?
-
Competitive landscape - Fitness is a crowded space. How might we stand apart?
-
Users - Who has unmet needs? Who would use our product?
-
Possible strategies for success - What would success look like?
To carry this out, I personally took action on:
-
Reviewing exploratory and secondary research - Previous research identified key experiences for our users
-
Conducting competitive research - Identifying competitors in fitness space using competitive and SWOT analysis
-
Defining user groups - Building from previous research, identify high-value participants
1
/ 4
The fruit of this process was an understanding of the pathways that connected directly to KEI metrics, avoided competitive overlap, and explored strategies that nudged cohabitating couples to engage in partnered fitness activities (as this partnering was a significant driver in secondary research).
Hypothesis to Strategy
Building toward goals & planning for testing
Understanding our targets, it was time to build and test outcomes:
To carry this out, I personally took action on:
-
Hypothesis testing with storyboards - feedback from users helped validate the problem and solution
-
Identifying key features for prototypes with Kano Cards - Organized the experience using MoSCoW
-
Mapping user flows- Strategically mapped out how a user might experience different journeys with the product
-
Wireframing - Working from the user flow created napkin sketches and then digital wireframes with annotations
-
Building prototypes - Implemented designs into both Figma and physical prototypes ready for testing
2
/ 4
Rapid hand-drawn
Wireframe
Figma Prototype
-
Pivot from insights and competitive research to action
-
Build plan for validation and user testing
-
Map out the solution and find key experiences
-
Begin creation of prototypes needed for testing
-
47% increase in habit adoption among cohabitating couples, BUT one has to start first
-
Nudge new couples toward success leveraging Jobs-To-Be-Done and KEIs uncovered in research
-
Game mechanisms have potential to enhance all KEIs: engagement, variety, partnership, balance of power
-
A hybrid product that combined physical and digital experiences would afford ease-of-use, tap into the fun of board games with multiple local users, and provide physical artifacts that could act as triggers and word-of mouth sale generation (as well as create an avenue in the couples gifts and retail markets).
Design Strategy
Measuring Risk
Testing for uncertainty and exposure in design solutions
With storyboarded product strategy, it fell to me to:
-
Identify, categorize, and prioritize risky assumptions
-
Plan methods, metrics, recruitment to test biggest risks
-
Map user experience to build testable prototypes
-
Conduct validation tests with user-aligned participants
-
Evaluate validation test and report findings, next steps
-
Couples would be interested in a gamified, fitness-focused product
-
The product will be easy and engaging enough to follow through on
-
The product will be valuable to users beyond default behaviors.
To carry this out, I personally took action on:
-
Affinity mapping risky assumptions - desirability, feasibility, viability risks prioritized by impact and uncertainty
-
Targeting biggest risks for testing - based on impact, uncertainty, accessibility of measuring risks
-
Planning success metrics and test methods - setting number-based quantifiers of success/failure
-
Implement recruitment strategy - finding high-value, user-aligned, accessible participants
-
Build testable prototypes - fake-front-door and Instagram ads, physical and digital hybrid product experiences
-
Conduct two concurring validation tests - collecting data in relation to success metrics
-
Analyze test results and report findings - comparing results with success metrics and considering next steps
3
/ 4
1. Open envelope of tiles.
Game mechanics balance choice, variety among partners.
3. Completed tiles go on fridge mounted progress board.
2. Scan chosen tiles. Use Google forms in place of app content.
Success Metrics
10% or less failure rate at completing all stages of their first adventure; at least 40% of users in the pilot will complete 2 or more adventures within 4 weeks.
Method
Data Collection
MVP with QR code magnetic tiles, google forms for app flow, magnetic progress board
4 week pilot with 6 participants- Google forms, pre/post surveys and interviews
Results
The pilot validation test failed both success metrics: 66% failure rate when completing first adventure (vs 10% goal); 33% completing two or more adventures in 4 weeks (vs 40% goal). Data gathered from exit interviews revealed that the product could leave users feeling overwhelmed on the first series of interactions. Additional feedback revealed usability issues, promise for the physical game mechanisms, and 2 of the 6 participants did lament not having more time with the prototype. KEIs in pre and post tests showed slight increase but larger quantities of data (participants, time) will be needed.
Recommendations
Because of the positive feedback for specific aspects of the experience and of concrete usability issues, a follow-up prototype of test is recommended with:
-
12 Gold-level participants (gathered from Fake Front Door test below)
-
Extreme streamlined first experience (specific recommendations made)
-
Specified usability corrections for prototypes
Prototype 1 - Minimum viable product
Will the product be engaging and intuitive to new users?
For testing the 3 biggest risks, I carried out two simultaneous prototype tests with participants:
-
Minimum Viable Product Prototype Test - measuring engagement, ease-of-use, and retention
-
Fake Front Door Prototype Test - measuring market desirability for such a unique product
2. Run ad for product on Instagram, linked to landing page.
1. Develop brand, mock ups, Instagram page, landing page.
3. Enter email for a preorder notification for product release
Success Metrics
At least .2% of Instagram users that see the ad will sign up for an Adventurous preorder notification on a landing page (email address required).
Method
Data Collection
Instagram account, ads that link to landing page, preorder email sign up
5 days - unique emails via Wix landing page form, Meta advertising data
Results
This study was conducted via Instagram and a Wix landing page over a period of 5 days with a budget of about $25. Validation testing goals of .2% of Instagram users targeted by our ads, clicking on the ad link, and then signing up with their email for a preorder notification on the landing page was exceeded, with .84% providing an email. The conclusion of this early test is that there is significant demand for this niche product, with 15 unique emails provided over just 5 days and a minuscule budget.
Email subscriptions exceeded target by 4x
Instagram ad interaction segments
Prototype 2 - Fake Front Door
Is there demand for a hybrid, gamified fitness product for couples?
Aligning with Users
Finding usability problems and prioritizing changes
Utilizing developed prototype assets, I set out to:
-
Plan usability research
-
Conduct usability tests
-
Build a roadmap of usability fixes
To carry this out, I personally took action on:
-
Identified key user flows for testing - examining important and complex flows
-
Wrote usability testing script - utilizing natural, unbiased tasks and testing templates
-
Finalized Figma prototypes - ensured prototype fidelity would net usability problems
-
Recruited participants - user-aligned with diverse technology skills (low skills capture more problems)
-
Conducted usability tests - in-person and remote, following scripted prompts and recording results
-
Analyzed results - clustering frequent problems, highlighting severe issues, highlighting SEQ score results
-
Prioritized changes - based on severity ratings (frequency and impact), created list of usability fixes
-
Shared results - provided findings, resources and collaborated on roadmap to address issues
4
/ 4
For what should be an easy, intuitive task, participants were lost when trying to add to an album.
Frequency 40%
Impact 3 /5
This step in the tutorial caused several participants cognitive burden as they had to read and process all of the text.
Frequency 80%
Impact 2 /5
Challenges were missed by participants as they already started activity. Others said they wish challenges were part of planning step.
Frequency 40%
Impact 4 /5
Placing tiles on the progress board was accompanied by high time on task, straining facial expressions.
Frequency 50%
Impact 3 /5
Suggestion: Consult other popular designs for already accepted solutions like a plus icon CTA.
Suggestion: Break this part of the tutorial into two steps: examine tiles and place tiles.
Suggestion: Challenges could be part of planning, adding a main goal to the adventure.
Suggestion: Provide a tutorial moment for this first task, number spots for clearer placement order, greater separation of banked tiles.
USABILITY PROBLEMS
22 total issues uncovered in analog and digital prototype testing
Outcomes
-
Strong demand for the niche product. Validation benchmarks exceeded with .84% of all those who viewed prototype ads, clicking and providing their email for notification
-
Poor usability and high onboarding failure. 66% of participants failed their goals when using product for the first time
-
Poor engagement, retention. Only 33% of participants completed two or more adventures in 4 weeks
-
Assets from testing, like gold-level participants from the fake-front door prototype emails, can be leveraged for more precise testing after making design and test tweaks.
-
22 usability issues discovered with just 5 participants, 8 high severity with major impact to experience. A prioritization list of fixes was created with consideration for frequency, impact, and investment to fix.
Reflection
There were many interesting layers to both the psychological drivers of the product solution (couples attempting to partner in fitness activities) and the design strategy (gamification, physical, digital hybrid product) that made this project a lot of fun for the curious.
-
Recruitment was tough! I had almost no budget and few contacts but was able to find networks, like social running groups, to tap into. This was my weakest element of this project. In the end, I did get the contact info of many gold-level participants from the fake-front door prototype (these users really want to try the product). I would have loved to utilize them in a second test.
-
Building Figma prototypes, sure but building physical prototypes?! Even more fun! It was hard to find just enough fidelity in such little time (and with lesser graphic design skills) but I made it work. I would have loved to work on a creative team here.
-
Usability testing was a great strength for me. Thinking through natural ways to approach tasks, avoiding bias, and conducting and analyzing the tests came to me naturally (of course, always room to improve).
With good recruitment and effectively written tasks (natural, don’t lead)
a single testing session with 5 users has an 85% chance of finding the problems that will effect 1 in 3 users.