Part 2: Experiment & Metric Design
Question 1: Propose and define the primary success metric. What are 2-3 additional
tracking metrics?
Primary Metric: User Engagement Rate
Definition: This metric measures the level of interaction users have with the
redesigned "Reporting" section. It encompasses various engagement activities such
as the time spent on the section, frequency of feature use, and the number of
reports generated.
Reasoning:
User engagement is often a leading indicator of success in software redesigns,
especially when new features are intended to enhance the user experience. By
tracking engagement, we can determine if users find the new features valuable and
easy to use.
We chose session duration and report generation frequency as they directly reflect
users’ interest in exploring the new tools. If these metrics show an increase, it
suggests that users are spending more time interacting with the section and making
use of the capabilities offered.
Feature interaction frequency was included to provide more granular insights into
which features are popular and which might need further refinement.
Considerations:
We anticipate that early engagement may be inflated due to the novelty effect as
users explore the new features. To address this, the engagement rate will be
tracked longitudinally over a 3-6 month period. This approach allows us to
distinguish between temporary curiosity and sustained interest, ensuring the
metric’s long-term validity.
Secondary Metric: Conversion Rate
Definition: The percentage of users who perform key actions (e.g., saving,
exporting, or sharing reports) after interacting with the redesigned features.
Reasoning:
While user engagement measures interaction levels, conversion rate provides insight
into whether these interactions translate into meaningful, goal-oriented actions.
By monitoring this metric, we can validate if the redesign not only captures
attention but also drives users toward desired outcomes (e.g., using advanced
features or exporting reports).
This metric is crucial because higher user engagement doesn’t necessarily guarantee
higher conversion. Users might explore the section out of curiosity but not follow
through with actions that have business value. By coupling engagement with
conversion, we get a fuller picture of success.
Additional Metrics:
Feature Usage Frequency: Tracks which specific features (e.g., filters, analytics
charts) are most/least used. This helps identify which elements of the redesign are
successful and which may need improvement or further development.
Reasoning: Understanding which features drive the most engagement allows the team
to optimize the platform, ensuring resources are directed toward enhancing high-
value tools while reevaluating or removing underused features.
User Satisfaction Score: Obtained through feedback surveys or ratings, this metric
gauges user sentiment and provides qualitative insights into user experience.
Reasoning: Even if quantitative metrics like engagement and conversion are
positive, user feedback offers valuable context that can highlight underlying
issues or provide suggestions for improvements.
