Personalization Data ROI report
The Personalization Data ROI report dashboard allows you to understand the impact of your Personalization Data implementation on ROI over time. You can also download a weekly detailed CSV file to serve as input for your own ROI analysis.
Why is it important?
Personalization is an established best practice in eCommerce. By presenting shoppers with an experience tailored to their interests, you can drive conversion and customer loyalty. Shoppers that view personalized product recommendations on a retail site are not only more likely to purchase a product, but they are also more likely to return to the site in the future.
Good personalization requires good, comprehensive data, and the customer journey seldom starts—or ends—at your website. BV Pixel captures product purchases as well as non-transactional events within our network of over 5,700 brand and retail sites. We track each customer’s journey within our network, on and off your site, and build individual shopper profiles in real time. We use these shopper profiles to recommend products or categories for each profiled visitor to your site.
A/B testing
The key to understanding the value of your Personalization Data implementation is to set up an A/B test. Each visitor to your site is randomly assigned to either a control or test experience upon entering the site. Once a visitor has been assigned to a control or test group, the visitor stays in that group for all future site visits throughout the duration of your experiment. Using this experimental framework, you can compare the groups’ behaviors to one another to accurately determine the impact of your Personalization Data implementation on revenue and transactions, among other metrics.
Control and test groups
The following describes the site experiences of your control and test groups, depending on your recommendations display solution:
Control group | Test group | |
---|---|---|
If you do not currently use third-party recommendations: | Will not be shown any recommendations consistently across the entire site |
Will have the opportunity to see a recommendations carousel if they scroll to the portion of the page where it is rendered Visitors with a Bazaarvoice shopper profile will be shown recommendations based on that profile; visitors without a Bazaarvoice shopper profile will be shown Bazaarvoice trending recommendations |
If you are already using third-party recommendations: | Will have the opportunity to see a recommendations carousel that is fully populated per your existing third-party recommendations provider if they scroll to the portion of the page where it is rendered |
Will have the opportunity to see a recommendations carousel if they scroll to the portion of the page where it is rendered Visitors with a Bazaarvoice shopper profile will be shown recommendations based on that profile; visitors without a Bazaarvoice shopper profile will be shown your current provider's recommendations |
Sample size and duration
To ensure confidence in the data, the sample size of both traffic and conversions needs to be statistically significant. While you are planning your A/B test, you should determine the sample size you need to reach statistical significance and estimate how many weeks you should be running your test based on that. See Optimizely Sample Size Calculator or VWO A/B Test Duration for help estimating these values.
The actual duration of your A/B test depends on a number of factors. Review these suggested stopping rules to help you determine a test duration. A/B tests should be run across 7-day weeks (Monday through Sunday) and not partial weeks. We will not show data from partial weeks in the dashboard.
Traffic distribution
It is important to set traffic distribution (for example, a 50/50 split between control and test groups) before starting an A/B test. Do not change this distribution while the test is running. In the event that you change the traffic distribution during your test, only new visitors are split according to the new traffic distribution, which can alter the results of your test. This is called Simpson’s Paradox .
You may want to start off with a traffic distribution of 90% of visitors in the control group and 10% of visitors in the test group to make sure everything is working properly. When you feel confident, you can stop that test and move towards a 50/50 split of your traffic. Always keep track of when you make test split changes.
Seasonality and external influencers
You should run A/B tests during a “normal” time of year—that is, when no major increases or decreases in conversion rate are expected due to seasonality, such as during a peak holiday shopping season.
You should also be mindful of other marketing activities that are occurring at the same time as the test. We recommend, when possible, that you pause these other activities before the test starts and while the test is underway.
Personalization Data ROI report dashboard
Access the dashboard
To access this report dashboard:
- Log in to the Bazaarvoice Portal at https://portal.bazaarvoice.com .
- Select Menu » Personalization.
Filter
Using the date range filter at the top-right of the dashboard, you can see metrics from different time frames of your A/B test. You can choose from the following date ranges:
- Last week
- Last 2 weeks
- Last 3 weeks
- Last 4 weeks
- Last 12 weeks
- Last 52 weeks
You can see the date range currently selected in the banner at the top of the dashboard.
Unique Visitors
Unique visitors are detected across tracked pages during the specified date range. Tracked pages include any page that loads Bazaarvoice analytics tagging, whether through the Bazaarvoice-hosted display or through BV Pixel. The Personalization Data ROI report deduplicates unique visitors weekly based on a full week from Monday at 12:00am to Sunday at 11:59pm.
For example, an online visitor, Drew, arrives at your site on during his lunch hour on Tuesday, August 28 at 12:13pm. Upon entering the site, he is automatically assigned to your test group. He has browsed your site and some other sites within the Bazaarvoice network before, and his shopper profile indicates that he is interested in high-top sneakers. He scrolls down your home page, where you have implemented a Bazaarvoice-powered recommendations carousel. Because he is in your test group and has a Bazaarvoice shopper profile, the carousel he sees is populated using Bazaarvoice Personalization Data. Drew clicks on a pair of sneakers that catches his eye. However, he decides not to buy anything from your site. He returns to your site again on Friday, August 31 to continue browsing. He does not return again to the site until the following week on Monday, September 3 at 12:47pm. Although he visited the site two times in Week 1 (Monday, August 27 to Sunday, September 2), he is only counted as one unique visitor in your Personalization Data ROI report calculations. He is counted again as a unique visitor in Week 2 (Monday, September 3 to Sunday, September 9), and he is counted as a single unique visitor each subsequent week he visits your site, no matter how many times he visits per week.
Attribution window
When a visitor scrolls to the part of the page where the recommendations carousel is or would be rendered or a visitor interacts with the recommendations carousel, the Personalization Data ROI report associates conversions that occur within 24 hours of meeting that criteria. These 24 hours are known as the attribution window.
For example, a test group customer might look at product recommendations one day but not purchase a product until the following day. As long as this purchase falls within 24 hours from the time the customer saw the product recommendations, Bazaarvoice attributes the purchase to those recommendations.
Conversion Impact report (CIR) versus Personalization Data ROI report
While the Conversion Impact report (CIR) and Personalization Data ROI report may seem similar upon first glance, there are some significant differences between the two:
- The Personalization Data ROI report utilizes an A/B test framework with a test and a control group that receive different site experiences. The CIR utilizes a cohort (or segment group) of shoppers in which everyone gets the same site experience, and shoppers are grouped based on their shopping behavior (such as saw content, interacted with content, etc.).
- For the Personalization Data ROI report, the day of attribution is the day of the visitor’s transaction. For the CIR, the day of attribution is the day of the visitor’s page view.
- The Personalization Data ROI report has a 1-day look-back, meaning a transaction that occurs within 24 hours of a visitor viewing a recommendations carousel is attributed to personalized recommendations. The look-back period for the CIR is 3 days.
- The Personalization Data ROI report provides time frames of 7-day full weeks (Monday through Sunday). The CIR provides daily and monthly reports.
Report for all pages
The first report in the dashboard displays aggregate performance metrics for all page types with a personalized recommendations carousel or a personalized Curations display. This section shows the following metrics:
Metric | Definition |
---|---|
Revenue Per Visitor (RPV) | The Revenue collected by a Client divided by the number of Visitors over that same time period. |
Average Order Value (AOV) | The average Order Total for a set of Visitors who made a transaction over a given period of time. |
Conversion Rate | The number of Conversions divided by the number of Visitors during the time period. |
Products Per Order | The average number of products sold per unique Order. |
The Control column displays the value for each metric for all users in your control group (visitors who cannot see a recommendations carousel rendered on the page or visitors who can see a recommendations carousel powered by your third-party recommendations provider).
The Test column displays the value for each metric for all users in your test group (visitors who have the opportunity to see a Bazaarvoice-powered recommendations carousel rendered on the page).
The Difference column shows the difference in each metric between the control and test groups (Test column - Control column).
Lastly, the %Lift column displays the percent change between the test and control groups by calculating the difference between the groups and dividing by the original group. Lift is calculated for Conversion Rate, Average Order Value, Revenue Per Visitor, and other metrics.
Below the dashboard table, you see some other useful metrics:
Total Visitors Who Saw Personalization—Total count of unique Visitors who viewed a personalized slot for the selected date range. This is measured when a shopper scrolls far enough to bring a Bazaarvoice Data-powered recommendations slot into view.
Percent Visitors Who Saw Personalization—Percent of unique Visitors who viewed a personalized slot for the selected date range. This is measured when a shopper scrolls far enough to bring a Bazaarvoice Data-powered recommendations slot into view.
Percent Visitors With A Shopper Profile—Percent of unique Visitors who have a Bazaarvoice shopper profile. This count represents the percentage of your shoppers that can receive Bazaarvoice Data-powered recommendations. Whether a Visitor was a Profiled Shopper or not will only be captured on pages with personalized content. (This metric only applies to Visitors in your test group.)
Percent Engaged Visitors—Percent of unique Visitors who Engaged with a personalized slot (by clicking or scrolling) for the selected date range. This is a leading indicator of the quality of your personalized content. (This metric only applies to Visitors in your test group.)
Hover over the icons in the dashboard for a brief explanation of each of the metrics above.
Weekly detail CSV file for all pages
Click Download CSV for all pages to download a more detailed CSV file with additional reported metrics.
Example scenarios
Allyson
Day 1 (Thursday, March 22, 12:08 PM): An online visitor, Allyson, arrives at your site. Upon entering the site, she is automatically assigned to your control group. She scrolls down the home page to where the personalized recommendations carousel would render for your test group. She then decides to click on a featured product on the home page, SUPER Speed Running Shoes. She looks at the product display page and scrolls down to read the Ratings & Reviews content. She decides to wait before making her purchase because she wants to do more research, so she does not place an order.
Day 2 (Friday, March 23, 10:33 PM): Allyson cannot stop thinking about the shoes she saw on your site the day before, and she decides to place an order for this product. Because Allyson makes a purchase within 24 hours of scrolling to the empty recommendations slot, her order is counted in the Control calculations for the ROI report. She is also counted as one unique visitor, as she visited the site twice within the same 7-day week. The day of her transaction, March 23, is counted as the day of attribution.
Joe
Day 1 (Thursday, March 22, 9:15 AM): A second online visitor, Joe, arrives at your site on Thursday. Upon entering the site, he is automatically assigned to your test group. Although there is a Bazaarvoice-powered recommendations carousel rendered at the bottom of the home page, Joe does not scroll to it. Instead, he goes directly to your site navigation menu to search for leather notebooks. After browsing the category page for Notebooks, he places an order for five college-ruled brown leather notebooks. Joe did not see a Bazaarvoice-powered recommendations carousel, so even though he made his purchase within a 24-hour period, his purchase cannot be attributed to personalized recommendations. Nevertheless, he still counts as a unique test group visitor for that week.
Jason
Day 1 (Saturday, March 31, 6:28 PM): A third online visitor, Jason, arrives at your site the following week on Saturday. Upon entering the site, he is automatically assigned to your test group. Jason regularly browses and purchases products from other sites within the Bazaarvoice network. He scrolls to your Bazaarvoice-powered recommendations carousel and clicks on a pair of multi-colored socks that interest him. He adds this product to his shopping cart, but he decides to come back later to make a purchase.
Day 2 (Tuesday, April 3, 5:31 PM): Jason returns to your site and purchases the socks in his shopping cart. Although he engaged with the personalized recommendations carousel and made a purchase, this transaction cannot be attributed to Personalization Data because it was made outside of the 24-hour attribution window. However, he does count as a unique visitor in both Week 1 (Monday, March 26 to Sunday, April 1) and Week 2 (Monday, April 2 to Sunday, April 8).
Reports for specific page types and Curations displays
The remainder of the reports on the page are segmented by page type as defined by your team during implementation, like “Home page” or “Category page.” If you have Curations displays with personalized content, all information related to your personalized displays will be rolled into its own segment.
Metrics are reported in aggregate at a page type or Curations display level. They are only reported for page types if you tag the page with a specific unique identifier (page type) in your Personalization Data implementation.
In addition to the metrics shown for all pages, a specific page dashboard also includes the following:
Attributable Revenue Impact—Revenue generated by test group visitors who saw Bazaarvoice-powered personalized recommendations for the time period you select using the date range filter. In other words, it is the revenue you can attribute to your Bazaarvoice-powered personalized recommendations carousel for your selected time frame.
The following formula is used to calculate Attributable Revenue Impact:
Multiply the daily estimated attributable revenue by 30 to calculate monthly estimated attributable revenue. Multiply the daily estimated attributable revenue by the number of days in a time frame to calculate the Attributable Revenue Impact for that time frame.
For example, if you have a $1 difference in RPV ($10 in RPV from a profiled test group and $9 from a non-profiled control group) and 250,000 profiled visitors in the past 100 days, then your calculation for daily Attributable Revenue Impact would be:
Yearly Estimated Revenue—Estimation of the revenue you can attribute to your Bazaarvoice-powered personalized recommendations carousel over a year. The following formula is used to calculate Yearly Estimated Revenue:
Weekly detail CSV file for specific page types
For each additional report in the dashboard, you can also click Download CSV for <page type> to download a more detailed file with additional reported metrics.
Definitions
This section provides definitions for the metrics and visitor segments included in the Weekly detail CSV file: