The Personalization Data ROI report dashboard allows you to understand the impact of your Personalization Data implementation on ROI over time. You can also download a weekly detailed CSV file to serve as input for your own ROI analysis.

Tip: Bazaarvoice offers both a Bazaarvoice-hosted recommendations carousel and a Personalization Data SDK to pass recommendations data to your existing recommendations display element.

Why is it important?

Personalization is an established best practice in eCommerce. By presenting shoppers with an experience tailored to their interests, you can drive conversion and customer loyalty. Shoppers that view personalized product recommendations on a retail site are not only more likely to purchase a product, but they are also more likely to return to the site in the future.

Good personalization requires good, comprehensive data, and the customer journey seldom starts—or ends—at your website. BV Pixel captures product purchases as well as non-transactional events within our network of over 5,700 brand and retail sites. We track each customer’s journey within our network, on and off your site, and build individual shopper profiles in real time. We use these shopper profiles to recommend products or categories for each profiled visitor to your site.

A/B testing

The key to understanding the value of your Personalization Data implementation is to set up an A/B test. Each visitor to your site is randomly assigned to either a control or test experience upon entering the site. Once a visitor has been assigned to a control or test group, the visitor stays in that group for all future site visits throughout the duration of your experiment. Using this experimental framework, you can compare the groups’ behaviors to one another to accurately determine the impact of your Personalization Data implementation on revenue and transactions, among other metrics.

Control and test groups

The following describes the site experiences of your control and test groups, depending on your recommendations display solution:

Control group Test group
If you do not currently use third-party recommendations: Will not be shown any recommendations consistently across the entire site

Will have the opportunity to see a recommendations carousel if they scroll to the portion of the page where it is rendered

Visitors with a Bazaarvoice shopper profile will be shown recommendations based on that profile; visitors without a Bazaarvoice shopper profile will be shown Bazaarvoice trending recommendations

If you are already using third-party recommendations: Will have the opportunity to see a recommendations carousel that is fully populated per your existing third-party recommendations provider if they scroll to the portion of the page where it is rendered

Will have the opportunity to see a recommendations carousel if they scroll to the portion of the page where it is rendered

Visitors with a Bazaarvoice shopper profile will be shown recommendations based on that profile; visitors without a Bazaarvoice shopper profile will be shown your current provider's recommendations

Note: Because the recommendations carousel may be rendered further down the page, it is important to remember that the Personalization Data ROI report only considers the visitors who scroll down the page and see the personalized recommendations slot, whether it is populated with personalized recommendations (test group), left empty (control group without a third-party recommendations provider), or populated with your third-party provider's recommendations (control group with a third-party recommendations provider).

Sample size and duration

To ensure confidence in the data, the sample size of both traffic and conversions needs to be statistically significant. While you are planning your A/B test, you should determine the sample size you need to reach statistical significance and estimate how many weeks you should be running your test based on that. See Optimizely Sample Size Calculator or VWO A/B Test Duration for help estimating these values.

The actual duration of your A/B test depends on a number of factors. Review these suggested stopping rules to help you determine a test duration. A/B tests should be run across 7-day weeks (Monday through Sunday) and not partial weeks. We will not show data from partial weeks in the dashboard.

Note: The Personalization Data ROI report uses a 95% confidence interval.

Traffic distribution

It is important to set traffic distribution (for example, a 50/50 split between control and test groups) before starting an A/B test. Do not change this distribution while the test is running. In the event that you change the traffic distribution during your test, only new visitors are split according to the new traffic distribution, which can alter the results of your test. This is called Simpson’s Paradox .

You may want to start off with a traffic distribution of 90% of visitors in the control group and 10% of visitors in the test group to make sure everything is working properly. When you feel confident, you can stop that test and move towards a 50/50 split of your traffic. Always keep track of when you make test split changes.

Seasonality and external influencers

You should run A/B tests during a “normal” time of year—that is, when no major increases or decreases in conversion rate are expected due to seasonality, such as during a peak holiday shopping season.

You should also be mindful of other marketing activities that are occurring at the same time as the test. We recommend, when possible, that you pause these other activities before the test starts and while the test is underway.

Tip: Contact your Bazaarvoice Implementation Engineer for help setting up your Personalization Data A/B test framework.

Personalization Data ROI report dashboard

Note: If you have not properly implemented the BV Pixel, the Personalization Data ROI report does not display report data.

Access the dashboard

To access this report dashboard:

  1. Log in to My Bazaarvoice .
  2. Select Menu » Personalization.

Filter

Using the date range filter at the top-right of the dashboard, you can see metrics from different time frames of your A/B test. You can choose from the following date ranges:

  • Last week
  • Last 2 weeks
  • Last 3 weeks
  • Last 4 weeks
  • Last 12 weeks
  • Last 52 weeks

You can see the date range currently selected in the banner at the top of the dashboard.

Note: You can only see date range options for which you have available reports. When Bazaarvoice has enough to data to compile a report for the next date range option, you can select that date range to view its report.

Unique Visitors

Unique visitors are detected across tracked pages during the specified date range. Tracked pages include any page that loads Bazaarvoice analytics tagging, whether through the Bazaarvoice-hosted display or through BV Pixel. The Personalization Data ROI report deduplicates unique visitors weekly based on a full week from Monday at 12:00am to Sunday at 11:59pm.

For example, an online visitor, Drew, arrives at your site on during his lunch hour on Tuesday, August 28 at 12:13pm. Upon entering the site, he is automatically assigned to your test group. He has browsed your site and some other sites within the Bazaarvoice network before, and his shopper profile indicates that he is interested in high-top sneakers. He scrolls down your home page, where you have implemented a Bazaarvoice-powered recommendations carousel. Because he is in your test group and has a Bazaarvoice shopper profile, the carousel he sees is populated using Bazaarvoice Personalization Data. Drew clicks on a pair of sneakers that catches his eye. However, he decides not to buy anything from your site. He returns to your site again on Friday, August 31 to continue browsing. He does not return again to the site until the following week on Monday, September 3 at 12:47pm. Although he visited the site two times in Week 1 (Monday, August 27 to Sunday, September 2), he is only counted as one unique visitor in your Personalization Data ROI report calculations. He is counted again as a unique visitor in Week 2 (Monday, September 3 to Sunday, September 9), and he is counted as a single unique visitor each subsequent week he visits your site, no matter how many times he visits per week.

Attribution window

When a visitor scrolls to the part of the page where the recommendations carousel is or would be rendered or a visitor interacts with the recommendations carousel, the Personalization Data ROI report associates conversions that occur within 24 hours of meeting that criteria. These 24 hours are known as the attribution window.

For example, a test group customer might look at product recommendations one day but not purchase a product until the following day. As long as this purchase falls within 24 hours from the time the customer saw the product recommendations, Bazaarvoice attributes the purchase to those recommendations.

Conversion Impact report (CIR) versus Personalization Data ROI report

While the Conversion Impact report (CIR) and Personalization Data ROI report may seem similar upon first glance, there are some significant differences between the two:

  • The Personalization Data ROI report utilizes an A/B test framework with a test and a control group that receive different site experiences. The CIR utilizes a cohort (or segment group) of shoppers in which everyone gets the same site experience, and shoppers are grouped based on their shopping behavior (such as saw content, interacted with content, etc.).
  • For the Personalization Data ROI report, the day of attribution is the day of the visitor’s transaction. For the CIR, the day of attribution is the day of the visitor’s page view.
  • The Personalization Data ROI report has a 1-day look-back, meaning a transaction that occurs within 24 hours of a visitor viewing a recommendations carousel is attributed to personalized recommendations. The look-back period for the CIR is 3 days.
  • The Personalization Data ROI report provides time frames of 7-day full weeks (Monday through Sunday). The CIR provides daily and monthly reports.

Report for all pages

The first report in the dashboard displays aggregate performance metrics for all page types with a personalized recommendations carousel or a personalized Curations display. This section shows the following metrics:

Metric Definition
Revenue Per Visitor (RPV) The Revenue collected by a Client divided by the number of Visitors over that same time period.
Average Order Value (AOV) The average Order Total for a set of Visitors who made a transaction over a given period of time.
Conversion Rate The number of Conversions divided by the number of Visitors during the time period.
Products Per Order The average number of products sold per unique Order.

The Control column displays the value for each metric for all users in your control group (visitors who cannot see a recommendations carousel rendered on the page or visitors who can see a recommendations carousel powered by your third-party recommendations provider).

The Test column displays the value for each metric for all users in your test group (visitors who have the opportunity to see a Bazaarvoice-powered recommendations carousel rendered on the page).

The Difference column shows the difference in each metric between the control and test groups (Test column - Control column).

Lastly, the %Lift column displays the percent change between the test and control groups by calculating the difference between the groups and dividing by the original group. Lift is calculated for Conversion Rate, Average Order Value, Revenue Per Visitor, and other metrics.

Note: Beside some %Lift values, you may see a icon that signals that the report does not have enough data yet to be statistically significant.

Below the dashboard table, you see some other useful metrics:

Total Visitors Who Saw Personalization—Total count of unique Visitors who viewed a personalized slot for the selected date range. This is measured when a shopper scrolls far enough to bring a Bazaarvoice Data-powered recommendations slot into view.

Percent Visitors Who Saw Personalization—Percent of unique Visitors who viewed a personalized slot for the selected date range. This is measured when a shopper scrolls far enough to bring a Bazaarvoice Data-powered recommendations slot into view.

Percent Visitors With A Shopper Profile—Percent of unique Visitors who have a Bazaarvoice shopper profile. This count represents the percentage of your shoppers that can receive Bazaarvoice Data-powered recommendations. Whether a Visitor was a Profiled Shopper or not will only be captured on pages with personalized content. (This metric only applies to Visitors in your test group.)

Percent Engaged Visitors—Percent of unique Visitors who Engaged with a personalized slot (by clicking or scrolling) for the selected date range. This is a leading indicator of the quality of your personalized content. (This metric only applies to Visitors in your test group.)

Note: The counts above are all deduplicated weekly. See the Unique Visitors section for more information on visitor deduplication.

Hover over the icons in the dashboard for a brief explanation of each of the metrics above.

Weekly detail CSV file for all pages

Click Download CSV for all pages to download a more detailed CSV file with additional reported metrics.

Example scenarios

Allyson

Day 1 (Thursday, March 22, 12:08 PM): An online visitor, Allyson, arrives at your site. Upon entering the site, she is automatically assigned to your control group. She scrolls down the home page to where the personalized recommendations carousel would render for your test group. She then decides to click on a featured product on the home page, SUPER Speed Running Shoes. She looks at the product display page and scrolls down to read the Ratings & Reviews content. She decides to wait before making her purchase because she wants to do more research, so she does not place an order.

Day 2 (Friday, March 23, 10:33 PM): Allyson cannot stop thinking about the shoes she saw on your site the day before, and she decides to place an order for this product. Because Allyson makes a purchase within 24 hours of scrolling to the empty recommendations slot, her order is counted in the Control calculations for the ROI report. She is also counted as one unique visitor, as she visited the site twice within the same 7-day week. The day of her transaction, March 23, is counted as the day of attribution.

Joe

Day 1 (Thursday, March 22, 9:15 AM): A second online visitor, Joe, arrives at your site on Thursday. Upon entering the site, he is automatically assigned to your test group. Although there is a Bazaarvoice-powered recommendations carousel rendered at the bottom of the home page, Joe does not scroll to it. Instead, he goes directly to your site navigation menu to search for leather notebooks. After browsing the category page for Notebooks, he places an order for five college-ruled brown leather notebooks. Joe did not see a Bazaarvoice-powered recommendations carousel, so even though he made his purchase within a 24-hour period, his purchase cannot be attributed to personalized recommendations. Nevertheless, he still counts as a unique test group visitor for that week.

Jason

Day 1 (Saturday, March 31, 6:28 PM): A third online visitor, Jason, arrives at your site the following week on Saturday. Upon entering the site, he is automatically assigned to your test group. Jason regularly browses and purchases products from other sites within the Bazaarvoice network. He scrolls to your Bazaarvoice-powered recommendations carousel and clicks on a pair of multi-colored socks that interest him. He adds this product to his shopping cart, but he decides to come back later to make a purchase.

Day 2 (Tuesday, April 3, 5:31 PM): Jason returns to your site and purchases the socks in his shopping cart. Although he engaged with the personalized recommendations carousel and made a purchase, this transaction cannot be attributed to Personalization Data because it was made outside of the 24-hour attribution window. However, he does count as a unique visitor in both Week 1 (Monday, March 26 to Sunday, April 1) and Week 2 (Monday, April 2 to Sunday, April 8).

Reports for specific page types and Curations displays

The remainder of the reports on the page are segmented by page type as defined by your team during implementation, like “Home page” or “Category page.” If you have Curations displays with personalized content, all information related to your personalized displays will be rolled into its own segment.

Metrics are reported in aggregate at a page type or Curations display level. They are only reported for page types if you tag the page with a specific unique identifier (page type) in your Personalization Data implementation.

Tip: In your Bazaarvoice-hosted recommendations carousel implementation code or in your Personalization SDK JavaScript, the free-form value you define as the page type on which you want to render a recommendations module is displayed on the ROI report section for that page. Make sure to name your page types something that any business owner, analyst, or stakeholder looking at the report can easily understand. For example, if you name your page type "12e7hq3", it would be much harder to know which page type's report you are looking at than if it were named something more descriptive, like "Product display page."

In addition to the metrics shown for all pages, a specific page dashboard also includes the following:

Attributable Revenue Impact—Revenue generated by test group visitors who saw Bazaarvoice-powered personalized recommendations for the time period you select using the date range filter. In other words, it is the revenue you can attribute to your Bazaarvoice-powered personalized recommendations carousel for your selected time frame.

The following formula is used to calculate Attributable Revenue Impact:

Multiply the daily estimated attributable revenue by 30 to calculate monthly estimated attributable revenue. Multiply the daily estimated attributable revenue by the number of days in a time frame to calculate the Attributable Revenue Impact for that time frame.

For example, if you have a $1 difference in RPV ($10 in RPV from a profiled test group and $9 from a non-profiled control group) and 250,000 profiled visitors in the past 100 days, then your calculation for daily Attributable Revenue Impact would be:

Yearly Estimated Revenue—Estimation of the revenue you can attribute to your Bazaarvoice-powered personalized recommendations carousel over a year. The following formula is used to calculate Yearly Estimated Revenue:

Caution: The Yearly Estimated Revenue metric could easily change due to the time frame you are viewing, seasonality, and other external factors influencing your A/B test. This is a projection based on observable data from your A/B test and not an exact calculation of the yearly estimated revenue you should expect from implementing Personalization Data.

Weekly detail CSV file for specific page types

For each additional report in the dashboard, you can also click Download CSV for <page type> to download a more detailed file with additional reported metrics.

Definitions

This section provides definitions for the metrics and visitor segments included in the Weekly detail CSV file:

Metric Definition
Attributable Revenue Impact Based on an A/B test, the difference in revenue between the control A group vs B test group, as this is the test group's impact on revenue. This is calculated by taking the difference in Revenue Per Visitor from the control vs test group and multiplying it by the number of Visitors in the test group. Because A/B tests may often not be live on a monthly or annual basis, this metric may also be divided by the number of days the A/B test was running and multiplied by 30 to get monthly or 365 to get annual Attributable Revenue Impact.
Average Order Value (AOV) The average Order Total for a set of Visitors who made a transaction over a given period of time.
Conversion A transactional commerce conversion that involves placing an order, and is usually accompanied by an exchange of currency between two parties.
Conversions The total number of times a Visitor had a Conversion. This may be across a client site or across the BV network.
Conversion Rate The number of Conversions divided by the number of Visitors during the time period.
Demand Sales The demand in sales which is all Orders placed on the Client's site. This does not account for product returns, rejected payments, out of stock products, etc.
Engaged An interaction type identifying that an Exposed Visitor actively interacted with the target content or product offering. This includes clicking, sorting, scrolling, etc.
Lift Percent change between 2 groups by calculating the difference between the groups and dividing by the original group. Groups may be: those who did not use Bazaarvoice and those who did use Bazaarvoice; Control (A) group and Test (B) group; Before and After Syndication, etc. Lift is calculated for Conversion Rate, Average Order Value, Revenue Per Visitor, and other metrics.
Order An online sale as defined by each transaction conversion event collected by the BV Pixel (with a unique order ID or loadid).
Percent of Visitors who are Shopper Profiled The percentage of Visitors (to select pages on a client's site with personalized content) that are Profiled Shoppers. Whether a Visitor was Profiled or not will only be captured on pages with personalized content.
Profiled Shopper A subset of EDR shoppers that have signals. For advertising and/or personalization purposes, these shoppers have been mapped to some targetable interest(s).
Recommendations Engagement Rate (of those who Saw) The percentage of Visitors who Engaged with recommendations when they saw recommendations.
Recommendations Engagement Rate (overall) The percentage of Visitors who Engaged with recommendations when they were on a page where recommendations was rendered.
Recommended Products Bought The total number of products that were recommended, viewed by a Visitor, and purchased inside the attribution window. (Note: This metric is only available for product recommendations, not category recommendations).
Revenue A transactional commerce conversion that involves placing an order, and is usually accompanied by an exchange of currency between two parties.
Revenue Per Visitor (RPV) The Revenue collected by a Client divided by the number of Visitors over that same time period.
Unique Products per Order The average number of Products Sold per unique Order.
Unique Products Sold The number of unique products sold in an Order. If OrderXYZ had ProductA with quantity=1 and ProductB with quantity=3, then Products Sold=2.
Visitor A Shopper that visits a site during a period of time.
Visitors All Shoppers that visit a site during a period of time.