Synthetic Data vs Real User Data: What Is Each Best For and How Do They Work Together?

We've had a lot of followers stop by and ask us for help understanding the difference between synthetic user data, and real user data. While we don't have all the answer and specific use cases will depend a bit on your product, I'll break it down into 3 parts to get you started and if you've got more questions after, you know how to find me!

When Is Synthetic Data Useful?

Synthetics are basically automated test scripts that run through your application and perform various test scenarios to determine if you site is working as expected. Because they are pre-recorded bots, they make a few assumptions that would be hard to make when looking at real user data. These assumptions allow you to focus on specific KPIs by removing some white noise. synthetic scenarios are often considered clean room or happy path tests.

Assumptions Made

1. Your buttons, product info, and overall funnel makes sense - If any part of your website would be confusing to a guest, your synthetic flow would not detect that because you recorded it t​o complete tasks. It'll go through each page, wait for it to load, and continue where real users might read something that gives them pause or sparks a question the page may not answer.
2. Your user wants to make a purchase - Synthetic users start browsing your site "knowing" they want to make a purchase so, they usually pick an item, choose a size or color, add the item tho their cart, then buy it. Retention, conversion, and general "shopping" around are not taken into account as a result.
3. Destructive behaviors don't exist - Scenarios like filtering product results until you get "no products found" are sometimes called destructive behaviors and they happen more often than you think. If a user filters for shoes that come in either red or green, then selects a pair of shoes only sold in white, the search results might come back empty. This causes user confusion and flow abandonment but, it's not detected by a synthetic.

Synthetic users allow you to understand the user experience from a technical side. If my flow makes sense and my content is on point, users will be happy with my site because the page load times are good, content is rich but well-compressed, and key flows have been up and operational all day. Because of that, frequent synthetic runs are a good way to see if key parts of your site are operational without having to wait for real users to try using them.

When is Real User Data Useful?

It's hard to not see the value of real user data upfront and with synthetics explained earlier in this write-up, this should be pretty straight-forward but, let's dive in anyways! Real users are best for understanding patterns and trends but, they can also be used for some real deep-dives into edge cases and even for 1 on 1 customer support.

Example Use Cases

1. User trends and patterns - Let's say your CDN is experiencing issues in the state of Texas but, you haven't been notified of that yet by your provider and hey, they might not even know. With the right tool in place, a spike in load times from users in Texas or a drop in conversions would quickly throw up a red flag and using real user data and comparing it to historical trends, you've just identified a problem.
2. Deep-dive analysis - For some reason, people using Google Chrome are 4 times more likely to buy your product than those using Safari...why? Using real user data, you can dive deep and after some hard work, you'll discover that a button that doesn't render correctly on Safari is preventing users from clicking through to the payment screen.
3. 1 on 1 customer support - users call in complaining about your site every now and then. Some might just be looking for a discount but, others are genuinely confused or frustrated. If that user can share their email address with you or maybe a customer ID, you can look for their session and walk through it with them, not only identifying the problem but helping them get what they need.

How Can Synthetics and Real Users Work Together?

Real user data is collected as application users interact with your website and takes into account every click, mouse-over, aimless back and forth...well, it takes everything into account. As a result, it's a lot to sift through or reliably digest without a good baseline. Synthetics establish that baseline for you by measuring the "happy path" for that same scenario. If a synthetic user gets through the flow without issue, the problem is not in the application itself but, maybe in the experience. Is the user confused? Is the right information being given? Is the price too high?