Why cloud-only synthetic monitoring is problematic

Many of the most sophisticated DevOps teams at companies like Google, Microsoft, and L’Oreal choose Catchpoint Synthetic Monitoring because of its ability to simulate end-user experiences accurately. Over 20 distinct tests, deployed from over 700 nodes across the globe, provide visibility not only into application performance but the entire delivery chain–spanning the world’s most popular public clouds, fiber backbones, ISP’s and mobile networks. This allows DevOps teams to pinpoint performance issues before they impact real users.

On November 1st Dynatrace announced they were sunsetting Synthetic Classic and requiring all Synthetic Classic customers to migrate to their new cloud-based synthetic monitoring locations by October 31, 2019. Dynatrace joins other monitoring providers like NewRelic and AppDynamics in having only synthetic monitoring nodes located inside cloud providers like AWS, Azure, and Linode.

Dynatrace’s sunset of Synthetic Classic and move to cloud-based synthetic monitoring locations will create monitoring blind spots for their customers. Frankly, we’re concerned for a few reasons. First, Dynatrace customers won’t have an accurate view of user experience. Second, moving synthetic monitoring nodes to the cloud saves monitoring providers like Dynatrace money. Are they passing these savings along to customers in the same way Catchpoint does for our cloud nodes? And third, the geographic coverage of cloud nodes leaves broad gaps around large population centers of users in regions like the south and the midwestern United States.

Cloud-only synthetic monitoring creates blind spots

At Catchpoint, we’ve documented why a cloud-only synthetic monitoring approach creates real problems for DevOps teams looking to monitor their digital experiences accurately and pinpoint performance problems before they impact real users. Simply put, cloud-only synthetic monitoring creates more noise and weaker signals.

Earlier this year, we published results from a study demonstrating the wide variance in cloud-only synthetic monitoring versus a more comprehensive approach. We fetched the base HTML of popular US websites from our Catchpoint AWS nodes, as well as Catchpoint backbone nodes running in traditional datacenters like Equinix, and connected to internet backbone providers like NTT, Level3, etc.

In almost every test, we saw the performance of tests which ran on the AWS nodes were faster than tests run on Backbone Nodes. The table below documents that in most cases, a website was faster by a significant percentage. The net effect is that this is not an accurate representation of end-user experience.

In more than 70% of test instances, we noticed tests that ran on AWS nodes were faster than the tests that ran on backbone nodes. This is because:

• Monitoring an application hosted on AWS from AWS is not entirely realistic because there is usually some form of dedicated network connection between data centers.
• Because of these dedicated connections, the network operators can tune their routing policy to more effectively send and receive traffic through adjacent autonomous systems. This practice is known as BGP traffic engineering.
• When the routing must happen within the same entity (AWS), it is in most cases, completely under their control.

Not everything should be monitored from the cloud

Monitoring from the cloud does not monitor performance along the entire chain of delivery for users and employees. For example, the following are not suitable for cloud-based monitoring:

• SLA measurements for services delivered to end users or within the same cloud
• SLA measurements for third-party providers or suppliers that are in the delivery chain (DNS, CDN, cloud providers, Adserving, etc.)
• Supplier/provider performance testing and validation for services like DNS, CDN, cloud, SaaS
• Benchmarking of consumer service delivery for your competitors or industry
• Network/ISP connectivity monitoring and alerting
• DNS availability, performance or validation of service based on geolocations

If you do want to monitor from the cloud, Catchpoint has the largest network of nodes

When it is appropriate to monitor from the cloud, we have you covered. We provide 111 Cloud Nodes deployed on the six major cloud providers: AWS, Azure, Google, IBM, Alibaba, and Tencent. Our Cloud Nodes cover 62 cities in 25 countries worldwide, the most extensive cloud provider coverage available. As part of Catchpoint’s holistic monitoring approach, there are more than 700 vantage points on internet backbone, broadband/ISP, last mile, and wireless infrastructure. It’s the largest and most diverse monitoring network in its category. And finally, if you are running a private or hybrid cloud, you can deploy Catchpoint Enterprise Nodes inside your private clouds to get full synthetic coverage across your entire service delivery chain.

A Final Question: Is it sufficient to use real user monitoring to augment cloud-only synthetic monitoring?

Real user monitoring (RUM) is an integral part of a comprehensive monitoring strategy. Correlating metrics from real users with synthetic monitoring rounds out the performance view of your services, giving you a combined set of data from both passive and active sources. Together, they can help your DevOps teams to easily identify performance problems with the highest impact on end users—and focus on what matters most. At Catchpoint, our Digital Experience Monitoring platform helps DevOps teams:

• Utilize AI-driven dashboards: Intuitive displays show relevant information on a single screen, helping you find answers fast.
• Break data down by highest impact: Maximize ROI by viewing page and data attributes that contribute to changes.
• Empower citizen data science: Aggregate disparate data into intelligent and actionable datasets that drive faster, data-driven decision making.

However, relying on Real User Monitoring, in place of a holistic monitoring strategy with an array of Synthetic Monitoring nodes, can delay identification of performance and availability issues. Real users are impacted. You end up with longer Mean Time to Repair (MTTR) and even more end users impacted negatively. Below is a real customer example of an API checkout failure that Catchpoint Synthetic Monitoring picked up well in advance of real user monitoring.

SafeSwitch: Avoid the blind spots of cloud-only synthetic monitoring

Migrate to Catchpoint with 100% Confidence and 0% Disruption

Dynatrace is requiring every Synthetic Classic customer to embark on a forced migration to a new platform in the next 12 months. This migration will result in Dynatrace customers being forced to adopt a cloud-only synthetic monitoring tool that creates more monitoring blind spots than their current Synthetic Classic solution.

There’s a better way for those seeking a holistic approach to Digital Experience Monitoring that creates more signal and less noise: Catchpoint SafeSwitch. The SafeSwitch program is designed to create a seamless, hassle-free experience to migrate from your existing monitoring solution to Catchpoint. Our focus is to help you achieve your monitoring goals, mitigate the potential risks involved with migration, with minimal effort from your organization.

During the SafeSwitch period, your existing test configurations will be replicated in the Catchpoint environment and will run concurrently with your legacy system. Training and review sessions will be scheduled as needed to ensure your organization is prepared to make the switch.

In most instances, your legacy system and the Catchpoint solution will run concurrently for at least 30 days. This time allows your organization to receive training, gain confidence in navigating the solution, and most importantly, see for yourself how Catchpoint’s Digital Experience Monitoring Platform produces more signal with less noise.

Conclusion

Accurately simulating the end user experience requires you to have nodes on local ISPs, mobile networks, backbones, and clouds that end users’ access when visiting a site or using an app. Our comprehensive variety of tests and node types eliminate blind spots and deliver the insights you need to understand the end-users experience and to identify and resolve issues quickly. The ability to monitor from more than 700 backbone, broadband, last mile, wireless, cloud, and corporate locations can help companies detect suboptimal experiences from anywhere in real time. This holistic monitoring strategy is necessary to ensure the speed, reachability, reliability, and availability of all your digital services.