Basic VS Complete Synthetic Monitoring Solution

Last week, I received a phone call from the Head of Observability at a large global enterprise firm. He has been in the monitoring space for almost three decades working with various organizations and has experience with most monitoring tools. The first thing he asked was, “I have used several synthetic monitoring solutions right from basic HTTP to comprehensive user journey capable tools. I would like to know how Catchpoint is different from the rest.”

He went on to cite a few names he was familiar with including SiteScope, Gomez, and Keynote, and clarified that he has been out of touch with the monitoring space over the last few years and what he said aligned with the vendors he named. Some of them don’t exist anymore or have been diluted and/or rebranded.

A lot has changed in the monitoring space with many new vendors and a range of features to choose from. Although all vendors promise a solution that addresses every type of monitoring concerns, not every synthetic monitoring tool is built equally. In this blog, we try to answer how vendors, including Catchpoint, differ and focus on what makes a synthetic monitoring solution complete.

What is Synthetic Monitoring?

Synthetic Monitoring is a methodology that has been around for close to three decades and can be considered a derivative of “mystery shoppers”. A mystery shopper is a paid consumer hired to shop in specific stores to collect data. Think of mystery shoppers as “undercover customers” sent in to observe, interact, and report on other customers and the store employees.

Mystery shoppers are typically employed by a third-party rather than the store or company itself. Shoppers are sent to a wide variety of service-based businesses – potentially anywhere customers are served or serviced. Mystery shoppers are not hired to give opinions but are hired to assess a business and its employees from the customer’s perspective.

The methodology found its way in the digital world with companies like Mercury, Silk, Keynote, Gomez, Webmetrics, Alertsite, etc. The primary focus was on testing webpages which then evolved into scripted web transactions or user journeys. Before we dive into the details of synthetic monitoring, let’s discuss the key tenets of synthetic monitoring:

1. Simulate what? User interaction with the webpage.
2. Simulate from where? From where the users are, a mystery shopper cannot evaluate a Starbucks outlet from his kitchen.
3. What metrics to capture and what analytical capabilities are required? Actionable metrics and signals that can help you diagnose a problem, answer questions and verify assumptions.
4. What is the end-goal? How is the data used and for what purpose? To drive sales, improve conversions, or scout new markets.
5. Who should be using it? How is the captured data consumed by different departments and how does it help your organization?

In an ideal scenario one would expect a synthetic monitoring solution to do all of the above and more:

1. Simulate what – Protocols, applications, services, user interactions, user journey, network, etc.
2. Simulate from everywhere – Every last-mile ISP, every backbone ISP, every data center, every cloud provider, every edge provider.
3. Metrics and analytics – Capture every single metric that provides insight into protocols, services, application, end-user experience, etc. Provide different analytical options to evaluate the data.
4. Goal – Help detect problems in every layer, application, 3rd party, provider, vendor, user experience, business impact, etc.
5. Users – Enable business, marketing, product management, developers, devOps, operations, network, etc.

But,

1. Not all Synthetic vendors focus on building a complete solution.
2. And not all enterprises utilize Synthetics to its fullest potential.

A complete Synthetic Monitoring Solution

To understand what defines a complete synthetic monitoring solution, we need to first understand some of the popular technology changes we have seen over the last decade.

Fig 1: Evolution of Monitoring Technology

All the advancement in technology has made it necessary to implement a comprehensive synthetic monitoring solution. The matrix below (Fig 2) covers all the pillars of synthetic monitoring and the key features that should be provided by these solutions.

Fig 2: Synthetic Monitoring Pillars and Key Features (click to enlarge)

If this is what it takes for a complete synthetic monitoring solution, how many vendors in the current market space actually offer this today? Not many to be honest!

Most synthetic monitoring vendors are compromising the depth and breadth of the solution they offer and this has diluted its capability to focus on specific use cases or functions. The matrix below (Fig 3) compares the features included/excluded by different vendors.

Fig 3: Monitoring Vendor-Feature Comparison (click to enlarge)

This trend of synthetic vendors focusing on specific functions or departments originates from the different monitoring mindsets that exist within organizations.

While there is nothing wrong with the business model of the siloed specialized synthetic monitoring solutions, it does create a lot of gaps in the monitoring maturity of an organization.

Is the monitoring vendors’ business model right for the enterprise that purchases it?

When Gomez and Keynote Synthetics were merged to form today’s Dynatrace Synthetics a lot of the features it offered were diluted to align with the developer-first mindset of Dynatrace. I have discussed how this approach has created a lot of monitoring blindspots in this blog. Most enterprises typically don’t prefer using multiple synthetic monitoring solutions across different teams.

Some common challenges faced by enterprises that use such a diluted synthetic monitoring solution are:

• End users (employees or customers) complain of poor application load time or poor user experience, but the root cause is unknown.
• Marketing or business teams are concerned about a drop in revenue or missed opportunities, but the monitoring solution is incapable of signaling or validating such concerns.
• Very high Mean Time To Innocence (MTTI) that leads to finger-pointing between teams within an organization.
• Developers are tasked with optimizing end-user experience but don’t have the right data or metrics to measure the actual end-user experience.
• Significant investments made in CDN services, but the ROI is mediocre. Read our blog on CDN performance to understand why this happens.
• Many micro-outages or frequent performance degradations tend to go unnoticed.
• Unable to hold vendors such as cloud/SaaS providers accountable for performance issues. The AWS incident from last year is a classic example.

Synthetic Monitoring for an Enterprise

When an enterprise is evaluating synthetic monitoring tools, they should ask themselves two important questions:

1. What type of synthetic monitoring solution do we need?
2. What is the purpose of using such a tool?
3. Is it for developers to test and monitor releases and follow best practices?
4. Is it for developers to truly understand the end-user experience?
5. Is it for Ops to measure and monitor SLA of all their infrastructure components?
6. Is it for the network team to measure and monitor network and services and ensure every user is able to reach their services?
7. Is it for business and marketing to understand the business impact of application performance and to help them plan better for market or business expansion?
8. Is it for centralized teams such as application support or performance engineers or SRE?
9. Is it to have a monitor of monitors as described by Adrian Cockcroft.
10. Is it all of the above?

These questions are crucial when determining the right tool for your organization.

Every organization is continuously focusing on improving their goals and practices. Usually, most organizations are in one of these three monitoring maturity phases (Fig 4).

Fig 4: Monitoring Maturity Phases (click to enlarge)

Choosing the Right Synthetic Monitoring Solution

The monitoring tools used are integral to the functioning of the five core components of an organization – application, technology, people, process, and tools.

If you are using a Synthetic Monitoring solution or plan to buy one, then take a step back and ask yourself – Why do you need one?

• To enable developers to detect problems even before it is rolled out to production – shift left?
• To continuously measure end-user experience in production and identify areas that can be optimized by developers?
• To monitor every component of your Infrastructure and quickly detect problems?
• To monitor and measure the SLA of your vendors such as DNS, CDN, Cloud, SAAS, PAAS, etc. to hold them accountable when needed?
• To enable the network team to quickly detect network or peering issues so they can proactively mitigate any issues or optimize the network path between the end users and their services?
• To enable SREs or monitoring teams to proactively identify bottlenecks and weak points in the application or the delivery chain?
• Finally, to enable better collaboration between business and technology groups by being a data-driven organization?

If the answer to all the above questions is a resounding YES, then you need a “comprehensive” synthetic monitoring solution. The decision is yours!