Monitor and Analyze Performance on AWS

Our goal at Catchpoint has always been to simplify things, make it more simple and scalable for our end users. In our constant endeavor to achieve scalability at lower costs, we introduced Webhooks. Webhooks are HTTP callbacks that are triggered by events specified in the code by the application developer; the event data is then recorded and pushed to an end-point URL. Webhooks are a powerful way of exchanging data.

Let’s take an example:

Imagine, you go to your neighborhood grocery store to pick up mangoes, but unfortunately today the shop is out of the mangoes that you want. The delivery boy who is supposed to bring the mangoes to the store, is stuck in traffic. So, the shop owner asks you to come back in an hour. You go back home and return to the shop after an hour, but the delivery boy has not yet arrived with the mangoes. The story repeats and after multiple such attempts you finally get your beloved mangoes. This is exactly how API’s work. You keep making requests, you keep wasting resources, but you won’t get data until it is available.

Now imagine, instead of you going to the shop, you leave your address with the shopkeeper and ask him to deliver mangoes as soon as they arrive, this will help you save your time, effort and resources. This is exactly how Webhooks work.

One of the advantages of Webhooks is, since there’s no request required for a Webhook, it sends the data to the endpoint as soon as it’s available and hence the data is made available in near real time

In this blog, we explore the idea of using Webhooks to analyze web performance data from Catchpoint in your Amazon Elastic Search service.

Step 1: Setting up the AWS instance:

Following are the steps to setup your own AWS instance.

1: Log in to your AWS Account.

2: Go to Services and select Elastic Search Service under the Analytics section.

3: Create a Domain and select the elastic search version and click next.

4: Configure the cluster as per your requirements and click next

5: Setup access policy, make sure you allow catchpoint Webhook domain/ip so that it can post data to elastic and then click next.

6: Confirm your changes and create the AWS ES instance. Wait for the Domain Status to change to Active, that’s when the service is ready to ingest data. Here endpoint URL and Kibana URL are important.

7: Create an index to push data to AWS ES.

For example: The following curl request will create an index called cpindex

curl -X PUT http://search-catchpoint-test-imwp2rphrbhue6k2gizw6c5afm.ap-south.es.amazonaws.com/cpindex  -H 'cache-control: no-cache' -H 'content-length: 225' -H 'content-type: application/json'

8: While sending data don’t forget to add a type to this index.

So, the new post URL will look like, here data is the AWS ES type.:

http://search-catchpoint-test-imwp2rphrbhue6k2gizw6c5afm.ap-south.es.amazonaws.com/cpindex/data

The following lines are taken from Elastic search official documentation:

For example, let’s assume you run a blogging platform and store all your data in a single index. In this index, you may define a type for user data, another type for blog data, and yet another type for comments data

Step 2:  Setting up the Webhook to post data to AWS ES instance endpoint.

1:  Select a test for which you want to enable Webhook. Go to advanced options in the test setup page and enable test data Webhook.

2: In the Webhook Settings, copy the AWS ES endpoint URL and save

Step 3: Verifying the setup

Once the test settings have been saved and a test run executed, you can view the index in AWS ES indices tab.

You should be able to view the results in Kibana.

That’s how you can leverage the power of Webhooks for your Performance analysis.

See how Google uses Catchpoint’s Webhooks to access their performance data in real time.