Message Tracing with WSO2 API Manager

The latest version of WSO2 API Manager can be downloaded from here. We could deploy API Manager in a distributed manner by separating Gateway, KeyManager, Publisher and Store servers. All the API requests are first comes to the gateway server. Then gateway calls to the key-validation server to validate the key. Only if key validation is successful, gateway allows for the back-end call.

It's important to have message details logging capability in gateway and key-manager servers, so that we can track the request and response key validation details for all requests. And it should have a same activity_id for a particular message in the key-manager and gateway servers. So that we can co-relate the same message at the key-manager and gateway nodes when doing the analysis.

Message tracing feature has been included in the API-Manager-1.6.0 release. How this message tracing feature track the same message in different servers is, when the request comes into a particular server, it adds a http header called "activityID" if this header is already not there in the request. Same is doing for the outgoing response message as well.  It checks the incoming message context and if the "activityID" header is available, then same activityID is added to the outgoing message as well. So that the messages can be co-relate using this activityID header. This feature can be used in any product to track the message. As an example say message flow is ESB -> Application-Server -> Data-Services-Server, then we can install this feature into each server and track the message.

Coming back to the API-Manager scenario, with this message tracing feature, you could log the message info in gateway and key-manager servers and/or publish the message tracing info into Business Activity Monitoring server (BAM) as streams. Once you published message info into BAM server, those streams are saved in to Cassandra keyspace and we could use them for analysis.

How to enable message tracing to log details in gateway and key-manager servers ?

1. Login to the Management Console of Gateway server. ( eg: https://localhost:9443/carbon )
Go to "Configure -> Message Tracing". This will give you the configuration UI as shown in bellow and Select the "Enable Message Tracing" option.

2. Now go to the "Configure -> Logging". In the prompt UI, give the "APIManagerExtensionHandler" as input value to "Filter Loggers by" field and click on "Contains" button. This will give you the following filter result and change the "org.wso2.carbon.apimgt.gateway.handlers.ext.APIManagerExtensionHandler" log level to "DEBUG" as shown.

3. Same way, go to the Management console of Key-Manager server and enable the "Massage Tracing" as we did in the step-1. Now go to the "Configure -> Logging" and set the log level to "DEBUG" for class "org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService" as we did in step-2.

Now, in each request call, we can see the message tracing logs in gateway and key-manager servers as shown bellow.

Gateway Server Logs:

Key-Manager Server Logs:

Note that same transaction_id has been printed in both gateway and key-manager servers for a given request.

We are done with logging. Further, if you need we could publish these tracing info in to BAM server as well.

How to configure message tracing to publish info into Business Activity Monitoring Server?

We need to have a remote BAM server running. You can download the latest version of BAM sever from here. Login to the management console of Gateway and Key-Manager severs and go to "Configure -> Message Tracing" and enable "Enable BAM Event Publishing" option and provide the BAM server details as shown bellow. (The following configuration is provided when BAM server is running with port offset 3).

Now, there will be streams publish by the gateway and key-manager servers to BAM server in each request. You can connect to the Cassandra server and check the published message info. Login to the management console of BAM server and go to "Tools > Cassandra Explorer > Connect to Cluster".

 Once you connect to the Cassandra server as mentioned above, you can see that there is a column family called "BAM_MESSAGE_TRACE" has been created. This column family contains all the message tracing data.

You can click on the "BAM_MESSAGE_TRACE" and explore the column family data.

Also, we can use "Activity Dashboard" comes with BAM server to analyse the data. Using this dashboard, we can filter the messages using certain criteria and it will group the messages that having same activity_id. This document describes how we can use the "Activity Dashboard".