Integrating WSO2 API Manager with BAM and CEP - (Real time data analyzing for WSO2 API Manager)

WSO2 API Manager provides complete solution for creating, publishing and managing all aspects of an API and its life cycle. One of the most common use case in API management is to statistic analyzing for APIs, like number of requests for a particular API within a given time period, number of subscribers for APIs, number of faulty requests happened for an API etc. At the same time, it's important to have real time data analyzing capability as well, like we may need to get a notification message if particular user exceed the throttling limit of an API etc.

WSO2 Complex Event Processing Server (WSO2CEP) is used to analyze the events and acts them on the real time. While WSO2 Business Activity Monitor (WSO2BAM server) is used to post process the events and store and analyze the data.

In this post, we are going to write a sample that will show how we can integrate the WSO2CEP with API Manager to do the real time event processing and same time we are going to integrate WSO2BAM as well into the same API Manager instance for statistic analysis.

Sample Scenario

API Manager will publish the events to CEP. CEP is doing two things.
1. Forward data to BAM to use for statistic calculations.
2. If same API invokes more than 5 times within 1 minute from the same application, this will send an email notification to a given email address.

Configure the Setup

First download the wso2am, wso2cep and wso2bam servers and extract them into somewhere in file system.

1. Configure WSO2 API Manager Server

The data-publisher in API Manager, publish events to CEP in each client request. Following configuration in api-manager.xml is used to do this. Usually when we configuring APIM with BAM, we provide BAM thrift port and url here. Instead of BAM server thrift port, we have provide CEP thrfit port/url here. (i.e 7613 ;because later in this sample, we are going to start the CEP server with offset 2).

i. Open the api-manager.xml (located in "{AM_HOME}/repository/conf/" directory) and modify the "<APIUsageTracking>" section as follows;

<APIUsageTracking>
 
        <Enabled>true<⁄Enabled>       
        <PublisherClass>
        org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher
        <⁄PublisherClass>

        <!--
            Thrift port of the remote CEP server.
        -->
        <ThriftPort>7613<⁄ThriftPort>

        <!--
            Server URL of the remote CEP server.
        -->
        <BAMServerURL>tcp:⁄⁄localhost:7613⁄<⁄BAMServerURL>

        <BAMUsername>admin<⁄BAMUsername>
        <BAMPassword>admin<⁄BAMPassword>
        <DataSourceName>jdbc⁄WSO2AM_STATS_DB<⁄DataSourceName>

        <GoogleAnalyticsTracking>
             <Enabled>false<⁄Enabled>
             <TrackingID>UA-XXXXXXXX-X<⁄TrackingID>
       <⁄GoogleAnalyticsTracking>

    <⁄APIUsageTracking>
 
        

ii. Open the master-datasources.xml file (located in "{AM_HOME}/repository/conf/datasources/" directory) and change the jdbc url path of WSO2AM_STATS_DB to point absolute path of BAM as given in following example. "WSO2AM_STATS_DB" is the data-source that pointed to BAM stats database. This is used for the statistic analyzing.

<datasource>
  <name>WSO2AM_STATS_DB<⁄name>
  <description>The datasource used for getting statistics to API Manager<⁄description>
  <jndiConfig>
        <name>jdbc⁄WSO2AM_STATS_DB<⁄name>
    <⁄jndiConfig>
  <definition type="RDBMS">
  <configuration>
         <url>jdbc:h2:⁄home⁄dinusha⁄BAM_CEP_AM⁄wso2bam-2.3.0⁄repository⁄database⁄APIMGTSTATS_DB;AUTO_SERVER=TRUE<⁄url>
         <username>wso2carbon<⁄username>
         <password>wso2carbon<⁄password>
         <driverClassName>org.h2.Driver<⁄driverClassName>
         <maxActive>50<⁄maxActive>
         <maxWait>60000<⁄maxWait>
         <testOnBorrow>true<⁄testOnBorrow>
         <validationQuery>SELECT 1<⁄validationQuery>
         <validationInterval>30000<⁄validationInterval>
    <⁄configuration>
 <⁄definition>
<⁄datasource>

We are done with the API Manager related configurations.

2. Configure WSO2 CEP Server

i. Open the carbon.xml file (located in {CEP_HOME}/repository/conf/ directory) and change the port offset parameter "<Offset>" to 2. So the CEP server is going to run in the 9445 https port.

ii. We are going to configure three brokers in CEP. A broker operates in between an event source and the CEP Engine, facilitating their connection. You can find the more details about CEP brokers from here. These brokers are going to use later by the buckets. Open the broker-manager-config.xml file (located in {CEP_HOME}/repository/conf/ directory) and add the following broker configurations.

<brokerConfiguraton name="externalAgentBroker" type="agent">
        <property name="receiverURL">tcp:⁄⁄localhost:7612<⁄property>
        <property name="authenticatorURL">ssl:⁄⁄localhost:7712<⁄property>
        <property name="username">admin<⁄property>
        <property name="password">admin<⁄property>
    <⁄brokerConfiguraton>

    <brokerConfiguraton name="emailBroker" type="e-mail">
        <property name="defaultSubject">Message from WSO2CEP<⁄property>
    <⁄brokerConfiguraton>

    <brokerConfiguraton name="APIM_Broker" type="agent">
        <property name="receiverURL">tcp:⁄⁄localhost:7613<⁄property>
        <property name="authenticatorURL">ssl:⁄⁄localhost:7713<⁄property>
        <property name="username">admin<⁄property>
        <property name="password">admin<⁄property>
    <⁄brokerConfiguraton>

iii. Total processing of received events and triggering of new events happens through the buckets. You can find more details about CEP buckets from here. We are going to create four buckets in CEP for this sample as follows.

APIMBucket.xml, APIMFaultBucket.xml, APIMResponseBucket.xml : These three buckets are used to stat publishing for BAM. Note that in all these three buckets have used "APIM_Broker" as the input broker which has pointed the broker in the CEP server and "externalAgentBroker" as the output broker which has pointed the broker in the BAM server.

APIMRealTimeBucket.xml : This bucket is use for show the real time data analyzing (Send an email). This bucket has used the "APIM_Broker" as the input broker and "emailBroker" as the output broker. This will read the some of the data from the request event stream ("org.wso2.apimgt.statistics.request/1.0.0") and do some calculations in the output and send an email using the "emailBroker".

You can download the bucket configurations from here and deploy them in  "{CEP_HOME}/repository/deployment/server/cepbuckets/" directory.


3. Configure WSO2 BAM Server

i. Open the carbon.xml file (located in {BAM_HOME}/repository/conf/ directory) and change the port offset parameter "<Offset>" to 1. So the BAM server is going to run in the 9444 https port.

ii. You can find the toolbox (API_Manager_Analytics.tbox) that used for statistic analysis in "{AM_HOME}/statistics/" directory. Copy this toolbox into the "{BAM_HOME}/repository/deployment/server/bam-toolbox/" directory.

iii. Open the master-datasources.xml file (located in "{BAM_HOME}/repository/conf/datasources/" directory) and add the following datasource configuration.

<datasource>
  <name>WSO2AM_STATS_DB<⁄name>
  <description>The datasource used for getting statistics to API Manager<⁄description>
  <jndiConfig>
        <name>jdbc⁄WSO2AM_STATS_DB<⁄name>
    <⁄jndiConfig>
  <definition type="RDBMS">
  <configuration>
         <url>jdbc:h2:repository⁄database⁄APIMGTSTATS_DB;AUTO_SERVER=TRUE<⁄url>
         <username>wso2carbon<⁄username>
         <password>wso2carbon<⁄password>
         <driverClassName>org.h2.Driver<⁄driverClassName>
         <maxActive>50<⁄maxActive>
         <maxWait>60000<⁄maxWait>
         <testOnBorrow>true<⁄testOnBorrow>
         <validationQuery>SELECT 1<⁄validationQuery>
         <validationInterval>30000<⁄validationInterval>
    <⁄configuration>
 <⁄definition>
<⁄datasource>

We are done with the all configuration changes. Now you can start the all three servers and verify the scenario. (There is a pre-configured setup with all these changes available to download here. If you are going to use it, please read the readme.txt provided there and change the data-sources and buckets to fit with our paths.)

Test and verify the setup

1. Login to the API Manager. Create and publish an API. Subscribe to it and invoke.
2. Stats should have shown in APIM stats page.
3. Invoke same API more than 5 times within 1 minute. Email notification should goes to email address defined in the APIMRealTimeBucket.xml.