Source

esbperformance / wso2-esb / README.TXT

Full commit
Updated for version 4.0.3

Copy the resources directory in this directory into the repository/deployment/server directory of the WSO2 ESB

    $ cd ~/resources/esbperformance/wso2-esb
    $ cp -R resources {WSO2ESB_HOME}/repository/deployment/server

e.g.
    $ cd ~/resources/esbperformance/wso2-esb
    $ cp -R resources /home/ubuntu/esbs/wso2esb-4.0.3/repository/deployment/server

Copy the synapse.xml into the {WSO2ESB_HOME}/repository/deployment/server/synapse-configs/default direcotry

    $ cp synapse.xml {WSO2ESB_HOME}/repository/deployment/server/synapse-configs/default

e.g.
    $ cp synapse.xml /home/ubuntu/esbs/wso2esb-4.0.3/repository/deployment/server/synapse-configs/default

Increase Heap to 2G by editing {WSO2ESB_HOME}/bin/wso2server.sh (e.g. /home/ubuntu/esbs/wso2esb-4.0.3/bin/wso2server.sh) to read as follows:

    -Xms2048m -Xmx2048m -XX:MaxPermSize=256m \

Disable the HTTP access logs by adding the following line into the lib/log4j.properties file
    log4j.logger.org.apache.synapse.transport.nhttp.access=WARN

Configure approximate worker threads to ~300 and socket timeout to 120 seconds by editing {WSO2ESB_HOME}/lib/core/WEB-INF/classes/nhttp.properties 
(e.g. /home/ubuntu/esbs/wso2esb-4.0.3/lib/core/WEB-INF/classes/nhttp.properties)

    http.socket.timeout=120000
    snd_t_core=20
    snd_t_max=150
    lst_t_core=20
    lst_t_max=150

Start the WSO2 ESB server.

    $ cd {WSO2ESB_HOME}/bin
    $ ./wso2server.sh

Service URLs
============

Direct Proxy
    url             : http://localhost:8280/services/DirectProxy
    wsdl-url        : http://localhost:8280/services/DirectProxy?wsdl

SOAP Body CBR Proxy
    url             : http://localhost:8280/services/CBRProxy
    wsdl-url        : http://localhost:8280/services/CBRProxy?wsdl

SOAP Header CBR Proxy
    url             : http://localhost:8280/services/CBRSOAPHeaderProxy
    wsdl-url        : http://localhost:8280/services/CBRSOAPHeaderProxy?wsdl

Transport Header CBR Proxy
    url             : http://localhost:8280/services/CBRTransportHeaderProxy
    wsdl-url        : http://localhost:8280/services/CBRTransportHeaderProxy?wsdl

XSLT Transformation Proxy
    url             : http://localhost:8280/services/XSLTProxy
    wsdl-url        : http://localhost:8280/services/XSLTProxy?wsdl

WS-Security Proxy
    url             : http://localhost:8280/services/SecureProxy
    wsdl-url        : http://localhost:8280/services/SecureProxy?wsdl

Passthrough Transport
=====================
For the WSO2 Passthrough Transport, add the following to the {WSO2ESB_HOME}/lib/core/WEB-INF/classes/passthru-http.properties 
(e.g. /home/ubuntu/esbs/wso2esb-4.0.3/lib/core/WEB-INF/classes/passthru-http.properties)

    http.socket.timeout=120000
    worker_pool_size_core=40
    worker_pool_size_max=300

Edit the {WSO2ESB_HOME}/repository/conf/axis2.xml (e.g. /home/ubuntu/esbs/wso2esb-4.0.3/repository/conf/axis2.xml) to enable and configure
the Message Relay transport for "text/xml" content types, and the Passthrough Transport on port 8281

1. Add the following under "<messageBuilders>" element as a child
    <messageBuilder contentType="text/xml" class="org.wso2.carbon.relay.BinaryRelayBuilder"/>

2. Add the following under "messageFormatters" element as a child
    <messageFormatter contentType="text/xml" class="org.apache.axis2.transport.http.SOAPMessageFormatter"/>

To test the Direct Proxy and CBR Transport Header Proxy scenarios, you may comment out the normal HTTP transport and replace them with the following.
(See http://wso2.org/library/articles/2012/03/wso2-esb-message-transfer-mechanisms-comparative-benchmarks/ for more information)

Add the following near the other <transportReceiver ...> elements
    <transportReceiver name="http" class="org.wso2.carbon.transport.passthru.PassThroughHttpListener">
        <parameter name="port">8280</parameter>
        <parameter name="non-blocking"> true</parameter>
    </transportReceiver>

Add the following near the other <transportSender ...> elements
    <transportSender name="http"  class="org.wso2.carbon.transport.passthru.PassThroughHttpSender">
        <parameter name="non-blocking" locked="false">true</parameter>
        <parameter name="warnOnHTTP500" locked="false">*</parameter>
    </transportSender>


Notes:
======

Normal - 1st try
----------------

Exceptions seen on the logs during the load test were many and the complete log can be found at /home/ubuntu/esbs/wso2esb-4.0.3/repository/logs/crash/wso2carbon.log. Some snippets follows

TID: [] [WSO2 ESB] [2012-08-04 08:48:41,657] ERROR {org.apache.synapse.core.axis2.Axis2Sender} -  Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Error in signature with X509Token
...
TID: [] [WSO2 ESB] [2012-08-04 08:48:49,276] ERROR {org.apache.synapse.transport.nhttp.ServerWorker} -  Error processing POST request  {org.apache.synapse.transport.nhttp.ServerWorker}
java.util.ConcurrentModificationException
...
TID: [] [WSO2 ESB] [2012-08-04 09:30:11,494]  WARN {org.apache.synapse.commons.jmx.ThreadingView} -  SYSTEM ALERT: 58.48484848484848% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
TID: [] [WSO2 ESB] [2012-08-04 09:30:11,503]  WARN {org.apache.synapse.commons.jmx.ThreadingView} -  SYSTEM ALERT: 60.45454545454546% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
...

Exceptions seen on the logs during the load test execution were many, and although we initially selected WSO2 for the final comparison, we noticed that the system got stuck halfway through the 1st iteration out of the three. The output of the top command was as follows and the system was not progressing although running at full CPU utilization.

Tasks: 178 total,   1 running, 177 sleeping,   0 stopped,   0 zombie
Cpu(s): 88.5%us,  0.1%sy,  0.0%ni,  5.4%id,  0.0%wa,  0.0%hi,  0.0%si,  6.0%st
Mem:   7132352k total,  3886384k used,  3245968k free,   219356k buffers
Swap:        0k total,        0k used,        0k free,   466728k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7382 ubuntu    20   0 2821m 2.3g  15m S  776 33.7 892:55.45 java   
32519 ubuntu    20   0 19276 1380  968 R    0  0.0   0:00.03 top    
    1 root      20   0 23744 1936 1300 S    0  0.0   0:00.41 init  

After many hours (~6.5hours), the system was still fully blocked on CPU as per EC2 monitoring and top. As no progress was being made, the test was abandoned.

Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
Cpu(s): 91.5%us,  0.0%sy,  0.0%ni,  2.5%id,  0.0%wa,  0.0%hi,  0.0%si,  6.0%st
Mem:   7132352k total,  3899800k used,  3232552k free,   219356k buffers
Swap:        0k total,        0k used,        0k free,   467920k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7382 ubuntu    20   0 2821m 2.3g  15m S  791 33.7   2815:14 java   
  858 ubuntu    20   0 19276 1356  968 R    0  0.0   0:00.07 top    
    1 root      20   0 23744 1936 1300 S    0  0.0   0:00.43 init     

Passthrough 
-----------

Exceptions seen on the logs during the load test were many and the complete log can be found at /home/ubuntu/esbs/wso2esb-4.0.3/repository/logs/wso2carbon.log from timestamp 2012-08-05 06:09:06 onwards. Some snippets follows

[2012-08-05 06:34:52,764]  WARN - SourceHandler Illegal incoming connection state: CLOSED . Possibly two send backs are happening for the same request
[2012-08-05 06:34:52,768]  WARN - SourceHandler Trying to write response body while the handler is in an inconsistent state REQUEST_READY
[2012-08-05 06:34:52,770]  WARN - PassThroughHttpListener System may be unstable: HTTP ListeningIOReactor encountered a runtime exception : null
java.lang.NullPointerException
	at org.wso2.carbon.transport.passthru.SourceHandler.outputReady(SourceHandler.java:179)
...
[2012-08-05 06:34:52,778]  WARN - SourceHandler Connection closed while writing the response: 127.0.0.1:51216->127.0.0.1:8280
...

Normal - Retry
--------------

We gave the WSO2 ESB one more retry, and this time the test completed.

Exceptions seen on the logs during the load test were many and the complete log can be found at /home/ubuntu/esbs/wso2esb-4.0.3/repository/logs/wso2carbon.log from timestamp 2012-08-05 06:57:22 onwards. Some snippets follows

TID: [] [WSO2 ESB] [2012-08-05 07:40:40,570]  WARN {org.apache.synapse.commons.jmx.ThreadingView} -  SYSTEM ALERT: 60.15151515151514% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
TID: [] [WSO2 ESB] [2012-08-05 07:40:40,577]  WARN {org.apache.synapse.commons.jmx.ThreadingView} -  SYSTEM ALERT: 60.30303030303028% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
...
...
TID: [] [WSO2 ESB] [2012-08-05 08:12:28,825] ERROR {org.apache.synapse.core.axis2.Axis2Sender} -  Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Error in signature with X509Token
        at org.apache.rampart.handler.RampartSender.invoke(RampartSender.java:76)
...
Caused by: org.apache.rampart.RampartException: Error in signature with X509Token
...
Caused by: org.apache.ws.security.WSSecurityException: Signature creation failed; nested exception is:
...
        ... 22 more
Caused by: java.util.ConcurrentModificationException
...