WebSphere Application Server (WAS) High Performance Extensible Logging (HPEL) - some tips and tricks

As we know, WebSphere Application Server generally supports two types of logging mechanisms - Basic and High Performance Extensible Logging (HPEL).  If your application(s) logs with high frequency, there is a real possibility that system will be benefited (performance wise), if you enable HPEL. On the performance benefit of HPEL, IBM documentation says,
"HPEL has been designed and tested to significantly outperform the existing basic log and trace facility. One result is that the application server can  run with trace enabled while causing less impact to performance than tracing the same components using basic logging. Another result is that applications that frequently write to the logs might run faster with HPEL ..."
Reference: https://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/ctrb_HPELOverview.html

I have recently worked on a project to change logging from Basic to HPEL and based on my experience, I'm listing few tips and tricks below.

1. Enabling HPEL:

If you have just few App Servers in your server farm, then you can do this task manually. However, if you need to enable this for dozens of App Servers then, it's better to automate the task using script. WAS InfoCenter provides some fragments of scripts here and there, but nothing that you can take and execute. Below is a simple Jython script that I put together. It enables or disables HPEL based on passed argument to all servers within the given WAS Cell. You can also exclude specific server(s) as needed. Below script excludes 'nodeagent' and 'dmgr'.

enableHPEL.py

import sys global AdminConfig global AdminControl action = sys.argv[0] if action == 'true': enableHPEL='true' enableRAS='false' else: enableHPEL='false' enableRAS='true' #endElse #Get Cell, nodes, and servers cellName = AdminControl.getCell( ) cellID = AdminConfig.getid("/Cell:"+cellName+"/" ) nodes = AdminConfig.list("Node", cellID ).split() for nodeID in nodes: nodeName = AdminConfig.showAttribute(nodeID, "name") servers = AdminConfig.list("Server", nodeID ).split() for serverID in servers: srvName = AdminConfig.showAttribute(serverID, "name" ) print "/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName if srvName == 'nodeagent' or srvName == 'dmgr': continue #endIf HPELService = AdminConfig.getid("/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName+"/HighPerformanceExtensibleLogging:/"); RASLogService = AdminConfig.getid("/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName+"/RASLoggingService:/"); if enableHPEL == 'true': print "Enabling HPELService for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName AdminConfig.modify(HPELService, "[[enable true]]") AdminConfig.modify(RASLogService, "[[enable false]]") #endIf if enableRAS == 'true': print "Enabling RASLogService for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName AdminConfig.modify(HPELService, "[[enable false]]") AdminConfig.modify(RASLogService, "[[enable true]]") #endIf print "Saving configuration for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName AdminConfig.save() print "Synchronizing configuration for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName Sync1 = AdminControl.completeObjectName('type=NodeSync,process=nodeagent,node='+nodeName+',*') synced = AdminControl.invoke(Sync1, 'sync') print "Synchronization completed: "+synced #endFor #endFor

Above script is for full WAS profile, for WAS Liberty profile you have to do it manually for one WAS liberty server and may copy the configuration to other Liberty servers. You can find details here: https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_confHPEL.html?cp=SSEQTP_8.5.5
Or use Eclipse WAS Liberty tool as outlined here: https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/t_edit_server_config.html?cp=SSEQTP_8.5.5

If you need to customize HPEL setting for full WAS profile further (like changing size of log repository, location of repository, buffering etc.)  using wsadmin scripting, you can follow the following link for Full WAS profile:
http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.base.doc/ae/ttrb_confHPELwsadmin.html
Similar options for WAS Liberty profile can be found here:
https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_confHPEL.html?cp=SSEQTP_8.5.5

You can execute the above mentioned script as follows using wsadmin.sh[bat].
Enable HPEL:
wsadmin -username <username> -password <password> -lang jython -f ~/enableHPEL.py true

Disable HPEL:
wsadmin -username <username> -password <password> -lang jython -f ~/enableHPEL.py false

Note: provided script changes the configuration using wsadmin's AdminConfig object, which requires App Server to be restarted in order to take effect. You can use AdminControl object to do the same in runtime and that takes effect immediately.

2. Usability:

One of the major concerns with HPEL I found was usability issue. Once HPEL is enabled, the System logs and traces are written in binary, and people can not just open the log files using any text editor as they used to do it. There is an option to write plain text log also, but by enabling that option we loose certain performance benefit of HPEL. Even though IBM provides pretty good tools ('logViewer.sh[bat]' with full WAS profile, and 'binaryLog' with WAS Liberty profile) either to convert the binary logs into text format or just monitor the logs in real time, it's not easy to convince people to change their way of doing things. It takes time. One catch though, IBM's tools require WAS installed in the machine where tools are supposed to run. Not everybody who needs to look logs has WAS installed and your log repository server where you copy your binary log repositories for processing also may not have WAS installed. So, alternative is to convert logs on the App Server, where its generated  (may not be a good practice as it uses CPU cycles of production server)  and provide text logs to whoever need to review them.
I thought that it would be really convenient if we can have a tool to process these binary logs without WAS. And indeed, with little customization, you can use IBM's tool without WAS, which I am going to explain here:

2.1. Preparing custom 'logViewer' tool so that it can be run independent of WAS:

2.1.A) Create following directory structure under /tmp and copy corresponding file from your WAS installation:

/tmp/hpelviewer bin logViewer.sh (copy from WAS_INSTALLED_DIR/bin directory) logViewer.bat (copy from WAS_INSTALLED_DIR/bin directory) lib bootstrap.jar (copy from WAS_INSTALLED_DIR/lib directory) com.ibm.hpel.logging.jar  (copy from WAS_INSTALLED_DIR/plugins directory) startup.jar  (copy from WAS_INSTALLED_DIR/lib directory) properties WsHeader  (copy from WAS_INSTALLED_DIR/properties directory) WsLevels.properties  (copy from WAS_INSTALLED_DIR/properties directory)


2.1.B) Once the files are copied, open the logViewer.sh from /tmp/hpelviewer/bin and change as follow (actual script lines are in blue):

#Set JAVA_HOME at the top: JAVA_HOME=/cygdrive/c/java/jdk1.7.0_45 #leave following line as it is: binDir=`dirname "$0"` #Comment the following line: #. "$binDir/setupCmdLine.sh" #Instead set the WAS_HOME: setWASHome() {     CUR_DIR=$(pwd) WAS_DIR="${binDir}"/.. cd "${WAS_DIR}" WAS_HOME=$(pwd) cd "${CUR_DIR}"   } setWASHome #Leave following lines as it is: if [ -f ${JAVA_HOME}/bin/java ]; then     JAVA_EXE="${JAVA_HOME}/bin/java" else     JAVA_EXE="${JAVA_HOME}/jre/bin/java" fi #comment the following lines as we need to make them compatible to Unix and Cygwin: #WAS_HEADER="-Dlogviewer.custom.header=${WAS_HOME}/properties/WsHeader" #WAS_LEVELS="-Dlogviewer.custom.levels=${WAS_HOME}/properties/WsLevels.properties" #Instead add the following lines: PATH=.:"$JAVA_HOME"/bin/:"$JAVA_HOME"/jre/bin:$PATH WAS_CLASSPATH="$WAS_HOME/properties:$WAS_HOME/lib/bootstrap.jar:$WAS_HOME/lib/com.ibm.hpel.logging.jar" WAS_HEADER_PROP="${WAS_HOME}/properties/WsHeader" WAS_LEVELS_PROP="${WAS_HOME}/properties/WsLevels.properties" WAS_LOGGING="-Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Djava.util.logging.configureByServer=true" # to be used as default log repository path LOG_ROOT="$WAS_HOME/logs" ## ## Determine the platform and convert path accordingly ## case $OSTYPE in   cygwin)     uname=CYGWIN_NT         ;;   *)     uname=`uname` esac case $uname in   CYGWIN_*) WAS_HEADER_PROP=$(cygpath -w "$WAS_HEADER_PROP") WAS_LEVELS_PROP=$(cygpath -w "$WAS_LEVELS_PROP") LOG_ROOT=$(cygpath -w "$LOG_ROOT") WAS_HOME=$(cygpath -w "$WAS_HOME")         WAS_CLASSPATH=$(cygpath -pw "$WAS_CLASSPATH")         PATH=$(cygpath -pw "$PATH")         ;;   *) ;; esac OSGI_INSTALL="-Dosgi.install.area=$WAS_HOME" WAS_HEADER="-Dlogviewer.custom.header=$WAS_HEADER_PROP" WAS_LEVELS="-Dlogviewer.custom.levels=$WAS_LEVELS_PROP" #Finally, modify the Java execution command and options as follow: ${JAVA_EXE} \ -Dwas.install.root="$WAS_HOME" \ -Duser.install.root="$WAS_HOME" \ -Dlog.repository.root="$LOG_ROOT" \ $WAS_LOGGING \ $WAS_HEADER \ $WAS_LEVELS \ -classpath $WAS_CLASSPATH  com.ibm.ws.bootstrap.WSLauncher \ com.ibm.ws.logging.hpel.viewer.LogViewer "$@"

2.1.C) Save the changes.

Note: In the above script 'cygpath' has been used to make it compatible with Cygwin. for more information about 'cygpath' see http://www.cygwin.com/cygwin-ug-net/cygpath.html

2.2. Preparing custom 'binaryLog' tool (for Liberty profile) so that it can be run independent of WAS:

2.2.A) Create following directory structure under /tmp and copy corresponding file from your WAS installation:

/tmp/hpelviewer bin binaryLog binaryLog.bat tools ws-binarylogviewer.jar lib com.ibm.ws.kernel.cmdline_1.0.10.jar com.ibm.ws.logging.hpel_1.0.10.jar

2.2.B) customization of binaryLog is easier as IBM has written it in in such a way that it can be run under Cygwin as well as other Unix like platform. Just add the following two lines (actual lines to be added are in blue) in the beginning of the script.

WLP_INSTALL_DIR=.. #optionally set JAVA_HOME JAVA_HOME=/cygdrive/c/java/jdk1.8.0

2.2.C) Save the file.

Note: follow similar tricks if you need to modify corresponding BAT files for Windows.
Once your customization is complete, just create a zip file (for example hpelviewer.zip) making hpelviewer as root directory of the zip,  and give it your users.
They can just extract the zip and and execute as follow. Only thing they need is Java installed on their machine.
Here is an exmple:

cd <hpelviewer_extract_dir>/hpelviewer/bin ./logViewer.sh -repositoryDir <repositoryDir>

Or:

cd <hpelviewer_extract_dir>/hpelviewer/bin ./binaryLog view  {serverName | repositoryPath}

Note: Since the property files, jar files and script files for WAS full profile and WAS Liberty profile are unique, you can actually create a single zip file by combining files for both. Here is how the file structure for combined  tool would look like:

hpelviewer bin logViewer.sh logViewer.bat binaryLog binaryLog.bat tools ws-binarylogviewer.jar lib bootstrap.jar com.ibm.hpel.logging.jar startup.jar com.ibm.ws.kernel.cmdline_1.0.10.jar com.ibm.ws.logging.hpel_1.0.10.jar properties WsHeader WsLevels.properties

Create a zip and give it to your users, so that they can use it to view binary logs generated by both WAS full profile or Liberty profile.

Happy HPEL !!!