How to Parse WebSphere Application Server Logs for Troubleshooting & Reporting

Note: if you haven't already, see Log Parsing, Analysis, Correlation, and Reporting Engine post first.

WebsphereLogParser parses IBM WebSphere Application Server SystemOut.log. This is one of the parsers included in the suite that I have posted. This particular parser script expects that the SystemOut log follows the default/basic message formats outlined by IBM in JVM log interpretation document. Since, SystemOut.log does not contain the WAS server name, in order to relate the data to corresponding WAS JVM, it is advisable that you put SystemOut logs for each WAS under corresponding directory, named after the WAS name. It is specially important when you are parsing logs from multiple WAS servers. Script takes directory name as WAS name for the purpose of reporting. For example, let's say, you have Application servers 'appSrv01, appSrv02, appSrv03 ... etc.), then put logs from each Application Server under corresponding directories like:

/tmp/appSrv01     SystemOut.log     SystemOut_2017.09.05.log     SystemErr.log  /tmp/appSrv02     SystemOut.log     SystemOut_2017.09.05.log     SystemErr.log

It parses both zipped file and or regular file. By default, it finds and processes following files in a given path:

SystemOut.log SystemOut.log.zip SystemOut.zip SystemOut_'$recYY'.'$rec0MM'.'$rec0DD'_.* SystemOut_'$recNYY'.'$recN0MM'.'$recN0DD'_.* Where: recYY is Year like 17 (17 represent year of 2017) rec0MM is Month like 01 (01 represent month of January) rec0DD is Day like 01 (01 represents the first day of a month) recNYY/recN0M/recN0DD = (recYY/rec0MM/rec0DD)+1 day

The naming suffix for historical files can be different from one environment to another. So, if you have different suffix for historical files, you need to tweak the find script. Currently it looks like this:

find $rootcontext -name "SystemOut*" -type f | \   egrep '(SystemOut.log$|SystemOut.log.zip$|SystemOut.zip$|SystemOut_'$recYY'.'$rec0MM'.'$rec0DD'_.*|SystemOut_'$recNYY'.'$recN0MM'.'$recN0DD'_.*)'

where $rootcontext is root path.

Review the actual script available in github - https://github.com/pppoudel/log-parser/blob/master/websphereLogParser.sh for details.

Note: script is written to parse the date format like '[4/23/17 8:13:22:137 EDT]' in SystemOut.log. If your SystemOut.log uses different date format, you may need to tweak the section of script which parses the date.

How to execute:

You can see all the available options, by just launching:
$> ./websphereLogParser.sh

Few examples are here:

# processing current day's logs $> ./websphereLogParser.sh --rootcontext <log-path> # processing yesterday's logs with historical report updates $> ./websphereLogParser.sh --rootcontext <log-path> --rpttype daily # processing any day's logs updates $> ./websphereLogParser.sh --rootcontext <log-path> --recorddate <date in (YYYY-MM-DD) format>

Output
Report/Output files:

• $rptDir/00_Alert.txt
• $rptDir/01_WASLogSummaryRpt.txt
• $rptDir/WASLogErrRpt_all.csv
• $rptDir/WASLogFilteredErrRpt.csv
• $rptDir/WASLogSummaryByErrCmpRpt.csv
• $rptDir/WASLogSummaryByErrClassRpt.csv
• $rptDir/WASLogSummaryByErrExpRpt.csv
• $rptDir/WASLogSummaryByErrMsgRpt.csv
• $rptDir/WASLogSummaryByWarnCmpRpt.csv
• $rptDir/WASLogSummaryByWarnClassRpt.csv
• $rptDir/WASLogSummaryByWarnExpRpt.csv
• $rptDir/WASLogSummaryByWarnMsgRpt.csv

Where $rptDir is report directory. Default value is $TMP/$recDate

History Report/Output files:

# These are historical reports. Each run will append record in existing report file.

• $pDir/RecycleHistoryRpt_all.csv
• $pDir/WASOutOfMemoryHistoryRpt.csv
• $pDir/WASTransactionTimeOutHistoryRpt.csv
• $pDir/WASSHungThreadHistoryRpt.csv

Where $pDir is parent of $rptDir.

See sample summary report in github - https://github.com/pppoudel/log-parser/blob/master/sample_reports/01_WASLogSummaryRpt.txt
See my other posts in this series

1. webAccessLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) access_log
2. webErrorLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) error_log
3. javaGCStatsParser.sh for parsing, analyzing and reporting Java verbose Garbage Collection (GC) log