How to Parse Java GC logs for Troubleshooting & Reporting

Note: if you haven't already, see Log Parsing, Analysis, Correlation, and Reporting Engine post first.

Java Garbage Collection (GC) log format may depend on Java version, Java Virtual Machine (JVM) settings, JVM providers etc. This particular parser has been tested with verbose GC output from WebSphere Application Server 8.5.x, configured to use IBM Java version 7.0.4.0 with the following JVM configuration.

<jvmEntries xmi:id="JavaVirtualMachine_12315382660776" verboseModeGarbageCollection="true" verboseModeJNI="false" initialHeapSize="8192" maximumHeapSize="8192" runHProf="false" hprofArguments="" debugMode="false" debugArgs="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" genericJvmArguments="-XX:MaxPermSize=2560m -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=16 -XX:-TraceClassUnloading -XX:+UseCompressedOops -XX:+AlwaysPreTouch -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 " executableJarFileName="" disableJIT="false">

If your log format is different, you may need to tweak the parser script a little bit.

Note: this parser is not designed to replace any of your existing parser, but rather to complement in terms of data gathering, and visualization. See, the sample summary report.

Also, it may generate alert messages like the one seen below:
12 : number of Full GC exceeds threshold of 6 for AppSrv04 on 2016-11-29 Old Generation Heap space after Full GC exceeded threshold of 4700000(K) for AppSrv03. There is possibility of OutOfMemory in near future because of Not sufficient Heap space 

Other than summary report and alert messages written in 00_Alert.txt, it also produces GCstats_all.csv. It is a pipe '|' delimitted file, which captures all relevant data for each day for each JVM. Data written in GCstats_all.csv can be imported to spread sheet like Excel to create graph, chart and table to present to  the management.
Review the actual script available in github - https://github.com/pppoudel/log-parser/blob/master/javaGCStatsParser.sh for details.

Note: script is written to parse the date format like '2017-05-25T08:11:50.666-0400' in native_stdout.log. If your native_stdout.log uses different date format, you may need to tweak the section of script which parses the date.

How to execute:

You can see all the available options, by just launching:
$> ./javaGCStatsParser.sh

Few examples are here:

# processing current day's logs $> ./javaGCStatsParser.sh --rootcontext <log-path> # processing yesterday's logs with historical report updates $> ./javaGCStatsParser.sh --rootcontext <log-path> --rpttype daily # processing any day's logs updates $> ./javaGCStatsParser.sh --rootcontext <log-path> --recorddate <date in (YYYY-MM-DD) format>

Output

Report/Output files:

• $rptDir/00_Alert.txt
• $rptDir/04_GCSummaryRpt.txt
• $rptDir/GCstatsRpt_all.csv

Where $rptDir is report directory. Default value is $TMP/$recDate

History Report/Output files:

# These are historical reports. Each run will append record in existing report file.

• $pDir/GCHistoryRpt_all.csv

Where $pDir is parent of $rptDir.

See sample summary report in github - https://github.com/pppoudel/log-parser/blob/master/sample_reports/04_GCSummaryRpt.txt
See my other posts in this series

1. websphereLogParser.sh for parsing, analyzing and reporting WebSphere Application Server (WAS) SystemOut.log
2. webAccessLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) access_log
3. webErrorLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) error_log