How to Parse Apache error_log for Troubleshooting & Reporting

Note: if you haven't already, see Log Parsing, Analysis, Correlation, and Reporting Engine post first.

Apache error_log can be useful while troubleshooting production problem. So parsing and analysing the content of this file regularly helps in maintaining the overall health of the system. If mod_mpmstats enabled, error_log also contains Multi-Processing Modules (MPM) stats data. MPM stats can be used for both troubleshooting and performance tuning. http://publib.boulder.ibm.com/httpserv/manual70/mod/mod_mpmstats.html provides more details about MPM stats.
Since, error_log does not contain the Web server name, in order to co-relate the data to corresponding Web server, it is advisable that you put error_log files for each Web server under corresponding directory, named after the Web server. It is specially important when you are parsing logs from multiple Web servers. Script takes directory name as Web server name for the purpose of reporting and analysis. For example, let's say, you have Web servers 'webSrv01, webSrv02, webSrv03 ... etc., then put logs from each Web server under corresponding directories as shown below:

/tmp/webSrv01     error_log     error_log_2017.09.05.log     access_log  /tmp/webSrv02     error_log     error_log_2017.09.05.log     access_log

The naming suffix for historical files can be different from one environment to another. So, if you have different suffix for historical files, you can tweak the find script. Currently the fragment of script that finds error_log looks like this:

find $rootcontext -name "error_log*" -type f | grep "$logFileName"

where $rootcontext is root path.

Review the actual script available in github - https://github.com/pppoudel/log-parser/blob/master/webErrorLogParser.sh for details.

Note: script is written to parse the date format like '[Thu Dec 14 08:13:08 2017]' in error_log. If your error_log uses different date format, you may need to tweak the section of script which parses the date.

How to execute:
You can see all the available options, by just launching:
$> ./webErrorLogParser.sh

See below for few examples:

# processing current day's logs $> ./webErrorLogParser.sh --rootcontext <log-path> # processing yesterday's logs with historical report updates $> ./webErrorLogParser.sh --rootcontext <log-path> --rpttype daily # processing any day's logs updates $> ./webErrorLogParser.sh --rootcontext <log-path> --recorddate <date in (YYYY-MM-DD) format>

Output
Report/Output files:

• $rptDir/00_Alert.txt
• $rptDir/03_WebErrorLogSummaryRpt.txt
• $rptDir/WebErrorLogMpmStatsRpt_all.csv
• $rptDir/WebErrorLogRpt_all.csv

Where $rptDir is report directory. Default value is $TMP/$recDate

History Report/Output files:

# These are historical reports. Each run will append record in existing report file.

• $pDir/RecycleHistoryRpt_all.csv
• $pDir/MPMStatsHistoryRpt.csv

Where $pDir is parent of $rptDir.

See sample summary report in github - https://github.com/pppoudel/log-parser/blob/master/sample_reports/03_WebErrorLogSummaryRpt.txt
And here is a sample MPM stats report https://github.com/pppoudel/log-parser/blob/master/sample_reports/WebErrorLogMpmStatsRpt_all.csv

See my other posts in this series

1. websphereLogParser.sh for parsing, analyzing and reporting WebSphere Application Server (WAS) SystemOut.log
2. webAccessLogParser.sh for parsing, analyzing and reporting Apache/IBM HTTP Server (IHS) access_log
3. javaGCStatsParser.sh for parsing, analyzing and reporting Java verbose Garbage Collection (GC) log