Possible Issues caused by CA Wily Introscope APM BRTM feature

After going live with CA APM on week-ends, right next Monday morning when site opened for business, help desk started getting string of complains. Either both or one of the following were reported:
1) Users experiencing spinning wheel when clicking submit on their web page.
2) Some users were losing their sessions and were prompted to re-login to their web application.

Initially, we were completely lost as we did not know what's causing this. Odd thing, not all users were experiencing the difficulties. And also server resources (CPU, memory) usage were normal. Performance was also normal for those users who did not experience the above mentioned issues.   After analyzing the web server access log, we were able to put some pattern based on the affected users' User Agent information. Some how, all the affected users were using Internet Explorer (IE) browsers.  IE 9/Trident 5.0, IE 10/Trident 6.0.  Even though more than 60% of our users were using Firefox (different versions), none of them reported any such issues, only IE users were affected. With this information in hand, we asked QA to test with different browser and they were able to reproduce the issues with IE and not with Firefox.

     Next step was to identify what was causing the issues. Again, looking into the Access log, we were able to identify that the request URL coming from those users just before they were kicked out had query parameter 'WilyCmd=cmdMetrics' like '/abc/def.jsp?WilyCmd=cmdMetrics'.
We also noticed that number of http(s) requests were considerably higher than usual. Basically almost each request url has duplicate request with the above mentioned query parameter appended. This information allowed us to pin-point the newly installed monitoring tool's BRTM Business Transaction Monitoring" feature (which injects Java Script into response header so that it's executed on user's browser to make another request with query parameter 'WilyCmd=cmdMetrics' appended), as primary suspect.  In our case, we would normally see in average 450,000 requests/day, but with this feature in place we recorded 750,000 requests in average per day.
You can read more about BRTM feature here:
https://wiki.ca.com/display/APMDEVOPS98/APM+for+Browser+Response+Time+Monitor

In the above mentioned link, you can also see a short statement about IE, it states, "... The CA BRTM JavaScript is loaded asynchronously so it does not block the loading and execution of any application JavaScript files and other components. However, Internet Explorer (6 through 9) does not wait on asynchronous loads before it generates the load event. This limitation can affect metric generation for Internet Explorer."

From our own investigation, we concluded that BRTM feature caused the above mentioned issues when using certain versions of IE browser. As a work-around, we had to turn off this feature until it is fixed by the vendor. Disabling BRTM feature is easy, you need to assign 'false' to 'introscope.agent.brtm.enabled' property in 'IntroscopeAgent.profile' file and restart the application server.

Lesson learned: until the issue is fixed, make sure you test your web application using all supported browsers before enabling this feature into your production environment.

ArcSight Logger Installation Issue

We were trying to install & configure ArcSight Logger on 64 bit Red Hat Enterprise Linux Server release 6.1 (Santiago) . We encountered strange issue.  We had issue with the versions of Loggers:

• ArcSight-logger-5.3.0.6684.0.bin
• ArcSight-logger-5.3.1.6838.0.bin 

Let me explain below what's the issue was and what custom solution we found after few hours/days of investigation. We had to do investigation ourselves, as Software Vendor did not provide any satisfactory solution, even though we had opened a ticket with them. Their answers were very generic, which did not lead to any solution.

Problem Details:

While installing the ArcSight Logger in console mode (with '-i console' option):

• Logger initialization failed while installing as “root” (we typically become root as 'sudo su -').  Error message, “The initialization of Logger software was unsuccessful. Reason: Platform initializer failed”.
• Logger installation and initialization went Okay, while installing as non-root user like “arcsight”.  Please note, while installing as non-root user, the logger can not listen on privileged SSL port "443" and it cannot create linux service for application. 

 Since, our requirement was logger to be accessible on Port 443, we had to install & initialize it as root user.

Note: Even though, “root” is being used during installation, Arcsight http processes are owned by non-privileged user ‘nobody’ for security reason.

Investigation details/result:

As  we found, in both cases above, the installation itself was successful, but it failed during initialzation. We found that it was failing while executing the script file: <Logger-Install-Dir>/current/arcsight/service/function in the following line:

runAsOtherUser() {

        curr_user=`whoami`

        if [ -f "$CURRENT_HOME/arcsight/service/user.config" -a "X$curr_user" = "Xroot" ]; then

                source $CURRENT_HOME/arcsight/service/user.config

                su -c "$1" $CUSTOM_NONROOT_USER

        else

                eval $1

        fi

}

We got the segmentation fault at this point, as seen in the <Logger-Install-Dir>/current/arcsight/logger/logs/logger_init_driver.log

../service/functions: line 159: 23437 Segmentation fault      (core dumped) su -c "$1" $CUSTOM_NONROOT_USER

and the actual reason seemed to be NOT finding the “NSS_3.12.9” libraries.  As we noticed the below the error message in /var/log/secure log:

su: PAM unable to dlopen(/lib64/security/pam_ldap.so): /opt/arcsight/logger/current/local/nss/lib/libnss3.so: version `NSS_3.12.9' not found (required by /lib64/libldap-2.4.so.2)

su: PAM adding faulty module: /lib64/security/pam_ldap.so

We checked the installed RPM packages related to the nss* on our Logger server, and found that we actually had slightly higher version (nss_3.14.3) installed,. See below:

nss-3.14.3-4.el6_4.x86_64

nss-pam-ldapd-0.7.5-18.2.el6_4.x86_64

nss-softokn-3.14.3-3.el6_4.x86_64

nss-softokn-freebl-3.14.3-3.el6_4.i686

nss-softokn-freebl-3.14.3-3.el6_4.x86_64

nss-sysinit-3.14.3-4.el6_4.x86_64

nss-tools-3.14.3-4.el6_4.x86_64

nss-util-3.14.3-3.el6_4.x86_64

since, it was not failing while using non root user, it's obvious that, nss related code library (located under <Logger-Install-Dir>/current/local/nss/lib/libnss3.so) is being used only while doing 'su *' in the script. And it's also obvious that ArcSight's libnss library code requires (hard coded ???) nss_3.12.9 version of nss libraries installed on the server.

Resolution:

1. I guess, the simple resolution of this problem would be to lower the version of nss libraries to 3.12.9, however, in our case it's not possible because of operation policy not to go with lower version and always maintain the latest version of libraries. Many of your organization may have similar policy in effect, so read below what (work-around ) option you have until software vendor provides the fix:
2. Here’s how we're able to successfully install and configure ArcSight Logger using the ‘root’ user without lowering the version of nss libraries. As previously discussed, the issue seemed to be incompatibility with the nss* libraries (software code seems to be hard-coded to require 3.12.9 version of nss libraries) and required some application specific script changes. 

         Run the installer as root in the console mode. As soon as you see the message, "Begin    Initialization ... The installation of Logger software was successfull ... Initialization will begin after pressing [Enter]...", open another terminal/command window, and remove the entry: <Logger-Install-Dir>/current/local/nss/lib from the LD_LIBRARY_PATH variable. Affected files:

•  /opt/arcsight/logger/current/arcsight/logger/bin/scripts/relative_paths_env.sh
• /opt/arcsight/logger/current/arcsight/service/functions
• /opt/arcsight/logger/current/arcsight/logger/bin/scripts/web.sh

Once, the files updated, save the files.

Hit the [Enter] from the first console window and continue the initialization/configuration.

It seemed that once you remove the nss/lib path entry from the LD_LIBRARY_PATH, the application uses the latest version of nss libraries installed/available on your server.