Showing posts with label Middleware. Show all posts
Showing posts with label Middleware. Show all posts

How to Make your WAS Static Content Deployment Shareable

   Make your static content (html, java script, Cascading Style Sheets (CSS) etc.) deployment on IBM WebSphere Application Server (WAS) shareable among different applications by taking advantage of the feature in IBM WAS's application deployment descriptor extension. This is specifically useful if several applications require access to a set of common static files. You can place these files in a directory (can be network file system) and configure each application so that the content deployed outside (external to Web Application Archive (WAR) is accessible to each application.
   Usually, the best practice (for performance reason) is to deploy the static content in a front-end Web Server (Apache, IBM HTTP Server (IHS), nginx etc). In this case, we don't have to worry about the application deployment extension feature of WAS. However, if you don't have front-end Web Server but still want to deploy the static content separately (outside of a WAR directory) or have to deal with some unique requirements like the one I'm going to describe below, you can utilise the application deployment extension feature of WAS.

   Here is what I am dealing with on one of the migration projects (migrating from front-end IHS to DataPower Gateway).
   In the old environment, IHS served the static content, which was deployed under it's DocumentRoot directory. The WAS plug-in for IHS filtered the requests (URLs) and only passed requests for Servlet or JSP to back-end WAS.
   Now, as part of the migration project, DataPower Gateway will act as a Proxy as well as load balancer. Even though, Datapower can cache the content and serve to end users, but there is no easy mechanism to deploy the static content there. So, it is decided that WAS is going to serve the static content along with Servlet and JSPs. However, the static content will be packaged and deployed separately.
   In addition to that, during the transition period, users must be able to access the application the usual way (request passes through the front-end IHS) as well as the new way (request passes through the front-end Datapower Gateway). This means, IHS should continue to serve the static content for the traffic coming through it and only pass Servlet or JSP related request to back-end WAS.
   Here is the requirements in point form:
  1. User traffic can pass through either one of the  available two distinct routes as shown in the diagram below.
  2. For user requests that pass through front-end DataPower Gateway to WAS, both application and static content are served by WAS. 
  3. For user requests that pass through IHS to WAS, static content is served by IHS and application is served by WAS. 
  4. Static content is packaged and deployed separately.

The diagram shows user traffic coming though two different routes. Static contents deployed and served accordingly. 

In order to fulfil the above mentioned requirements, we need to deploy the static content as follows:
  1. Deploy static content under DocumentRoot directory of IHS and configure WAS plug-in for IHS so that IHS serves static content from within the DocumentRoot directory and only passes requests for dynamic content (like Servelt, JSP etc) to back-end WAS.
  2. Deploy static content outside of the WAR directory (can be network file system) that is accessible to WAS web container. This is achieved through proper configuration in application deployment descriptor extension file, namely ibm-web-ext.xmi or ibm-web-ext.xml files.
The main focus of this post is to explain how to do the 2nd option (i.e. configure the application deployment descriptor extension file) above. However, we'll also do a quick overview of the 1st option. And we'll do it using a real example. We'll use DefaultApplication.ear, that ships with WAS and play with it. First however, let's have quick overview of these extensions.

ibm-web-ext.xmi vs ibm-web-ext.xml

  • ibm-web-ext.xmi: IBM extension for Java Enterprise Edition (EE) application deployment descriptor and compatible with pre-Java EE 5 application deployment
  • ibm-web-ext.xml: IBM extension for Java EE application deployment descriptor and compatible with Java EE 5 or later application deployment
For more details on compatibility, refer to supported configurations paragraph from https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tweb_jspengine.html
extendedDocumentRoot, and fileServingEnabled are two important properties in ibm-web-ext.xmi or ibm-web-ext.xml that need to be enabled and configured properly in order to support the file-serving (by Servlet) when an application requires access to files that exist outside of the application web application archive (WAR) directory.

extendedDocumentRoot - specifies one or more directory paths (comma separated if more than one) outside of WAR from which static files and Java ServerPages (JSP) files can be served. For details, see extendedDocumentRoot section from https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cweb_flserv.html

Note: to serve static files from an extended document root directory, you must enable file serving by assigning 'true' value to fileServingEnabled attribute.
Note: there are some other attributes related to extendedDocumentRoot, however, we are not going to discuss about them in this post. Refer to below URLs for details:
   Now, it's time to go through a real example and understand the concept. As a working example, we take DefaultApplication.ear application that ships with IBM WAS, update ibm-web-ext.xmi file accordingly and deploy its static content outside of the WAR directory. I'll also show an example of WAS plug-in for IHS. DataPower Gateway configuration is out of scope for this blog post.
Note: WAS does not support the modification of deployment descriptor extension parameters through the Administrative Console or wsadmin scripting.
   I've put together an Ant based scripting solution, that can update the file serving and extended document root properties in one or more WAR modules within one or more Enterprise Application Archive (EAR). I'll share the scripting solution as well.

Here are the steps to deploy static content outside of WAR

1. Prepare EAR/WAR file and deploy static content.
Since, DefaultApplication.ear is built as pre-JEE 5 compatible, we'll need to update the  ibm-web-ext.xmi that is located under WEB-INF directory of it's web module DefaultWebApplication.war.

Here is look of DefaultApplication.ear before the changes. As you can see ibm-web-ext.xmi file located under WEB-INF directory has attribute fileServingEnabled="false".
DefaultApplication.ear Archive view

DefaultApplication.ear/DefaultWebApplication.war Archive view




As a part of our update, we'll remove the index.html and banner.gif from DefaultWebApplication.war, update the ibm-web-ext.xmi and deploy the EAR file and static content.

Here is how the updated ibm-web-ext.xmi will look like after the changes:

<?xml version="1.0" encoding="UTF-8"?>
<webappext:WebAppExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" 
  xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi" 
xmlns:commonext.localtran="commonext.localtran.xmi" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmi:id="WebApp_ID_Ext" reloadInterval="3" reloadingEnabled="true" 
fileServingEnabled="true" directoryBrowsingEnabled="false" 
serveServletsByClassnameEnabled="true" preCompileJSPs="false" 
autoRequestEncoding="false" autoResponseEncoding="false">
  <defaultErrorPage xsi:nil="true"/>
  <additionalClassPath xsi:nil="true"/>
  <webApp href="WEB-INF/web.xml#WebApp_ID"/>
  <extendedServlets xmi:id="ServletExtension_1">
    <extendedServlet href="WEB-INF/web.xml#Servlet_1"/>
  </extendedServlets>
  <extendedServlets xmi:id="ServletExtension_2">
    <markupLanguages xmi:id="MarkupLanguage_1" name="HTML" mimeType="text/html" errorPage="Page_1" defaultPage="Page_2">
      <pages xmi:id="Page_2" name="Hello.page" uri="/HelloHTML.jsp"/>
      <pages xmi:id="Page_1" name="Error.page" uri="/HelloHTMLError.jsp"/>
    </markupLanguages>
    <markupLanguages xmi:id="MarkupLanguage_2" name="VXML" mimeType="text/x-vxml" errorPage="Page_3" defaultPage="Page_4">
      <pages xmi:id="Page_4" name="Hello.page" uri="/HelloVXML.jsp"/>
      <pages xmi:id="Page_3" name="Error.page" uri="/HelloVXMLError.jsp"/>
    </markupLanguages>
    <markupLanguages xmi:id="MarkupLanguage_3" name="WML" mimeType="vnd.wap.wml" errorPage="Page_5" defaultPage="Page_6">
      <pages xmi:id="Page_6" name="Hello.page" uri="/HelloWML.jsp"/>
      <pages xmi:id="Page_5" name="Error.page" uri="/HelloWMLError.jsp"/>
    </markupLanguages>
    <extendedServlet href="WEB-INF/web.xml#Servlet_2"/>
  </extendedServlets>
  <extendedServlets xmi:id="ServletExtension_3">
    <extendedServlet href="WEB-INF/web.xml#Servlet_3"/>
    <localTransaction xmi:id="LocalTransaction_1" unresolvedAction="Rollback"/>
  </extendedServlets>
  <fileServingAttributes name="extendedDocumentRoot" value="/opt/ibm/static_contents/" xmi:id="FileServingAttribute_1"/>
</webappext:WebAppExtension>
As seen (highlighted above), fileServingEnabled is changed to "true" and fileServingAttribute "extendedDocumentRoot" has been added with value "/opt/ibm/static_contents"
Once, the ibm-web-ext.xmi is updated, we remove the static content from WAR file and place them under the /opt/ibm/static_contents directory on the server where WAS web container will be able to access them. Recreate the EAR file and Deploy into WAS. Here is how the deployment of static content and EAR looks on server where WAS is running:

Listing of /opt/ibm/static_content:
$> cd /opt/ibm/static_contents
$> ls -la
drwxrwxr-x 2 wasadmin wasadmin 4096 Sep  7 10:28 .
drwxr-xr-x 7 wasadmin wasadmin 4096 Sep  7 10:27 ..
-rw-r--r-- 1 wasadmin wasadmin 3798 Sep  7 10:28 banner.gif
-rw-r--r-- 1 wasadmin wasadmin  667 Sep  7 10:28 index.html

Listing under Deployed DefaultApplication.ear.ear/DefaultWebApplication.war. As you see, static contents are not there.
$> cd /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/installedApps/ubuntuwas9Cell01/DefaultApplication.ear.ear/DefaultWebApplication.war $> ls -la
drwxr-xr-x 4 wasadmin wasadmin 4096 Sep  7 11:03 .
drwxr-xr-x 5 wasadmin wasadmin 4096 Sep  7 11:03 ..
-rw-r--r-- 1 wasadmin wasadmin  742 Aug 20  2015 auth_error.jsp
-rw-r--r-- 1 wasadmin wasadmin  393 Aug 20  2015 HelloHTMLError.jsp
-rw-r--r-- 1 wasadmin wasadmin  849 Aug 20  2015 HelloHTML.jsp
-rw-r--r-- 1 wasadmin wasadmin  461 Aug 20  2015 HelloVXMLError.jsp
-rw-r--r-- 1 wasadmin wasadmin  365 Aug 20  2015 HelloVXML.jsp
-rw-r--r-- 1 wasadmin wasadmin  525 Aug 20  2015 HelloWMLError.jsp
-rw-r--r-- 1 wasadmin wasadmin  476 Aug 20  2015 HelloWML.jsp
-rw-r--r-- 1 wasadmin wasadmin 3284 Aug 20  2015 HitCount.jsp
drwxr-xr-x 2 wasadmin wasadmin 4096 Sep  7 11:03 META-INF
drwxr-xr-x 4 wasadmin wasadmin 4096 Sep  7 11:03 WEB-INF     

Note: If you want to change the ibm-web-ext.xmi or ibm-web-ext.xml file of pre-deployed/running application manually for testing purpose, you need to do it in two locations:
  1. under <was-install-dir>/profiles/<profile-name>/installedApps/<cell-name>/<app-ear-dir>/<app-war-dir>/WEB-INF
  2. under <was-install-dir>/profiles/<profile-name>/config/cells/<cell-name>/applications/<app-ear-dir>/deployments/<app-dir>/<app-war-dir>/WEB-INF
Also, note that if it is network deployment environment, then as soon as you do synchronisation from master configuration repository, your manual changes will be overwritten.

2. Test and Verify
Let's access our default application using the URL: http://<host>:<port>/index.html. As long as we can see the index page with the listing of Snoop, Hello and Hitcount servlets, it is safe to say that the static content is served from the directory defined by extendedDocRoot property of fileServingAttributes, because our repackaged and redeployed application does not contain any static content. However, there is another (more official) way to do validate it as well.
   We can enable WAS WebContainer Trace to confirm where static content is served from. Here is the trace spec. Detail about the WebContainer trace can be found here.

*=info:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all

You can enable the run-time trace as follows:
Launch and access the WAS administrative console and navigate to:

  • Troubleshooting > Logging and tracing > <server-name> Change log detail levels
  • Select "Runtime" tab 
  • Replace the existing "*=info" with the above mentioned trace spec into "Select components and specify a log detail level. ..." text box
  • Click "OK"
There is no need to restart the server. Once the trace is enabled, type the application URL (http://<host>:<port>/index.html) again on your browser and access it. As soon as the index page is rendedered on the browser, look the trace file for your WAS. You'll see something like below:
[9/7/17 11:14:06:498 EDT] 000000a0 servlet       1 com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl handleRequest   request--->/index.html<---
[9/7/17 11:14:06:498 EDT] 000000a0 SRTServletReq 1 com.ibm.ws.webcontainer.srt.SRTServletRequest getWebAppDispatcherContext
[9/7/17 11:14:06:498 EDT] 000000a0 SRTServletReq 1 com.ibm.ws.webcontainer.srt.SRTServletRequest getAttribute this->com.ibm.ws.webcontainer.srt.SRTServletRequest@178b0ff6:  name --> javax.servlet.include.request_uri
[9/7/17 11:14:06:498 EDT] 000000a0 servlet       1 com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl handleRequest relative uri -->[/index.html]
[9/7/17 11:14:06:498 EDT] 000000a0 servlet       1 com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl handleRequest Request path=[/index.html]
[9/7/17 11:14:06:498 EDT] 000000a0 util          > com.ibm.wsspi.webcontainer.util.FileSystem uriCaseCheck ENTRY file canpath=/opt/ibm/static_contents/index.html, matchString=/index.html, checkWEBINF=true
[9/7/17 11:14:06:498 EDT] 000000a0 util          < com.ibm.wsspi.webcontainer.util.FileSystem uriCaseCheck : result=true RETURN
[9/7/17 11:14:06:499 EDT] 000000a0 servlet       > com.ibm.ws.webcontainer.servlet.FileServletWrapper handleRequest com.ibm.ws.webcontainer.servlet.StaticFileServletWrapperImpl@11621b65 ,request-> com.ibm.ws.webcontainer.srt.SRTServletRequest@178b0ff6 ,response-> com.ibm.ws.webcontainer.srt.SRTServletResponse@d3200815 ENTRY
[9/7/17 11:14:06:499 EDT] 000000a0 SRTServletReq 1 com.ibm.ws.webcontainer.srt.SRTServletRequest getRequestURI  uri --> /index.html
[9/7/17 11:14:06:499 EDT] 000000a0 servlet       1 com.ibm.ws.webcontainer.servlet.FileServletWrapper handleRequest   request--->/index.html<---     

As you can see from the highlighted (in yellow) text above, WAS is able to match and route /index.html to /opt/ibm/static_contents/index.html

Serving Static Content From IHS

Since, it is a very general type of deployment, I'm not going to elaborate this topic in detail, but just highlight few things in the context of our requirements above. 
As per the requirement above, users whose application traffic passes through IHS, the static content is served by IHS.  
Below are the steps:

1) Copy/Deploy static content under directory defined by DocumentRoot in httpd.conf.

Listing of /opt/ibm/HTTPServer/htdocs:
$> cd /opt/ibm/HTTPServer/htdocs
$> ls -la
drwxrwxr-x 2 wasadmin wasadmin 4096 Sep  7 10:28 .
drwxr-xr-x 7 wasadmin wasadmin 4096 Sep  7 10:27 ..
-rw-r--r-- 1 wasadmin wasadmin 3798 Sep  7 10:28 banner.gif
-rw-r--r-- 1 wasadmin wasadmin  667 Sep  7 10:28 index.html

Now, we need to make sure that the WAS plug-in for IHS filters the request correctly so that static content is served directly from IHS.

Fragment of (original) generated plugin-cfg.xml, which passes every request to back-end WAS.
<UriGroup Name="default_host_sso_clus_URIs">
   <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/*"/>
</UriGroup>  

Here is a fragment of modified plugin-cfg.xml, which filters the request.  It enforces IHS to serve the content not explicitly listed in given UriGroup.
<UriGroup Name="default_host_sso_clus_URIs">
   <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/snoop/*"/>
   <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/hello"/>
   <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/hitcount"/>
</UriGroup>

2. Test and Verify

Now, let's access our application. This time, we'll use IHS's host and port. IHS in this case is listening on port 80. So, I access it using http://<host>/index.html
Since, I've enabled the trace in my plugin-cfg.xml file, I can see that the index.html and banner.gif is served from IHS itself. See below:

[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='localhost'; uri='/index.html'
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - DEBUG: ws_common: websphereShouldHandleRequest: NOT config->odrEnabled(reqInfo(ef699b0))
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - TRACE: ws_common: webspherePortNumberForMatching: Using logical.
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - TRACE: ws_common: websphereVhostMatch: Comparing '*:80' to 'localhost:80' in VhostGroup: default_host
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - TRACE: ws_common: websphereVhostMatch: Found a match '*:80' to 'localhost:80' in VhostGroup: default_host with score 1, exact match 0
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9081' to 'localhost:80' in VhostGroup: default_host
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9080' to 'localhost:80' in VhostGroup: default_host
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - TRACE: ws_common: websphereUriMatch: Failed to match: /index.html
[Thu Sep 07 14:19:13 2017] 00007774 00009d24 - DETAIL: ws_common: websphereShouldHandleRequest: No route found  
....
....
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='localhost'; uri='/banner.gif'
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - DEBUG: ws_common: websphereShouldHandleRequest: NOT config->odrEnabled(reqInfo(ef679f8))
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - TRACE: ws_common: webspherePortNumberForMatching: Using logical.
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - TRACE: ws_common: websphereVhostMatch: Comparing '*:80' to 'localhost:80' in VhostGroup: default_host
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - TRACE: ws_common: websphereVhostMatch: Found a match '*:80' to 'localhost:80' in VhostGroup: default_host with score 1, exact match 0
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9081' to 'localhost:80' in VhostGroup: default_host
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9080' to 'localhost:80' in VhostGroup: default_host
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - TRACE: ws_common: websphereUriMatch: Failed to match: /banner.gif
[Thu Sep 07 14:19:13 2017] 00007774 0000a368 - DETAIL: ws_common: websphereShouldHandleRequest: No route found   
As you can see from the highlighted text above it has failed to match the URI /index.html and /banner.gif  as expected. However, the browser is able to render the page, means the content is served by IHS.

Above, I explained the concept of extended document root and how to use it in certain scenarios. As I mentioned above, you need to update the IBM extension file for application deployment descriptor to utilise the extended document root feature, however, WAS does not support the modification of deployment descriptor extension parameters through the Administrative Console or wsadmin scripting, so you have to do it manually. It is OK to manually update, if it is just a one or two web modules, what if you have 10 or hundreds of web modules? I have put together Ant based script just to do that. You can get everything that is required to run the script from GitLab project here:  https://gitlab.com/pppoudel/public_shared/tree/master/extdocroot
You can run the script either using extDocRootUpdater.sh (Unix) or extDocRootUpdater.cmd (Windows). Included README.txt explains some property values that you need to update as per your environment. Here is the content of the README.txt

Purpose:
Use this script to update EAR/WAR file(s) to enable the file serving and add extendedDocumentRoot feature. 
Script updates ibm-web-ext.xml, and/or ibm-web-ext.xmi file(s) in all Web modules inside each EAR file.

Setup:
Script is written using Apache Ant framework. So, it requires Apache Ant and few additional libraries found under ${basedir}/lib
1) ant-contrib-1.0b4.jar - Refer to https://sourceforge.net/projects/ant-contrib/files/ant-contrib/1.0b3/
2) xmltask-1.16c v1.16.jar - Refer to http://www.oopsconsultancy.com/software/xmltask/

Property file: pkgupdate.properties
Open it and update the value of following properties as necessary:

1) installableapps.loc
It is the location where you put all your EAR/WAR files to be updated.  
installableapps.loc=C:/IBM/installableApps

4) failOnError
If failOnError is true, it stops processing further even one update fails. If you want to continue the processing assign 'false' value.
For example:
failOnError=false

5) ext.doc.root, and append.ctxroot
It is a runtime property. The value will be used to formulate the value for extendedDocumentRoot. Here is how the value is formed.
if 'append.ctxroot=true'; then
extendedDocumentRoot=${ext.doc.root}+<root-context>
Note: root-context value is extracted by script from the application.xml
For example:
ext.doc.root=/opt/ibm/static_contents
and root-context=/myapp
then
extendedDocumentRoot=/opt/ibm/static_contents/myapp

if 'append.ctxroot=false'; then
extendedDocumentRoot=${ext.doc.root}

How to run:
> Set JAVA_HOME and ANT_HOME appropriately in extDocRootUpdater.sh or extDocRootUpdater.cmd then 
> launch command prompt (windows) or Shell (Unix/Linux)
> cd to extdocroot directory
> execute extDocRootUpdater.sh or extDocRootUpdater.cmd     

Hope, you found this blog post useful while deciding how to manage and deploy static content for your WAS based application serving platform.

DataPower Front-End for WAS Cluster with Load Balancing and SSO

In this blog post, I'm guiding through simple and easy to follow steps to create a Proof of Concept (POC) environment with Datapower (DP) as font-end and  application serving environment hosted on WebSphere Application Server (WAS) based cluster with Single Sign On (SSO) enabled. DP will also load-balance requests to WAS cluster members. For details of DP supported features as front-end, refer to https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/cwsv_topology.html

Assumptions:
  1. In order for you to follow the steps outlined below, you have good working knowledge of IBM WAS and DataPower.
  2. You have WAS ND (version 8.5.X), and IBM DataPower Gateway (version 7.5.X) installed and ready to create Application Serving environment.  For your reference, I'm using the following:
    •    WAS 8.5.5.7 Network Deployment.
    •    IBM DataPower Gateway Virtual Edition (Firmware: IDG.7.5.1.0).
In a nutshell, here is what we'll do as part of this exercise:

On WAS side:

  • Create a cluster with two members (application servers).
  • Create security domain and assign scope to the cluster level
  • Create and configure an external trusted realm for the security domain. It is because, we will not have any user registry configured (for Application authentication) and we'll trust identifies authenticated by DP.
  • Enable Application security on the security domain
  • Deploy a sample application on the WAS cluster
  • Map security roles and authorize any user belonging to External Trusted Realm.
  • Configure LTPA, export LTPA key and customize (as we are using External Trusted Realm) it.
  • Export SSL certificate.

On DP side:

  • Import LTPA key, import WAS SSL certificate.
  • Create DP objects (including Crypto, SSL profiles, Password aliase etc).
  • Create Load Balancer Group
  • Create Multi-Protocol Gateway (MPGW) service etc.

Let's start working on WAS side:

Note:
Once you make configuration changes on WAS either using WAS admin console or wsadmin script,
you need to save the changes and synchronise the changes with the nodes (in Network Deployment (ND). Since it's essential part for every change or set of changes, I'm not going to list these two steps after every change below. But you need to make sure to save and synchronise. You can use
the wsadmin Jython script below to save and synchronize the changes with nodes:
# Save the configuration/change
AdminConfig.save()
# Synchronise the changes with the nodes
dmgrObj=AdminControl.queryNames('WebSphere:type=DeploymentManager,*')
AdminControl.invoke(dmgrObj, 'multiSync', '[false]', '[java.lang.Boolean]')

1.0) Create a WAS cluster containing two members. In this example, we will create a WAS cluster named "sso_clus" with two members "ssoSrv1" and "ssoSrv2". For more info on creating WAS cluster, refer to https://www.ibm.com/support/knowledgecenter/SSRMWJ_6.0.0.10/com.ibm.isim.doc/installing/tsk/tsk_ic_ins_was_85_cluster.htm
Note: You may need to change text in italic as per your configuration.
# Using WAS Admin Console: Servers --> Clusters --> WebSphere application server clusters
# Using wsadmin script:
AdminTask.createCluster('[-clusterConfig [-clusterName sso_clus -preferLocal true]]')

# Add 1st cluster member 'ssoSrv1' using default server template
AdminTask.createClusterMember('[-clusterName sso_clus -memberConfig [-memberNode myWASHostNode01 -memberName ssoSrv1 -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateName default -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope cluster]]')

# Add  2nd cluster member 'ssoSrv2'
AdminTask.createClusterMember('[-clusterName sso_clus -memberConfig [-memberNode myWASHostNode01 -memberName ssoSrv2 -memberWeight 2 -genUniquePorts true -replicatorEntry false]]')

2.0) Enable SSO (if not already). Here you need to know request(s) from what domain(s) you are supporting, you can have multiple domains defined here. For detail how domain works refer to: https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tsec_msso.html. For this example, we are going to use 'UseDomainFromURL'. See the above URL for details.
# Configure Single Sign On (SSO)
AdminTask.configureSingleSignon('-enable true -requiresSSL true -domainName UseDomainFromURL -attributePropagation true')

# Make sure LTPA is active authentication mechanism:
AdminTask.setActiveAuthMechanism('-authMechanismType LTPA')

3.0) Disable automatic LTPA key generation option as follows:
  Access WAS Admin console and navigate to Security > Global security > LTPA
   > Click on link "Key set groups"
   > Click on link "CellLTPAKeySetGroup"
   > Uncheck "Automatically generate keys"
  Apply, save and synchronise.

4.0) Create security domain. Creating security domain is encouraged rather than modifying the Global security configuration, because, you may have other application serving environment (s) managed under this Cell and they may rely on Global security and any change may affect them.
For this example, we name our new security domain 'mySSOLTPADomain' and we'll use the Global security as template (basically copying the Global security) to create the new security domain and modifying few things on newly created security domain:
Using WAS Admin Console: Security > Global security > Security domains > New...
# Using wsadmin script:
AdminTask.copySecurityDomainFromGlobalSecurity('-securityDomainName mySSOLTPADomain')


5.0) Customise the newly created security domain as per our requirement.
5.1) Since, we will not have any user registry configured on WAS for application authentication, and we'll trust the credential authenticated by DP, we'll create a trusted external realm called 'DatapowerRealm' and add that to newly created security domain. For detail about trusted external realm, Refer to https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tsec_sec_realm_trust.html
Using WAS Admin console: Security domains > mySSOLTPADomain > Federated repositories > Trusted authentication realms - inbound > New...
Using wsadmin script:
AdminTask.addTrustedRealms('[-securityDomainName mySSOLTPADomain -communicationType inbound -realmList DatapowerRealm]')
# And only trust the realm(s) that is explicitly indicated (not all)
Using WAS Console: Security domains > mySSOLTPADomain > Federated repositories > Trusted authentication realms - inbound
Using wsadmin script:
AdminTask.configureTrustedRealms('[-communicationType inbound -securityDomainName mySSOLTPADomain -trustAllRealms false]')
Using WAS Admin console: Security domains > mySSOLTPADomain
Using wsadmin script:
AdminTask.configureAppWIMUserRegistry('[-securityDomainName mySSOLTPADomain -verifyRegistry true ]')
# Enable Application Security
Using WAS Admin console: Security domains > mySSOLTPADomain > Application security > Customize for this domain > check Enable Application security
Using wsadmin script:
AdminTask.setAppActiveSecuritySettings('[-securityDomainName mySSOLTPADomain -appSecurityEnabled true]')
# Here, we are also increasing the LTPA timeout
Using WAS Console: Security domains > mySSOLTPADomain > Authentication Mechanism Attributes > Customize for this domain
Using wsadmin script:
AdminTask.setLTPATimeout('[-securityDomainName mySSOLTPADomain -timeout 220 ]')
# And, we are going to assign the scope of this security domain to cluster 'sso_clus' scope
Using WAS Admin console: Security domains > mySSOLTPADomain
Using wsadmin script:
AdminTask.mapResourceToSecurityDomain('[-securityDomainName mySSOLTPADomain -resourceName Cell=:ServerCluster=sso_clus]')

 
6.0) Once you finish setting the security related configuration, you can run the following command to get information related to active security domain.
Using wsadmin script:
AdminTask.getActiveSecuritySettings('-securityDomainName testDomain')

7.0) Deploy the application. For this exercise, we'll be using DefaultApplication that comes with WAS installation. Below command installs the app on sso_clus.
 Note: for clarity, the below command is broken into multiple lines. Run as a single line.
# AdminApp.install('/apps/IBM/WebSphereND/AppServer/installableApps/DefaultApplication.ear',
   '[-distributeApp -appname DefaultSSOApplication -createMBeansForResources -cluster sso_clus -MapModulesToServers [[ "Increment EJB module" Increment.jar,META-INF/ejb-jar.xml WebSphere:cluster=sso_clus ][ "Default Web Application" DefaultWebApplication.war,WEB-INF/web.xml WebSphere:cluster=sso_clus ]]]')
 
8.0) Security role to user/group mapping:
As outlined in the beginning, for this POC , we will map "All Role" to Special subject, which is "All Authenticated in Trusted Realms". Follow the step below using WebSphere Admin console:
Navigate to Applications > Application Types > WebSphere enterprise application
Select DefaultSSOApplication.ear > Security role to user/group mapping.
Select "All Role" and Select "All Authenticated in Trusted Realms" from Map Special Subjects drop down list. and apply, save and synchronise.

9.0) Export LTPA key:
# Using WAS console: Security > Global security > LTPA
# Using wsadmin script:
AdminTask.exportLTPAKeys('[-ltpaKeyFile file:/tmp/Myltpa.key -password Your-Password ]')

10.0) Since, we are using external trusted realm, you need to open (open it in VI like text editor, so that it does not insert any unwanted character) the exported LTPA key and modify the following:
# Original:
com.ibm.websphere.ltpa.Realm=defaultWIMFileBasedRealm
# Updated:
com.ibm.websphere.ltpa.Realm=DatapowerRealm

11.0) Install ODCInfo application. ODCInfo application will be deployed and run on WAS Deployment Manager server. It captures the WAS cluster information and DP will be able to retrieve it dynamically and load balance the traffic. You can download it from IBM Fix Central. Search for "ResourceKit.710" and download the ResourceKit.710.zip. You need to logon using your IBM credentials to download it. For details on ODCInfo application installation and configuration, refer to https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.1.0/com.ibm.dp.doc/lbg_enablingretrievalwlminfo.html
11.1) Extract ResourceKit.710.zip on Deploymen Manager server. In this example, I've extracted it under ~/ResourceKit.
11.2) Installing the OSGi bundle. Basically copy the com.ibm.datapower.odc.osgi.jar to plugins directory of WAS.
Note: my WAS_HOME is /apps/IBM/WebSphere/AppServer

$>cp ~/ResourceKit/AO/com.ibm.datapower.odc.osgi/dist/com.ibm.datapower.odc.osgi.jar /apps/IBM/WebSphere/AppServer/plugins

11.3) Run following command to clear and recreate the OSGi cache. Details: https://www.ibm.com/support/knowledgecenter/en/SSAW57_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rxml_osgicfginit_script.html
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./osgiCfgInit.sh
   OSGi profile cache successfully cleaned for /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile.
   OSGi server cache successfully cleaned for /apps/IBM/WebSphere/AppServer/profiles/<myDmgrProfile>/servers/dmgr.
# Start the OSGI console (optional):
$>./osgiConsole.sh
#(Optional) run the OSGI diagnostic
diag com.ibm.datapower.odc.osgi
# Verify that a message states: No unresolved constraints.
$>cd /apps/IBM/WebSphere/AppServer/profiles/<myDmgrProfile>/bin
$>./osgiConsole.sh
   osgi> diag com.ibm.datapower.odc.osgi
   websphere@plugins/com.ibm.datapower.odc.osgi.jar [5]
  No unresolved constraints.

11.4) Install ODCInfo Web application and verify:
# ./wsadmin.sh -f script_path/ODCInfoDeploy.jacl dmgr_server_name dmgr_node_name path_to_war_file ODCInfo
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./wsadmin.sh -f ~/ResourceKit/AO/scripts/ODCInfoDeploy.jacl dmgr myDmgrNode ~/ResourceKit/AO/com.ibm.datapower.odc/ODCInfo_ND61.war ODCInfo
   ...
   ADMA5013I: Application ODCInfo installed successfully.
   ODCInfoDeploy: saving the configuration


# Verify installation:
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./wsadmin.sh -f ~/ResourceKit/AO/scripts/ODCInfoCheckInstall.jacl myCell01 dmgr ODCInfo
    ...
   ADMA5072I: Distribution status check completed for application ODCInfo.
   Application ODCInfo is deployed successfully.
 
# Start the ODCInfo application:
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./wsadmin.sh -f ~/ResourceKit/AO/scripts/ODCInfoStart.jacl myCell01 myDmgrNode ODCInfo
   ...
   WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: ...
   ODCInfoStart.jacl: ODCInfo started
 
11.5) Verify that the ODCInfo is able to retrieve the cluster info:
   http://<DMGR_IP>:<DMGR_HTTP_PORT>/ODCInfo/ODCInfo?c=sso_clus
Important note:
If you are not able to access the cluster info using the above mentioned URL, it’s most probably because of following reason.
By default ‘dmgr’ server’s both (secured, and non-secured) webcontainer transport ports are assigned as host aliases under ‘admin_host’ virtual host. However by default, ODCInfo uses ‘default_host’ virtual host as it cannot use ‘admin_host’, what you need to do is remove non-secured webcontainer port of dmgr from ‘admin_host’ virtual host, and add it as an alias to ‘default_host’virtual host. Save & synchronise the configuration and restart the Deployment Manager.

12.0) Extract SSL certificate from WAS.For details on SSL key related wsadmin commands, refer to: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rxml_atpersonalcert.html#rxml_atpersonalcert__cmd5

# Here is how you can export the Cell or node cert from WAS:
# List available keystores using wsadmin script:
AdminTask.listKeyStores('[-all true -keyStoreUsage SSLKeys ]')

# List personal certificates:
AdminTask.listPersonalCertificates('[-keyStoreName NodeDefaultKeyStore -keyStoreScope (cell):myCell01:(node):myWASNode01 ]')

# Extract the certificates
# Using WAS Admin console: SSL certificate and key management > Key stores and certificates > NodeDefaultKeyStore > Personal certificates > Extract certificate

# Using wsadmin script:
AdminTask.extractCertificate('[-certificateFilePath /tmp/localWASNodeCertKS.crt -base64Encoded true -certificateAlias default -keyStoreName NodeDefaultKeyStore -keyStoreScope (cell):myCell01:(node):myWASNode01 ]')

Steps to be performed on DP side:


1.0) Upload the LTPA key to DP:
 You can use DP CLI scp command. In following example, I'm copying MySSOLTPA.key located at /tmp directory on MyWAShost to Datapower cert: folder.
Note: make sure you are in config mode in Datapower while running scp command.
copy scp://<username>@<MyWAShost>//tmp/MySSOLTPA.key cert:///MySSOLTPA.key
For detail on using scp/sftp commands from DP CLI, refer to: http://www-01.ibm.com/support/docview.wss?uid=swg21250655

2.0) Upload/import SSL cert  (extracted from WAS) to DP. Use CLI option as explained in step #1.0.

3.0) Configure password alias for LTPA key using DP Console.
 Navigate to Objects > Configuration Management > Policy> Password Map Alias
Click "Add" button
 Name: mySSOLTPAPWAlias
Password: <your password>
Note: Make sure this password matches exactly to the password that you used while exporting the    LTPA key from WAS
 Apply and Save Configuration.

4.0) Generate Crypto Key to be used as server identity and server certificate (we will use self-signed for this POC) in front side handler.
Navigate to Administration --> Miscellaneous > Crypto Tools with options:
key type: RSA
Generate Self-Signed Certificate
Generate Key and Certificate Objects
Object Name: 'mySSOCryptoKeyObj' using the Crypto tool
For password alias, you can use the same one that was created for LTPA key or create new one.
For details, refer to: https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.5.0/com.ibm.dp.doc/cryptotool_generatingkeyscertificates.html

Here is the execution result:
Action completed successfully.
     Generated private key in "cert:///mySSOCryptoKeyObj-privkey.pem"
     Generated Certificate Signing Request in "temporary:///mySSOCryptoKeyObj.csr"
     Generated Self-Signed Certificate in "cert:///mySSOCryptoKeyObj-sscert.pem" and exported a copy in "temporary:///mySSOCryptoKeyObj-sscert.pem"
     Generated a Crypto Key object named "mySSOCryptoKeyObj" and a Crypto Certificate object named "mySSOCryptoKeyObj"
 
5.0) Create/Configure Crypto certificate:
Navigate to Objects > Crypto Certificate
Click "Add"
Name: LocalWASCert
File Name: localWASNodeCert.crt
Note: for file name, make sure to select the file that was extracted from WAS and imported to DP.
Password Alias: none

6.0) Create SSL Client Profile:
Objects > Crypto Configuration > SSL Client Profile
Click "Add"
Name: mySSOWASSSLClntProfile
In the Credential section, you don't need Identification credentials as this SSL client profile is only be used to validate certificate presented by WAS during secure back-end communication.
enable "Validate server certificate"
Validation credential click on "+" button and create Crypto Validation Credentials
Name: LocalWASCertValCred
Certificate: LocalWASCert
 
7.0) Create SSL Server Profile.
Navigate to Objects > Crypto Configuration > SSL Server Profile
Click on "Add" button.
Name: myLTPASSOSrvProfile
Since, it's not participating into mutual SSL auth with client, disable "Request client authentication" option.
   In Identification credential, click on "+"
   Create/Configure Identification Credential:
   Name:myDPIdCred
   Crypto Key: mySSOCryptoKeyObj
   Note: mySSOCryptoKeyObj was created as part of the crypto key generation using Crypto tool in previous step.
 
8.0) Create Load Blancer Group
Navigate to Object > Network Settings > Load Balancer Group and Add new LB Group.
For this POC, we name it as "myWASLBG"
Select "on" for "Retrieve Workload Management Information"
Workload Management Retrieval: WebSphere Cell
WebSphere Cell: click on "+" sign and provide name and enter WAS Deployment Manager information
Notes:

  • "Deployment Manager Port number" should be the one that is part of the 'default_host' virtual host in WAS. In this POC, we used the non-secured (http) port.
  • SSL Profile - can be ignored in this screen as the WAS cluster information will be retrieved through non SSL port.

Back on the Load Balancer Group configuration page, the "Workload Management Group Name" should be the actual WAS cluster name. For this POC, we have created "sso_clus" WAS cluster, and will enter it here. Protocol: HTTPS.

Note: protocol in this page, identifies the actual (application related) communication between DP and WAS.
 
9.0) Verify the Load Balancer Group status:
Navigate to Status > IP-Network > Load Balancer
   Make sure your Load Balancer Group listed with WAS cluster members.
 
10.0) Create Multi-Protocol Gateway (MPGW) service, define policy etc... In order to make this simple, for this POC, we use template provided by IBM.
10.1) Navigate to "Blueprint Console" and click on "Patterns" symbol on left hand navigation panel.
10.2) Click on "Web application with Form-based authentication and authorization" link.
10.3) Read the information and instruction carefully. It explains in detail here how the DP policies are setup for this template and how it's going to work once deployed. Once you understand, Click on "Deploy..." button.
It'll prompt for few inputs:
  •    Service name: myFormBasedLTPASSO
  •    Back-end end point details: https://myWASLBG
  •    Note: "myWASLBG" is the name of Load Balancer Group that we created in step #8.0.
  •    SSL Client Profile: mySSOWASSSLClntProfile
  •    Note: "mySSOWASSSLClntProfile" is the name of SSL Client Profile that we created in step #6.0 .
  •    Front-end end point details: choose the IP for front end or leave default and choose port that is available
  •    SSL Server type: Server Profile
  •    SSL Server Profile: myLTPASSOSrvProfile
  •    Note: myLTPASSOSrvProfile is the name of SSL Server Profile that we created in step # 7.
  •    For step 4 (Authenticate with LTPA), and step 5 (Generated LTPA Token),
  •    For LTPA key file: MySSOLTPA.key
  •    For Key file password alias:mySSOLTPAPWAlias
Note:
  • MySSOLTPA.key was exported from WAS, customised, and imported to DP.
  • mySSOLTPAPWAlias was created in step #3.0.
Once all the information entered correctly, click on "Deploy Pattern" button. It may take a minute or so to fully deploy the pattern.

11.0) Once, it is successfully deployed, you'll see the "myFormBasedLTPASSO" under Multi-Protocol Gateway Service list. You need to change few things:
11.1) Navigate to "Multi-Protocol Gateway" and click on the MPGW gateway link just created. Look for the "XML Manager" setting, most probably, by default, the "default" XML Manager has been set.
Click on the '+' sign to create another XML Manager with Load Balancer Group.
Provide the name for XML Manager and select the LB group "myWASLBG" that you've created in step #8.
11.2) Also, by default, DP uses "AAAInfo.xml" as a credential storage. If you need to add/edit user, password or credential mapping information, you can edit this file or use another one.

12.0) Save the configuration and you're ready to test.

Testing:

1.0) Make sure WAS cluster is up and all DP objects and services running.
2.0) Access the page: https://<DP_Front_Side_Handler_IP>:<DP_Front_Side_Handler_Port>/snoop
   you'll be prompted for Username and Password.

   Note: Use one of the usernames and corresponding password listed in AAAInfo.xml.
   Once login is successful, you'll get the snoop page served by one of the WAS cluster member.

3.0) Verify the DP Load Balancer configuration is working properly:
   > Each time, you access the snoop page, look at the "Server name" and "Server Port" and make sure you're hitting all cluster members.

Troubleshooting:

1.0) Your SSO is not working (prompted for credential again) and you see following message in WAS SystemOut.log:

[23/05/17 14:35:14:841 EDT] 000000a0 LTPAServerObj W   SECJ0371W: Validation of the LTPA token failed because the token expired with the following info: Token expiration Date: Tue May 23 10:44:58 EDT 2017, current Date: Tue May 23 14:35:14 EDT 2017 Token attributes:  username=user:<realm>/<user>.. This warning might indicate expected behavior. Please refer to technote at http://www-01.ibm.com/support/docview.wss?uid=swg21594981.

Resolution:
It can be caused by few things, and one of them is time not in synchronized between DP and WAS servers. So, first thing you want to check and make sure is the time setting in DP and WAS server are in sync.

2.0) Looked like your SSO with LTPA is working (because you see your request hitting the WAS server(s), but you can't see intended page and you see the authorisation error in browser and WAS SystemOut.log shows the following:

[23/05/17 15:10:54:474 EDT] 000000a0 WebCollaborat A   SECJ0129E: Authorization failed for user user:<realm>/<user>:<realm> while invoking GET on default_host://snoop, Authorization failed, Not granted any of the required roles: All Role

Resolution:
It is most probably, because the application security has been enabled, but no User role assigned.
On WAS Admin console, navigate to "Applications > Application Types > WebSphere enterprise applications"
Double click on the application link .
Click on "Security role to user/group mapping"
When role mapping page appears, do one of the following based on your requirement:
   1) Map either User or Group for the role.
   2) Map Special Subjects, for the example, since we are trusting the users authenticated by DP and we've created trusted external realm, we'll select option "All Authenticated in Trusted Realms" option.
Save and Synchronise.
And try your test again.

Hope, you'll find this post helpful !!!

Few Tips on Secure vs. Non-Secure WAS Web Server Plug-in Connection

In this blog, I am focusing only on one aspect of WebSphere Application Server (WAS) Web Server plug-in to Application Server connection i.e. secure vs. non-secure, what has changed in recent versions of WAS and any tips and tricks.
By default, when plug-in configuration file (plugin-cfg.xml) is generated, it creates configuration for both secure (HTTPS) and non-secure (HTTP) transport channels for the communication to
the application server(s). Below is fragment of plugin-cfg.xml:
...
<ServerCluster CloneSeparatorChange="false" ... >
      <Server ConnectTimeout="0" ExtendedHandshake="false" ...>
<!-- non-secure (HTTP) transport channel -->
             <Transport Hostname="MyHost" Port="9080" Protocol="http"/>
<!-- secure (HTTPS) transport channel -->
<Transport Hostname="MyHost" Port="9443" Protocol="https">
                   <p
roperty Name="keyring" Value="c:\IBM\Plugins\plugin-key.kdb"/>
                   <Property Name="stashfile" Value="c:\IBM\Plugins\plugin-key.sth"/>
              </Transport>

       </Server>
  </ServerCluster>
...

When you have two transport channels defined, a lot of time it creates confusion because people may not know which communication channel is actually going to be used or whether they really need both channels in their configuration?
When you have both channels defined (just like shown in the example above), if the incoming traffic is secure (HTTPS), it automatically chooses the secure channel (MyHost:9443 in example) to create connection to the back-end application server, but if the incoming traffic is non-secure (HTTP), it by default chooses non-secure (HTTP) (MyHost:9080 in example).

  • What if reverse case scenario? 
  • Or only one type of back-end connection is defined/available and opposite type of incoming traffic is encountered? 

Looks like there are some recent changes and gotcha here.


Incoming secure (HTTPS) traffic and and only non-secure (HTTP) transport channel defined/available:

  • version 8.5.5.0 and later:


 In version 8.5.5.0 or latter, in this particular case, plug-in won't create any connection to the    application server, because it interprets this situation as a security risk and request fails. If plugin trace is enabled, you'll see something like below in the plugin.log:

[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereFindTransport: Nosecure transports available
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find a transport
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ESI: getResponse: failed to get response: rc = 4
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - DEBUG: ESI: esiHandleRequest: failed to get response
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - DEBUG: ESI: esiRequestUrlStackDestroy
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereHandleRequest: Failed to handle request

However, you can use the plug-in custom property UseInsecure=true in the plugin-cfg.xml file; In this case plugin will use non-secure (HTTP) transport channel to establish connection to the application server despite the secure incoming request. You can add custom property in two wasys:
1. You can directly modify the plugin-cfg.xml as follows:
<Config ASDisableNagle="false" AcceptAllContent="true" AppServerPortPreference="HostHeader" ... UseInsecure="true">
...
</Config>
2. Or add this property through WebSphere Administration Console (Servers > Web Servers > Web_server_name > Plug-in properties > Custom properties page) and regenerate the plugin-cfg.xml.

Once the UseInsecure=true custom property becomes effective, the above mentioned scenario can create connection successfully. Below are some relevant lines from plugin.log (trace enabled)

[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - TRACE: ws_common: websphereFindTransport: Finding the transport for server 021313E0
websphereFindTransport: Setting the transport(case 3): OND2C00981304.cihs.ad.gov.on.ca on 080
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereExecute: Executing the transaction with the app server reqInfo is e671f78 useExistingStream=0, client->stream=00000000
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: Getting the stream to the app server (keepalive 28)
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: calling blocking connect
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: Setting socket to non-block for ServerIOTimeout over HTTP
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: socket 3564 connected to OND2C00981304.cihs.ad.gov.on.ca:9080 timeout=900
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: lib_stream: openStream: Opening the  stream soc=3564

  • Previous versions:


By default, in previous versions of WAS plug-in for web server, if the web server plug-in received secure (HTTPS) request but could not create secure connection to the application server (either secure transport channel not defined or secure connection could not be established),
it would create a non-secure (HTTP) connection (if one is defined and available). If HTTP transport not defined, then no connection would be created.
So, the behaviour in older version of WAS that defaulted HTTPS to HTTP when secure connection was not available, was a real problem from security prospective. Read more about the problem as documented here:
http://www-01.ibm.com/support/docview.wss?uid=swg1PM85452.

As mentioned above, in WAS version 8.5.5.0 and later, it has been fixed and it only defaults from HTTPS to HTTP if it is explicitly configured to default when HTTPS connection can not be
established. There still seems to be some logging issue in version 8.5.5 with UseInsecure="true" and fix is available in fix pack 8.5.5.2. Read detail here: http://www-01.ibm.com/support/docview.wss?uid=swg1PM96173

Incoming HTTP traffic and only secure (HTTPS) transport channel to back end configured/available:

   In this case, there should be no issue as long as keyring that contains the certificate of back-end application server and stash file properly configured for the transport channel.

Note: In order to minimize confusion, if you are sure only secure (HTTPS) connection should be allowed for your implementation, you can simply comment out the non-secure transport channel configuration in the plugin-cfg.xml or vice-versa.

Hope these tips help to eradicate (if any) confusion you to might have related to plug-in to application server connection from security perspective.

Read more about the Web server plug-in connections in IBM Knowledge Center here: http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cwsv_plugin_connections.html?lang=en.

WebSphere Application Server (WAS) High Performance Extensible Logging (HPEL) - some tips and tricks

As we know, WebSphere Application Server generally supports two types of logging mechanisms - Basic and High Performance Extensible Logging (HPEL).  If your application(s) logs with high frequency, there is a real possibility that system will be benefited (performance wise), if you enable HPEL. On the performance benefit of HPEL, IBM documentation says,
"HPEL has been designed and tested to significantly outperform the existing basic log and trace facility. One result is that the application server can  run with trace enabled while causing less impact to performance than tracing the same components using basic logging. Another result is that applications that frequently write to the logs might run faster with HPEL ..."
Reference: https://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/ctrb_HPELOverview.html

I have recently worked on a project to change logging from Basic to HPEL and based on my experience, I'm listing few tips and tricks below.


1. Enabling HPEL:

If you have just few App Servers in your server farm, then you can do this task manually. However, if you need to enable this for dozens of App Servers then, it's better to automate the task using script. WAS InfoCenter provides some fragments of scripts here and there, but nothing that you can take and execute. Below is a simple Jython script that I put together. It enables or disables HPEL based on passed argument to all servers within the given WAS Cell. You can also exclude specific server(s) as needed. Below script excludes 'nodeagent' and 'dmgr'.

enableHPEL.py
import sys
global AdminConfig
global AdminControl

action = sys.argv[0]

if action == 'true':
enableHPEL='true'
enableRAS='false'
else:
enableHPEL='false'
enableRAS='true'
#endElse


#Get Cell, nodes, and servers
cellName = AdminControl.getCell( )
cellID = AdminConfig.getid("/Cell:"+cellName+"/" )
nodes = AdminConfig.list("Node", cellID ).split()

for nodeID in nodes:
nodeName = AdminConfig.showAttribute(nodeID, "name")
servers = AdminConfig.list("Server", nodeID ).split()
for serverID in servers:
srvName = AdminConfig.showAttribute(serverID, "name" )
print "/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
if srvName == 'nodeagent' or srvName == 'dmgr':
continue
#endIf
HPELService = AdminConfig.getid("/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName+"/HighPerformanceExtensibleLogging:/");
RASLogService = AdminConfig.getid("/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName+"/RASLoggingService:/");
if enableHPEL == 'true':
print "Enabling HPELService for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
AdminConfig.modify(HPELService, "[[enable true]]")
AdminConfig.modify(RASLogService, "[[enable false]]")
#endIf
if enableRAS == 'true':
print "Enabling RASLogService for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
AdminConfig.modify(HPELService, "[[enable false]]")
AdminConfig.modify(RASLogService, "[[enable true]]")
#endIf
print "Saving configuration for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
AdminConfig.save()
print "Synchronizing configuration for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
Sync1 = AdminControl.completeObjectName('type=NodeSync,process=nodeagent,node='+nodeName+',*')
synced = AdminControl.invoke(Sync1, 'sync')
print "Synchronization completed: "+synced
#endFor
#endFor

Above script is for full WAS profile, for WAS Liberty profile you have to do it manually for one WAS liberty server and may copy the configuration to other Liberty servers. You can find details here: https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_confHPEL.html?cp=SSEQTP_8.5.5
Or use Eclipse WAS Liberty tool as outlined here: https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/t_edit_server_config.html?cp=SSEQTP_8.5.5

If you need to customize HPEL setting for full WAS profile further (like changing size of log repository, location of repository, buffering etc.)  using wsadmin scripting, you can follow the following link for Full WAS profile:
http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.base.doc/ae/ttrb_confHPELwsadmin.html
Similar options for WAS Liberty profile can be found here:
https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_confHPEL.html?cp=SSEQTP_8.5.5

You can execute the above mentioned script as follows using wsadmin.sh[bat].
Enable HPEL:
wsadmin -username <username> -password <password> -lang jython -f ~/enableHPEL.py true

Disable HPEL:
wsadmin -username <username> -password <password> -lang jython -f ~/enableHPEL.py false

Note: provided script changes the configuration using wsadmin's AdminConfig object, which requires App Server to be restarted in order to take effect. You can use AdminControl object to do the same in runtime and that takes effect immediately.


2. Usability:

One of the major concerns with HPEL I found was usability issue. Once HPEL is enabled, the System logs and traces are written in binary, and people can not just open the log files using any text editor as they used to do it. There is an option to write plain text log also, but by enabling that option we loose certain performance benefit of HPEL. Even though IBM provides pretty good tools ('logViewer.sh[bat]' with full WAS profile, and 'binaryLog' with WAS Liberty profile) either to convert the binary logs into text format or just monitor the logs in real time, it's not easy to convince people to change their way of doing things. It takes time. One catch though, IBM's tools require WAS installed in the machine where tools are supposed to run. Not everybody who needs to look logs has WAS installed and your log repository server where you copy your binary log repositories for processing also may not have WAS installed. So, alternative is to convert logs on the App Server, where its generated  (may not be a good practice as it uses CPU cycles of production server)  and provide text logs to whoever need to review them.
I thought that it would be really convenient if we can have a tool to process these binary logs without WAS. And indeed, with little customization, you can use IBM's tool without WAS, which I am going to explain here:

2.1. Preparing custom 'logViewer' tool so that it can be run independent of WAS:

2.1.A) Create following directory structure under /tmp and copy corresponding file from your WAS installation:

/tmp/hpelviewer
bin
logViewer.sh (copy from WAS_INSTALLED_DIR/bin directory)
logViewer.bat (copy from WAS_INSTALLED_DIR/bin directory)
lib
bootstrap.jar (copy from WAS_INSTALLED_DIR/lib directory)
com.ibm.hpel.logging.jar  (copy from WAS_INSTALLED_DIR/plugins directory)
startup.jar  (copy from WAS_INSTALLED_DIR/lib directory)
properties
WsHeader  (copy from WAS_INSTALLED_DIR/properties directory)
WsLevels.properties  (copy from WAS_INSTALLED_DIR/properties directory)


2.1.B) Once the files are copied, open the logViewer.sh from /tmp/hpelviewer/bin and change as follow (actual script lines are in blue):
#Set JAVA_HOME at the top:
JAVA_HOME=/cygdrive/c/java/jdk1.7.0_45
#leave following line as it is:
binDir=`dirname "$0"`
#Comment the following line:
#. "$binDir/setupCmdLine.sh"
#Instead set the WAS_HOME:
setWASHome()
{
    CUR_DIR=$(pwd)
WAS_DIR="${binDir}"/..
cd "${WAS_DIR}"
WAS_HOME=$(pwd)
cd "${CUR_DIR}"  
}
setWASHome

#Leave following lines as it is:
if [ -f ${JAVA_HOME}/bin/java ]; then
    JAVA_EXE="${JAVA_HOME}/bin/java"
else
    JAVA_EXE="${JAVA_HOME}/jre/bin/java"
fi


#comment the following lines as we need to make them compatible to Unix and Cygwin:
#WAS_HEADER="-Dlogviewer.custom.header=${WAS_HOME}/properties/WsHeader"
#WAS_LEVELS="-Dlogviewer.custom.levels=${WAS_HOME}/properties/WsLevels.properties"
#Instead add the following lines:
PATH=.:"$JAVA_HOME"/bin/:"$JAVA_HOME"/jre/bin:$PATH
WAS_CLASSPATH="$WAS_HOME/properties:$WAS_HOME/lib/bootstrap.jar:$WAS_HOME/lib/com.ibm.hpel.logging.jar"
WAS_HEADER_PROP="${WAS_HOME}/properties/WsHeader"
WAS_LEVELS_PROP="${WAS_HOME}/properties/WsLevels.properties"
WAS_LOGGING="-Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Djava.util.logging.configureByServer=true"
# to be used as default log repository path
LOG_ROOT="$WAS_HOME/logs"

##
## Determine the platform and convert path accordingly
##
case $OSTYPE in
  cygwin)
    uname=CYGWIN_NT
        ;;
  *)
    uname=`uname`
esac

case $uname in
  CYGWIN_*)
WAS_HEADER_PROP=$(cygpath -w "$WAS_HEADER_PROP")
WAS_LEVELS_PROP=$(cygpath -w "$WAS_LEVELS_PROP")
LOG_ROOT=$(cygpath -w "$LOG_ROOT")
WAS_HOME=$(cygpath -w "$WAS_HOME")
        WAS_CLASSPATH=$(cygpath -pw "$WAS_CLASSPATH")
        PATH=$(cygpath -pw "$PATH")
        ;;
  *)
;;
esac
OSGI_INSTALL="-Dosgi.install.area=$WAS_HOME"
WAS_HEADER="-Dlogviewer.custom.header=$WAS_HEADER_PROP"
WAS_LEVELS="-Dlogviewer.custom.levels=$WAS_LEVELS_PROP"

#Finally, modify the Java execution command and options as follow:

${JAVA_EXE} \
-Dwas.install.root="$WAS_HOME" \
-Duser.install.root="$WAS_HOME" \
-Dlog.repository.root="$LOG_ROOT" \
$WAS_LOGGING \
$WAS_HEADER \
$WAS_LEVELS \
-classpath $WAS_CLASSPATH  com.ibm.ws.bootstrap.WSLauncher \
com.ibm.ws.logging.hpel.viewer.LogViewer "$@"

2.1.C) Save the changes.

Note: In the above script 'cygpath' has been used to make it compatible with Cygwin. for more information about 'cygpath' see http://www.cygwin.com/cygwin-ug-net/cygpath.html

2.2. Preparing custom 'binaryLog' tool (for Liberty profile) so that it can be run independent of WAS:

2.2.A) Create following directory structure under /tmp and copy corresponding file from your WAS installation:
/tmp/hpelviewer
bin
binaryLog
binaryLog.bat
tools
ws-binarylogviewer.jar
lib
com.ibm.ws.kernel.cmdline_1.0.10.jar
com.ibm.ws.logging.hpel_1.0.10.jar

2.2.B) customization of binaryLog is easier as IBM has written it in in such a way that it can be run under Cygwin as well as other Unix like platform. Just add the following two lines (actual lines to be added are in blue) in the beginning of the script.
WLP_INSTALL_DIR=..
#optionally set JAVA_HOME
JAVA_HOME=/cygdrive/c/java/jdk1.8.0

2.2.C) Save the file.

Note: follow similar tricks if you need to modify corresponding BAT files for Windows.
Once your customization is complete, just create a zip file (for example hpelviewer.zip) making hpelviewer as root directory of the zip,  and give it your users.
They can just extract the zip and and execute as follow. Only thing they need is Java installed on their machine.
Here is an exmple:
cd <hpelviewer_extract_dir>/hpelviewer/bin
./logViewer.sh -repositoryDir <repositoryDir>
Or:
cd <hpelviewer_extract_dir>/hpelviewer/bin
./binaryLog view  {serverName | repositoryPath}

Note: Since the property files, jar files and script files for WAS full profile and WAS Liberty profile are unique, you can actually create a single zip file by combining files for both. Here is how the file structure for combined  tool would look like:

hpelviewer
bin
logViewer.sh
logViewer.bat
binaryLog
binaryLog.bat
tools
ws-binarylogviewer.jar
lib
bootstrap.jar
com.ibm.hpel.logging.jar
startup.jar
com.ibm.ws.kernel.cmdline_1.0.10.jar
com.ibm.ws.logging.hpel_1.0.10.jar
properties
WsHeader
WsLevels.properties

Create a zip and give it to your users, so that they can use it to view binary logs generated by both WAS full profile or Liberty profile.

Happy HPEL !!!