How to Setup Front-End Web Server for WebSphere Liberty Cluster

I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on setting up Front-End Web Server. Below listed are all the posts in this series:

1. How to Create and Configure WebSphere Liberty Cluster End-to-End
2. How to Deploy Application in WebSphere Liberty Cluster
3. How to Setup Front-End Web Server for WebSphere Liberty Cluster

In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server

Note: in order to complete all the steps in this blog post, first you need to complete required steps in blog posts 1, and 2 in this series as stated above.

IHS powered by Apache with Plug-in for WebSphere Application Server (WAS) uses pre-configured workload management (WLM) policies to dispatch web requests to the appropriate cluster members and their containers. Having front-end web server like IHS also helps to boost security. To prevent single point of failure at the web server level, deploy an additional (backup or active) web server.
If you like to compare/review any other available front-end option, refer to Selecting a front end for your WebSphere Application Server topology.

Install IHS and IHS Plug-in for WAS 

Note: In the example topology above IHS and Plug-in is installed and configured on Machine: 05.

Providing installation instruction of IHS/Apache is out of scope for this post. This post assumes that you have Apache/IHS instance available to configure and integrate with WLP server. If you need to install IHS and WAS plug-in for IHS, refer to following:

For IHS:
https://www.ibm.com/support/knowledgecenter/en/SS7JFU_8.5.5/com.ibm.websphere.ihs.doc/ihs/tihs_silentinstall.html.

For Plug-in:
Installing the Web Server Plug-ins using the command line

Generate WAS plugin for IHS.

Generating WAS plug-in for IHS (plugin-cfg.xml) is easy but somehow trickier in WLP. Starting version 16.0.0.3, pluginUtility command comes with WLP, which makes it easy to generate plugin-cfg.xml. In fact from version 16.0.0.3, plugin-cfg.xml for each WLP server is generated automatically (triggered by different server events). You can see something like this in messages.log
com.ibm.ws.webcontainer.osgi.mbeans.PluginGenerator I SRVE9103I: A configuration file for a web server plugin was automatically generated for this server at /opt/ibm/wlp/usr/servers/wlpSrv02/logs/state/plugin-cfg.xml.

See Automatic generation of the plugin-cfg.xml file in IBM Knowledge Center for more details.
And with the help of pluginUtility, we can either merge individual plugin-cfg.xml or generate a brand new merged plugin-cfg.xml. As part of this exercise, we are going to generate merged plugin-cfg.xml. See command in action below:

$> cd /opt/ibm/wlp/bin $> ./pluginUtility generate \   --server=wasadmin:<replace_with_your_password>@waslibctlr01:9443 \   --cluster=wlpCluster \   --targetPath=/tmp/plugin-cfg.xml The remote target server waslibctlr01:9443 will be used to generate the webserver plugin configuration file. SSL trust has not been established with the target server. Certificate chain information: Certificate [0] Subject DN: CN=waslibctlr01, OU=wlpCntlr, O=ibm, C=us ... Do you want to accept the above certificate chain? (y/n) y The generation of the webserver plugin configuration file was successful for the target collective controller.

As per input command above, it generates plugin-cfg.xml under /tmp, which we will review later.
For pluginUtility command details and all available options, refer to pluginUtility command page at IBM Knowledge Center.

Note: If you are using WLP version 8.5.5.x where pluginUtility command is not available, you can call the generateClusterPluginConfig operation on the ClusterManager MBean to generate a merged plugin-cfg.xml file for all started cluster members. For working code example, refer to topic Generating a merged plug-in configuration for Liberty servers at IBM Knowledge Center.

Troubleshooting: If your plugin-cfg.xml file generation fails with following error: The generation of the webserver plugin configuration file failed for the target collective controller. Analyze the logs of the target collective controller to diagnose the problem. And you see following message in your Collective Controller messages.log ... FFDC1015I: An FFDC Incident has been created: "java.net.NoRouteToHostException: No route to host (Host unreachable) com.ibm.ws.collective.repository.internal.ClusterManager 323" at ffdc_17.11.11_18.08.48.0.log ... It is possibly caused by the firewall on the server machine(s). Make sure to open port(s) (listed in the server.xml) for communications on all member server(s) as well as controller.

Here is how our generated plugin-cfg.xml looks like:

<?xml version="1.0" encoding="UTF-8"?> <!-- This config file was generated by plugin's merge tool v1.0.0.2 on 2017.11.13 at 16:50:31 GMT --> <Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">   <Log LogLevel="Debug" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>   <Property Name="ESIEnable" Value="true"/>   <Property Name="ESIMaxCacheSize" Value="1024"/>   <Property Name="ESIInvalidationMonitor" Value="false"/>   <Property Name="ESIEnableToPassCookies" Value="false"/>   <Property Name="PluginInstallRoot" Value="/opt/IBM/WebSphere/Plugins"/>   <!-- Server Clusters -->   <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="Shared_2_Cluster_0" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">     <Server CloneID="8b2ad3bf-c30d-4c7e-9fab-ec856dfc20b7" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_defaultServer_1" ServerIOTimeout="900" WaitForContinue="false">       <Transport Hostname="waslibmem01" Port="9081" Protocol="http"/>       <Transport Hostname="waslibmem01" Port="9444" Protocol="https">         <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>         <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>       </Transport>     </Server>     <Server CloneID="9c9bb9b9-d4b9-4bf8-98ef-336e3c5c436d" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_defaultServer_0" ServerIOTimeout="900" WaitForContinue="false">       <Transport Hostname="waslibmem02" Port="9081" Protocol="http"/>       <Transport Hostname="waslibmem02" Port="9444" Protocol="https">         <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>         <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>       </Transport>     </Server>     <PrimaryServers>       <Server Name="default_node_defaultServer_1"/>       <Server Name="default_node_defaultServer_0"/>     </PrimaryServers>   </ServerCluster>   <!-- Virtual Host Groups -->   <VirtualHostGroup Name="/cell/sharedCell_2/vHostGroup/shared_host_0">     <VirtualHost Name="*:443"/>     <VirtualHost Name="*:80"/>   </VirtualHostGroup>   <!-- URI Groups -->   <UriGroup Name="/cell/sharedCell_2/application/default_host_defaultServer_default_node_Cluster_URIs">     <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/IBMJMXConnectorREST/*"/>     <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ibm/api/*"/>     <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/JavaHelloWorldApp/*"/>   </UriGroup>   <!-- Routes -->   <Route ServerCluster="Shared_2_Cluster_0" UriGroup="/cell/sharedCell_2/application/default_host_defaultServer_default_node_Cluster_URIs" VirtualHostGroup="/cell/sharedCell_2/vHostGroup/shared_host_0"/> </Config>

As you can see, I've highlighted few lines above and let's discuss about those.

1. pluginUtility does not create keystore related files (plugin-key.kdb and plugin-key.sth) referenced in the plugin-cfg.xml file. Later in the post, we will talk how to create these file manually.
2. Generated plugin-cfg.xml has default values. If you need to generate plugin-cfg.xml with certain values, define pluginConfiguration element (see below) in server.xml and regenerate the plugin-cfg.xml.
   
   

   <pluginConfiguration webserverPort="80"   webserverSecurePort="443"   sslKeyringLocation="path/to/sslkeyring"   sslStashfileLocation="path/to/stashfile"   sslCertlabel="definedbyuser"/>

   
   For all available configuration attributes of pluginConfiguration, see IBM Knowledge Center

Establishing Secure communication between IHS/Plug-in and WLP servers

As you can see in the generated plugin-cfg.xml, it has both http and https transport definitions. Https transport references the plugin-key.kdb/sth files, however, these files are not generated by pluginUtility. So, we need to create them manually. Below, you'll find two ways to create them:

1) Create key database (kdb) file and import Signer Certificate. For this purpose, you can use 'gskcmd' or 'IKEYMAN' that are installed as part of IHS. Below I'll show how to use 'gskcmd'.

$> cd /opt/IBM/HTTPServer/bin # creates .kdb,sth, and .rdb files $> ./gskcmd -keydb -create -db /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \   -pw <replace_with_your_password> \   -type kdb -expire 7300 -stash # Add signer certificate. If you are using WAS generated keys on WAS Liberty, # member root certificate can be extracted from trust.jks file under  # ${server.config.dir}/resources/security directory. $> ./gskcmd -cert -add -db /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \   -pw <replace_with_your_password> \   -file /tmp/myWASSigner.crt # List the added certificates $> ./gskcmd -cert -list -db /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \    -pw <replace_with_your_password>

2) Convert JKS file into KDB file. The following command takes jks file as an input and creates plugin-key.kdb, plugin-key.rdb, and plugin-key.sth files.
If you are using WAS generated keys, you can find signer certificate in trust.jks file on any member server under ${server.config.dir}/resources/security directory.

$> ./gskcmd -keydb -convert -db /tmp/trust.jks \   -pw <replace_with_your_password> \   -new_format kdb \   -target /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \   -stash

IHS Certificate

In order to secure communication between the client/browser and IHS, you need to have either CA signed certificate (preferred) or self-signed certificate for IHS. For this exercise, we are going to use self-signed certificate. Please note that if your front-end communication is not secure, then back-end communication (from plug-in to WAS WLP) is also going to be not secure (by default). If you like to know more detail on how it works, see Few Tips on Secure vs. Non-Secure WAS Web Server Plug-in Connection.
Below command shows how to create a self-signed certificate using gskcmd:

$> cd /opt/IBM/HTTPServer/bin # run gskcmd to create kdb file to store private and public keys for IHS $> ./gskcmd -keydb -create -db /opt/IBM/HTTPServer/ssl/ihs.kdb \   -pw <replace_with_your_password> \   -type kdb -expire 7300 -stash # create self signed cert: $> ./gskcmd -cert -create -db /opt/IBM/HTTPServer/ssl/ihs.kdb \   -label ihs_cert -pw <replace_with_your_password> \   -type kdb -size 2048 -expire 7300 -default_cert yes \   -dn "CN=waslibhihs01, OU=WebSphere, O=SysGenius, C=CA"

IHS and plug-in Configuration

Now, we have everything in order to configure IHS and WAS plug-in for IHS. This configuration is very straight forward and can be done manually. For details refer to https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_admin_webserver_plugin.html

Verify content of plugin-cfg.xml

•  Make sure .kdb/.rdb/.sth file location is correct. 
• log file location is correct. 
• Both (http and https) transport definitions (host and port) are correct and resolvable from IHS server. You can do a telnet test for both http and https transport to verify. 

Update plugin-cfg.xml if required.

Update httpd.conf

1) Add plug-in related configuration in httpd.conf:

LoadModule was_ap22_module /opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

2) Add configuration for front-end secure communication

• Uncomment/enable the SSL module configuration directive.
• Create an SSL virtual host stanza in the httpd.conf file using the following examples and directives.

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so Listen 443 <VirtualHost *:443>    SSLEnable </VirtualHost> SSLDisable KeyFile "/opt/IBM/HTTPServer/ssl/ihs.kdb"

Start IHS:

#Start IHS: $> /opt/IBM/HTTPServer/bin $> ./apachectl -k start # See if it's started $> ps -ef | grep httpd $> ps -ef | grep httpd root 9 1 0 14:39 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start nobody 13 9 0 14:39 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start nobody 14 9 0 14:39 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start

Review the error_log (under /opt/IBM/HTTPServer/logs) and make sure it does not have any error and it loaded plug-in successfully

[Sat Nov 18 14:39:21.325566 2017] [ibm_ssl:notice] [pid 9:tid 139878623614720] Using GSKit version 8.0.50.77 [Sat Nov 18 14:39:21.391142 2017] [was_ap24:notice] [pid 9:tid 139878623614720] --------------------------------------------------- [Sat Nov 18 14:39:21.391164 2017] [was_ap24:notice] [pid 9:tid 139878623614720] WebSphere Plugins loaded. [Sat Nov 18 14:39:21.391167 2017] [was_ap24:notice] [pid 9:tid 139878623614720] Bld version: 9.0.0.4 [Sat Nov 18 14:39:21.391170 2017] [was_ap24:notice] [pid 9:tid 139878623614720] Bld date: Apr 11 2017, 00:11:26 [Sat Nov 18 14:39:21.391172 2017] [was_ap24:notice] [pid 9:tid 139878623614720] Webserver: IBM_HTTP_Server/9.0.0.4 (Unix) [Sat Nov 18 14:39:21.391174 2017] [was_ap24:notice] [pid 9:tid 139878623614720] --------------------------------------------------- [Sat Nov 18 14:39:21.391234 2017] [:notice] [pid 9:tid 139878623614720] Using config file /opt/IBM/HTTPServer/conf/httpd.conf [Sat Nov 18 14:39:21.391840 2017] [mpm_event:notice] [pid 9:tid 139878623614720] CoreDumpDirectory not set; core dumps may not be written for child process crashes [Sat Nov 18 14:39:21.391841 2017] [mpm_event:notice] [pid 9:tid 139878623614720] AH00489: IBM_HTTP_Server/9.0.0.4 (Unix) configured -- resuming normal operations

And from http_plugin.log

[18/Nov/2017:14:39:21.39104] 00000009 07ac1700 - PLUGIN: Plugins loaded. [18/Nov/2017:14:39:21.39105] 00000009 07ac1700 - PLUGIN: --------------------System Information----------------------- [18/Nov/2017:14:39:21.39106] 00000009 07ac1700 - PLUGIN: Bld version: 9.0.0.4 [18/Nov/2017:14:39:21.39107] 00000009 07ac1700 - PLUGIN: Bld date: Apr 11 2017, 00:11:45 [18/Nov/2017:14:39:21.39107] 00000009 07ac1700 - PLUGIN: Webserver: IBM_HTTP_Server/9.0.0.4 (Unix) [18/Nov/2017:14:39:21.39108] 00000009 07ac1700 - PLUGIN: OS : Linux x86_64 [18/Nov/2017:14:39:21.39109] 00000009 07ac1700 - PLUGIN: Hostname = waslibhihs01 [18/Nov/2017:14:39:21.39110] 00000009 07ac1700 - PLUGIN: NOFILES = hard: 65536, soft: 65536 [18/Nov/2017:14:39:21.39110] 00000009 07ac1700 - PLUGIN: MAX COREFILE SZ = hard: INFINITE, soft: INFINITE [18/Nov/2017:14:39:21.39111] 00000009 07ac1700 - PLUGIN: DATA = hard: INFINITE, soft: INFINITE [18/Nov/2017:14:39:21.39112] 00000009 07ac1700 - PLUGIN: --------------------------------------------------------------

Access the application through IHS:

https://waslibhihs01/JavaHelloWorldApp

Hurray! Here is JavaHelloWorldApp

And, check the access_log, you'll see something like this:

192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/ HTTP/1.1" 200 705 13237 192.168.56.109:9444 + 192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/style.css HTTP/1.1" 200 1157 973 + 192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/SimpleServlet HTTP/1.1" 200 12 15625 192.168.56.108:9444 + 192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/ HTTP/1.1" 200 705 13397 192.168.56.109:9444 + 192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/SimpleServlet HTTP/1.1" 200 12 12708 192.168.56.108:9444 +

If you encounter any issue, make sure Web Server Plug-in, WebSphere Liberty and IHS (powered by Apache) are compatible. Refer to Supported combinations of IBM HTTP Server, WebSphere Application Server, and the WebSphere WebServer Plug-in for more details.

Thank you coming so far! It is the end of this series!


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below:

• How to Create and Configure WebSphere Liberty Cluster End-to-End
• How to Deploy Application in WebSphere Liberty Cluster
• How to Setup Front-End Web Server for WebSphere Liberty Cluster (this post)
• Using Docker Secrets with IBM WebSphere Liberty Profile Application Server
• How to use WLP passwordUtilities feature for encryption/decryption