How to use Docker Swarm Configs service with WebSphere Liberty Profile


   In order to make your dockerized application portable, you can externalize (Docker container using configuration from outside of Docker image) configuration that changes from one environment to another (from DEV to QA, UAT, Prod etc.). This helps to maintain a generic docker image for your dockerized application and also get rid of most of the bind-mount configuration files and/or environment variables used by your container. Following Docker Swarm services are extremely useful in externalizing the configuration:
  • Docker Secrets (available in Docker 1.13 and higher version)
  • Docker Configs (available in Docker 17.06 and higher version)
You can use Docker Secrets to externalize configuration that are confidential in nature, and Docker Configs for general configuration that has potential to be changed from one environment to another.
In this blog post, I will use Dockerized application powered by WebSphere Application Server Liberty Profile (WLP) to show how to use Docker Configs service to externalize server.xml. You can look my other blog - Using Docker Secrets with IBM WebSphere Liberty Profile Application Server, to learn how to use Docker Secrets.


So, here is my server.xml for my WLP application to be used in this post as an example.

<server description="TestWLPApp">
   <featuremanager>
      <feature>javaee-7.0</feature>
      <feature>localConnector-1.0</feature>
      <feature>ejbLite-3.2</feature>
      <feature>jaxrs-2.0</feature>
      <feature>jpa-2.1</feature>
      <feature>jsf-2.2</feature>
      <feature>json-1.0</feature>
      <feature>cdi-1.2</feature>
      <feature>ssl-1.0</feature>
   </featuremanager>
   <include location="/run/secrets/app_enc_key.xml"/>
   <httpendpoint host="*" httpport="9080" httpsport="9443" id="defaultHttpEndpoint"/>
   <ssl clientauthenticationsupported="true" id="defaultSSLConfig" keystoreref="defaultKeyStore"     truststoreref="defaultTrustStore"/>
   <keystore id="defaultKeyStore" location="/run/secrets/keystore.jks" password="{aes}ANGkm5cIca4hoPMh4EUeA4YYqVPAbo4HIqlB9zOCXp1n"/>
   <keystore id="defaultTrustStore" location="/run/secrets/truststore.jks" password="{aes}ANGkm5cIca4hoPMh4EUeA4YYqVPAbo4HIqlB9zOCXp1n"/>
   <applicationmonitor updatetrigger="mbean"/>
   <datasource id="wlpappDS" jndiname="wlpappDS">
      <jdbcdriver libraryref="OracleDBLib"/>
      <properties.oracle password="{aes}AAj/El4TFm/8+9UFzWu5kCtURUiDIV/XKbGY/lT2SVKFij/+H38b11uhjh+Peo/rBA==" url="jdbc:oracle:thin:@192.168.xx.xxx:1752:WLPAPPDB" user="wlpappuser"/>
   </datasource>  
    <library id="OracleDBLib">
       <fileset dir="/apps/wlpapp/shared_lib" includes="ojdbc6-11.2.0.1.0.jar"/>
    </library>
    <webapplication contextRoot="wlpappctx" id="wlpapp" location="/apps/wlpapp/war/wlptest.war" name="wlpapp"/>
</server>

As you can see in above server.xml, the following items were created as Docker Secrets:


  • <include location="/run/secrets/app_enc_key.xml"/>

  • <keystore id="defaultKeyStore" location="/run/secrets/keystore.jks" ...

  • <keystore id="defaultTrustStore" location="/run/secrets/truststore.jks" ...


See, Create Docker Secrets paragraph of  Using Docker Secrets with IBM WebSphere Liberty Profile Application Server to create these confidential configuration items.

Once confidential configuration items are created using Docker Secrets, follow the steps below to create general configuration items using Docker Configs.
  1. Connect to Docker UCP using client bundle. 
  2. Create configuration item for server.xml using docker config create ...command.
    Important: both the client and daemon API must both be at least at version 1.30 to use this command.

    $> docker config create dev_wlp_server_config_v1.0 /mnt/nfs/dockershared/wlpapp/server.xml_v1.0

    9i5edohyzyrvopuz988caxw4r

    Note: here dev_wlp_server_config_v1.0 is configuration item name which gets configuration from /mnt/nfs/dockershared/wlpapp/server.xml_v1.0. I've decided to version the configuration item, so that in future if I need to update the configuration, it becomes easier.

  3. verify that the configuration item created

     $> docker config ls

    ID                        NAME                       CREATED        UPDATED
    9i5edohyzyrvopuz988caxw4r dev_wlp_server_config_v1.0 18 seconds ago 18 seconds ago
    geuerj6t98d8eeu8nqvvxgtw9 com.docker.license-0       5 days ago     5 days ago
    vdzwhpe91iptvuiro654u3lue com.docker.ucp.config-1    5 days ago     5 days ago

  4. Use configuration item. Below example shows using YAML.

    docker-compose.yml
    version: "3.3"
    services:
       wlpappsrv: 

          image: 192.168.56.102/osboxes/wlptest:1.0
          networks:
             - my_hrm_network
          secrets:
             - keystore.jks
             - truststore.jks
             - app_enc_key.xml
          ports:
             - 9080
             - 9443
          configs:
             - source: dev_wlp_server_config_v1.0
               target: /opt/ibm/wlp/usr/servers/defaultServer/server.xml
               mode: 0444

         deploy:
            placement:
               constraints: [node.role == worker]
               mode: replicated
               replicas: 4
               resources:
                  limits:
                     memory: 2048M
               restart_policy:
                  condition: on-failure
                  max_attempts: 3
                  window: 6000s
               labels:
                  - "com.docker.ucp.mesh.http.9080=external_route=http://mydockertest.com:8080,internal_port=9080"
                  - "com.docker.ucp.mesh.http.9443=external_route=sni://mydockertest.com:8443,internal_port=9443"
    networks:
       my_hrm_network:
          external:
             name: my_hrm_network
    secrets:
       keystore.jks:
          external: true
       truststore.jks:
          external: true
       app_enc_key.xml:
          external: true
    configs:
        dev_wlp_server_config_v1.0:
         external: true

    Note: if you don't want to create configuration item in advance (step #2 above), you can also specify configuration file in the YAML file itself. Replace external: true in the above example with file: /mnt/nfs/dockershared/wlpapp/server.xml_v1.0

    If you want to use use docker service create ... command, instead of YAML file, here is how you can use config

    docker service create \
     --name wlpappsrv \
     --config  source=dev_wlp_server_config_v1.0,target=/opt/ibm/wlp/usr/servers/defaultServer/server.xml,mode=0444 \
     ... \
     192.168.56.102/osboxes/wlptest:1.0

  5. Validate compose file:
    $> docker-compose -f docker-compose.yml config
    WARNING: Some services (opal) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm. WARNING: Some services (opal) use the 'configs' key, which will be ignored. Compose does not support 'configs' configuration - use `docker stack deploy` to deploy to a swarm.

  6. Deploy the service as a stack:
    $> docker stack deploy --compose-file docker-compose.yml dev_WLPAPP


How to refresh/update or rotate configuration


Configuration item created by Docker Configs service is immutable, however, there is a way to rotate configuration. Let's say, you need to update some configuration value in server.xml, like you have to reference to new version of JDBC driver.  See the steps below:
  1. Create another configuration item that references to updated server.xml
    $>docker config create dev_wlp_server_config_v2.0 \
      /mnt/nfs/var/dockershared/dev_PAL/server.xml_v2.0



    o4173tet99vuwuz1fma4dqd2j

  2. Update the service so that it references to the newly created configuration item
    $>docker service update \
     --config-rm dev_wlp_server_config_v1.0 \
     --config-add  source=dev_wlp_server_config_v2.0,target=/opt/ibm/wlp/usr/servers/defaultServer/server.xml
    \
     wlpappsrv

  3. [optional] Once the service is fully updated, you can remove the old configuration item:
    $> docker config rm dev_wlp_server_config_v1.0

  4. [optional] If you need to see which configuration item is attached to the service, you can run 'docker service inspect <service-name>' command.
    $>docker service inspect wlpappsrv

    ...
    "Configs": [
      {
        "File": {
          "Name": "/opt/ibm/wlp/usr/servers/defaultServer/server.xml",
          "UID": "0",
          "GID": "0",
          "Mode": 292
        },
         "ConfigID": "o4173tet99vuwuz1fma4dqd2j",
         "ConfigName": "dev_wlp_server_config_v2.0"
      }
    ]
    ...


For more information about Docker Swarm Configs service, review the following Docker documentations:

How to Deploy Application in WebSphere Liberty Cluster



I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on application deployment. Below listed are all the posts in this series:
  1. How to Create and Configure WebSphere Liberty Cluster End-to-End
  2. How to Deploy Application in WebSphere Liberty Cluster
  3. How to Setup Front-End Web Server for WebSphere Liberty Cluster
In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server

Note: in order to complete all the steps in this blog post, first you need to complete required steps outlined in How to Create and Configure WebSphere Liberty Cluster End-to-End

Deploying an application in WLP server or cluster

There are few ways to deploy an application into WLP server or cluster. Application can be simply dropped into a pre-defined dropins (${server.config.dir}/dropins) directory, or by adding an application definition entry to the server configuration.
In this blog post, we will explore script driven, remote application deployment for distributed platforms (multiple WLP servers or cluster managed by WLP Collective Controller) by utilizing secure JMX REST connection. Refer to Configuring secure JMX connection to Liberty for more details.
We'll be using manageAppOnCluster.py script and libraries available in IBM developerWorks to manage application(s) on WLP cluster.

Download manageAppOnCluster.zip

You can download it from https://developer.ibm.com/wasdev/downloads/#asset/scripts-jython-Install_or_uninstall_an_application_to_cluster. Here I have downloaded and extracted manageAppOnCluster.zip in Deployment/Tool server (machine: 01 in example topology) under /opt/workspace/wlpDeployment directory.

Download and Install Jython. 

You can download Jython from http://www.jython.org/downloads.html
General installation instruction can be found at https://wiki.python.org/jython/InstallationInstructions
Console based installation is very straight forward. See below, I'm installing standalone Jython under /opt/jython directory on Machine: 01

# Create /opt/jython directory
$> sudo -p mkdir /opt/jython

# Change the ownership if needed
$> sudo chown R wasadmin:wasgrp /opt/jython

# Install Jython
$> java -jar jython-installer-2.7.0.jar --console

Welcome to Jython !
You are about to install Jython version 2.7.0
(at any time, answer c to cancel the installation)
For the installation process, the following languages are available: English, German
Please select your language [E/g] >>> E
Do you want to read the license agreement now ? [y/N] >>> N
Do you accept the license agreement ? [Y/n] >>> Y
The following installation types are available:
   1. All (everything, including sources)
   2. Standard (core, library modules, demos and examples, documentation)
   3. Minimum (core)
   9. Standalone (a single, executable .jar)
Please select the installation type [ 1 /2/3/9] >>> 9
Please enter the target directory >>> /opt/jython Your java version to start Jython is: Oracle Corporation / 1.8.0_102
Your operating system version is: Linux / 3.10.0-514.el7.x86_64
Summary:
  - standalone
Please confirm copying of files to directory /opt/jython [Y/n] >>> Y
  10 %
20 %
  ...
  ...
Packing standalone jython.jar ...
  90 %
  100 %
Do you want to show the contents of README ? [y/N] >>> N
Congratulations! You successfully installed Jython 2.7.0 to directory /opt/jython.

You'll also need restConnector.jar and restConnector.py in order to be able to make remote JMX REST connection. If you have WLP installed on the machine, you should be good as these files are installed as part of WLP installation. If you don't have WLP installed on the machine, or you don't want to install WLP, you can just copy these files from another machine where it is installed. Easiest way is to tar the content of entire ${wlp.install.dir}/clients directory and copy it to Deployment/Tool server where you can untar it.
Here, I'm going to create a wlpclient.tar file on Machine: 02, as WLP is installed there and copy to Machine: 01.

$> cd /opt/ibm/wlp
$> tar -cvf /tmp/wlpclient.tar clients/
tar: Removing leading `/' from member names
clients/
clients/restConnector.jar
clients/jython/
clients/jython/restConnector.py
clients/jython/README

Copy the wlpclient.tar to Deployment/Tool server and extract it.

# Create directory /opt/ibm/wlp in Machine 02 (if it doesn't exist)
$> sudo mkdir /opt/ibm/wlp

# Change ownership as required.
$> sudo chown -R wasadmin:wasgrp

# Extract under /opt/ibm/wlp
$> tar -xvf wlpclient.tar -C /opt/ibm/wlp

In order to create secure connection, you'll also need signer certificate from the Collective Controller. Just copy trust.jks file (located under ${server.config.dir}/resources/security/ directory) from Collective Controller server and put into Deployment/tool server.
Now, for convenient, create a script file (name it wlpMgmt.sh), which look something like this:

wlpMgmt.sh
#! /bin/bash
# wlpMgmt.sh
export JYTHONPATH=/opt/ibm/wlp/clients/jython/restConnector.py
java -cp /opt/jython/jython.jar:/opt/ibm/wlp/clients/restConnector.jar:/opt/ibm/wlp/clients/jython/:/opt/workspace/wlpDeployment/lib/ org.python.util.jython "$@"


Prepare your application

For this exercise, I'm using a sample JavaHelloWorldApp.war application downloaded from https://github.com/IBM-Bluemix/java-helloworld/tree/master/target
I've put the application under /opt/workspace/wlpDeployment/apps directory in Machine:1.

Deploy the Application

Here, we are going to use wlpMgmt.sh (created above) to setup environment and call Jython and execute manageAppOnCluster.py to create secure connection to Liberty Controller and deploy application remotely.


$> /opt/workspace/wlpDeployment


$> ./wlpMgmt.sh manageAppOnCluster.py \
   --install=/opt/workspace/wlpDeployment/apps/JavaHelloWorldApp.war \
   --truststore=/opt/workspace/wlpDeployment/resources/security/trust.jks \
   --truststorePassword=<replace_with_your_password> \
   --host=waslibclnt01 \
   --port=9443 \
   --user=wasadmin \
   --password=<replace_with_your_password> \
   --clusterName=wlpCluster \

Connecting to the server...
Successfully connected to the server "waslibclnt01:9443"
Uploading application JavaHelloWorldApp.war to waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Updating server config for waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Complete
Uploading application JavaHelloWorldApp.war to waslibmem02,/opt/ibm/wlp/usr,wlpSrv02
Updating server config for waslibmem02,/opt/ibm/wlp/usr,wlpSrv02
Complete


Verify application deployed successfully

  1. Verify application binary is copied to each member server (in this case Machine: 03wlpSrv01, and Machine: 04wlpSrv02). By default, it is copied to ${server.config.dir}/apps directory of each cluster member server. Check and make sure it is copied. 
  2. Verify that server.xml of each member server is updated. In my case, it has added the following line in the member server's server.xml:

    <application name="JavaHelloWorldApp" location="${server.config.dir}/apps/JavaHelloWorldApp.war"/>

  3. Access application and make sure it is available. In our case, it is accessible as:
    • Machine: 03 URL: http://waslibmem01:9081/JavaHelloWorldApp/
    • Machine: 04 URL: http://waslibmem02:9081/JavaHelloWorldApp/

Note: If for any reason you need to undeploy/uninstall the application, here is how to do it:


$> ./appMgmt.sh manageAppOnCluster.py \
  --uninstall=JavaHelloWorldApp.war \
  --truststore=/opt/workspace/wlpDeployment/resources/security/trust.jks \
  --truststorePassword=<replace_with_your_password> \
  --host=waslibctlr01 \
  --port=9443 \
  --user=wasadmin \
  --password=<replace_with_your_password>; \
  --clusterName=wlpCluster
Connecting to the server...
Successfully connected to the server "waslibctlr01:9443"
Removing application JavaHelloWorldApp.war from waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Updating server config for waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Complete
Removing application JavaHelloWorldApp.war from waslibmem01,/opt/ibm/wlp/usr,wlpSrv02
Updating server config for waslibmem01,/opt/ibm/wlp/usr,wlpSrv02
Complete

uninstall removes the application binary as well as <application> definition from server.xml.

Next: proceed to Setup Front-End Web Server for WebSphere Liberty Cluster -->


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below:


How to Setup Front-End Web Server for WebSphere Liberty Cluster



I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on setting up Front-End Web Server. Below listed are all the posts in this series:
  1. How to Create and Configure WebSphere Liberty Cluster End-to-End
  2. How to Deploy Application in WebSphere Liberty Cluster
  3. How to Setup Front-End Web Server for WebSphere Liberty Cluster
In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server

Note: in order to complete all the steps in this blog post, first you need to complete required steps in blog posts 1, and 2 in this series as stated above.

IHS powered by Apache with Plug-in for WebSphere Application Server (WAS) uses pre-configured workload management (WLM) policies to dispatch web requests to the appropriate cluster members and their containers. Having front-end web server like IHS also helps to boost security. To prevent single point of failure at the web server level, deploy an additional (backup or active) web server.
If you like to compare/review any other available front-end option, refer to Selecting a front end for your WebSphere Application Server topology.


Install IHS and IHS Plug-in for WAS 

Note: In the example topology above IHS and Plug-in is installed and configured on Machine: 05.

Providing installation instruction of IHS/Apache is out of scope for this post. This post assumes that you have Apache/IHS instance available to configure and integrate with WLP server. If you need to install IHS and WAS plug-in for IHS, refer to following:

For IHS:
https://www.ibm.com/support/knowledgecenter/en/SS7JFU_8.5.5/com.ibm.websphere.ihs.doc/ihs/tihs_silentinstall.html.

For Plug-in:
Installing the Web Server Plug-ins using the command line

Generate WAS plugin for IHS.

Generating WAS plug-in for IHS (plugin-cfg.xml) is easy but somehow trickier in WLP. Starting version 16.0.0.3, pluginUtility command comes with WLP, which makes it easy to generate plugin-cfg.xml. In fact from version 16.0.0.3, plugin-cfg.xml for each WLP server is generated automatically (triggered by different server events). You can see something like this in messages.log
com.ibm.ws.webcontainer.osgi.mbeans.PluginGenerator I SRVE9103I: A configuration file for a web server plugin was automatically generated for this server at /opt/ibm/wlp/usr/servers/wlpSrv02/logs/state/plugin-cfg.xml.

See Automatic generation of the plugin-cfg.xml file in IBM Knowledge Center for more details.
And with the help of pluginUtility, we can either merge individual plugin-cfg.xml or generate a brand new merged plugin-cfg.xml. As part of this exercise, we are going to generate merged plugin-cfg.xml. See command in action below:

$> cd /opt/ibm/wlp/bin
$> ./pluginUtility generate \
  --server=wasadmin:<replace_with_your_password>@waslibctlr01:9443 \
  --cluster=wlpCluster \
  --targetPath=/tmp/plugin-cfg.xml

The remote target server waslibctlr01:9443 will be used to generate the webserver plugin configuration file.

SSL trust has not been established with the target server.

Certificate chain information:
Certificate [0]
Subject DN: CN=waslibctlr01, OU=wlpCntlr, O=ibm, C=us
...
Do you want to accept the above certificate chain? (y/n) y

The generation of the webserver plugin configuration file was successful for the target collective controller.

As per input command above, it generates plugin-cfg.xml under /tmp, which we will review later.
For pluginUtility command details and all available options, refer to pluginUtility command page at IBM Knowledge Center.

Note: If you are using WLP version 8.5.5.x where pluginUtility command is not available, you can call the generateClusterPluginConfig operation on the ClusterManager MBean to generate a merged plugin-cfg.xml file for all started cluster members. For working code example, refer to topic Generating a merged plug-in configuration for Liberty servers at IBM Knowledge Center.

Troubleshooting:
If your plugin-cfg.xml file generation fails with following error:
The generation of the webserver plugin configuration file failed for the target collective controller.
Analyze the logs of the target collective controller to diagnose the problem.

And you see following message in your Collective Controller messages.log
... FFDC1015I: An FFDC Incident has been created: "java.net.NoRouteToHostException: No route to host (Host unreachable) com.ibm.ws.collective.repository.internal.ClusterManager 323" at ffdc_17.11.11_18.08.48.0.log
...
It is possibly caused by the firewall on the server machine(s). Make sure to open port(s) (listed in the server.xml) for communications on all member server(s) as well as controller.

Here is how our generated plugin-cfg.xml looks like:
<?xml version="1.0" encoding="UTF-8"?>
<!-- This config file was generated by plugin's merge tool v1.0.0.2 on 2017.11.13 at 16:50:31 GMT -->
<Config ASDisableNagle="false" AcceptAllContent="false" AppServerPortPreference="HostHeader" ChunkedResponse="false" FIPSEnable="false" IISDisableNagle="false" IISPluginPriority="High" IgnoreDNSFailures="false" RefreshInterval="60" ResponseChunkSize="64" SSLConsolidate="false" TrustedProxyEnable="false" VHostMatchingCompat="false">
  <Log LogLevel="Debug" Name="/opt/IBM/WebSphere/Plugins/logs/webserver1/http_plugin.log"/>
  <Property Name="ESIEnable" Value="true"/>
  <Property Name="ESIMaxCacheSize" Value="1024"/>
  <Property Name="ESIInvalidationMonitor" Value="false"/>
  <Property Name="ESIEnableToPassCookies" Value="false"/>
  <Property Name="PluginInstallRoot" Value="/opt/IBM/WebSphere/Plugins"/>
  <!-- Server Clusters -->
  <ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="Shared_2_Cluster_0" PostBufferSize="0" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60" ServerIOTimeoutRetry="-1">
    <Server CloneID="8b2ad3bf-c30d-4c7e-9fab-ec856dfc20b7" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_defaultServer_1" ServerIOTimeout="900" WaitForContinue="false">
      <Transport Hostname="waslibmem01" Port="9081" Protocol="http"/>
      <Transport Hostname="waslibmem01" Port="9444" Protocol="https">
        <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>
        <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>
      </Transport>
    </Server>
    <Server CloneID="9c9bb9b9-d4b9-4bf8-98ef-336e3c5c436d" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="default_node_defaultServer_0" ServerIOTimeout="900" WaitForContinue="false">
      <Transport Hostname="waslibmem02" Port="9081" Protocol="http"/>
      <Transport Hostname="waslibmem02" Port="9444" Protocol="https">
        <Property Name="keyring" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb"/>
        <Property Name="stashfile" Value="/opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.sth"/>
      </Transport>
    </Server>
    <PrimaryServers>
      <Server Name="default_node_defaultServer_1"/>
      <Server Name="default_node_defaultServer_0"/>
    </PrimaryServers>
  </ServerCluster>
  <!-- Virtual Host Groups -->
  <VirtualHostGroup Name="/cell/sharedCell_2/vHostGroup/shared_host_0">
    <VirtualHost Name="*:443"/>
    <VirtualHost Name="*:80"/>
  </VirtualHostGroup>
  <!-- URI Groups -->
  <UriGroup Name="/cell/sharedCell_2/application/default_host_defaultServer_default_node_Cluster_URIs">
    <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/IBMJMXConnectorREST/*"/>
    <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/ibm/api/*"/>
    <Uri AffinityCookie="JSESSIONID" AffinityURLIdentifier="jsessionid" Name="/JavaHelloWorldApp/*"/>
  </UriGroup>
  <!-- Routes -->
  <Route ServerCluster="Shared_2_Cluster_0" UriGroup="/cell/sharedCell_2/application/default_host_defaultServer_default_node_Cluster_URIs" VirtualHostGroup="/cell/sharedCell_2/vHostGroup/shared_host_0"/>
</Config>

As you can see, I've highlighted few lines above and let's discuss about those.
  1. pluginUtility does not create keystore related files (plugin-key.kdb and plugin-key.sth) referenced in the plugin-cfg.xml file. Later in the post, we will talk how to create these file manually.
  2. Generated plugin-cfg.xml has default values. If you need to generate plugin-cfg.xml with certain values, define pluginConfiguration element (see below) in server.xml and regenerate the plugin-cfg.xml.

    <pluginConfiguration webserverPort="80"
      webserverSecurePort="443"
      sslKeyringLocation="path/to/sslkeyring"
      sslStashfileLocation="path/to/stashfile"
      sslCertlabel="definedbyuser"/>

    For all available configuration attributes of pluginConfiguration, see IBM Knowledge Center


Establishing Secure communication between IHS/Plug-in and WLP servers

As you can see in the generated plugin-cfg.xml, it has both http and https transport definitions. Https transport references the plugin-key.kdb/sth files, however, these files are not generated by pluginUtility. So, we need to create them manually. Below, you'll find two ways to create them:

1) Create key database (kdb) file and import Signer Certificate. For this purpose, you can use 'gskcmd' or 'IKEYMAN' that are installed as part of IHS. Below I'll show how to use 'gskcmd'.

$> cd /opt/IBM/HTTPServer/bin
# creates .kdb,sth, and .rdb files
$> ./gskcmd -keydb -create -db /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \
  -pw <replace_with_your_password> \
  -type kdb -expire 7300 -stash

# Add signer certificate. If you are using WAS generated keys on WAS Liberty,

# member root certificate can be extracted from trust.jks file under 
# ${server.config.dir}/resources/security directory.

$> ./gskcmd -cert -add -db /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \
  -pw <replace_with_your_password> \
  -file /tmp/myWASSigner.crt


# List the added certificates
$> ./gskcmd -cert -list -db /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \
   -pw <replace_with_your_password>

2) Convert JKS file into KDB file. The following command takes jks file as an input and creates plugin-key.kdb, plugin-key.rdb, and plugin-key.sth files.
If you are using WAS generated keys, you can find signer certificate in trust.jks file on any member server under ${server.config.dir}/resources/security directory.

$> ./gskcmd -keydb -convert -db /tmp/trust.jks \
  -pw <replace_with_your_password> \
  -new_format kdb \
  -target /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-key.kdb \
  -stash


IHS Certificate

In order to secure communication between the client/browser and IHS, you need to have either CA signed certificate (preferred) or self-signed certificate for IHS. For this exercise, we are going to use self-signed certificate. Please note that if your front-end communication is not secure, then back-end communication (from plug-in to WAS WLP) is also going to be not secure (by default). If you like to know more detail on how it works, see Few Tips on Secure vs. Non-Secure WAS Web Server Plug-in Connection.
Below command shows how to create a self-signed certificate using gskcmd:

$> cd /opt/IBM/HTTPServer/bin

# run gskcmd to create kdb file to store private and public keys for IHS
$> ./gskcmd -keydb -create -db /opt/IBM/HTTPServer/ssl/ihs.kdb \
  -pw <replace_with_your_password> \
  -type kdb -expire 7300 -stash


# create self signed cert:
$> ./gskcmd -cert -create -db /opt/IBM/HTTPServer/ssl/ihs.kdb \
  -label ihs_cert -pw <replace_with_your_password> \
  -type kdb -size 2048 -expire 7300 -default_cert yes \
  -dn "CN=waslibhihs01, OU=WebSphere, O=SysGenius, C=CA"

IHS and plug-in Configuration

Now, we have everything in order to configure IHS and WAS plug-in for IHS. This configuration is very straight forward and can be done manually. For details refer to https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_admin_webserver_plugin.html

Verify content of plugin-cfg.xml
  •  Make sure .kdb/.rdb/.sth file location is correct. 
  • log file location is correct. 
  • Both (http and https) transport definitions (host and port) are correct and resolvable from IHS server. You can do a telnet test for both http and https transport to verify. 
Update plugin-cfg.xml if required.



Update httpd.conf

1) Add plug-in related configuration in httpd.conf:

LoadModule was_ap22_module /opt/IBM/WebSphere/Plugins/bin/64bits/mod_was_ap22_http.so
WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml

2) Add configuration for front-end secure communication

  • Uncomment/enable the SSL module configuration directive.
  • Create an SSL virtual host stanza in the httpd.conf file using the following examples and directives.

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 443

<VirtualHost *:443>
   SSLEnable
</VirtualHost>

SSLDisable


KeyFile "/opt/IBM/HTTPServer/ssl/ihs.kdb"

Start IHS:

#Start IHS:
$> /opt/IBM/HTTPServer/bin

$> ./apachectl -k start

# See if it's started $> ps -ef | grep httpd
$> ps -ef | grep httpd
root 9 1 0 14:39 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody 13 9 0 14:39 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start
nobody 14 9 0 14:39 ? 00:00:00 /opt/IBM/HTTPServer/bin/httpd -d /opt/IBM/HTTPServer -k start

Review the error_log (under /opt/IBM/HTTPServer/logs) and make sure it does not have any error and it loaded plug-in successfully

[Sat Nov 18 14:39:21.325566 2017] [ibm_ssl:notice] [pid 9:tid 139878623614720] Using GSKit version 8.0.50.77
[Sat Nov 18 14:39:21.391142 2017] [was_ap24:notice] [pid 9:tid 139878623614720] ---------------------------------------------------
[Sat Nov 18 14:39:21.391164 2017] [was_ap24:notice] [pid 9:tid 139878623614720] WebSphere Plugins loaded.
[Sat Nov 18 14:39:21.391167 2017] [was_ap24:notice] [pid 9:tid 139878623614720] Bld version: 9.0.0.4
[Sat Nov 18 14:39:21.391170 2017] [was_ap24:notice] [pid 9:tid 139878623614720] Bld date: Apr 11 2017, 00:11:26
[Sat Nov 18 14:39:21.391172 2017] [was_ap24:notice] [pid 9:tid 139878623614720] Webserver: IBM_HTTP_Server/9.0.0.4 (Unix)
[Sat Nov 18 14:39:21.391174 2017] [was_ap24:notice] [pid 9:tid 139878623614720] ---------------------------------------------------
[Sat Nov 18 14:39:21.391234 2017] [:notice] [pid 9:tid 139878623614720] Using config file /opt/IBM/HTTPServer/conf/httpd.conf
[Sat Nov 18 14:39:21.391840 2017] [mpm_event:notice] [pid 9:tid 139878623614720] CoreDumpDirectory not set; core dumps may not be written for child process crashes
[Sat Nov 18 14:39:21.391841 2017] [mpm_event:notice] [pid 9:tid 139878623614720] AH00489: IBM_HTTP_Server/9.0.0.4 (Unix) configured -- resuming normal operations

And from http_plugin.log

[18/Nov/2017:14:39:21.39104] 00000009 07ac1700 - PLUGIN: Plugins loaded.
[18/Nov/2017:14:39:21.39105] 00000009 07ac1700 - PLUGIN: --------------------System Information-----------------------
[18/Nov/2017:14:39:21.39106] 00000009 07ac1700 - PLUGIN: Bld version: 9.0.0.4
[18/Nov/2017:14:39:21.39107] 00000009 07ac1700 - PLUGIN: Bld date: Apr 11 2017, 00:11:45
[18/Nov/2017:14:39:21.39107] 00000009 07ac1700 - PLUGIN: Webserver: IBM_HTTP_Server/9.0.0.4 (Unix)
[18/Nov/2017:14:39:21.39108] 00000009 07ac1700 - PLUGIN: OS : Linux x86_64
[18/Nov/2017:14:39:21.39109] 00000009 07ac1700 - PLUGIN: Hostname = waslibhihs01
[18/Nov/2017:14:39:21.39110] 00000009 07ac1700 - PLUGIN: NOFILES = hard: 65536, soft: 65536
[18/Nov/2017:14:39:21.39110] 00000009 07ac1700 - PLUGIN: MAX COREFILE SZ = hard: INFINITE, soft: INFINITE
[18/Nov/2017:14:39:21.39111] 00000009 07ac1700 - PLUGIN: DATA = hard: INFINITE, soft: INFINITE
[18/Nov/2017:14:39:21.39112] 00000009 07ac1700 - PLUGIN: --------------------------------------------------------------


Access the application through IHS:

https://waslibhihs01/JavaHelloWorldApp

Hurray! Here is JavaHelloWorldApp

And, check the access_log, you'll see something like this:

192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/ HTTP/1.1" 200 705 13237 192.168.56.109:9444 +
192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/style.css HTTP/1.1" 200 1157 973 +
192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/SimpleServlet HTTP/1.1" 200 12 15625 192.168.56.108:9444 +
192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/ HTTP/1.1" 200 705 13397 192.168.56.109:9444 +
192.168.56.1 - - [18/Nov/2017:14:40:17 +0000] "GET /JavaHelloWorldApp/SimpleServlet HTTP/1.1" 200 12 12708 192.168.56.108:9444 +

If you encounter any issue, make sure Web Server Plug-in, WebSphere Liberty and IHS (powered by Apache) are compatible. Refer to Supported combinations of IBM HTTP Server, WebSphere Application Server, and the WebSphere WebServer Plug-in for more details.

Thank you coming so far! It is the end of this series!


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below:


How to Create and Configure WebSphere Liberty Cluster End-to-End



Because of it's simplicity, light-weight, and scalability, popularity and adoption of WebSphere Application Server Liberty Profile (WLP) server is  growing. Recently, I was involved in one of the projects, that utilised the WLP Collective and Clustering capabilities. In order to share the knowledge, I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on setting up a WebSpere Liberty Collective and a Cluster. Below listed are all the posts in this series:
  1. How to Create and Configure WebSphere Liberty Cluster End-to-End
  2. How to Deploy Application in WebSphere Liberty Cluster
  3. How to Setup Front-End Web Server for WebSphere Liberty Cluster
In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server


WLP Installation 

There are several ways to install WLP. If you like to explore and compare best ways for your particular situation, take a look at IBM Knowledge Center documentation.

Check WLP System Requirements:

First, you need to decide what version of WLP is needed for your particular situation. I'm using version 17.0.2 to implement this example topology. System requirement for WLP 17.0.2 can be found here.  For this exercise, I will be using Virtual Machines with CentOS Linux release 7.3.1611.

Note: Before proceeding with the WLP installation, make sure you have supported version of Java installed in the machine. 

Here is what I have for this exercise:
$> cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
$> uname -a
Linux waslibctlr01 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$> ./java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build pxa6480sr4fp5-20170421_01(SR4 FP5))
IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64 Compressed References 20170419_344392 (JIT enabled, AOT enabled)
J9VM - R28_20170419_1004_B344392
JIT - tr.r14.java_20170419_344392
GC - R28_20170419_1004_B344392_CMPRSS
J9CL - 20170419_344392)
JCL - 20170420_01 based on Oracle jdk8u131-b11

Let's start with the WLP installation.
Important: first, I will install WLP in Machine: 02 by using downloaded WLP Jar installer, add all the required features and then package it and distribute to other servers.

This IBM Knowledge Center documentation explains in detail how to download and extract the WLP installer Jar. As shown below, I'm installing it under /opt/ibm directory:


# Create directory /opt/ibm
$> sudo mkdir -p /opt/ibm

# Change ownership if required
$> sudo chown -R wasadmin:wasgrp /opt/ibm

java -jar wlp-<edition>-all-<fix_pack>.jar --acceptLicense /opt/ibm

Note: If you have multiple Java/JRE installed on your machine and you need to setup proper Java/JRE for your WLP environment, there are few ways that you can accomplished this. See Customizing the Liberty environment.

Once WLP is intalled and Java Runtime is configured, then check the version.

$> cd /opt/ibm/wlp/bin

$> ./productInfo version
Product name: WebSphere Application Server
Product version: 17.0.0.2


Install Features

Here, I'm installing features required for Collective, Cluster, SSL, restConnector, and localConnector. All the required features can be installed once or one at a time. For all available features refer to Liberty Features at IBM Knowledge Center.

$> ./installUtility install collectiveController-1.0 collectiveMember-1.0 clusterMember-1.0 websocket-1.1 restConnector-2.0 ssl-1.0 localConnector-1.0 adminCenter-1.0

Establishing a connection to the configured repositories ...
This process might take several minutes to complete.

Successfully connected to all configured repositories.
Preparing assets for installation. This process might take several minutes to complete.

Additional Features Terms & Conditions:
By clicking on the "I agree" button , Licensee agrees that the program
code, samples, updates, fixes and related licensed materials such as
...
...
production workloads, simulating production workloads or testing
scalability of any code, application or system.

Select [1] I Agree, or [2] I do not Agree: 1

Step 1 of 12: Downloading clusterMember-1.0 ...
Step 2 of 12: Installing clusterMember-1.0 ...
Step 3 of 12: Downloading collectiveController-1.0 ...
Step 4 of 12: Installing collectiveController-1.0 ...
...
...
All assets were successfully installed.

Start product validation...
Product validation completed successfully.

Since, all required features are installed, it's time to package the installation as server package and distribute to other servers for installation. In order to  do this, I'm going to create a default WLP server and create a package. For more information about server packaging, refer to IBM Knowledge Center documentation. Below, I'm showing server packaging as a Jar file, because it preserves (mostly) the execute permission for the scripts in bin directory.

$> cd /opt/ibm/wlp/bin

# Create defaultServer
$> ./server create
Server defaultServer created.

# Create server package
$> ./server package defaultServer --archive=/tmp/wlp_install.jar --include=all
Packaging server defaultServer.
Server defaultServer package complete in /tmp/wlp_install.jar.

Now, copy the generated wlp_install.jar to other machines (Machine: 03, and Machine: 04 for this exercise) where WLP installation is required. Once wlp_install.jar is copied, proceed with installation. As shown below, we have copied the wlp_install.jar into /tmp and going to install WLP under /opt/ibm directory.

If you need to see the installation options, you can just run the --help as shown below:

java -jar wlp_install.jar --help

Usage
java -jar wlp_install.jar [Options] [install location]

Options
--acceptLicense
Automatically indicate acceptance of license terms and conditions.
--verbose
Display detailed information during archive extraction.
--viewLicenseAgreement
View license agreement.
--viewLicenseInfo
View license information.

Let's launch the installation on Machine: 03 and Machine: 04


# Create /opt/ibm directory and assign ownership to proper user:group
$> sudo mkdir -p /opt/ibm
$> chown -R <Your-WAS-user>:<Your-WAS-group> /opt/ibm

# Install WLP
$> java -jar wlp_install.jar --acceptLicense /opt/ibm

Once the installation is completed, verify the version and validate the product as shown below:

# Verify version
$> ./productInfo version
Product name: WebSphere Application Server
Product version: 17.0.0.2

# Validate installed product
$> ./productInfo validate
Start product validation...
Validating feature: adminCenter-1.0... PASS!
...
Validating feature: clusterMember-1.0... PASS!
Validating feature: collectiveController-1.0... PASS!
Validating feature: collectiveMember-1.0... PASS!
...
Validating feature: restConnector-1.0... PASS!
Validating feature: restConnector-2.0... PASS!
...
Validating feature: ssl-1.0... PASS!
...
Product validation completed successfully.

Troubleshooting:
If you get message ./productInfo: line 187: /opt/ibm/wlp/java/java/bin/java: Permission denied
while executing script, check and make sure the script files have execute permission. Jar packaging usually preserves the permission, but if you have Java that is installed under ${wlp.install.dir}/java directory and packaged as part of the installer, files under /java/bin and java/jre/bin may not preserve their execute permission. So, go ahead and add execute permission.


Setup Collective Controller

Once the WLP is installed on all servers that will be part of WLP Collective, let's start setting up Collective Controller. If you need to understand the concept of Liberty Collective, refer to IBM Knowledge Center article: Liberty: Collective architecture

Note: in given example topology (diagram 1.0), Collective Controller is setup on Machine: 02.

# Create server wlpCntlr, which will be configured as Collective Controller.
$> ./server create wlpCntlr
Server wlpCntlr created.

# Configure wlpCntlr as Collective Controller.
# Option --createConfigFile=<file-location> outputs the configuration in that file.


$> ./collective create wlpCntlr --keystorePassword=<replace_with_your_password> --createConfigFile=/opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml


Creating required certificates to establish a collective...
This may take a while.
Successfully generated the controller root certificate.
Successfully generated the member root certificate.
Successfully generated the server identity certificate.
Successfully generated the HTTPS certificate.

Successfully set up collective controller configuration for wlpCntlr.

Add the following lines to the server.xml to enable:

<include location="${server.config.dir}/wlpcntlr_include.xml" />

Please ensure administrative security is configured for the server.
An administrative user is required to join members to the collective.

As per our collective create ... command, /opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml is created with all required configuration for WLP Collective Controller. There are few manual updates required, before we can start our wlpCntlr server. Here they are:

1) Open server.xml and include wlpcntlr_include.xml in it:

# Here I'm using ${server.config.dir}/wlpcntlr_include.xml instead of /opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml
# Refer to Liberty Directory locations and properties
<include location="${server.config.dir}/wlpcntlr_include.xml"/>

2) Open server.xml and make sure to specify host attribute with proper value for httpEndPoint element. See below for example.

<httpEndpoint id="defaultHttpEndpoint"
httpPort="9080"
httpsPort="9443" host="*"/>

3) Open generated include file. In our case it is wlpcntlr_include.xml and update quickStartSecurity element with correct values for userName and userPassword.

<quickStartSecurity userName="<replace-with-your-userName>" userPassword="<replace-with-your-password>" />

If you need more explanation about WLP Collective Controller configuration, refer to Configuring a Liberty collective chapter in IBM Knowledge Center.

Note: I've also added following features in WLP Collective Controller server configuration:
  • adminCenter-1.0: provides Web Based GUI for collective/cluster/server and application management and monitoring.  See Administering Liberty using Admin Center
  • websocket-1.1:  protocol that enables a client and a server application to communicate by using one full duplex connection. See WebSocker for details.
  • restConnector-1.0: provides a secure JMX connector that can be used locally/remotely. Refer to Connecting to Liberty by using JMX for details.
  • localConnector-1.0: provides a local JMX connector that is built into the JVM.
Below is complete listing of server.xml and wlpcntlr_include.xml for wlpCntlr  (Collective Controller) server.

server.xml:
<?xml version="1.0" encoding="UTF-8"?>
<server description="CollectiveController">
   <!-- Enable features -->
   <featureManager>
      <feature>adminCenter-1.0</feature>
      <feature>websocket-1.1</feature>
      <feature>restConnector-1.0</feature>
      <feature>localConnector-1.0</feature>
   </featureManager>
   <include location="${server.config.dir}/wlpcntlr_include.xml" />
   <!-- To access this server from a remote client add a host attribute e.g. host="*" -->
   <httpEndpoint id="defaultHttpEndpoint"
      httpPort="9080"
      httpsPort="9443" host="*"/>
</server>

wlpcntlr_include.xml
<?xml version="1.0" encoding="UTF-8" ?>
<server description="This file was generated by the 'collective create' command on 2017-10-29 12:59:19 GMT.">
   <featureManager>
      <feature>collectiveController-1.0</feature>
   </featureManager>

   <!-- Define the host name for use by the collective.
    If the host name needs to be changed, the server should be
    removed from the collective and re-joined or re-replicated. -->
   <variable name="defaultHostName" value="waslibctlr01" />

   <!-- TODO: Set the security configuration for Administrative access -->
   <quickStartSecurity userName="wasadmin" userPassword="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs=" />

   <!-- clientAuthenticationSupported set to enable bidirectional trust -->
   <ssl id="defaultSSLConfig"
       keyStoreRef="defaultKeyStore"
       trustStoreRef="defaultTrustStore"
       clientAuthenticationSupported="true" />

   <!-- inbound (HTTPS) keystore -->
   <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/security/key.jks" />

   <!-- inbound (HTTPS) truststore -->
   <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/security/trust.jks" />

   <!-- server identity keystore -->
   <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/collective/serverIdentity.jks" />

   <!-- collective trust keystore -->
   <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/collective/collectiveTrust.jks" />

   <!-- collective root signers keystore -->
   <keyStore id="collectiveRootKeys" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/collective/rootKeys.jks" />

</server>

Based on from where and how you are accessing your Collective Controller server and how many firewall(s) in between, you may need to open firewall port(s) to allow communication. Below is command that I needed to run to open ports 9080 and 9443 so that CentOS allowed the incoming requests.
# Check to see if port 9443 and 9080 open
$> sudo firewall-cmd --zone=public --query-port=9443/tcp
no
$> sudo firewall-cmd --zone=public --query-port=9080/tcp
no
# Open port 9080 and 9443
$> sudo firewall-cmd --zone=public --permanent --add-port=9443/tcp
success
$> sudo firewall-cmd --zone=public --permanent --add-port=9080/tcp
success
# recycle firewall-cmd
$> sudo firewall-cmd --reload

Start WLP Collective Controller server & Verify

Start:
#Start the server
$>cd /opt/ibm/wlp/bin


$> ./server start wlpCntlr
Starting server wlpCntlr.
Server wlpCntlr started with process ID 3840.

Check the messages.log. By default, it is located under ${server.config.dir}/logs directory. Make sure log file doesn't have any error and you see the message something like 'CWWKX6011I: The collective controller is ready, and can accept requests...".
If you have added feature adminCenter-1.0, you should also see a message something like ' CWWKT0016I: Web application available (default_host): http://waslibctlr01:9080/adminCenter/, in the log file

Access the Admin Center. Launch your browser and type the Admin Center URL seen on your log file. If it is http port, it will be redirected to https port automatically. Use the same userName and userPassword that you defined above in the quickStartSecurity element. See below screen shot of Admin Center landing page

Admin Center landing page

Create & Configure Collective and Cluster Member

You can have one or more WLP Collective and/or Cluster member server(s) in the same host (Vertical clustering/scaling) or different host (Horizontal clustering/scaling) or mix of Vertical and Horizontal clustering/scaling. In this exercise, as defined in the diagram 1.0, I'm doing Horizontal clustering/scaling with two member servers residing on two separate hosts.


Create 1st Collective and Cluster Member 

Below commands are executed in Machine: 03

# Create member server wlpSrv01:
$> cd /opt/ibm/wlp/bin
$> ./server create wlpSrv01
Server wlpSrv01 created.

# Add member to Collective:
# Note: execute this command from the host where this member server resides (Machine: 01):

$> cd /opt/ibm/wlp/bin

$> ./collective join wlpSrv01 \
  --host=waslibctlr01 \
  --port=9443 \
  --user=wasadmin \
  --password=<replace_with_your_password> \
  --keystorePassword=<replace_with_your_password> \
  --createConfigFile=/opt/ibm/wlp/usr/servers/wlpSrv01/wlpsrv01_include.xml

Joining the collective with target controller waslibctlr01:9443...
This may take a while.

SSL trust has not been established with the target server.

Certificate chain information:
Certificate [0]
Subject DN: CN=waslibctlr01, OU=wlpCntlr, O=ibm, C=us
Issuer DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective
...

Certificate [1]
Subject DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective
Issuer DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective
...

Do you want to accept the above certificate chain? (y/n) y
Successfully completed MBean request to the controller.

Successfully joined the collective for server wlpSrv01.

Add the following lines to the server.xml to enable:

Troubleshooting:
If you encounter issues like one of these (below) while joining the WLP server to the Collective:
  1. CWWKX0229E: There was a problem with the user credentials provided. The server responded with code 401 and message 'Unauthorized'
  2. CWWKX0229E: There was a problem with the user credentials provided. The server responded with code 403 and message 'Forbidden'
  • Error with response code 401 is mainly because of unsuccessful authentication. Check and make sure 'user' and 'password' values specified in collective join ... command match with the corresponding values specified in quickStartSecurity or basicRegistry element for Collective Controller. 
  • Error with response code 403 comes because the authenticated user does not have required permission. In this case make sure: you are only using either quickStartSecurity or basicRegistry If you defined both, remove one).
  • In case if you are using basicRegistry, make sure to provide administrator role to the user being used in collective join .... Something like:

<quickStartSecurity userName="wasadmin" userPassword="<replace-with-your-password>"/>

OR

<basicRegistry id="basic" realm="WebRealm">
   <user name="wasadmin" password="<replace-with-your-password>" />
</basicRegistry>
<administrator-role>
   <user>wasadmin</user>
</administrator-role>

For detail refer to Setting up BasicRegistry and role mapping on Liberty.

If collective join ... commands completes successfully, (as per our command input) it generates  /opt/ibm/wlp/usr/servers/wlpSrv01/wlpsrv01_include.xml with all required configuration for WLP Collective Member. There are few manual updates required, before we can start our wlpSrv01 server. Here they are:

1) Open server.xml for wlpSrv01 and include wlpsrv01_include.xml in it:
<include location="${server.config.dir}/wlpsrv01_include.xml" />

2) Define Cluster and Cluster Member:
Open server.xml for Collective Member server, and add the following:
<-- Declares this server as a member of defined cluster -->
<-- Add following line under <featureManager> element. -->

<feature>clusterMember-1.0</feature>

<-- Defines cluster wlpCluster -->

<clusterMember name="wlpCluster"/>

<-- Following provides the write access to Collective Controller under ${server.config.dir} of Member server. It is needed for deployment and configuration managment. For more information, refer to IBM Knowledge Center Document.-->

<remoteFileAccess>
   <writeDir>${server.config.dir}</writeDir>
</remoteFileAccess>

For more information on setting WLP Cluster, refer to Configuring a Liberty server cluster.
Below is complete listing of server.xml and wlpsrv01_include.xml for wlpSrv01  (Collective and Cluster Member server) server.

server.xml
<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

 <!-- Enable features -->
 <featureManager>
  <feature>webProfile-7.0</feature>
  <feature>restConnector-1.0</feature>
  <feature>localConnector-1.0</feature>
 </featureManager>
 <include location="${server.config.dir}/wlpsrv01_include.xml" />

 <!-- To access this server from a remote client add a host attribute, e.g. host="*" -->
 <httpEndpoint id="defaultHttpEndpoint"
  httpPort="9081"
  httpsPort="9444" host="*"/>

 <!-- Automatically expand WAR files and EAR files -->
 <!-- <applicationManager autoExpand="true"/> -->
</server>

wlpSrv01_include.xml
<?xml version="1.0" encoding="UTF-8" ?>
<server description="This file was generated by the 'collective join' command ...">
  <featureManager>
    <feature>collectiveMember-1.0</feature>
    <feature>clusterMember-1.0</feature>
  </featureManager>
  <clusterMember name="wlpCluster"/>
  <remoteFileAccess>
    <writeDir>${server.config.dir}</writeDir>
  </remoteFileAccess>

  <!-- Define the host name for use by the collective.
  If the host name needs to be changed, the server should be
  removed from the collective and re-joined or re-replicated. -->
  <variable name="defaultHostName" value="waslibmem01" />

  <!-- Connection to the collective controller -->
  <collectiveMember controllerHost="waslibctlr01"
    controllerPort="9443" />

  <!-- clientAuthenticationSupported set to enable bidirectional trust -->
  <ssl id="defaultSSLConfig"
    keyStoreRef="defaultKeyStore"
    trustStoreRef="defaultTrustStore"
    clientAuthenticationSupported="true" />
  <!-- inbound (HTTPS) keystore -->
  <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/key.jks" />

  <!-- inbound (HTTPS) truststore -->
  <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/trust.jks" />

  <!-- server identity keystore -->
  <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/serverIdentity.jks" />

  <!-- collective truststore -->
  <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/collectiveTrust.jks" />

</server>

Note:
1) Make sure both HTTP and HTTPS ports that server listens are open on firewall(s).
2) In order to avoid port conflict, if you have more than one WLP server running on same host, make sure listen ports are unique for each of them. Usually by default WLP assigns 9080 (http), and 9443 (https).

Create 2nd Collective and Cluster Member

Below commands are executed in Machine: 04

Follow the same steps outlined in creating 1st Collective and Cluster Member . Here, I'm assigning name wlpSrv02 for my 2nd server.
Below is complete listing of server.xml and wlpsrv02_include.xml for wlpSrv02  (Collective and Cluster Member server) server.

Server.xml
<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

 <!-- Enable features -->
 <featureManager>
  <feature>webProfile-7.0</feature>
  <feature>restConnector-1.0</feature>
  <feature>localConnector-1.0</feature>
 </featureManager>
 <include location="${server.config.dir}/wlpsrv02_include.xml" />

 <!-- To access this server from a remote client add a host attribute, e.g. host="*" -->
 <httpEndpoint id="defaultHttpEndpoint"
  httpPort="9081"
  httpsPort="9444" host="*"/>

 <!-- Automatically expand WAR files and EAR files -->
 <!-- <applicationManager autoExpand="true"/> -->
</server>

wlpSrv02_include.xml
<?xml version="1.0" encoding="UTF-8" ?>
<server description="This file was generated by the 'collective join' command ...">
  <featureManager>
    <feature>collectiveMember-1.0</feature>
    <feature>clusterMember-1.0</feature>
  </featureManager>
  <clusterMember name="wlpCluster"/>
  <remoteFileAccess>
    <writeDir>${server.config.dir}</writeDir>
  </remoteFileAccess>

  <!-- Define the host name for use by the collective.
  If the host name needs to be changed, the server should be
  removed from the collective and re-joined or re-replicated. -->
  <variable name="defaultHostName" value="waslibmem02" />

  <!-- Connection to the collective controller -->
  <collectiveMember controllerHost="waslibctlr01"
    controllerPort="9443" />

  <!-- clientAuthenticationSupported set to enable bidirectional trust -->
  <ssl id="defaultSSLConfig"
    keyStoreRef="defaultKeyStore"
    trustStoreRef="defaultTrustStore"
    clientAuthenticationSupported="true" />
  <!-- inbound (HTTPS) keystore -->
  <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/key.jks" />

  <!-- inbound (HTTPS) truststore -->
  <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/trust.jks" />

  <!-- server identity keystore -->
  <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/serverIdentity.jks" />

  <!-- collective truststore -->
  <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/collectiveTrust.jks" />

</server>



Note: Make sure all members in the cluster use the same LTPA keys.
For secure applications to function properly in a WLP clustered environment, each member of a given cluster must use the same LTPA key. For simplicity, Choose the LTPA key file from one of the cluster members and copy it to all of the other members in the cluster. The default LTPA key file is ${server.ouput.dir}/resources/security/ltpa.keys. For more information, refer to Creating a Liberty cluster with security considerations.

Start Member Server(s)

#Start wlpSrv01 on Machine: 03

$> ./server start wlpSrv01

Starting server wlpSrv01.
Server wlpSrv01 started with process ID 6639.

#Start wlpSrv02 on Machine: 04:

$> ./server start wlpSrv02

Starting server wlpSrv02.
Server wlpSrv02 started with process ID 5002.

Make sure member servers started without any issue and joined the Collective and Cluster. Easy way to do it is to check the messages.log. Check and make sure no errors there and verify messages something like below:
Successful start: ... A CWWKF0011I: The server wlpSrv02 is ready to run a smarter planet.
Successfully joined the Collective:... I CWWKX8112I: The server's host information was successfully published to the collective repository.
Successfully joined the Cluster:... CWWKX7400I: The ClusterMember MBean is available.

Once the collective member successfully join the collective and successfully started, further administrative work can be carried out using Admin Center Console.

Troubleshooting:
If you are enabling <feature>javaee-7.0</feature> which along with other services also supports Enterprise JavaBeans (EJB) Lite, and Java Message Service (JMS). You may see following errors depending upon your configuration:

1) CWWKS9582E:
... ibm.ws.transport.iiop.security.AbstractCsiv2SubsystemFactory E CWWKS9582E: The [defaultSSLConfig] sslRef attributes required by the orb element with the defaultOrb id have not been resolved within 10 seconds. As a result, the applications will not start. Ensure that you have included a keyStore element and that Secure Sockets Layer (SSL) is configured correctly.

Root cause: Usually SSL key is not available for secure iiop listener.
Resolution: Define SSL config:
If using default key store ${server.config.dir}/resources/security/key.jks, configure as follows:

<keyStore password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="/>

OR, if defining custom key.jks and trust.jks, configure as follows:

<ssl id="defaultSSLConfig"
  keyStoreRef="defaultKeyStore"
  trustStoreRef="defaultTrustStore"
  clientAuthenticationSupported="true" />
<keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="   location="${server.config.dir}/resources/security/key.jks" />
<keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="   location="${server.config.dir}/resources/security/trust.jks" />


2) CWWKO0221E:
... com.ibm.ws.tcpchannel.internal.TCPPort E CWWKO0221E: TCP Channel wasJmsEndpoint329 initialization did not succeed. The socket bind did not succeed for host localhost and port 7276. The port might already be in use.

Root cause: default port for JMS listener is in use. Most probably you have more than one WLP instances running on the same host and both listening on same listener port for JMS.

Resolution: Change the default wasJmsPort as follows:

<wasJmsEndpoint id="InboundJmsCommsEndpoint" host="*" wasJmsPort="9011" wasJmsSSLPort="9100" /> 

Refer to https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_msg_multi_server.html for details.


3) CWWKS9580E:
... ibm.ws.transport.iiop.security.config.ssl.yoko.SocketFactory E CWWKS9580E: The server socket could not be opened on localhost:2,809. The exception message is Address already in use.
... com.ibm.ws.logging.internal.impl.IncidentImpl I FFDC1015I: An FFDC Incident has been created: "org.omg.CORBA.INITIALIZE: : vmcid: 0x0 minor code: 0x0 completed: No com.ibm.ws.transport.iiop.internal.ORBWrapperInternal 109" at ffdc_17.12.04_15.43.42.0.log


Root cause: default port for iiop is in use. Most probably you have more than one WLP instances running on the same host and both listening on same port for IIOP.
Resolution: change iiop ports as follows:

<iiopEndpoint host="*" id="defaultIiopEndpoint" iiopPort="2709">
   <iiopsOptions iiopsPort="2815" /> 
</iiopEndpoint>

Next: proceed to application deployment -->


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below: