How to Create and Configure WebSphere Liberty Cluster End-to-End

Because of it's simplicity, light-weight, and scalability, popularity and adoption of WebSphere Application Server Liberty Profile (WLP) server is  growing. Recently, I was involved in one of the projects, that utilised the WLP Collective and Clustering capabilities. In order to share the knowledge, I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on setting up a WebSpere Liberty Collective and a Cluster. Below listed are all the posts in this series:

1. How to Create and Configure WebSphere Liberty Cluster End-to-End
2. How to Deploy Application in WebSphere Liberty Cluster
3. How to Setup Front-End Web Server for WebSphere Liberty Cluster

In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server

WLP Installation 

There are several ways to install WLP. If you like to explore and compare best ways for your particular situation, take a look at IBM Knowledge Center documentation.

Check WLP System Requirements:

First, you need to decide what version of WLP is needed for your particular situation. I'm using version 17.0.2 to implement this example topology. System requirement for WLP 17.0.2 can be found here.  For this exercise, I will be using Virtual Machines with CentOS Linux release 7.3.1611.

Note: Before proceeding with the WLP installation, make sure you have supported version of Java installed in the machine. 

Here is what I have for this exercise:

$> cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) $> uname -a Linux waslibctlr01 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux $> ./java -version java version "1.8.0" Java(TM) SE Runtime Environment (build pxa6480sr4fp5-20170421_01(SR4 FP5)) IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64 Compressed References 20170419_344392 (JIT enabled, AOT enabled) J9VM - R28_20170419_1004_B344392 JIT - tr.r14.java_20170419_344392 GC - R28_20170419_1004_B344392_CMPRSS J9CL - 20170419_344392) JCL - 20170420_01 based on Oracle jdk8u131-b11

Let's start with the WLP installation.
Important: first, I will install WLP in Machine: 02 by using downloaded WLP Jar installer, add all the required features and then package it and distribute to other servers.

This IBM Knowledge Center documentation explains in detail how to download and extract the WLP installer Jar. As shown below, I'm installing it under /opt/ibm directory:

# Create directory /opt/ibm $> sudo mkdir -p /opt/ibm # Change ownership if required $> sudo chown -R wasadmin:wasgrp /opt/ibm java -jar wlp-<edition>-all-<fix_pack>.jar --acceptLicense /opt/ibm

Note: If you have multiple Java/JRE installed on your machine and you need to setup proper Java/JRE for your WLP environment, there are few ways that you can accomplished this. See Customizing the Liberty environment.

Once WLP is intalled and Java Runtime is configured, then check the version.

$> cd /opt/ibm/wlp/bin $> ./productInfo version Product name: WebSphere Application Server Product version: 17.0.0.2

Install Features

Here, I'm installing features required for Collective, Cluster, SSL, restConnector, and localConnector. All the required features can be installed once or one at a time. For all available features refer to Liberty Features at IBM Knowledge Center.

$> ./installUtility install collectiveController-1.0 collectiveMember-1.0 clusterMember-1.0 websocket-1.1 restConnector-2.0 ssl-1.0 localConnector-1.0 adminCenter-1.0 Establishing a connection to the configured repositories ... This process might take several minutes to complete. Successfully connected to all configured repositories. Preparing assets for installation. This process might take several minutes to complete. Additional Features Terms & Conditions: By clicking on the "I agree" button , Licensee agrees that the program code, samples, updates, fixes and related licensed materials such as ... ... production workloads, simulating production workloads or testing scalability of any code, application or system. Select [1] I Agree, or [2] I do not Agree: 1 Step 1 of 12: Downloading clusterMember-1.0 ... Step 2 of 12: Installing clusterMember-1.0 ... Step 3 of 12: Downloading collectiveController-1.0 ... Step 4 of 12: Installing collectiveController-1.0 ... ... ... All assets were successfully installed. Start product validation... Product validation completed successfully.

Since, all required features are installed, it's time to package the installation as server package and distribute to other servers for installation. In order to  do this, I'm going to create a default WLP server and create a package. For more information about server packaging, refer to IBM Knowledge Center documentation. Below, I'm showing server packaging as a Jar file, because it preserves (mostly) the execute permission for the scripts in bin directory.

$> cd /opt/ibm/wlp/bin # Create defaultServer $> ./server create Server defaultServer created. # Create server package $> ./server package defaultServer --archive=/tmp/wlp_install.jar --include=all Packaging server defaultServer. Server defaultServer package complete in /tmp/wlp_install.jar.

Now, copy the generated wlp_install.jar to other machines (Machine: 03, and Machine: 04 for this exercise) where WLP installation is required. Once wlp_install.jar is copied, proceed with installation. As shown below, we have copied the wlp_install.jar into /tmp and going to install WLP under /opt/ibm directory.

If you need to see the installation options, you can just run the --help as shown below:

java -jar wlp_install.jar --help Usage java -jar wlp_install.jar [Options] [install location] Options --acceptLicense Automatically indicate acceptance of license terms and conditions. --verbose Display detailed information during archive extraction. --viewLicenseAgreement View license agreement. --viewLicenseInfo View license information.

Let's launch the installation on Machine: 03 and Machine: 04

# Create /opt/ibm directory and assign ownership to proper user:group $> sudo mkdir -p /opt/ibm $> chown -R <Your-WAS-user>:<Your-WAS-group> /opt/ibm # Install WLP $> java -jar wlp_install.jar --acceptLicense /opt/ibm

Once the installation is completed, verify the version and validate the product as shown below:

# Verify version $> ./productInfo version Product name: WebSphere Application Server Product version: 17.0.0.2 # Validate installed product $> ./productInfo validate Start product validation... Validating feature: adminCenter-1.0... PASS! ... Validating feature: clusterMember-1.0... PASS! Validating feature: collectiveController-1.0... PASS! Validating feature: collectiveMember-1.0... PASS! ... Validating feature: restConnector-1.0... PASS! Validating feature: restConnector-2.0... PASS! ... Validating feature: ssl-1.0... PASS! ... Product validation completed successfully.

Troubleshooting: If you get message ./productInfo: line 187: /opt/ibm/wlp/java/java/bin/java: Permission denied while executing script, check and make sure the script files have execute permission. Jar packaging usually preserves the permission, but if you have Java that is installed under ${wlp.install.dir}/java directory and packaged as part of the installer, files under /java/bin and java/jre/bin may not preserve their execute permission. So, go ahead and add execute permission.

Setup Collective Controller

Once the WLP is installed on all servers that will be part of WLP Collective, let's start setting up Collective Controller. If you need to understand the concept of Liberty Collective, refer to IBM Knowledge Center article: Liberty: Collective architecture

Note: in given example topology (diagram 1.0), Collective Controller is setup on Machine: 02.

# Create server wlpCntlr, which will be configured as Collective Controller. $> ./server create wlpCntlr Server wlpCntlr created. # Configure wlpCntlr as Collective Controller. # Option --createConfigFile=<file-location> outputs the configuration in that file. $> ./collective create wlpCntlr --keystorePassword=<replace_with_your_password> --createConfigFile=/opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml Creating required certificates to establish a collective... This may take a while. Successfully generated the controller root certificate. Successfully generated the member root certificate. Successfully generated the server identity certificate. Successfully generated the HTTPS certificate. Successfully set up collective controller configuration for wlpCntlr. Add the following lines to the server.xml to enable: <include location="${server.config.dir}/wlpcntlr_include.xml" /> Please ensure administrative security is configured for the server. An administrative user is required to join members to the collective.

As per our collective create ... command, /opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml is created with all required configuration for WLP Collective Controller. There are few manual updates required, before we can start our wlpCntlr server. Here they are:

1) Open server.xml and include wlpcntlr_include.xml in it:

# Here I'm using ${server.config.dir}/wlpcntlr_include.xml instead of /opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml # Refer to Liberty Directory locations and properties <include location="${server.config.dir}/wlpcntlr_include.xml"/>

2) Open server.xml and make sure to specify host attribute with proper value for httpEndPoint element. See below for example.

<httpEndpoint id="defaultHttpEndpoint" httpPort="9080" httpsPort="9443" host="*"/>

3) Open generated include file. In our case it is wlpcntlr_include.xml and update quickStartSecurity element with correct values for userName and userPassword.

<quickStartSecurity userName="<replace-with-your-userName>" userPassword="<replace-with-your-password>" />

If you need more explanation about WLP Collective Controller configuration, refer to Configuring a Liberty collective chapter in IBM Knowledge Center.

Note: I've also added following features in WLP Collective Controller server configuration:

• adminCenter-1.0: provides Web Based GUI for collective/cluster/server and application management and monitoring.  See Administering Liberty using Admin Center
• websocket-1.1:  protocol that enables a client and a server application to communicate by using one full duplex connection. See WebSocker for details.
• restConnector-1.0: provides a secure JMX connector that can be used locally/remotely. Refer to Connecting to Liberty by using JMX for details.
• localConnector-1.0: provides a local JMX connector that is built into the JVM.

Below is complete listing of server.xml and wlpcntlr_include.xml for wlpCntlr  (Collective Controller) server.

server.xml:

<?xml version="1.0" encoding="UTF-8"?> <server description="CollectiveController">    <!-- Enable features -->    <featureManager>       <feature>adminCenter-1.0</feature>       <feature>websocket-1.1</feature>       <feature>restConnector-1.0</feature>       <feature>localConnector-1.0</feature>    </featureManager>    <include location="${server.config.dir}/wlpcntlr_include.xml" />    <!-- To access this server from a remote client add a host attribute e.g. host="*" -->    <httpEndpoint id="defaultHttpEndpoint"       httpPort="9080"       httpsPort="9443" host="*"/> </server>

wlpcntlr_include.xml

<?xml version="1.0" encoding="UTF-8" ?> <server description="This file was generated by the 'collective create' command on 2017-10-29 12:59:19 GMT.">    <featureManager>       <feature>collectiveController-1.0</feature>    </featureManager>    <!-- Define the host name for use by the collective.     If the host name needs to be changed, the server should be     removed from the collective and re-joined or re-replicated. -->    <variable name="defaultHostName" value="waslibctlr01" />    <!-- TODO: Set the security configuration for Administrative access -->    <quickStartSecurity userName="wasadmin" userPassword="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs=" />    <!-- clientAuthenticationSupported set to enable bidirectional trust -->    <ssl id="defaultSSLConfig"        keyStoreRef="defaultKeyStore"        trustStoreRef="defaultTrustStore"        clientAuthenticationSupported="true" />    <!-- inbound (HTTPS) keystore -->    <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="        location="${server.config.dir}/resources/security/key.jks" />    <!-- inbound (HTTPS) truststore -->    <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="        location="${server.config.dir}/resources/security/trust.jks" />    <!-- server identity keystore -->    <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="        location="${server.config.dir}/resources/collective/serverIdentity.jks" />    <!-- collective trust keystore -->    <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="        location="${server.config.dir}/resources/collective/collectiveTrust.jks" />    <!-- collective root signers keystore -->    <keyStore id="collectiveRootKeys" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="        location="${server.config.dir}/resources/collective/rootKeys.jks" /> </server>

Based on from where and how you are accessing your Collective Controller server and how many firewall(s) in between, you may need to open firewall port(s) to allow communication. Below is command that I needed to run to open ports 9080 and 9443 so that CentOS allowed the incoming requests.

# Check to see if port 9443 and 9080 open $> sudo firewall-cmd --zone=public --query-port=9443/tcp no $> sudo firewall-cmd --zone=public --query-port=9080/tcp no # Open port 9080 and 9443 $> sudo firewall-cmd --zone=public --permanent --add-port=9443/tcp success $> sudo firewall-cmd --zone=public --permanent --add-port=9080/tcp success # recycle firewall-cmd $> sudo firewall-cmd --reload

Start WLP Collective Controller server & Verify

Start:

#Start the server $>cd /opt/ibm/wlp/bin $> ./server start wlpCntlr Starting server wlpCntlr. Server wlpCntlr started with process ID 3840.

Check the messages.log. By default, it is located under ${server.config.dir}/logs directory. Make sure log file doesn't have any error and you see the message something like 'CWWKX6011I: The collective controller is ready, and can accept requests...".
If you have added feature adminCenter-1.0, you should also see a message something like ' CWWKT0016I: Web application available (default_host): http://waslibctlr01:9080/adminCenter/, in the log file

Access the Admin Center. Launch your browser and type the Admin Center URL seen on your log file. If it is http port, it will be redirected to https port automatically. Use the same userName and userPassword that you defined above in the quickStartSecurity element. See below screen shot of Admin Center landing page

Admin Center landing page

Create & Configure Collective and Cluster Member

You can have one or more WLP Collective and/or Cluster member server(s) in the same host (Vertical clustering/scaling) or different host (Horizontal clustering/scaling) or mix of Vertical and Horizontal clustering/scaling. In this exercise, as defined in the diagram 1.0, I'm doing Horizontal clustering/scaling with two member servers residing on two separate hosts.

Create 1st Collective and Cluster Member 

Below commands are executed in Machine: 03

# Create member server wlpSrv01: $> cd /opt/ibm/wlp/bin $> ./server create wlpSrv01 Server wlpSrv01 created. # Add member to Collective: # Note: execute this command from the host where this member server resides (Machine: 01): $> cd /opt/ibm/wlp/bin $> ./collective join wlpSrv01 \   --host=waslibctlr01 \   --port=9443 \   --user=wasadmin \   --password=<replace_with_your_password> \   --keystorePassword=<replace_with_your_password> \   --createConfigFile=/opt/ibm/wlp/usr/servers/wlpSrv01/wlpsrv01_include.xml Joining the collective with target controller waslibctlr01:9443... This may take a while. SSL trust has not been established with the target server. Certificate chain information: Certificate [0] Subject DN: CN=waslibctlr01, OU=wlpCntlr, O=ibm, C=us Issuer DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective ... Certificate [1] Subject DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective Issuer DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective ... Do you want to accept the above certificate chain? (y/n) y Successfully completed MBean request to the controller. Successfully joined the collective for server wlpSrv01. Add the following lines to the server.xml to enable:

Troubleshooting: If you encounter issues like one of these (below) while joining the WLP server to the Collective: CWWKX0229E: There was a problem with the user credentials provided. The server responded with code 401 and message 'Unauthorized' CWWKX0229E: There was a problem with the user credentials provided. The server responded with code 403 and message 'Forbidden' Error with response code 401 is mainly because of unsuccessful authentication. Check and make sure 'user' and 'password' values specified in collective join ... command match with the corresponding values specified in quickStartSecurity or basicRegistry element for Collective Controller.  Error with response code 403 comes because the authenticated user does not have required permission. In this case make sure: you are only using either quickStartSecurity or basicRegistry If you defined both, remove one). In case if you are using basicRegistry, make sure to provide administrator role to the user being used in collective join .... Something like: <quickStartSecurity userName="wasadmin" userPassword="<replace-with-your-password>"/> OR <basicRegistry id="basic" realm="WebRealm">    <user name="wasadmin" password="<replace-with-your-password>" /> </basicRegistry> <administrator-role>    <user>wasadmin</user> </administrator-role> For detail refer to Setting up BasicRegistry and role mapping on Liberty.

If collective join ... commands completes successfully, (as per our command input) it generates  /opt/ibm/wlp/usr/servers/wlpSrv01/wlpsrv01_include.xml with all required configuration for WLP Collective Member. There are few manual updates required, before we can start our wlpSrv01 server. Here they are:

1) Open server.xml for wlpSrv01 and include wlpsrv01_include.xml in it:

<include location="${server.config.dir}/wlpsrv01_include.xml" />

2) Define Cluster and Cluster Member:
Open server.xml for Collective Member server, and add the following:

<-- Declares this server as a member of defined cluster --> <-- Add following line under <featureManager> element. --> <feature>clusterMember-1.0</feature> <-- Defines cluster wlpCluster --> <clusterMember name="wlpCluster"/> <-- Following provides the write access to Collective Controller under ${server.config.dir} of Member server. It is needed for deployment and configuration managment. For more information, refer to IBM Knowledge Center Document.--> <remoteFileAccess>    <writeDir>${server.config.dir}</writeDir> </remoteFileAccess>

For more information on setting WLP Cluster, refer to Configuring a Liberty server cluster.
Below is complete listing of server.xml and wlpsrv01_include.xml for wlpSrv01  (Collective and Cluster Member server) server.

server.xml

<?xml version="1.0" encoding="UTF-8"?> <server description="new server">  <!-- Enable features -->  <featureManager>   <feature>webProfile-7.0</feature>   <feature>restConnector-1.0</feature>   <feature>localConnector-1.0</feature>  </featureManager>  <include location="${server.config.dir}/wlpsrv01_include.xml" />  <!-- To access this server from a remote client add a host attribute, e.g. host="*" -->  <httpEndpoint id="defaultHttpEndpoint"   httpPort="9081"   httpsPort="9444" host="*"/>  <!-- Automatically expand WAR files and EAR files -->  <!-- <applicationManager autoExpand="true"/> --> </server>

wlpSrv01_include.xml

<?xml version="1.0" encoding="UTF-8" ?> <server description="This file was generated by the 'collective join' command ...">   <featureManager>     <feature>collectiveMember-1.0</feature>     <feature>clusterMember-1.0</feature>   </featureManager>   <clusterMember name="wlpCluster"/>   <remoteFileAccess>     <writeDir>${server.config.dir}</writeDir>   </remoteFileAccess>   <!-- Define the host name for use by the collective.   If the host name needs to be changed, the server should be   removed from the collective and re-joined or re-replicated. -->   <variable name="defaultHostName" value="waslibmem01" />   <!-- Connection to the collective controller -->   <collectiveMember controllerHost="waslibctlr01"     controllerPort="9443" />   <!-- clientAuthenticationSupported set to enable bidirectional trust -->   <ssl id="defaultSSLConfig"     keyStoreRef="defaultKeyStore"     trustStoreRef="defaultTrustStore"     clientAuthenticationSupported="true" />   <!-- inbound (HTTPS) keystore -->   <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/security/key.jks" />   <!-- inbound (HTTPS) truststore -->   <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/security/trust.jks" />   <!-- server identity keystore -->   <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/collective/serverIdentity.jks" />   <!-- collective truststore -->   <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/collective/collectiveTrust.jks" /> </server>

Note:
1) Make sure both HTTP and HTTPS ports that server listens are open on firewall(s).
2) In order to avoid port conflict, if you have more than one WLP server running on same host, make sure listen ports are unique for each of them. Usually by default WLP assigns 9080 (http), and 9443 (https).

Create 2nd Collective and Cluster Member

Below commands are executed in Machine: 04

Follow the same steps outlined in creating 1st Collective and Cluster Member . Here, I'm assigning name wlpSrv02 for my 2nd server.
Below is complete listing of server.xml and wlpsrv02_include.xml for wlpSrv02  (Collective and Cluster Member server) server.

Server.xml

<?xml version="1.0" encoding="UTF-8"?> <server description="new server">  <!-- Enable features -->  <featureManager>   <feature>webProfile-7.0</feature>   <feature>restConnector-1.0</feature>   <feature>localConnector-1.0</feature>  </featureManager>  <include location="${server.config.dir}/wlpsrv02_include.xml" />  <!-- To access this server from a remote client add a host attribute, e.g. host="*" -->  <httpEndpoint id="defaultHttpEndpoint"   httpPort="9081"   httpsPort="9444" host="*"/>  <!-- Automatically expand WAR files and EAR files -->  <!-- <applicationManager autoExpand="true"/> --> </server>

wlpSrv02_include.xml

<?xml version="1.0" encoding="UTF-8" ?> <server description="This file was generated by the 'collective join' command ...">   <featureManager>     <feature>collectiveMember-1.0</feature>     <feature>clusterMember-1.0</feature>   </featureManager>   <clusterMember name="wlpCluster"/>   <remoteFileAccess>     <writeDir>${server.config.dir}</writeDir>   </remoteFileAccess>   <!-- Define the host name for use by the collective.   If the host name needs to be changed, the server should be   removed from the collective and re-joined or re-replicated. -->   <variable name="defaultHostName" value="waslibmem02" />   <!-- Connection to the collective controller -->   <collectiveMember controllerHost="waslibctlr01"     controllerPort="9443" />   <!-- clientAuthenticationSupported set to enable bidirectional trust -->   <ssl id="defaultSSLConfig"     keyStoreRef="defaultKeyStore"     trustStoreRef="defaultTrustStore"     clientAuthenticationSupported="true" />   <!-- inbound (HTTPS) keystore -->   <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/security/key.jks" />   <!-- inbound (HTTPS) truststore -->   <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/security/trust.jks" />   <!-- server identity keystore -->   <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/collective/serverIdentity.jks" />   <!-- collective truststore -->   <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="     location="${server.config.dir}/resources/collective/collectiveTrust.jks" /> </server>

Note: Make sure all members in the cluster use the same LTPA keys.
For secure applications to function properly in a WLP clustered environment, each member of a given cluster must use the same LTPA key. For simplicity, Choose the LTPA key file from one of the cluster members and copy it to all of the other members in the cluster. The default LTPA key file is ${server.ouput.dir}/resources/security/ltpa.keys. For more information, refer to Creating a Liberty cluster with security considerations.

Start Member Server(s)

#Start wlpSrv01 on Machine: 03 $> ./server start wlpSrv01 Starting server wlpSrv01. Server wlpSrv01 started with process ID 6639. #Start wlpSrv02 on Machine: 04: $> ./server start wlpSrv02 Starting server wlpSrv02. Server wlpSrv02 started with process ID 5002.

Make sure member servers started without any issue and joined the Collective and Cluster. Easy way to do it is to check the messages.log. Check and make sure no errors there and verify messages something like below:
Successful start: ... A CWWKF0011I: The server wlpSrv02 is ready to run a smarter planet.
Successfully joined the Collective:... I CWWKX8112I: The server's host information was successfully published to the collective repository.
Successfully joined the Cluster:... CWWKX7400I: The ClusterMember MBean is available.

Once the collective member successfully join the collective and successfully started, further administrative work can be carried out using Admin Center Console.

Troubleshooting: If you are enabling <feature>javaee-7.0</feature> which along with other services also supports Enterprise JavaBeans (EJB) Lite, and Java Message Service (JMS). You may see following errors depending upon your configuration: 1) CWWKS9582E: ... ibm.ws.transport.iiop.security.AbstractCsiv2SubsystemFactory E CWWKS9582E: The [defaultSSLConfig] sslRef attributes required by the orb element with the defaultOrb id have not been resolved within 10 seconds. As a result, the applications will not start. Ensure that you have included a keyStore element and that Secure Sockets Layer (SSL) is configured correctly. Root cause: Usually SSL key is not available for secure iiop listener. Resolution: Define SSL config: If using default key store ${server.config.dir}/resources/security/key.jks, configure as follows: <keyStore password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="/> OR, if defining custom key.jks and trust.jks, configure as follows: <ssl id="defaultSSLConfig"   keyStoreRef="defaultKeyStore"   trustStoreRef="defaultTrustStore"   clientAuthenticationSupported="true" /> <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="   location="${server.config.dir}/resources/security/key.jks" /> <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="   location="${server.config.dir}/resources/security/trust.jks" /> 2) CWWKO0221E: ... com.ibm.ws.tcpchannel.internal.TCPPort E CWWKO0221E: TCP Channel wasJmsEndpoint329 initialization did not succeed. The socket bind did not succeed for host localhost and port 7276. The port might already be in use. Root cause: default port for JMS listener is in use. Most probably you have more than one WLP instances running on the same host and both listening on same listener port for JMS. Resolution: Change the default wasJmsPort as follows: <wasJmsEndpoint id="InboundJmsCommsEndpoint" host="*" wasJmsPort="9011" wasJmsSSLPort="9100" />  Refer to https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_msg_multi_server.html for details. 3) CWWKS9580E: ... ibm.ws.transport.iiop.security.config.ssl.yoko.SocketFactory E CWWKS9580E: The server socket could not be opened on localhost:2,809. The exception message is Address already in use. ... com.ibm.ws.logging.internal.impl.IncidentImpl I FFDC1015I: An FFDC Incident has been created: "org.omg.CORBA.INITIALIZE: : vmcid: 0x0 minor code: 0x0 completed: No com.ibm.ws.transport.iiop.internal.ORBWrapperInternal 109" at ffdc_17.12.04_15.43.42.0.log Root cause: default port for iiop is in use. Most probably you have more than one WLP instances running on the same host and both listening on same port for IIOP. Resolution: change iiop ports as follows: <iiopEndpoint host="*" id="defaultIiopEndpoint" iiopPort="2709">    <iiopsOptions iiopsPort="2815" />  </iiopEndpoint>

Next: proceed to application deployment -->


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below:

• How to Create and Configure WebSphere Liberty Cluster End-to-End (this post)
• How to Deploy Application in WebSphere Liberty Cluster
• How to Setup Front-End Web Server for WebSphere Liberty Cluster
• Using Docker Secrets with IBM WebSphere Liberty Profile Application Server
• How to use WLP passwordUtilities feature for encryption/decryption

How to use WLP passwordUtilities feature for encryption/decryption

Recently one of my colleagues asked if there was an easy way to utilise existing IBM WebSphere Application Server  (WAS) provided library to encrypt and decrypt password (or any sensitive information for that matter) from within the Java application deployed on WebSphere Liberty Profile (WLP). After doing some digging around, I found very useful WAS Liberty feature called "passwordUtilities-1.0". Class "com.ibm.websphere.crypto.PasswordUtil" is part of this feature, which provides set of static methods to deal with text encryption/decryption using default 'xor' or 'aes' algorithms. In this post, I'll walk you through, how to use it in you application.
Note: looks like feature "passwordUtilities-1.0" is available only from WLP version 8.5.5.9. Here is how you can check the version of your installed WLP product:

$> cd /opt/ibm/wlp/bin $> ./productInfo version Product name: WebSphere Application Server Product version: 17.0.0.2 Product edition: BASE_ILAN

If you are using WLP version prior to 8.5.5.9, however, still need to use com.ibm.websphere.crypto.PasswordUtil, there is some work-around, which I'll explain later in this post.
Note: If you need to verify whether or not "passwordUtilities-1.0" feature is available for your version of WLP, you can run "installUtility find passwordUtilities*". The feature is supported, if it appears in search result.

$> cd /opt/ibm/wlp/bin $> ./installUtility find passwordUtilities* Establishing a connection to the configured repositories ... ... Searching assets. This process might take several minutes to complete. feature : passwordUtilities-1.0 : Password Utilities

1) Steps for WLP 8.5.5.9 and Later that Support passwordUtilities-1.0 features

1.1) Make sure "passwordUtilities-1.0" is installed.

$> cd /opt/ibm/wlp/bin $> ./installUtility install passwordutilities-1.0 Establishing a connection to the configured repositories ... This process might take several minutes to complete. ... ... Step 5 of 8: Downloading passwordUtilities-1.0 ... Step 6 of 8: Installing passwordUtilities-1.0 ... ... All assets were successfully installed. Start product validation... Product validation completed successfully.

1.2) Make sure feature "passwordUtilities-1.0" is listed under <featureManager> in the server.xml

<featureManager>    ...    <feature>passwordUtilities-1.0</feature> </featureManager>

Note: If you forgot to install the "passwordUtilities-1.0" (step #1), but declared this feature in server.xml (step #2), you'll see an error message in the messages.log like this [10/7/17 20:56:42:807 BST] 0000001f com.ibm.ws.kernel.feature.internal.FeatureManager  E CWWKF0042E: A feature definition cannot  be found for the  passwordutilities-1.0 feature.  Try running the command, bin/installUtility install passwordutilities-1.0,  to install the feature. ...

1.3) [optional] Define encryption key. 

This step is required only if you are using 'aes' encryption/decryption and want to override the default encryption key. The default encryption key can be overridden by setting the wlp.password.encryption.key property. Property can be defined either in server.xml or bootstrap.properties file. However, as a best security practice, define this property in an external XML or properties file and include it in server.xml or bootstrap.properties accordingly. This ensures that the file containing the key is not included in the server dump or package command output. Here we define it in app_enc_key.xml file and put it outside of WLP configuration directory.

/opt/secrets/app_enc_key.xml

<server>     <variable name="wlp.password.encryption.key" value="replaceM3"/>    </server>

Now, include /opt/secrets/app_enc_key.xml in server.xml

<?xml version="1.0" encoding="UTF-8"?> <server description="new server">     <featureManager>         <feature>webProfile-7.0</feature>         <feature>passwordUtilities-1.0</feature>     </featureManager>     <include location="/opt/secrets/my_enc_key.xml"/>     <httpEndpoint id="defaultHttpEndpoint"                   httpPort="9080"                   httpsPort="9443" host="*"/>     <applicationManager autoExpand="true"/>     <application context-root="pwdencdemo" id="pwdencdemo" location="/opt/ibm/apps/pwdencdemo.war" name="pwdencdemo" type="war"/> </server>

1.4) Use PasswordUtil methods in the application.

Put together your application that has encoding/decoding (xor algorithm based) or more robust (aes algorithm based) encryption/decryption. You can use public static method of com.ibm.websphere.crypto.PasswordUtil class. See example below:

String myplainTxtPwd="t3mp_pwD"; // xor encoding String xorEncodedVal=com.ibm.websphere.crypto.PasswordUtil.passwordEncode(myplainTxtPwd, "xor")); // return xorEncodedVal value: {xor}K2wyLwAvKBs= // aes encryption String aesEncodedVal=com.ibm.websphere.crypto.PasswordUtil.passwordEncode(myplainTxtPwd, "aes")); // return aesEncodedVal value: {aes}AM99D1D9JLThDLt32ZCn3VBj27kuwNp5MorW6Ig/7h5h // xor decoding String xorDecodedVal=com.ibm.websphere.crypto.PasswordUtil.passwordDecode("{xor}K2wyLwAvKBs=")); // return xorDecodedVal value: t3mp_pwD // aes decryption String aesDecodedVal=com.ibm.websphere.crypto.PasswordUtil.passwordDecode("{aes}AM99D1D9JLThDLt32ZCn3VBj27kuwNp5MorW6Ig/7h5h")); // return aesDecodedVal value: t3mp_pwD

For full details of all available methods see Java documentation of  com.ibm.websphere.crypto.PasswordUtil. 

I have created a GitLab project, which contains server.xml and a simple web application pwdencdemo.war, which you can deploy into your WLP server and test the working of encryption/decryption.

2) Steps for WLP 8.5.5.8 and Prior version.

2.1) Define Java library and configure classloader for the application. 

Since "passwordUtilities-1.0" feature is not available in WLP 8.5.5.8 or earlier versions, in order to use public methods of com.ibm.websphere.crypto.PasswordUtil, we need to make sure this class and any other dependent classes are accessible for the application in run time. The easiest way to do it is to define a Java library (that includes com.ibm.ws.crypto.passwordutil_<version>.jar, com.ibm.ws.kernel.service_<version>.jar, and  com.ibm.ws.logging_<version>.jar), include in the application itself, and configure classloader for the application.  These jar files come with WLP product and located under ${wlp.install.dir}/lib directory. See below for example:

<application context-root="pwdencdemo" id="pwdencdemo" location="/opt/ibm/apps/pwdencdemo.war" name="pwdencdemo" type="war">     <classloader>        <privatelibrary>           <fileset dir="${wlp.install.dir}/lib" includes="com.ibm.ws.crypto.passwordutil_*.jar,com.ibm.ws.kernel.service_*.jar,com.ibm.ws.logging_*.jar">           </fileset>        </privatelibrary>     </classloader>  </application>

See IBM Knowledge Center documentation for more detail on configuring Java library.
Note: Make sure your library configuration is correct and all listed jar files are available in the given location, otherwise, you may get java.lang.NoClassDefFoundError: com/ibm/websphere/crypto/PasswordUtil exception.

2.2) Follow steps #1.3 and #1.4 from from the previous section.

Hope, this post will be helpful for your current or next WLP based project. If you are deploying your application in Docker container, visit my blog <<Using Docker Secrets with IBM WebSphere Liberty Profile Application Server>>

Note: If you just need to encode/encrypt password to put into your server.xml or bootstrap.properties, you can use securityUtility tool that comes with WLP product. Below is a simple example:

$> cd /opt/ibm/wlp/bin $> ./securityUtility encode --encoding=aes --key=replaceM3 t3mp_pwD

For more detail on securityUtility , refer to https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/rwlp_command_securityutil.html


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below:

• Using Docker Secrets with IBM WebSphere Liberty Profile Application Server
• How to use WLP passwordUtilities feature for encryption/decryption (this post)
• How to Create and Configure WebSphere Liberty Cluster End-to-End
• How to Deploy Application in WebSphere Liberty Cluster
• How to Setup Front-End Web Server for WebSphere Liberty Cluster