Showing posts with label Cluster. Show all posts
Showing posts with label Cluster. Show all posts

How to Deploy Application in WebSphere Liberty Cluster



I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on application deployment. Below listed are all the posts in this series:
  1. How to Create and Configure WebSphere Liberty Cluster End-to-End
  2. How to Deploy Application in WebSphere Liberty Cluster
  3. How to Setup Front-End Web Server for WebSphere Liberty Cluster
In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server

Note: in order to complete all the steps in this blog post, first you need to complete required steps outlined in How to Create and Configure WebSphere Liberty Cluster End-to-End

Deploying an application in WLP server or cluster

There are few ways to deploy an application into WLP server or cluster. Application can be simply dropped into a pre-defined dropins (${server.config.dir}/dropins) directory, or by adding an application definition entry to the server configuration.
In this blog post, we will explore script driven, remote application deployment for distributed platforms (multiple WLP servers or cluster managed by WLP Collective Controller) by utilizing secure JMX REST connection. Refer to Configuring secure JMX connection to Liberty for more details.
We'll be using manageAppOnCluster.py script and libraries available in IBM developerWorks to manage application(s) on WLP cluster.

Download manageAppOnCluster.zip

You can download it from https://developer.ibm.com/wasdev/downloads/#asset/scripts-jython-Install_or_uninstall_an_application_to_cluster. Here I have downloaded and extracted manageAppOnCluster.zip in Deployment/Tool server (machine: 01 in example topology) under /opt/workspace/wlpDeployment directory.

Download and Install Jython. 

You can download Jython from http://www.jython.org/downloads.html
General installation instruction can be found at https://wiki.python.org/jython/InstallationInstructions
Console based installation is very straight forward. See below, I'm installing standalone Jython under /opt/jython directory on Machine: 01

# Create /opt/jython directory
$> sudo -p mkdir /opt/jython

# Change the ownership if needed
$> sudo chown R wasadmin:wasgrp /opt/jython

# Install Jython
$> java -jar jython-installer-2.7.0.jar --console

Welcome to Jython !
You are about to install Jython version 2.7.0
(at any time, answer c to cancel the installation)
For the installation process, the following languages are available: English, German
Please select your language [E/g] >>> E
Do you want to read the license agreement now ? [y/N] >>> N
Do you accept the license agreement ? [Y/n] >>> Y
The following installation types are available:
   1. All (everything, including sources)
   2. Standard (core, library modules, demos and examples, documentation)
   3. Minimum (core)
   9. Standalone (a single, executable .jar)
Please select the installation type [ 1 /2/3/9] >>> 9
Please enter the target directory >>> /opt/jython Your java version to start Jython is: Oracle Corporation / 1.8.0_102
Your operating system version is: Linux / 3.10.0-514.el7.x86_64
Summary:
  - standalone
Please confirm copying of files to directory /opt/jython [Y/n] >>> Y
  10 %
20 %
  ...
  ...
Packing standalone jython.jar ...
  90 %
  100 %
Do you want to show the contents of README ? [y/N] >>> N
Congratulations! You successfully installed Jython 2.7.0 to directory /opt/jython.

You'll also need restConnector.jar and restConnector.py in order to be able to make remote JMX REST connection. If you have WLP installed on the machine, you should be good as these files are installed as part of WLP installation. If you don't have WLP installed on the machine, or you don't want to install WLP, you can just copy these files from another machine where it is installed. Easiest way is to tar the content of entire ${wlp.install.dir}/clients directory and copy it to Deployment/Tool server where you can untar it.
Here, I'm going to create a wlpclient.tar file on Machine: 02, as WLP is installed there and copy to Machine: 01.

$> cd /opt/ibm/wlp
$> tar -cvf /tmp/wlpclient.tar clients/
tar: Removing leading `/' from member names
clients/
clients/restConnector.jar
clients/jython/
clients/jython/restConnector.py
clients/jython/README

Copy the wlpclient.tar to Deployment/Tool server and extract it.

# Create directory /opt/ibm/wlp in Machine 02 (if it doesn't exist)
$> sudo mkdir /opt/ibm/wlp

# Change ownership as required.
$> sudo chown -R wasadmin:wasgrp

# Extract under /opt/ibm/wlp
$> tar -xvf wlpclient.tar -C /opt/ibm/wlp

In order to create secure connection, you'll also need signer certificate from the Collective Controller. Just copy trust.jks file (located under ${server.config.dir}/resources/security/ directory) from Collective Controller server and put into Deployment/tool server.
Now, for convenient, create a script file (name it wlpMgmt.sh), which look something like this:

wlpMgmt.sh
#! /bin/bash
# wlpMgmt.sh
export JYTHONPATH=/opt/ibm/wlp/clients/jython/restConnector.py
java -cp /opt/jython/jython.jar:/opt/ibm/wlp/clients/restConnector.jar:/opt/ibm/wlp/clients/jython/:/opt/workspace/wlpDeployment/lib/ org.python.util.jython "$@"


Prepare your application

For this exercise, I'm using a sample JavaHelloWorldApp.war application downloaded from https://github.com/IBM-Bluemix/java-helloworld/tree/master/target
I've put the application under /opt/workspace/wlpDeployment/apps directory in Machine:1.

Deploy the Application

Here, we are going to use wlpMgmt.sh (created above) to setup environment and call Jython and execute manageAppOnCluster.py to create secure connection to Liberty Controller and deploy application remotely.


$> /opt/workspace/wlpDeployment


$> ./wlpMgmt.sh manageAppOnCluster.py \
   --install=/opt/workspace/wlpDeployment/apps/JavaHelloWorldApp.war \
   --truststore=/opt/workspace/wlpDeployment/resources/security/trust.jks \
   --truststorePassword=<replace_with_your_password> \
   --host=waslibclnt01 \
   --port=9443 \
   --user=wasadmin \
   --password=<replace_with_your_password> \
   --clusterName=wlpCluster \

Connecting to the server...
Successfully connected to the server "waslibclnt01:9443"
Uploading application JavaHelloWorldApp.war to waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Updating server config for waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Complete
Uploading application JavaHelloWorldApp.war to waslibmem02,/opt/ibm/wlp/usr,wlpSrv02
Updating server config for waslibmem02,/opt/ibm/wlp/usr,wlpSrv02
Complete


Verify application deployed successfully

  1. Verify application binary is copied to each member server (in this case Machine: 03wlpSrv01, and Machine: 04wlpSrv02). By default, it is copied to ${server.config.dir}/apps directory of each cluster member server. Check and make sure it is copied. 
  2. Verify that server.xml of each member server is updated. In my case, it has added the following line in the member server's server.xml:

    <application name="JavaHelloWorldApp" location="${server.config.dir}/apps/JavaHelloWorldApp.war"/>

  3. Access application and make sure it is available. In our case, it is accessible as:
    • Machine: 03 URL: http://waslibmem01:9081/JavaHelloWorldApp/
    • Machine: 04 URL: http://waslibmem02:9081/JavaHelloWorldApp/

Note: If for any reason you need to undeploy/uninstall the application, here is how to do it:


$> ./appMgmt.sh manageAppOnCluster.py \
  --uninstall=JavaHelloWorldApp.war \
  --truststore=/opt/workspace/wlpDeployment/resources/security/trust.jks \
  --truststorePassword=<replace_with_your_password> \
  --host=waslibctlr01 \
  --port=9443 \
  --user=wasadmin \
  --password=<replace_with_your_password>; \
  --clusterName=wlpCluster
Connecting to the server...
Successfully connected to the server "waslibctlr01:9443"
Removing application JavaHelloWorldApp.war from waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Updating server config for waslibmem01,/opt/ibm/wlp/usr,wlpSrv01
Complete
Removing application JavaHelloWorldApp.war from waslibmem01,/opt/ibm/wlp/usr,wlpSrv02
Updating server config for waslibmem01,/opt/ibm/wlp/usr,wlpSrv02
Complete

uninstall removes the application binary as well as <application> definition from server.xml.

Next: proceed to Setup Front-End Web Server for WebSphere Liberty Cluster -->


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below:


How to Create and Configure WebSphere Liberty Cluster End-to-End



Because of it's simplicity, light-weight, and scalability, popularity and adoption of WebSphere Application Server Liberty Profile (WLP) server is  growing. Recently, I was involved in one of the projects, that utilised the WLP Collective and Clustering capabilities. In order to share the knowledge, I'm writing a series of blog posts on How to Create and Configure WebSphere Liberty Cluster End-to-End. This particular post focuses on setting up a WebSpere Liberty Collective and a Cluster. Below listed are all the posts in this series:
  1. How to Create and Configure WebSphere Liberty Cluster End-to-End
  2. How to Deploy Application in WebSphere Liberty Cluster
  3. How to Setup Front-End Web Server for WebSphere Liberty Cluster
In order to explain it better, I've created an example topology of WebSphere Liberty Profile (WLP) Collective with a Collective Controller and two Collective/Cluster member servers. Example topology contains IBM HTTP Server (IHS) as a Front-End and also a Deployment/Tool server. See diagram 1.0 for details.

Diagram 1.0
Example  Topology: WLP Collective with Front-End & Deployment Server


WLP Installation 

There are several ways to install WLP. If you like to explore and compare best ways for your particular situation, take a look at IBM Knowledge Center documentation.

Check WLP System Requirements:

First, you need to decide what version of WLP is needed for your particular situation. I'm using version 17.0.2 to implement this example topology. System requirement for WLP 17.0.2 can be found here.  For this exercise, I will be using Virtual Machines with CentOS Linux release 7.3.1611.

Note: Before proceeding with the WLP installation, make sure you have supported version of Java installed in the machine. 

Here is what I have for this exercise:
$> cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
$> uname -a
Linux waslibctlr01 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$> ./java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build pxa6480sr4fp5-20170421_01(SR4 FP5))
IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64 Compressed References 20170419_344392 (JIT enabled, AOT enabled)
J9VM - R28_20170419_1004_B344392
JIT - tr.r14.java_20170419_344392
GC - R28_20170419_1004_B344392_CMPRSS
J9CL - 20170419_344392)
JCL - 20170420_01 based on Oracle jdk8u131-b11

Let's start with the WLP installation.
Important: first, I will install WLP in Machine: 02 by using downloaded WLP Jar installer, add all the required features and then package it and distribute to other servers.

This IBM Knowledge Center documentation explains in detail how to download and extract the WLP installer Jar. As shown below, I'm installing it under /opt/ibm directory:


# Create directory /opt/ibm
$> sudo mkdir -p /opt/ibm

# Change ownership if required
$> sudo chown -R wasadmin:wasgrp /opt/ibm

java -jar wlp-<edition>-all-<fix_pack>.jar --acceptLicense /opt/ibm

Note: If you have multiple Java/JRE installed on your machine and you need to setup proper Java/JRE for your WLP environment, there are few ways that you can accomplished this. See Customizing the Liberty environment.

Once WLP is intalled and Java Runtime is configured, then check the version.

$> cd /opt/ibm/wlp/bin

$> ./productInfo version
Product name: WebSphere Application Server
Product version: 17.0.0.2


Install Features

Here, I'm installing features required for Collective, Cluster, SSL, restConnector, and localConnector. All the required features can be installed once or one at a time. For all available features refer to Liberty Features at IBM Knowledge Center.

$> ./installUtility install collectiveController-1.0 collectiveMember-1.0 clusterMember-1.0 websocket-1.1 restConnector-2.0 ssl-1.0 localConnector-1.0 adminCenter-1.0

Establishing a connection to the configured repositories ...
This process might take several minutes to complete.

Successfully connected to all configured repositories.
Preparing assets for installation. This process might take several minutes to complete.

Additional Features Terms & Conditions:
By clicking on the "I agree" button , Licensee agrees that the program
code, samples, updates, fixes and related licensed materials such as
...
...
production workloads, simulating production workloads or testing
scalability of any code, application or system.

Select [1] I Agree, or [2] I do not Agree: 1

Step 1 of 12: Downloading clusterMember-1.0 ...
Step 2 of 12: Installing clusterMember-1.0 ...
Step 3 of 12: Downloading collectiveController-1.0 ...
Step 4 of 12: Installing collectiveController-1.0 ...
...
...
All assets were successfully installed.

Start product validation...
Product validation completed successfully.

Since, all required features are installed, it's time to package the installation as server package and distribute to other servers for installation. In order to  do this, I'm going to create a default WLP server and create a package. For more information about server packaging, refer to IBM Knowledge Center documentation. Below, I'm showing server packaging as a Jar file, because it preserves (mostly) the execute permission for the scripts in bin directory.

$> cd /opt/ibm/wlp/bin

# Create defaultServer
$> ./server create
Server defaultServer created.

# Create server package
$> ./server package defaultServer --archive=/tmp/wlp_install.jar --include=all
Packaging server defaultServer.
Server defaultServer package complete in /tmp/wlp_install.jar.

Now, copy the generated wlp_install.jar to other machines (Machine: 03, and Machine: 04 for this exercise) where WLP installation is required. Once wlp_install.jar is copied, proceed with installation. As shown below, we have copied the wlp_install.jar into /tmp and going to install WLP under /opt/ibm directory.

If you need to see the installation options, you can just run the --help as shown below:

java -jar wlp_install.jar --help

Usage
java -jar wlp_install.jar [Options] [install location]

Options
--acceptLicense
Automatically indicate acceptance of license terms and conditions.
--verbose
Display detailed information during archive extraction.
--viewLicenseAgreement
View license agreement.
--viewLicenseInfo
View license information.

Let's launch the installation on Machine: 03 and Machine: 04


# Create /opt/ibm directory and assign ownership to proper user:group
$> sudo mkdir -p /opt/ibm
$> chown -R <Your-WAS-user>:<Your-WAS-group> /opt/ibm

# Install WLP
$> java -jar wlp_install.jar --acceptLicense /opt/ibm

Once the installation is completed, verify the version and validate the product as shown below:

# Verify version
$> ./productInfo version
Product name: WebSphere Application Server
Product version: 17.0.0.2

# Validate installed product
$> ./productInfo validate
Start product validation...
Validating feature: adminCenter-1.0... PASS!
...
Validating feature: clusterMember-1.0... PASS!
Validating feature: collectiveController-1.0... PASS!
Validating feature: collectiveMember-1.0... PASS!
...
Validating feature: restConnector-1.0... PASS!
Validating feature: restConnector-2.0... PASS!
...
Validating feature: ssl-1.0... PASS!
...
Product validation completed successfully.

Troubleshooting:
If you get message ./productInfo: line 187: /opt/ibm/wlp/java/java/bin/java: Permission denied
while executing script, check and make sure the script files have execute permission. Jar packaging usually preserves the permission, but if you have Java that is installed under ${wlp.install.dir}/java directory and packaged as part of the installer, files under /java/bin and java/jre/bin may not preserve their execute permission. So, go ahead and add execute permission.


Setup Collective Controller

Once the WLP is installed on all servers that will be part of WLP Collective, let's start setting up Collective Controller. If you need to understand the concept of Liberty Collective, refer to IBM Knowledge Center article: Liberty: Collective architecture

Note: in given example topology (diagram 1.0), Collective Controller is setup on Machine: 02.

# Create server wlpCntlr, which will be configured as Collective Controller.
$> ./server create wlpCntlr
Server wlpCntlr created.

# Configure wlpCntlr as Collective Controller.
# Option --createConfigFile=<file-location> outputs the configuration in that file.


$> ./collective create wlpCntlr --keystorePassword=<replace_with_your_password> --createConfigFile=/opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml


Creating required certificates to establish a collective...
This may take a while.
Successfully generated the controller root certificate.
Successfully generated the member root certificate.
Successfully generated the server identity certificate.
Successfully generated the HTTPS certificate.

Successfully set up collective controller configuration for wlpCntlr.

Add the following lines to the server.xml to enable:

<include location="${server.config.dir}/wlpcntlr_include.xml" />

Please ensure administrative security is configured for the server.
An administrative user is required to join members to the collective.

As per our collective create ... command, /opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml is created with all required configuration for WLP Collective Controller. There are few manual updates required, before we can start our wlpCntlr server. Here they are:

1) Open server.xml and include wlpcntlr_include.xml in it:

# Here I'm using ${server.config.dir}/wlpcntlr_include.xml instead of /opt/ibm/wlp/usr/servers/wlpCntlr/wlpcntlr_include.xml
# Refer to Liberty Directory locations and properties
<include location="${server.config.dir}/wlpcntlr_include.xml"/>

2) Open server.xml and make sure to specify host attribute with proper value for httpEndPoint element. See below for example.

<httpEndpoint id="defaultHttpEndpoint"
httpPort="9080"
httpsPort="9443" host="*"/>

3) Open generated include file. In our case it is wlpcntlr_include.xml and update quickStartSecurity element with correct values for userName and userPassword.

<quickStartSecurity userName="<replace-with-your-userName>" userPassword="<replace-with-your-password>" />

If you need more explanation about WLP Collective Controller configuration, refer to Configuring a Liberty collective chapter in IBM Knowledge Center.

Note: I've also added following features in WLP Collective Controller server configuration:
  • adminCenter-1.0: provides Web Based GUI for collective/cluster/server and application management and monitoring.  See Administering Liberty using Admin Center
  • websocket-1.1:  protocol that enables a client and a server application to communicate by using one full duplex connection. See WebSocker for details.
  • restConnector-1.0: provides a secure JMX connector that can be used locally/remotely. Refer to Connecting to Liberty by using JMX for details.
  • localConnector-1.0: provides a local JMX connector that is built into the JVM.
Below is complete listing of server.xml and wlpcntlr_include.xml for wlpCntlr  (Collective Controller) server.

server.xml:
<?xml version="1.0" encoding="UTF-8"?>
<server description="CollectiveController">
   <!-- Enable features -->
   <featureManager>
      <feature>adminCenter-1.0</feature>
      <feature>websocket-1.1</feature>
      <feature>restConnector-1.0</feature>
      <feature>localConnector-1.0</feature>
   </featureManager>
   <include location="${server.config.dir}/wlpcntlr_include.xml" />
   <!-- To access this server from a remote client add a host attribute e.g. host="*" -->
   <httpEndpoint id="defaultHttpEndpoint"
      httpPort="9080"
      httpsPort="9443" host="*"/>
</server>

wlpcntlr_include.xml
<?xml version="1.0" encoding="UTF-8" ?>
<server description="This file was generated by the 'collective create' command on 2017-10-29 12:59:19 GMT.">
   <featureManager>
      <feature>collectiveController-1.0</feature>
   </featureManager>

   <!-- Define the host name for use by the collective.
    If the host name needs to be changed, the server should be
    removed from the collective and re-joined or re-replicated. -->
   <variable name="defaultHostName" value="waslibctlr01" />

   <!-- TODO: Set the security configuration for Administrative access -->
   <quickStartSecurity userName="wasadmin" userPassword="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs=" />

   <!-- clientAuthenticationSupported set to enable bidirectional trust -->
   <ssl id="defaultSSLConfig"
       keyStoreRef="defaultKeyStore"
       trustStoreRef="defaultTrustStore"
       clientAuthenticationSupported="true" />

   <!-- inbound (HTTPS) keystore -->
   <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/security/key.jks" />

   <!-- inbound (HTTPS) truststore -->
   <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/security/trust.jks" />

   <!-- server identity keystore -->
   <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/collective/serverIdentity.jks" />

   <!-- collective trust keystore -->
   <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/collective/collectiveTrust.jks" />

   <!-- collective root signers keystore -->
   <keyStore id="collectiveRootKeys" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
       location="${server.config.dir}/resources/collective/rootKeys.jks" />

</server>

Based on from where and how you are accessing your Collective Controller server and how many firewall(s) in between, you may need to open firewall port(s) to allow communication. Below is command that I needed to run to open ports 9080 and 9443 so that CentOS allowed the incoming requests.
# Check to see if port 9443 and 9080 open
$> sudo firewall-cmd --zone=public --query-port=9443/tcp
no
$> sudo firewall-cmd --zone=public --query-port=9080/tcp
no
# Open port 9080 and 9443
$> sudo firewall-cmd --zone=public --permanent --add-port=9443/tcp
success
$> sudo firewall-cmd --zone=public --permanent --add-port=9080/tcp
success
# recycle firewall-cmd
$> sudo firewall-cmd --reload

Start WLP Collective Controller server & Verify

Start:
#Start the server
$>cd /opt/ibm/wlp/bin


$> ./server start wlpCntlr
Starting server wlpCntlr.
Server wlpCntlr started with process ID 3840.

Check the messages.log. By default, it is located under ${server.config.dir}/logs directory. Make sure log file doesn't have any error and you see the message something like 'CWWKX6011I: The collective controller is ready, and can accept requests...".
If you have added feature adminCenter-1.0, you should also see a message something like ' CWWKT0016I: Web application available (default_host): http://waslibctlr01:9080/adminCenter/, in the log file

Access the Admin Center. Launch your browser and type the Admin Center URL seen on your log file. If it is http port, it will be redirected to https port automatically. Use the same userName and userPassword that you defined above in the quickStartSecurity element. See below screen shot of Admin Center landing page

Admin Center landing page

Create & Configure Collective and Cluster Member

You can have one or more WLP Collective and/or Cluster member server(s) in the same host (Vertical clustering/scaling) or different host (Horizontal clustering/scaling) or mix of Vertical and Horizontal clustering/scaling. In this exercise, as defined in the diagram 1.0, I'm doing Horizontal clustering/scaling with two member servers residing on two separate hosts.


Create 1st Collective and Cluster Member 

Below commands are executed in Machine: 03

# Create member server wlpSrv01:
$> cd /opt/ibm/wlp/bin
$> ./server create wlpSrv01
Server wlpSrv01 created.

# Add member to Collective:
# Note: execute this command from the host where this member server resides (Machine: 01):

$> cd /opt/ibm/wlp/bin

$> ./collective join wlpSrv01 \
  --host=waslibctlr01 \
  --port=9443 \
  --user=wasadmin \
  --password=<replace_with_your_password> \
  --keystorePassword=<replace_with_your_password> \
  --createConfigFile=/opt/ibm/wlp/usr/servers/wlpSrv01/wlpsrv01_include.xml

Joining the collective with target controller waslibctlr01:9443...
This may take a while.

SSL trust has not been established with the target server.

Certificate chain information:
Certificate [0]
Subject DN: CN=waslibctlr01, OU=wlpCntlr, O=ibm, C=us
Issuer DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective
...

Certificate [1]
Subject DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective
Issuer DN: OU=controllerRoot, O=5e134d2b-d60c-4c1d-b97d-a44041e28fb8, DC=com.ibm.ws.collective
...

Do you want to accept the above certificate chain? (y/n) y
Successfully completed MBean request to the controller.

Successfully joined the collective for server wlpSrv01.

Add the following lines to the server.xml to enable:

Troubleshooting:
If you encounter issues like one of these (below) while joining the WLP server to the Collective:
  1. CWWKX0229E: There was a problem with the user credentials provided. The server responded with code 401 and message 'Unauthorized'
  2. CWWKX0229E: There was a problem with the user credentials provided. The server responded with code 403 and message 'Forbidden'
  • Error with response code 401 is mainly because of unsuccessful authentication. Check and make sure 'user' and 'password' values specified in collective join ... command match with the corresponding values specified in quickStartSecurity or basicRegistry element for Collective Controller. 
  • Error with response code 403 comes because the authenticated user does not have required permission. In this case make sure: you are only using either quickStartSecurity or basicRegistry If you defined both, remove one).
  • In case if you are using basicRegistry, make sure to provide administrator role to the user being used in collective join .... Something like:

<quickStartSecurity userName="wasadmin" userPassword="<replace-with-your-password>"/>

OR

<basicRegistry id="basic" realm="WebRealm">
   <user name="wasadmin" password="<replace-with-your-password>" />
</basicRegistry>
<administrator-role>
   <user>wasadmin</user>
</administrator-role>

For detail refer to Setting up BasicRegistry and role mapping on Liberty.

If collective join ... commands completes successfully, (as per our command input) it generates  /opt/ibm/wlp/usr/servers/wlpSrv01/wlpsrv01_include.xml with all required configuration for WLP Collective Member. There are few manual updates required, before we can start our wlpSrv01 server. Here they are:

1) Open server.xml for wlpSrv01 and include wlpsrv01_include.xml in it:
<include location="${server.config.dir}/wlpsrv01_include.xml" />

2) Define Cluster and Cluster Member:
Open server.xml for Collective Member server, and add the following:
<-- Declares this server as a member of defined cluster -->
<-- Add following line under <featureManager> element. -->

<feature>clusterMember-1.0</feature>

<-- Defines cluster wlpCluster -->

<clusterMember name="wlpCluster"/>

<-- Following provides the write access to Collective Controller under ${server.config.dir} of Member server. It is needed for deployment and configuration managment. For more information, refer to IBM Knowledge Center Document.-->

<remoteFileAccess>
   <writeDir>${server.config.dir}</writeDir>
</remoteFileAccess>

For more information on setting WLP Cluster, refer to Configuring a Liberty server cluster.
Below is complete listing of server.xml and wlpsrv01_include.xml for wlpSrv01  (Collective and Cluster Member server) server.

server.xml
<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

 <!-- Enable features -->
 <featureManager>
  <feature>webProfile-7.0</feature>
  <feature>restConnector-1.0</feature>
  <feature>localConnector-1.0</feature>
 </featureManager>
 <include location="${server.config.dir}/wlpsrv01_include.xml" />

 <!-- To access this server from a remote client add a host attribute, e.g. host="*" -->
 <httpEndpoint id="defaultHttpEndpoint"
  httpPort="9081"
  httpsPort="9444" host="*"/>

 <!-- Automatically expand WAR files and EAR files -->
 <!-- <applicationManager autoExpand="true"/> -->
</server>

wlpSrv01_include.xml
<?xml version="1.0" encoding="UTF-8" ?>
<server description="This file was generated by the 'collective join' command ...">
  <featureManager>
    <feature>collectiveMember-1.0</feature>
    <feature>clusterMember-1.0</feature>
  </featureManager>
  <clusterMember name="wlpCluster"/>
  <remoteFileAccess>
    <writeDir>${server.config.dir}</writeDir>
  </remoteFileAccess>

  <!-- Define the host name for use by the collective.
  If the host name needs to be changed, the server should be
  removed from the collective and re-joined or re-replicated. -->
  <variable name="defaultHostName" value="waslibmem01" />

  <!-- Connection to the collective controller -->
  <collectiveMember controllerHost="waslibctlr01"
    controllerPort="9443" />

  <!-- clientAuthenticationSupported set to enable bidirectional trust -->
  <ssl id="defaultSSLConfig"
    keyStoreRef="defaultKeyStore"
    trustStoreRef="defaultTrustStore"
    clientAuthenticationSupported="true" />
  <!-- inbound (HTTPS) keystore -->
  <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/key.jks" />

  <!-- inbound (HTTPS) truststore -->
  <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/trust.jks" />

  <!-- server identity keystore -->
  <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/serverIdentity.jks" />

  <!-- collective truststore -->
  <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/collectiveTrust.jks" />

</server>

Note:
1) Make sure both HTTP and HTTPS ports that server listens are open on firewall(s).
2) In order to avoid port conflict, if you have more than one WLP server running on same host, make sure listen ports are unique for each of them. Usually by default WLP assigns 9080 (http), and 9443 (https).

Create 2nd Collective and Cluster Member

Below commands are executed in Machine: 04

Follow the same steps outlined in creating 1st Collective and Cluster Member . Here, I'm assigning name wlpSrv02 for my 2nd server.
Below is complete listing of server.xml and wlpsrv02_include.xml for wlpSrv02  (Collective and Cluster Member server) server.

Server.xml
<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

 <!-- Enable features -->
 <featureManager>
  <feature>webProfile-7.0</feature>
  <feature>restConnector-1.0</feature>
  <feature>localConnector-1.0</feature>
 </featureManager>
 <include location="${server.config.dir}/wlpsrv02_include.xml" />

 <!-- To access this server from a remote client add a host attribute, e.g. host="*" -->
 <httpEndpoint id="defaultHttpEndpoint"
  httpPort="9081"
  httpsPort="9444" host="*"/>

 <!-- Automatically expand WAR files and EAR files -->
 <!-- <applicationManager autoExpand="true"/> -->
</server>

wlpSrv02_include.xml
<?xml version="1.0" encoding="UTF-8" ?>
<server description="This file was generated by the 'collective join' command ...">
  <featureManager>
    <feature>collectiveMember-1.0</feature>
    <feature>clusterMember-1.0</feature>
  </featureManager>
  <clusterMember name="wlpCluster"/>
  <remoteFileAccess>
    <writeDir>${server.config.dir}</writeDir>
  </remoteFileAccess>

  <!-- Define the host name for use by the collective.
  If the host name needs to be changed, the server should be
  removed from the collective and re-joined or re-replicated. -->
  <variable name="defaultHostName" value="waslibmem02" />

  <!-- Connection to the collective controller -->
  <collectiveMember controllerHost="waslibctlr01"
    controllerPort="9443" />

  <!-- clientAuthenticationSupported set to enable bidirectional trust -->
  <ssl id="defaultSSLConfig"
    keyStoreRef="defaultKeyStore"
    trustStoreRef="defaultTrustStore"
    clientAuthenticationSupported="true" />
  <!-- inbound (HTTPS) keystore -->
  <keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/key.jks" />

  <!-- inbound (HTTPS) truststore -->
  <keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/security/trust.jks" />

  <!-- server identity keystore -->
  <keyStore id="serverIdentity" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/serverIdentity.jks" />

  <!-- collective truststore -->
  <keyStore id="collectiveTrust" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="
    location="${server.config.dir}/resources/collective/collectiveTrust.jks" />

</server>



Note: Make sure all members in the cluster use the same LTPA keys.
For secure applications to function properly in a WLP clustered environment, each member of a given cluster must use the same LTPA key. For simplicity, Choose the LTPA key file from one of the cluster members and copy it to all of the other members in the cluster. The default LTPA key file is ${server.ouput.dir}/resources/security/ltpa.keys. For more information, refer to Creating a Liberty cluster with security considerations.

Start Member Server(s)

#Start wlpSrv01 on Machine: 03

$> ./server start wlpSrv01

Starting server wlpSrv01.
Server wlpSrv01 started with process ID 6639.

#Start wlpSrv02 on Machine: 04:

$> ./server start wlpSrv02

Starting server wlpSrv02.
Server wlpSrv02 started with process ID 5002.

Make sure member servers started without any issue and joined the Collective and Cluster. Easy way to do it is to check the messages.log. Check and make sure no errors there and verify messages something like below:
Successful start: ... A CWWKF0011I: The server wlpSrv02 is ready to run a smarter planet.
Successfully joined the Collective:... I CWWKX8112I: The server's host information was successfully published to the collective repository.
Successfully joined the Cluster:... CWWKX7400I: The ClusterMember MBean is available.

Once the collective member successfully join the collective and successfully started, further administrative work can be carried out using Admin Center Console.

Troubleshooting:
If you are enabling <feature>javaee-7.0</feature> which along with other services also supports Enterprise JavaBeans (EJB) Lite, and Java Message Service (JMS). You may see following errors depending upon your configuration:

1) CWWKS9582E:
... ibm.ws.transport.iiop.security.AbstractCsiv2SubsystemFactory E CWWKS9582E: The [defaultSSLConfig] sslRef attributes required by the orb element with the defaultOrb id have not been resolved within 10 seconds. As a result, the applications will not start. Ensure that you have included a keyStore element and that Secure Sockets Layer (SSL) is configured correctly.

Root cause: Usually SSL key is not available for secure iiop listener.
Resolution: Define SSL config:
If using default key store ${server.config.dir}/resources/security/key.jks, configure as follows:

<keyStore password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="/>

OR, if defining custom key.jks and trust.jks, configure as follows:

<ssl id="defaultSSLConfig"
  keyStoreRef="defaultKeyStore"
  trustStoreRef="defaultTrustStore"
  clientAuthenticationSupported="true" />
<keyStore id="defaultKeyStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="   location="${server.config.dir}/resources/security/key.jks" />
<keyStore id="defaultTrustStore" password="{xor}LTovMz48OgAoNis3ACYwKi0ALz4sLCgwLTs="   location="${server.config.dir}/resources/security/trust.jks" />


2) CWWKO0221E:
... com.ibm.ws.tcpchannel.internal.TCPPort E CWWKO0221E: TCP Channel wasJmsEndpoint329 initialization did not succeed. The socket bind did not succeed for host localhost and port 7276. The port might already be in use.

Root cause: default port for JMS listener is in use. Most probably you have more than one WLP instances running on the same host and both listening on same listener port for JMS.

Resolution: Change the default wasJmsPort as follows:

<wasJmsEndpoint id="InboundJmsCommsEndpoint" host="*" wasJmsPort="9011" wasJmsSSLPort="9100" /> 

Refer to https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_msg_multi_server.html for details.


3) CWWKS9580E:
... ibm.ws.transport.iiop.security.config.ssl.yoko.SocketFactory E CWWKS9580E: The server socket could not be opened on localhost:2,809. The exception message is Address already in use.
... com.ibm.ws.logging.internal.impl.IncidentImpl I FFDC1015I: An FFDC Incident has been created: "org.omg.CORBA.INITIALIZE: : vmcid: 0x0 minor code: 0x0 completed: No com.ibm.ws.transport.iiop.internal.ORBWrapperInternal 109" at ffdc_17.12.04_15.43.42.0.log


Root cause: default port for iiop is in use. Most probably you have more than one WLP instances running on the same host and both listening on same port for IIOP.
Resolution: change iiop ports as follows:

<iiopEndpoint host="*" id="defaultIiopEndpoint" iiopPort="2709">
   <iiopsOptions iiopsPort="2815" /> 
</iiopEndpoint>

Next: proceed to application deployment -->


Looks like you're really interested in WebSphere Liberty Profile, see my other related blog posts below: