Using Docker Secrets with IBM WebSphere Liberty Profile Application Server


In this blog post, I'm discussing how to utilize Docker Secrets (a Docker Swarm service feature) to manage sensitive data (like password encryption keys, SSH private keys, SSL certificates etc.) for Dockerized application powered by IBM WebSphere Liberty Profile (WLP) application server. Docker Secrets helps to centrally manage these sensitive information while in rest or in transit (encrypted and securely transmitted to only those containers that need access to it and has explicit access to it). It is out of scope for this post to go deep into Docker secretes, however, if you need to familiarize yourself with Docker Secretes, refer to https://docs.docker.com/engine/swarm/secrets/.
Note: if you like to know how to program encryption/decryption within your Java application using passwordutilities-1.0 feature of WLP, see my blog How to use WLP passwordUtilities feature for encryption/decryption
I'm going to write this post in a tutorial style, so that anyone interested to try can follow the steps.

Pre-requisite: In order to follow the steps outlined here, you have to have following:

  1. Good working knowledge of Docker
  2. Configured Docker Swarm environment (using Docker 1.13 or higher version) with at least one manager and one worker node or Docker Datacenter with Universal Control Plane (UCP) having manager node, worker node(s). It's good to have a separate Docker client node, so that you can remotely connect to manager and execute commands. 
  3. Good working knowledge of IBM WebSphere Liberty Profile (https://developer.ibm.com/wasdev/blog/2013/03/29/introducing_the_liberty_profile/).

Here is brief description, how we are going to utilize Docker Secretes with WLP.

  1. Password encryption key that is used to encrypt password for WLP KeyStore, TrustStore and any other password(s) used by WLP applications will be externalized and stored as Docker Secretes.
  2. Private key such as one stored in KeyStore (used to enable secure communication in WLP) will be externalized and stored as Docker Secretes.

Here are some obvious benefits:

  1. Centrally manage all sensitive data. Since Docker enforces access control, only people with enough/right privilege(s) will have access to sensitive data.
  2. Only those container(s) and service(s) will have access to private/sensitive data which has explicit access as per need basis.
  3. Private information remains private while in rest or in transit.
  4. New Docker image created by 'docker commit' will not contain any sensitive data and also dump/package created by WLP server dump or package command, will not contain encryption key as it's externalized. See more insights about WLP password encryption here: https://www.ibm.com/support/knowledgecenter/en/SS7K4U_8.5.5/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/cwlp_pwd_encrypt.html and managing Docker Secrets here: https://docs.docker.com/engine/swarm/secrets/

Enough talk, now, let's start the real work. Below are the major steps that we'll carry out:

  1. Create Docker secrets for following that is being used by WLP:
    • KeyStore
    • Truststore
    • Password Encryption key
  2. Build Docker image based on websphere-liberty:webProfile7
  3. Create network
  4. Put together docker-compose.xml for deployment.
  5. Deploy application as Docker service.


Create Docker Secrets

Here, we're going to use Docker Commandline (CLI) and we'll execute Docker command  from Docker client node remotely. You need have following three environment variables correctly setup in order to execute command remotely. Refer to https://docs.docker.com/engine/reference/commandline/cli/#description for detail.
  • DOCKER_TLS_VERIFY
  • DOCKER_CERT_PATH
  • DOCKER_HOST
If you are using Docker Datacenter, you can use GUI based UCP Admin Console to create the same. Note: label com.docker.ucp.access.label="<value>" is not mandatory unless you have access constraint defined. For detail refer to Authentication and authorization
1) Create Docker Secrete with name keystore.jks, which basically is key database that stores private key to be used by WLP.

#Usage: docker secret create [OPTIONS] SECRET file|- 
#Create a secret from a file or STDIN as content 

$> docker secret create keystore.jks /mnt/nfs/dockershared/wlpapp/keystore.jks --label com.docker.ucp.access.label="dev"
 idc9em1u3fki8k0z77ol91sh4 

2) Following command creates secret called truststore.jks using physical Java keystore file which contains trust certificates

$> docker secret create truststore.jks /mnt/nfs/dockershared/wlpapp/truststore.jks --label com.docker.ucp.access.label="dev"
w8qs1o7pwrvl96nuamv97sb9t

Finally create the Docker secret call app_enc_key.xml, which basically refers to the fragment of xml wich contains definintion of password encryption key

$> docker secret create app_enc_key.xml /mnt/nfs/dockershared/wlpapp/app_enc_key.xml --label com.docker.ucp.access.label="dev"
kj3hcw4ss71hnudfgr6g32mxm

Note: Docker secrets are available under '/run/secrets/' at runtime to any container which has explicit access to that secret.
Here is how the /mnt/nfs/dockershared/wlpapp/app_enc_key.xml look like:

<server> 
   <variable name="wlp.password.encryption.key" value="#replaceMe#">
   </variable>
</server>

Note: Make sure to replace the string '#replaceMe#' with your own password encryption key.

Let's check and make sure all our secrets are properly created and listed:

$> docker secret ls
ID                        NAME             CREATED        UPDATED
idc9em1u3fki8k0z77ol91sh4 keystore.jks     3 hours ago    3 hours ago
kj3hcw4ss71hnudfgr6g32mxm app_enc_key.xml  21 seconds ago 21 seconds ago
w8qs1o7pwrvl96nuamv97sb9t truststore.jks   3 hours ago    3 hours ago


Building Docker Image:

Now, let's first encrypt our keystore and trusstore passwords using the pre-defined encryption key and put together the server.xml for WLP server. We are going to use securityUtility tool that ships with IBM WLP to encrypt our password.
Note: make sure your password encryption key matches to the one that is defined by 'wlp.password.encryption.key' property in app_enc_key.xml.
Here I'm encoding my example password '#myStrongPassw0rd#' using encryption key '#replaceMe#' with encoding option 'aes'.
Please note that encoding option 'xor' ignores the encryption key and uses default.

$> cd /opt/ibm/wlp/bin
$> ./securityUtility encode #myStrongPassw0rd# --encoding=aes --key=#replaceMe#
{aes}AAj/El4TFm/8+9UFzWu5kCtURUiDIV/XKbGY/lT2SVKFij/+H38b11uhjh+Peo/rBA==

Now, we have our Docker secrets created and we have encrypted our password. It's time to put together our server.xml for WLP application server and build the Docker image. Here is how my server.xml looke like.

<server description="TestWLPApp">
   <featuremanager> 
      <feature>javaee-7.0</feature> 
      <feature>localConnector-1.0</feature> 
      <feature>ejbLite-3.2</feature> 
      <feature>jaxrs-2.0</feature> 
      <feature>jpa-2.1</feature> 
      <feature>jsf-2.2</feature> 
      <feature>json-1.0</feature> 
      <feature>cdi-1.2</feature> 
      <feature>ssl-1.0</feature> 
   </featuremanager> 
   <include location="/run/secrets/app_enc_key.xml"/> 
   <httpendpoint host="*" httpport="9080" httpsport="9443" id="defaultHttpEndpoint"/> 
   <ssl clientauthenticationsupported="true" id="defaultSSLConfig" keystoreref="defaultKeyStore"     truststoreref="defaultTrustStore"/> 
   <keystore id="defaultKeyStore" location="/run/secrets/keystore.jks" password="{aes}ANGkm5cIca4hoPMh4EUeA4YYqVPAbo4HIqlB9zOCXp1n"/> 
   <keystore id="defaultTrustStore" location="/run/secrets/truststore.jks" password="{aes}ANGkm5cIca4hoPMh4EUeA4YYqVPAbo4HIqlB9zOCXp1n"/> 
   <applicationmonitor updatetrigger="mbean"/> 
   <datasource id="wlpappDS" jndiname="wlpappDS"> 
      <jdbcdriver libraryref="OracleDBLib"/> 
      <properties.oracle password="{aes}AAj/El4TFm/8+9UFzWu5kCtURUiDIV/XKbGY/lT2SVKFij/+H38b11uhjh+Peo/rBA==" url="jdbc:oracle:thin:@192.168.xx.xxx:1752:WLPAPPDB" user="wlpappuser"/>
   </datasource>  
    <library id="OracleDBLib"> 
       <fileset dir="/apps/wlpapp/shared_lib" includes="ojdbc6-11.2.0.1.0.jar"/>
    </library> 
    <webapplication contextRoot="wlpappctx" id="wlpapp" location="/apps/wlpapp/war/wlptest.war" name="wlpapp"/> 
</server>

As you can see, the location of defaultKeyStore, defaultTrustStore, and app_enc_key.xml is pointing to directory '/run/secrets'. It is, as mentioned before, because all private data created by Docker Secrets will be available for the assigned services under '/run/secrets' of the corresponding container.

Now let's put together Dockerfile.

FROM websphere-liberty:webProfile7
COPY /mnt/nfs/dockershared/wlpapp/server.xml /opt/ibm/wlp/usr/servers/defaultServer/
RUN installUtility install --acceptLicense defaultServer
COPY /mnt/nfs/dockershared/wlpapp/wlptest.war /apps/wlpapp/war/
COPY /mnt/nfs/dockershared/wlpapp/ojdbc6-11.2.0.1.0.jar /apps/wlpapp/shared_lib/
CMD ["/opt/ibm/java/jre/bin/java","-javaagent:/opt/ibm/wlp/bin/tools/ws-javaagent.jar","-Djava.awt.headless=true","-jar","/opt/ibm/wlp/bin/tools/ws-server.jar","defaultServer"]

Note: above, I'm copying my server.xml into /opt/ibm/wlp/usr/servers/defaultServer/ before running the installUtility as I'm adding few features required by my application including, ssl-1.0.

Finally, we're going to build the Docker image.

$> docker build -t 192.168.56.102/osboxes/wlptest:1.0 .
Sending build context to Docker daemon 56.9 MB
Step 1/7 : FROM websphere-liberty:webProfile7
---> c035090355f5
...
Step 4/7 : RUN installUtility install --acceptLicense defaultServer
---> Running in 2bce0d02e253
Checking for missing features required by the server ...
...
Successfully built 07fef794348e

Note: 192.168.56.102 is my local Docker Trusted Registry (DTR).

Once, the image is successfully built, make sure the image is available on all nodes of Docker Swarm. I'm not going show details how you distribute the image.
> If you are using DTR, You can first push the image to the registry (using 'docker push ...', then connect to Docker Swarm host and execute 'docker pull ...'),
> Other option is to use 'docker save ...' to save the image as tar file then load the image into Swarm using 'docker load ...'.
 Here, I'm deploying this into Docker Datacenter which has two UCP worker nodes, one UCP manager node and DTR node. I'm also going to use the HTTP routing mesh (HRM), and User defined Overlay networks in swarm mode.
Note: User defined Docker network and HRM are NOT necessary to utilize the Docker secrets.


Create Overlay network:

$> docker network create -d overlay --label com.docker.ucp.access.label="dev" --label com.docker.ucp.mesh.http=true my_hrm_network
naf8hvyx22n6lsvb4bq43z968

Note: Label 'com.docker.ucp.mesh.http=true' is required while creating network in order to utilize HRM.

Put together docker-compose.yml

Here is my compose file. Your may look different.

version: "3.1"
services:
   wlpappsrv: 

      image: 192.168.56.102/osboxes/wlptest:1.0
      volumes:
         - /mnt/nfs/dockershared/wlpapp/server.xml:/opt/ibm/wlp/usr/servers/defaultServer/server.xml
      networks:
         - my_hrm_network
      secrets:
         - keystore.jks
         - truststore.jks
         - app_enc_key.xml
      ports:
         - 9080
         - 9443
     deploy:
        placement:
           constraints: [node.role == worker]
           mode: replicated
           replicas: 4
           resources:
              limits:
                 memory: 2048M
           restart_policy:
              condition: on-failure
              max_attempts: 3
              window: 6000s
           labels:
              - "com.docker.ucp.mesh.http.9080=external_route=http://mydockertest.com:8080,internal_port=9080"
              - "com.docker.ucp.mesh.http.9443=external_route=sni://mydockertest.com:8443,internal_port=9443"
              - "com.docker.ucp.access.label=dev"
networks:
   my_hrm_network:
      external:
         name: my_hrm_network
secrets:
   keystore.jks:
      external: true
   truststore.jks:
      external: true
   app_enc_key.xml:
      external: true

Few notes about the docker-compose.yml
  1. Volume definition that maps server.xml in the container with the one in the NFS file system is optional. This mapping gives additional flexibility to update the server.xml. You can achieve similar or even better flexibility/portability by using Docker Swarm Config service. See my blog post - How to use Docker Swarm Configs service with WebSphere Liberty Profile for details.
  2. The secrets definition under service 'wlpappsrv' refers to the secrets definition in the root level, which in it turns refers to externally defined secret.
  3. "com.docker.ucp.mesh.http." labels are totally optional and only required if you are using HRM. 
  4. "com.docker.ucp.access.label" is also optional and required only if you have defined access constraints.
  5. Since, I'm using Swarm and HRM, I don't need to explicitly map the internal container ports to host port. If you need to map, you can use something like below for your port definition:
    ports:
       - 9080:9080
       - 9443:9443
  6. You may encounter situation that your container application is not able to access the secrets created under /run/secrets. It may be related to bug #31006. In order to resolve the issue use 'mode: 0444' while defining your secrets. Something like this:
    secrets:
       - source: keystore.jks
         mode: 0444
       ...   

Deploy the service 

Here I'm using "docker stack deploy..." to deploy the service:
$> docker stack deploy --compose-file docker-compose.yml dev_WLPAPP

Note: In certain cases, you may get "secrets Additional property secrets is not allowed", error message. In order to resolve, make sure your compose file version to 3.1. In my case, where it's working fine, I've Docker version 17.03.2-ee4, API version: 1.27, Docker-Compose version 1.14.0.

Once the service is deployed. You can list it using 'docker service ls ..." command

$> docker service ls
ID           NAME                 MODE       REPLICAS IMAGE
28xhhnbcnhfg dev_WLPAPP_wlpappsrv replicated 4/4      192.168.56.102/osboxes/wlptest:1.0

And list the replicated containers:
$> docker ps
CONTAINER ID IMAGE                              COMMAND CREATED STATUS PORTS NAMES
2052806bbae3 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk01/dev_WLPAPP_wlpappsrv.3.m7apci6i1ks218ddnv4qsdbwv
541cf0f39b6e 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk01/dev_WLPAPP_wlpappsrv.4.wckec2jcjbmrhstftajh2zotr
ccdd7275fd7f 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk02/dev_WLPAPP_wlpappsrv.2.oke0fz2sifs5ej0vy63250wo9
7d5668a4d851 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk02/dev_WLPAPP_wlpappsrv.1.r9gi0qllnh8r9u8popqg5mg5b

And here is what the WLP messages.log shows (taken from one of the containers log file):
********************************************************************************
product = WebSphere Application Server 17.0.0.2 (wlp-1.0.17.cl170220170523-1818)
wlp.install.dir = /opt/ibm/wlp/
server.output.dir = /opt/ibm/wlp/output/defaultServer/
java.home = /opt/ibm/java/jre
java.version = 1.8.0
java.runtime = Java(TM) SE Runtime Environment (pxa6480sr4fp7-20170627_02 (SR4 FP7))
os = Linux (3.10.0-514.el7.x86_64; amd64) (en_US)
process = 1@e086b8c54a8d
********************************************************************************
[7/24/17 19:44:29:275 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager A CWWKE0001I: The server defaultServer has been launched.
...
[7/24/17 19:44:30:533 UTC] 00000017 com.ibm.ws.config.xml.internal.XMLConfigParser A CWWKG0028A: Processing included configuration resource: /run/secrets/app_enc_key.xml
[7/24/17 19:44:31:680 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager I CWWKE0002I: The kernel started after 2.763 seconds
[7/24/17 19:44:31:990 UTC] 0000001f com.ibm.ws.kernel.feature.internal.FeatureManager I CWWKF0007I: Feature update started.
[7/24/17 19:44:45:877 UTC] 00000017 com.ibm.ws.security.ready.internal.SecurityReadyServiceImpl I CWWKS0007I: The security service is starting...
[7/24/17 19:44:47:262 UTC] 00000028 com.ibm.ws.security.token.ltpa.internal.LTPAKeyInfoManager I CWWKS4103I: Creating the LTPA keys. This may take a few seconds.
[7/24/17 19:44:47:295 UTC] 00000017 ibm.ws.security.authentication.internal.jaas.JAASServiceImpl I CWWKS1123I: The collective authentication plugin with class name NullCollectiveAuthenticationPlugin has been activated.
[7/24/17 19:44:48:339 UTC] 00000028 com.ibm.ws.security.token.ltpa.internal.LTPAKeyInfoManager A CWWKS4104A: LTPA keys created in 1.065 seconds. LTPA key file: /opt/ibm/wlp/output/defaultServer/resources/security/ltpa.keys
[7/24/17 19:44:48:365 UTC] 00000028 com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask I CWWKS4105I: LTPA configuration is ready after 1.107 seconds.
[7/24/17 19:44:57:514 UTC] 00000017 com.ibm.ws.app.manager.internal.monitor.DropinMonitor A CWWKZ0058I: Monitoring dropins for applications.
[7/24/17 19:44:57:651 UTC] 0000003f com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel defaultHttpEndpoint has been started and is now listening for requests on host * (IPv6) port 9080.
[7/24/17 19:44:57:675 UTC] 0000003f com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started and is now listening for requests on host * (IPv6) port 9443.
[7/24/17 19:44:57:947 UTC] 00000017 com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel wasJmsEndpoint302 has been started and is now listening for requests on host localhost (IPv4: 127.0.0.1) port 7276.
[7/24/17 19:44:57:951 UTC] 00000017 com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel wasJmsEndpoint302-ssl has been started and is now listening for requests on host localhost (IPv4: 127.0.0.1) port 7286.
...

As you can see (messages in blue), it's able to include the configuration from /run/secrets/app_enc_key.xml and it also shows that defaultHttpEndpoint-ssl has been started and listening on port 9443; meaning that it's able to successfully load and open the /run/secrets/keystore.jks and /run/secrets/truststore.jks files using the encrypted password with encryption key defined in /run/secrets/app_enc_key.xml.

Now, it's time to access the application. In my case, since, I'm using HRM, I will access it as: https://mydockertest.com:8443/wlpappctx
If you are not using HRM; you may access it using:
https://<docker-container-host>:9443/<application-context>


Example using Load-Balancer


If you have a load-balancer in front and want to set-up a  pass-through SSL, you can use SNI: aka SSL routing. Below is simple example using ha-proxy. You can also refer to HA-Proxy documentation here for details.  

Here is haproxy.cfg for our example PoC:
# /etc/haproxy/haproxy.cfg, version 1.7
global
   maxconn 4096

defaults
   timeout connect 5000ms
   timeout client 50000ms
   timeout server 50000ms

frontend frontend_ssl_tcp
   bind *:8443
   mode tcp
   tcp-request inspect-delay 5s
   tcp-request content accept if { req_ssl_hello_type 1 }
   default_backend bckend_ssl_default

backend bckend_ssl_default
   mode tcp
   balance roundrobin
   server worker1 192.168.56.103:8443 check
   server worker2 192.168.56.104:8443 check           

Here is a Dockerfile for custom image:
FROM haproxy:1.7
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
Build the image:
Note: execute the 'docker build ...' command from the same directory where Dockerbuild file is located.

$> docker build -t my_haproxy:1.7 .
Once you build the image and start the ha-proxy container like below:
$> docker run -d --name ddchaproxy -p 8443:8443 my_haproxy:1.7

Note: In this case ha-proxy is listening on port 8443.

Access the application:

https://mydockertest.com:8443//wlpappctx

Note: Make sure mydockertest.com resolves to the IP address of ha-proxy.


Looks like you're really into Docker, see my other related blog posts below:

Sharing My Reading List

I love reading. I read every day at least around two hours - partly thanks to my commute time to work. I take YRT to work, it's comfortable seating gives me opportunity to indulge in reading while commuting. Listed below are some of the books that I have read in the past and liked or currently been reading.

Personal Development/Self Help/Finance/Management

Moving Forward - Taking The Lead in Your Life
- by Dave Pelzer
This is really a great book. It helped me to get rid of all (almost) craps from my mind.
The Four doors
- by Richard Paul Evans
One of the stories in this book (about a beggar), gave me a thunder shock and hit hard right in the centre of my greatest weakness
- which is afraid to try. Not even trying is the biggest failure in our life.

The Secret Letters Of The Monk Who Sold His Ferrari
- by Robin Sharma
WHAT DOESN'T KILL US
- by Scott Carney
This amazing book, which greatly inspired me to practice Wim Hof's breathing techniques and also lead me to Pranayama Yoga. These days, showering in cold water, practicing Wim Hof and Pranayama breathings techniques are part of my daily routine and I'm feeling great and strong. Robin Sharma's reading list introduced me this book, so many thanks to Robin.
The Fred factor : how passion in your work and life can turn the ordinary into the extraordinary
- by Mark Sanborn.
Let The Elephants Run
- by David Usher
Peak: Secrets from the New Science of Expertise
- by Robert Pool Anders Ericsson
The Freaks Shall Inherit the Earth: Entrepreneurship for Weirdos, Misfits, and World Dominators
- by Chris Brogan
Outliers: The Story of Success
- by Malcolm Gladwell
David and Goliath: Underdogs, Misfits, and the Art of Battling Giants
- by Malcolm Gladwell
Millionaire Teacher: The Nine Rules of Wealth You Should Have Learned in School
- by Andrew Hallam
This is one of the best books I have ever read in personal finance, investment etc.
Spark: The Revolutionary New Science of Exercise and the Brain
- by John J. Ratey, and Eric Hagerman
Fail Fast or Win Big: The Start-Up Plan for Starting Now
- by Bernhard Schroeder
The Hour Between Dog and Wolf: Risk Taking, Gut Feelings and the Biology of Boom
- by John Coates


Literature/Fiction/Non-Fiction/General


The Heart Goes Last
- by Margaret Atwood
Hemingway in Love: His Own Story
- by A.E. Hotchner
Jonathan Livingston Seagull
- by Richard Bach
I've got the recommendation for this book from Robin Sharma's reading list. Thanks Robin!
Superintelligence: Paths, Dangers, Strategies Reprint Edition
- by Nick Bostrom
Nexus
- by Ramez Naam
The Black Swan: The Impact of the Highly Improbable
- by Nassim Nicholas Taleb


I also read some great Nepalese books


Karnali Blues (कर्नाली ब्लुज)
- by Buddhi Sagar
Palpasa Cafe (पल्पसा क्याफे)
- by Narayan Wagle
Seto Dharti(सेतो धरती)
- by Amar Neupane
Prayogshala (प्रयोगशाला, नेपाली सङ्क्रमणमा दिल्ली, दरबार र माओवादी)
- by Sudheer Sharma
Jiwan Kada Ki Phool (जीवन काँडा कि फूल)
- by Jhamak Kumari Ghimire

Antarman ko Yatra (अन्तर्मनको यात्रा)
- by Jagadish Ghimire

DataPower Front-End for WAS Cluster with Load Balancing and SSO

In this blog post, I'm guiding through simple and easy to follow steps to create a Proof of Concept (POC) environment with Datapower (DP) as font-end and  application serving environment hosted on WebSphere Application Server (WAS) based cluster with Single Sign On (SSO) enabled. DP will also load-balance requests to WAS cluster members. For details of DP supported features as front-end, refer to https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/cwsv_topology.html

Assumptions:
  1. In order for you to follow the steps outlined below, you have good working knowledge of IBM WAS and DataPower.
  2. You have WAS ND (version 8.5.X), and IBM DataPower Gateway (version 7.5.X) installed and ready to create Application Serving environment.  For your reference, I'm using the following:
    •    WAS 8.5.5.7 Network Deployment.
    •    IBM DataPower Gateway Virtual Edition (Firmware: IDG.7.5.1.0).
In a nutshell, here is what we'll do as part of this exercise:

On WAS side:

  • Create a cluster with two members (application servers).
  • Create security domain and assign scope to the cluster level
  • Create and configure an external trusted realm for the security domain. It is because, we will not have any user registry configured (for Application authentication) and we'll trust identifies authenticated by DP.
  • Enable Application security on the security domain
  • Deploy a sample application on the WAS cluster
  • Map security roles and authorize any user belonging to External Trusted Realm.
  • Configure LTPA, export LTPA key and customize (as we are using External Trusted Realm) it.
  • Export SSL certificate.

On DP side:

  • Import LTPA key, import WAS SSL certificate.
  • Create DP objects (including Crypto, SSL profiles, Password aliase etc).
  • Create Load Balancer Group
  • Create Multi-Protocol Gateway (MPGW) service etc.

Let's start working on WAS side:

Note:
Once you make configuration changes on WAS either using WAS admin console or wsadmin script,
you need to save the changes and synchronise the changes with the nodes (in Network Deployment (ND). Since it's essential part for every change or set of changes, I'm not going to list these two steps after every change below. But you need to make sure to save and synchronise. You can use
the wsadmin Jython script below to save and synchronize the changes with nodes:
# Save the configuration/change
AdminConfig.save()
# Synchronise the changes with the nodes
dmgrObj=AdminControl.queryNames('WebSphere:type=DeploymentManager,*')
AdminControl.invoke(dmgrObj, 'multiSync', '[false]', '[java.lang.Boolean]')

1.0) Create a WAS cluster containing two members. In this example, we will create a WAS cluster named "sso_clus" with two members "ssoSrv1" and "ssoSrv2". For more info on creating WAS cluster, refer to https://www.ibm.com/support/knowledgecenter/SSRMWJ_6.0.0.10/com.ibm.isim.doc/installing/tsk/tsk_ic_ins_was_85_cluster.htm
Note: You may need to change text in italic as per your configuration.
# Using WAS Admin Console: Servers --> Clusters --> WebSphere application server clusters
# Using wsadmin script:
AdminTask.createCluster('[-clusterConfig [-clusterName sso_clus -preferLocal true]]')

# Add 1st cluster member 'ssoSrv1' using default server template
AdminTask.createClusterMember('[-clusterName sso_clus -memberConfig [-memberNode myWASHostNode01 -memberName ssoSrv1 -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateName default -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope cluster]]')

# Add  2nd cluster member 'ssoSrv2'
AdminTask.createClusterMember('[-clusterName sso_clus -memberConfig [-memberNode myWASHostNode01 -memberName ssoSrv2 -memberWeight 2 -genUniquePorts true -replicatorEntry false]]')

2.0) Enable SSO (if not already). Here you need to know request(s) from what domain(s) you are supporting, you can have multiple domains defined here. For detail how domain works refer to: https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tsec_msso.html. For this example, we are going to use 'UseDomainFromURL'. See the above URL for details.
# Configure Single Sign On (SSO)
AdminTask.configureSingleSignon('-enable true -requiresSSL true -domainName UseDomainFromURL -attributePropagation true')

# Make sure LTPA is active authentication mechanism:
AdminTask.setActiveAuthMechanism('-authMechanismType LTPA')

3.0) Disable automatic LTPA key generation option as follows:
  Access WAS Admin console and navigate to Security > Global security > LTPA
   > Click on link "Key set groups"
   > Click on link "CellLTPAKeySetGroup"
   > Uncheck "Automatically generate keys"
  Apply, save and synchronise.

4.0) Create security domain. Creating security domain is encouraged rather than modifying the Global security configuration, because, you may have other application serving environment (s) managed under this Cell and they may rely on Global security and any change may affect them.
For this example, we name our new security domain 'mySSOLTPADomain' and we'll use the Global security as template (basically copying the Global security) to create the new security domain and modifying few things on newly created security domain:
Using WAS Admin Console: Security > Global security > Security domains > New...
# Using wsadmin script:
AdminTask.copySecurityDomainFromGlobalSecurity('-securityDomainName mySSOLTPADomain')


5.0) Customise the newly created security domain as per our requirement.
5.1) Since, we will not have any user registry configured on WAS for application authentication, and we'll trust the credential authenticated by DP, we'll create a trusted external realm called 'DatapowerRealm' and add that to newly created security domain. For detail about trusted external realm, Refer to https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tsec_sec_realm_trust.html
Using WAS Admin console: Security domains > mySSOLTPADomain > Federated repositories > Trusted authentication realms - inbound > New...
Using wsadmin script:
AdminTask.addTrustedRealms('[-securityDomainName mySSOLTPADomain -communicationType inbound -realmList DatapowerRealm]')
# And only trust the realm(s) that is explicitly indicated (not all)
Using WAS Console: Security domains > mySSOLTPADomain > Federated repositories > Trusted authentication realms - inbound
Using wsadmin script:
AdminTask.configureTrustedRealms('[-communicationType inbound -securityDomainName mySSOLTPADomain -trustAllRealms false]')
Using WAS Admin console: Security domains > mySSOLTPADomain
Using wsadmin script:
AdminTask.configureAppWIMUserRegistry('[-securityDomainName mySSOLTPADomain -verifyRegistry true ]')
# Enable Application Security
Using WAS Admin console: Security domains > mySSOLTPADomain > Application security > Customize for this domain > check Enable Application security
Using wsadmin script:
AdminTask.setAppActiveSecuritySettings('[-securityDomainName mySSOLTPADomain -appSecurityEnabled true]')
# Here, we are also increasing the LTPA timeout
Using WAS Console: Security domains > mySSOLTPADomain > Authentication Mechanism Attributes > Customize for this domain
Using wsadmin script:
AdminTask.setLTPATimeout('[-securityDomainName mySSOLTPADomain -timeout 220 ]')
# And, we are going to assign the scope of this security domain to cluster 'sso_clus' scope
Using WAS Admin console: Security domains > mySSOLTPADomain
Using wsadmin script:
AdminTask.mapResourceToSecurityDomain('[-securityDomainName mySSOLTPADomain -resourceName Cell=:ServerCluster=sso_clus]')

 
6.0) Once you finish setting the security related configuration, you can run the following command to get information related to active security domain.
Using wsadmin script:
AdminTask.getActiveSecuritySettings('-securityDomainName testDomain')

7.0) Deploy the application. For this exercise, we'll be using DefaultApplication that comes with WAS installation. Below command installs the app on sso_clus.
 Note: for clarity, the below command is broken into multiple lines. Run as a single line.
# AdminApp.install('/apps/IBM/WebSphereND/AppServer/installableApps/DefaultApplication.ear',
   '[-distributeApp -appname DefaultSSOApplication -createMBeansForResources -cluster sso_clus -MapModulesToServers [[ "Increment EJB module" Increment.jar,META-INF/ejb-jar.xml WebSphere:cluster=sso_clus ][ "Default Web Application" DefaultWebApplication.war,WEB-INF/web.xml WebSphere:cluster=sso_clus ]]]')
 
8.0) Security role to user/group mapping:
As outlined in the beginning, for this POC , we will map "All Role" to Special subject, which is "All Authenticated in Trusted Realms". Follow the step below using WebSphere Admin console:
Navigate to Applications > Application Types > WebSphere enterprise application
Select DefaultSSOApplication.ear > Security role to user/group mapping.
Select "All Role" and Select "All Authenticated in Trusted Realms" from Map Special Subjects drop down list. and apply, save and synchronise.

9.0) Export LTPA key:
# Using WAS console: Security > Global security > LTPA
# Using wsadmin script:
AdminTask.exportLTPAKeys('[-ltpaKeyFile file:/tmp/Myltpa.key -password Your-Password ]')

10.0) Since, we are using external trusted realm, you need to open (open it in VI like text editor, so that it does not insert any unwanted character) the exported LTPA key and modify the following:
# Original:
com.ibm.websphere.ltpa.Realm=defaultWIMFileBasedRealm
# Updated:
com.ibm.websphere.ltpa.Realm=DatapowerRealm

11.0) Install ODCInfo application. ODCInfo application will be deployed and run on WAS Deployment Manager server. It captures the WAS cluster information and DP will be able to retrieve it dynamically and load balance the traffic. You can download it from IBM Fix Central. Search for "ResourceKit.710" and download the ResourceKit.710.zip. You need to logon using your IBM credentials to download it. For details on ODCInfo application installation and configuration, refer to https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.1.0/com.ibm.dp.doc/lbg_enablingretrievalwlminfo.html
11.1) Extract ResourceKit.710.zip on Deploymen Manager server. In this example, I've extracted it under ~/ResourceKit.
11.2) Installing the OSGi bundle. Basically copy the com.ibm.datapower.odc.osgi.jar to plugins directory of WAS.
Note: my WAS_HOME is /apps/IBM/WebSphere/AppServer

$>cp ~/ResourceKit/AO/com.ibm.datapower.odc.osgi/dist/com.ibm.datapower.odc.osgi.jar /apps/IBM/WebSphere/AppServer/plugins

11.3) Run following command to clear and recreate the OSGi cache. Details: https://www.ibm.com/support/knowledgecenter/en/SSAW57_7.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rxml_osgicfginit_script.html
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./osgiCfgInit.sh
   OSGi profile cache successfully cleaned for /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile.
   OSGi server cache successfully cleaned for /apps/IBM/WebSphere/AppServer/profiles/<myDmgrProfile>/servers/dmgr.
# Start the OSGI console (optional):
$>./osgiConsole.sh
#(Optional) run the OSGI diagnostic
diag com.ibm.datapower.odc.osgi
# Verify that a message states: No unresolved constraints.
$>cd /apps/IBM/WebSphere/AppServer/profiles/<myDmgrProfile>/bin
$>./osgiConsole.sh
   osgi> diag com.ibm.datapower.odc.osgi
   websphere@plugins/com.ibm.datapower.odc.osgi.jar [5]
  No unresolved constraints.

11.4) Install ODCInfo Web application and verify:
# ./wsadmin.sh -f script_path/ODCInfoDeploy.jacl dmgr_server_name dmgr_node_name path_to_war_file ODCInfo
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./wsadmin.sh -f ~/ResourceKit/AO/scripts/ODCInfoDeploy.jacl dmgr myDmgrNode ~/ResourceKit/AO/com.ibm.datapower.odc/ODCInfo_ND61.war ODCInfo
   ...
   ADMA5013I: Application ODCInfo installed successfully.
   ODCInfoDeploy: saving the configuration


# Verify installation:
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./wsadmin.sh -f ~/ResourceKit/AO/scripts/ODCInfoCheckInstall.jacl myCell01 dmgr ODCInfo
    ...
   ADMA5072I: Distribution status check completed for application ODCInfo.
   Application ODCInfo is deployed successfully.
 
# Start the ODCInfo application:
$>cd /apps/IBM/WebSphere/AppServer/profiles/myDmgrProfile/bin
$>./wsadmin.sh -f ~/ResourceKit/AO/scripts/ODCInfoStart.jacl myCell01 myDmgrNode ODCInfo
   ...
   WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: ...
   ODCInfoStart.jacl: ODCInfo started
 
11.5) Verify that the ODCInfo is able to retrieve the cluster info:
   http://<DMGR_IP>:<DMGR_HTTP_PORT>/ODCInfo/ODCInfo?c=sso_clus
Important note:
If you are not able to access the cluster info using the above mentioned URL, it’s most probably because of following reason.
By default ‘dmgr’ server’s both (secured, and non-secured) webcontainer transport ports are assigned as host aliases under ‘admin_host’ virtual host. However by default, ODCInfo uses ‘default_host’ virtual host as it cannot use ‘admin_host’, what you need to do is remove non-secured webcontainer port of dmgr from ‘admin_host’ virtual host, and add it as an alias to ‘default_host’virtual host. Save & synchronise the configuration and restart the Deployment Manager.

12.0) Extract SSL certificate from WAS.For details on SSL key related wsadmin commands, refer to: https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rxml_atpersonalcert.html#rxml_atpersonalcert__cmd5

# Here is how you can export the Cell or node cert from WAS:
# List available keystores using wsadmin script:
AdminTask.listKeyStores('[-all true -keyStoreUsage SSLKeys ]')

# List personal certificates:
AdminTask.listPersonalCertificates('[-keyStoreName NodeDefaultKeyStore -keyStoreScope (cell):myCell01:(node):myWASNode01 ]')

# Extract the certificates
# Using WAS Admin console: SSL certificate and key management > Key stores and certificates > NodeDefaultKeyStore > Personal certificates > Extract certificate

# Using wsadmin script:
AdminTask.extractCertificate('[-certificateFilePath /tmp/localWASNodeCertKS.crt -base64Encoded true -certificateAlias default -keyStoreName NodeDefaultKeyStore -keyStoreScope (cell):myCell01:(node):myWASNode01 ]')

Steps to be performed on DP side:


1.0) Upload the LTPA key to DP:
 You can use DP CLI scp command. In following example, I'm copying MySSOLTPA.key located at /tmp directory on MyWAShost to Datapower cert: folder.
Note: make sure you are in config mode in Datapower while running scp command.
copy scp://<username>@<MyWAShost>//tmp/MySSOLTPA.key cert:///MySSOLTPA.key
For detail on using scp/sftp commands from DP CLI, refer to: http://www-01.ibm.com/support/docview.wss?uid=swg21250655

2.0) Upload/import SSL cert  (extracted from WAS) to DP. Use CLI option as explained in step #1.0.

3.0) Configure password alias for LTPA key using DP Console.
 Navigate to Objects > Configuration Management > Policy> Password Map Alias
Click "Add" button
 Name: mySSOLTPAPWAlias
Password: <your password>
Note: Make sure this password matches exactly to the password that you used while exporting the    LTPA key from WAS
 Apply and Save Configuration.

4.0) Generate Crypto Key to be used as server identity and server certificate (we will use self-signed for this POC) in front side handler.
Navigate to Administration --> Miscellaneous > Crypto Tools with options:
key type: RSA
Generate Self-Signed Certificate
Generate Key and Certificate Objects
Object Name: 'mySSOCryptoKeyObj' using the Crypto tool
For password alias, you can use the same one that was created for LTPA key or create new one.
For details, refer to: https://www.ibm.com/support/knowledgecenter/SS9H2Y_7.5.0/com.ibm.dp.doc/cryptotool_generatingkeyscertificates.html

Here is the execution result:
Action completed successfully.
     Generated private key in "cert:///mySSOCryptoKeyObj-privkey.pem"
     Generated Certificate Signing Request in "temporary:///mySSOCryptoKeyObj.csr"
     Generated Self-Signed Certificate in "cert:///mySSOCryptoKeyObj-sscert.pem" and exported a copy in "temporary:///mySSOCryptoKeyObj-sscert.pem"
     Generated a Crypto Key object named "mySSOCryptoKeyObj" and a Crypto Certificate object named "mySSOCryptoKeyObj"
 
5.0) Create/Configure Crypto certificate:
Navigate to Objects > Crypto Certificate
Click "Add"
Name: LocalWASCert
File Name: localWASNodeCert.crt
Note: for file name, make sure to select the file that was extracted from WAS and imported to DP.
Password Alias: none

6.0) Create SSL Client Profile:
Objects > Crypto Configuration > SSL Client Profile
Click "Add"
Name: mySSOWASSSLClntProfile
In the Credential section, you don't need Identification credentials as this SSL client profile is only be used to validate certificate presented by WAS during secure back-end communication.
enable "Validate server certificate"
Validation credential click on "+" button and create Crypto Validation Credentials
Name: LocalWASCertValCred
Certificate: LocalWASCert
 
7.0) Create SSL Server Profile.
Navigate to Objects > Crypto Configuration > SSL Server Profile
Click on "Add" button.
Name: myLTPASSOSrvProfile
Since, it's not participating into mutual SSL auth with client, disable "Request client authentication" option.
   In Identification credential, click on "+"
   Create/Configure Identification Credential:
   Name:myDPIdCred
   Crypto Key: mySSOCryptoKeyObj
   Note: mySSOCryptoKeyObj was created as part of the crypto key generation using Crypto tool in previous step.
 
8.0) Create Load Blancer Group
Navigate to Object > Network Settings > Load Balancer Group and Add new LB Group.
For this POC, we name it as "myWASLBG"
Select "on" for "Retrieve Workload Management Information"
Workload Management Retrieval: WebSphere Cell
WebSphere Cell: click on "+" sign and provide name and enter WAS Deployment Manager information
Notes:

  • "Deployment Manager Port number" should be the one that is part of the 'default_host' virtual host in WAS. In this POC, we used the non-secured (http) port.
  • SSL Profile - can be ignored in this screen as the WAS cluster information will be retrieved through non SSL port.

Back on the Load Balancer Group configuration page, the "Workload Management Group Name" should be the actual WAS cluster name. For this POC, we have created "sso_clus" WAS cluster, and will enter it here. Protocol: HTTPS.

Note: protocol in this page, identifies the actual (application related) communication between DP and WAS.
 
9.0) Verify the Load Balancer Group status:
Navigate to Status > IP-Network > Load Balancer
   Make sure your Load Balancer Group listed with WAS cluster members.
 
10.0) Create Multi-Protocol Gateway (MPGW) service, define policy etc... In order to make this simple, for this POC, we use template provided by IBM.
10.1) Navigate to "Blueprint Console" and click on "Patterns" symbol on left hand navigation panel.
10.2) Click on "Web application with Form-based authentication and authorization" link.
10.3) Read the information and instruction carefully. It explains in detail here how the DP policies are setup for this template and how it's going to work once deployed. Once you understand, Click on "Deploy..." button.
It'll prompt for few inputs:
  •    Service name: myFormBasedLTPASSO
  •    Back-end end point details: https://myWASLBG
  •    Note: "myWASLBG" is the name of Load Balancer Group that we created in step #8.0.
  •    SSL Client Profile: mySSOWASSSLClntProfile
  •    Note: "mySSOWASSSLClntProfile" is the name of SSL Client Profile that we created in step #6.0 .
  •    Front-end end point details: choose the IP for front end or leave default and choose port that is available
  •    SSL Server type: Server Profile
  •    SSL Server Profile: myLTPASSOSrvProfile
  •    Note: myLTPASSOSrvProfile is the name of SSL Server Profile that we created in step # 7.
  •    For step 4 (Authenticate with LTPA), and step 5 (Generated LTPA Token),
  •    For LTPA key file: MySSOLTPA.key
  •    For Key file password alias:mySSOLTPAPWAlias
Note:
  • MySSOLTPA.key was exported from WAS, customised, and imported to DP.
  • mySSOLTPAPWAlias was created in step #3.0.
Once all the information entered correctly, click on "Deploy Pattern" button. It may take a minute or so to fully deploy the pattern.

11.0) Once, it is successfully deployed, you'll see the "myFormBasedLTPASSO" under Multi-Protocol Gateway Service list. You need to change few things:
11.1) Navigate to "Multi-Protocol Gateway" and click on the MPGW gateway link just created. Look for the "XML Manager" setting, most probably, by default, the "default" XML Manager has been set.
Click on the '+' sign to create another XML Manager with Load Balancer Group.
Provide the name for XML Manager and select the LB group "myWASLBG" that you've created in step #8.
11.2) Also, by default, DP uses "AAAInfo.xml" as a credential storage. If you need to add/edit user, password or credential mapping information, you can edit this file or use another one.

12.0) Save the configuration and you're ready to test.

Testing:

1.0) Make sure WAS cluster is up and all DP objects and services running.
2.0) Access the page: https://<DP_Front_Side_Handler_IP>:<DP_Front_Side_Handler_Port>/snoop
   you'll be prompted for Username and Password.

   Note: Use one of the usernames and corresponding password listed in AAAInfo.xml.
   Once login is successful, you'll get the snoop page served by one of the WAS cluster member.

3.0) Verify the DP Load Balancer configuration is working properly:
   > Each time, you access the snoop page, look at the "Server name" and "Server Port" and make sure you're hitting all cluster members.

Troubleshooting:

1.0) Your SSO is not working (prompted for credential again) and you see following message in WAS SystemOut.log:

[23/05/17 14:35:14:841 EDT] 000000a0 LTPAServerObj W   SECJ0371W: Validation of the LTPA token failed because the token expired with the following info: Token expiration Date: Tue May 23 10:44:58 EDT 2017, current Date: Tue May 23 14:35:14 EDT 2017 Token attributes:  username=user:<realm>/<user>.. This warning might indicate expected behavior. Please refer to technote at http://www-01.ibm.com/support/docview.wss?uid=swg21594981.

Resolution:
It can be caused by few things, and one of them is time not in synchronized between DP and WAS servers. So, first thing you want to check and make sure is the time setting in DP and WAS server are in sync.

2.0) Looked like your SSO with LTPA is working (because you see your request hitting the WAS server(s), but you can't see intended page and you see the authorisation error in browser and WAS SystemOut.log shows the following:

[23/05/17 15:10:54:474 EDT] 000000a0 WebCollaborat A   SECJ0129E: Authorization failed for user user:<realm>/<user>:<realm> while invoking GET on default_host://snoop, Authorization failed, Not granted any of the required roles: All Role

Resolution:
It is most probably, because the application security has been enabled, but no User role assigned.
On WAS Admin console, navigate to "Applications > Application Types > WebSphere enterprise applications"
Double click on the application link .
Click on "Security role to user/group mapping"
When role mapping page appears, do one of the following based on your requirement:
   1) Map either User or Group for the role.
   2) Map Special Subjects, for the example, since we are trusting the users authenticated by DP and we've created trusted external realm, we'll select option "All Authenticated in Trusted Realms" option.
Save and Synchronise.
And try your test again.

Hope, you'll find this post helpful !!!

My 2nd Year of 10K Run

What a day - rainy,  but beautiful and cheerful! 15,000 Cyclists, Runners and Walkers stormed the Gardiner Expressway and Don Valley Parkway (@TO_DVP) today braving the rain for Ride for Heart event (www.rideforheart.ca). It was still raining in Toronto when we're at start line of 10KM run, but by the time, we started running, mother nature  smiled a little bit and slowly stopped pouring. It felt amazingly good running on traffic free Gardiner Expressway while watching downtown Toronto with fellow participants - all happy and cheering. I completed the race in just few seconds less of one hour (my actual time recorded by BIB and provided by www.sportstats.ca was 59 minutes and 23 seconds). This post is to Thank YOU ALL who supported me and contributed for the good cause (Life-Saving research in the field of Heart and Stroke) and my family who waked up early in the morning and went with me to Ontario Place (http://www.ontarioplace.com/) to cheer for me. Because of your help, I was able to raise $500.00 for Heart and Stroke Foundation. Over all, today's event raised about 6 Million Dollar for Heart and Stroke related research. That's really great! Thank you All!