Port Conflict and Resolution in WebSphere Application Server and Liberty Profile

Port conflict is one of the frequent issues that integrator and administrator encounter. In this post, I'm going to discuss how to resolve port conflict in WebSphere Application Server Traditional (WASt) and WebSphere Liberty Profile (WLP). Recently, while doing a Proof of Concept (PoC) work, I had to install and launch a WLP server on the same host where WASt was running and I received following error in the WLP's messages.log:

[4/15/18 16:59:00:048 BST] 00000017 ibm.ws.transport.iiop.security.config.ssl.yoko.SocketFactory E CWWKS9580E: The server socket could not be opened on localhost:2,809.  The exception message is Address already in use (Bind failed).
[3/19/18 9:42:51:037 EDT] 00000017 LogService-271-com.ibm.ws.management.j2ee.mejb               E CWWKE0701E: [com.ibm.ws.management.j2ee.mejb.service.ManagementEJBService(77)] The setServerStarted method has thrown an exception Bundle:com.ibm.ws.management.j2ee.mejb(id=271) java.lang.IllegalStateException: com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeException: com.ibm.ws.exception.RuntimeError: java.lang.IllegalStateException: The orb is not available
        at com.ibm.ws.ejbcontainer.osgi.internal.EJBContainerImpl.startSystemModule(EJBContainerImpl.java:230)
        at com.ibm.ws.management.j2ee.mejb.service.ManagementEJBService.startManagementEJB(ManagementEJBService.java:161)
        ...
        Caused by: com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeException: com.ibm.ws.exception.RuntimeError: java.lang.IllegalStateException: The orb is not available
        at com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeImpl.startSystemModule(EJBRuntimeImpl.java:968)
        at com.ibm.ws.ejbcontainer.osgi.internal.EJBContainerImpl.startSystemModule(EJBContainerImpl.java:228)
        ... 39 more
Caused by: com.ibm.ws.exception.RuntimeError: java.lang.IllegalStateException: The orb is not available
        at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.startModule(AbstractEJBRuntime.java:587)
        at com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeImpl.startSystemModule(EJBRuntimeImpl.java:964)
        ... 40 more
Caused by: java.lang.IllegalStateException: The orb is not available
        at com.ibm.ws.ejbcontainer.remote.internal.EJBRemoteRuntimeImpl.bind(EJBRemoteRuntimeImpl.java:189)

As per error message above, the WLP servr could not start object request broker (ORB) service, because port 2809 is in use. Note: WLP (by default) uses port 2809 for ORB.
Next step is to find out which process is (already) using this port, you can use netstat -ntlp | grep <port> like netstat -ntlp | grep 2809.
Once you find out the PID (process id) that is listening on that particular port, you can get more detail about the process by using ps -ef | grep <PID> In my case the port 2809 was being used by WASt Nodeagent process.
Now, I had two choices: either change the WASt Nodeagent process's port number or change the IIOP Endpoint's port in WLP server. I have tested both and below I'm going to show both options.

Change the iiopEndpoint port for WLP server

It is simple, just open the server.xml for the WLP server and add following lines. Here I'm changing iiopport to 2709 and iiopsport to 9403.
Note: Make sure those new ports are not currently being used by any process. You can find out whether they are currently being used using command: netstat -na | egrep '(2709|9403)'


<iiopendpoint id="defaultIiopEndpoint" iiopport="2709">    <iiopsoptions iiopsport="9403" sslref="defaultSSLConfig">    </iiopsoptions> </iiopendpoint>

Refer to https://www.ibm.com/support/knowledgecenter/en/SS7K4U_liberty/com.ibm.websphere.wlp.zseries.doc/ae/rwlp_portnums.html to find out WLP default port numbers.
Restart the WLP server after updating the port number in server.xml and review the messages.log again to make sure port binding errors gone.

Update/Change the port on WASt side

You can change/update port on WASt side two ways. Either using Administration console or by using wsadmin command. To find out more details about the PortManagement command group for the AdminTask object, refer to https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rxml_atportmgt.html

Using wsadmin command:

1. Connect to Dmgr
   

   $> ./wsadmin.sh WASX7209I: Connected to process "dmgr" on node ubuntuwas9CellManager01 using SOAP connector; The type of process is: DeploymentManager WASX7031I: For help, enter: "print Help.help()"

2. Find out the current port in question. In my case, it is for nodeagent
   

   wsadmin>AdminTask.listServerPorts('nodeagent', '[-nodeName ubuntuwas9Node02]')  u'[[IPC_CONNECTOR_ADDRESS [[[host localhost] [node ubuntuwas9Node02] [server nodeagent] [port 9629] ]]] ]\n[[CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 9201] ]]] ]\n[[XDAGENT_PORT [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 7062] ]]] ]\n[[OVERLAY_UDP_LISTENER_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 11003] ]]] ]\n[[DCS_UNICAST_ADDRESS [[[host *] [node ubuntuwas9Node02] [server nodeagent] [port 9353] ]]] ]\n[[NODE_DISCOVERY_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 7272] ]]] ]\n[[BOOTSTRAP_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 2809] ]]] ]\n[[NODE_IPV6_MULTICAST_DISCOVERY_ADDRESS [[[host ff01::1] [node ubuntuwas9Node02] [server nodeagent] [port 5001] ]]] ]\n[[SAS_SSL_SERVERAUTH_LISTENER_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 9901] ]]] ]\n[[SOAP_CONNECTOR_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 8878] ]]] ]\n[[NODE_MULTICAST_DISCOVERY_ADDRESS [[[host 232.133.104.73] [node ubuntuwas9Node02] [server nodeagent] [port 5000] ]]] ]\n[[ORB_LISTENER_ADDRESS [[[host ubuntuwas9][node ubuntuwas9Node02] [server nodeagent] [port 9900] ]]] ]\n[[CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 9202] ]]] ]\n[[OVERLAY_TCP_LISTENER_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 11004] ]]] ]'

3. As seen from command output above, the nodeagent is listening on port 2809, let's change  it to 2709.
   

   wsadmin>AdminTask.modifyServerPort ('nodeagent', '[-nodeName ubuntuwas9Node02 -endPointName BOOTSTRAP_ADDRESS -port 2709 -modifyShared true]') u'[[BOOTSTRAP_ADDRESS [[[host ubuntuwas9] [node ubuntuwas9Node02] [server nodeagent] [port 2709] ]]] ]'

4. Save the changes:
   

   wsadmin>AdminConfig.save()

5. If it's a federated environment, make sure to synchronise the configuration with the node(s). If your Nodeagent is currently stopped, go to the host machine where Nodeagent is installed and run syncNode.sh <dmgr-host> as shown below:
   

   wasadmin@ubuntuwas9:/opt/ibm/WebSphere/AppServer/profiles/AppSrv01/bin$ ./syncNode.sh ubuntuwas9  ADMU0116I: Tool information is being logged in file /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/logs/syncNode.log ADMU0128I: Starting tool with the AppSrv01 profile ADMU0401I: Begin syncNode operation for node ubuntuwas9Node02 with Deployment Manager ubuntuwas9: 8879 ADMU0016I: Synchronizing configuration between node and cell. ADMU0402I: The configuration for node ubuntuwas9Node02 has been synchronized with Deployment Manager ubuntuwas9: 8879

   
   If your nodeagent is running. You can use the following command to sync:
   
   

   wsadmin>dmgrObj=AdminControl.queryNames('WebSphere:type=DeploymentManager,*') wsadmin>AdminControl.invoke(dmgrObj, 'multiSync', '[false]', '[java.lang.Boolean]')

6. (Re)start the Nodeagent:
   if nodeagent is stopped, start the nodeagent:
   Note: you can use the stop[start]Node.sh command from profile_root/bin directory. ./startNode.sh
   if it is currently running, stop and start it:
   ./stopNode.sh
./startNode.sh


Using WASt Administration console:

1. Access WASt Administration console: https://<host>:<port>/admin
2. Once logged in, on the left panel, navigate to "SystemAdministration" and click on "Node agents"
3. On the right hand side, click on "Ports" link under "Additional Properties". You can see a table that contains port name and port numbers.
4. Click on "Details" button beside the table in order to update any port.
5. Select any and click on any port name link.
6. Editable screen appears, update the port and click "Apply"
7. Make sure to Synchronise the change with the nodes.
8. Restart the affected server. (Nodeagent in this case).

Hope, this post will be helpful in your next port conflict resolution!