SysGenius-e-Log: Docker EE

Docker Enterprise Edition (EE) 2.0 has introduced integrated Kubernetes orchestration engine along with SWARM. Since Kubernetes is installed and configured as part of the of the upgrade to Docker EE 2.0 and Universal Control Plane (UCP) 3.x, it saves a lot of time which otherwise is needed to install and setup Kubernetes environment.

In this blog post, I'm discussing the upgrade process (not going to go through each step though. Because official Docker documentation is detail enough for this) and going to direct you to the right documentation and also discuss few issues that I encountered during the upgrade and how I resolved them.

Planning for Upgrade

1) Prerequisite check for hardware/software - Docker recommends at least 8 GB of physical memory available on UCP and Docker Trusted Registry (DTR) nodes and 4 GB for other worker nodes. See details hardware and software requirement here: https://docs.docker.com/ee/ucp/admin/install/system-requirements/

2) Firewall ports - since Kubernetes master and worker nodes will be part of the upgraded environment, additional ports required for Kubernetes need to open. Details on port used can be found here: https://docs.docker.com/ee/ucp/admin/install/system-requirements/#ports-used. I put together few lines of shell script to open firewall ports (uses firewall-cmd utility). Use/modify it as needed.

openFWPortsForDockerEE.sh


     #!/bin/sh

     # openFWPortsForDockerEE.sh

     # Opens required ports for Docker EE 2.0/UCP 3.x

     # Ref:

     # https://docs.docker.com/ee/ucp/admin/install/system-requirements/#ports-used

     # https://docs.docker.com/datacenter/ucp/2.1/guides/admin/install/system-requirements/#network-requirements

     tcp_ports="179,443,80,2375,2376,2377,2380,4001,4443,4789,6443,6444,7001,7946,8080,10250,12376-12387"

     udp_ports="4789,7946"

     

     openFW() {

         IFS=",";

         for _port in $1; do

             echo "Opening ${_port}/$2";

             sudo firewall-cmd --permanent --zone=public --add-port=${_port}/$2;

         done

         IFS=" ";

     }

     

     openFW "${tcp_ports}" tcp;

     openFW "${udp_ports}" udp;

     

     # Recycle firewall

     sudo firewall-cmd --reload;

Backup Docker EE

You need to backup Docker Swarm, UCP, and DTR . Please follow this document (https://docs.docker.com/ee/backup/) for backup.

Upgrade Docker Engine

Very well documented step by step process can be found here: https://docs.docker.com/ee/upgrade/#upgrade-docker-engine

Upgrade UCP

UCP can be upgraded from UCP Web user interface (Web UI) or command line interface (CLI). Both options are documented here: https://docs.docker.com/ee/ucp/admin/install/upgrade/#use-the-cli-to-perform-an-upgrade.

Note: If all possible try to use CLI instead of Web UI. I had upgraded my personal DEV environment using CLI and did not encounter any issue, however, one of my colleagues initially tried to use Web UI and had issue. The upgrade process went forever, and failed.

Note: If you have less than 4 GB of memory, you'll get warning during the upgrade. It may complete successfully (as you see below) or may fail. So, it is best practice to fulfil the minimum requirement whenever possible. Below is output from my UCP 3.0 upgrade:


     $> sudo docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:3.0.0 upgrade --interactive



INFO[0000] Your engine version 17.06.2-ee-10, build 66261a0 (3.10.0-514.el7.x86_64) is compatible

FATA[0000] Your system does not have enough memory.  UCP suggests a minimum of 4.00 GB, but you only have 2.92 GB.  You may have unexpected errors.  You may proceed by specifying the '--force-minimums' fla         g, but you may experience scale and performance problems as a result

[osboxes@centosddcucp scripts]$ sudo docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:3.0.0 upgrade --interactive --force-minimums

INFO[0000] Your engine version 17.06.2-ee-10, build 66261a0 (3.10.0-514.el7.x86_64) is compatible

WARN[0000] Your system does not have enough memory.  UCP suggests a minimum of 4.00 GB, but you only have 2.92 GB.  You may have unexpected errors.

WARN[0002] Your system uses devicemapper.  We can not accurately detect available storage space.  Please make sure you have at least 3.00 GB available in /var/lib/docker

INFO[0006] Upgrade the UCP 3.0.0 installation on this cluster to 3.0.0 for UCP ID: nufs9fb696bs6rm4kxaauewly

INFO[0006] Once this operation completes, all nodes in this cluster will be upgraded.

Do you want proceed with the upgrade? (y/n): y

INFO[0017] Pulling required images... (this may take a while)

INFO[0017] Pulling docker/ucp-interlock:3.0.0

INFO[0048] Pulling docker/ucp-compose:3.0.0

INFO[0130] Pulling docker/ucp-dsinfo:3.0.0

INFO[0183] Pulling docker/ucp-interlock-extension:3.0.0

WARN[0000] Your system does not have enough memory.  UCP suggests a minimum of 4.00 GB, but you only have 2.92 GB.  You may have unexpected errors.

WARN[0002] Your system uses devicemapper.  We can not accurately detect available storage space.  Please make sure you have at least 3.00 GB available in /var/lib/docker

INFO[0007] Checking for version compatibility

INFO[0007] Updating configuration for Interlock service

INFO[0038] Updating configuration for existing UCP service

INFO[0141] Waiting for cluster to finish upgrading

INFO[0146] Success! Please log in to the UCP console to verify your system.

Note: You may also find your upgrade to UCP 3.x process getting stuck while updating ucp-kv, just like we had in one of our environments. The symptom and resolution are documented here: https://success.docker.com/article/upgrade-to-ucp-3-gets-stuck-updating-ucp-kv

After the Upgrade

If you run 'docker ps' after upgrade on UCP host, all UCP related processes (like docker/ucp-*) should be of version '3.x', if you notice any of those processes still in version '2.x', meaning upgrade is not quite successful. You can also run 'docker version' and make sure the output shows 'ucp/3.x'

If your upgrade is successful, after the upgrade, you are going to notice few things right way, some of them are listed below:

1) UCP Web UI looks different now. You are going to see Kubernetes and related resources standing out as the first class citizen.

2) You may also notice that your application is not accessible any more even though corresponding service(s) may seem to be running (specifically, if you used HTTP Routing Mesh (HRM) before the upgrade). We encountered an issue (related to HRM) in our DEV environment. Before the upgrade, we had something like this configuration (fragment from our yaml file):


version: "3.1"

services:

   testsvc:

      ...

      ...

      ports:

         - "9080"

         - "9443"

      deploy:

         ...

         ...

         labels:

            - "com.docker.ucp.mesh.http.9080=external_route=http://testsvc.devdte.com:8080,internal_port=9080"

            - "com.docker.ucp.mesh.http.9443=external_route=sni://testsvc.devdte.com:8443,internal_port=9443"

...

...

As shown above, internal port 9080 is mapped to external port 8080 (http) and internal port 9443 is mapped to external port 8443 (https) and 'testsvc.devdte.com' is configured as a host. And our routing mesh setting looked like as shown below:

Before the upgrade, the above configuration allowed us to access the service as shown below:

http://testsvc.devdte.com:8080/xxx
or
https://testsvc.devdte.com:/8443/xxx

However, after the upgrade, we could access the application only on port 8443. If you encounter similar issue, refer to Layer 7 routing upgrade for more details.

3) Another interesting issue we encountered after the upgrade was related to HTTP header parameter being rejected. One of our applications relied on HTTP header parameter and the parameter had a underscore '_' (something like 'user_name'). After the upgrade, suddenly, application started responding with HTTP status code 502. After investigation, we found out that the Nginx - that's a part of Layer 7 routing solution, was silently rejecting this parameter because it had underscore '_'. Refer to my blog How to override Kubernetes Ingress-Nginx-Controller and Docker UCP Layer 7 Routing Configuration for details.

4) Lastly, if you are planning to use Kubernetes orchestration and 'kubectl' utility to connect to Kubernetes master, you need to download your client certificates bundle again. env.sh/env.cmd has been updated to set Kubernetes cluster, context and credentials configuration so that 'kubectl' command can securely establish connection to Kubernetes master and be able to communicate. Refer to CLI based access and Install the Kubernetes CLI for more details. Once you have installed 'kubectl' and downloaded and extracted client certificates bundle, test connectivity to Kubernetes master as follows:


# Change directory to the folder where you extracted you client certificates bundle 

# and run following command to set kubernetes context, credentials and cluster configuration 


$> eval "$(<env.sh)" 

Cluster "ucp_ddcucphost:6443_ppoudel" set.

User "ucp_ddcucphost:6443_ppoudel" set.

Context "ucp_ddcucphost:6443_ppoudel" created.



# Confirm the connection to UCP. You should see something like this:



$> kubectl config current-context

ucp_ddcucphost:6443_ppoudel



# Inspect Kubernetes resources



$> kubectl get all

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE

service/kubernetes   ClusterIP   10.96.0.1            443/TCP   6d

SysGenius-e-Log

Pages

Upgrade to Docker EE 2.0 and UCP 3.x for Choice of Swarm or Kubernetes Orchestration

Planning for Upgrade

Backup Docker EE

Upgrade Docker Engine

Upgrade UCP

After the Upgrade