Upgrade to Docker EE 2.0 and UCP 3.x for Choice of Swarm or Kubernetes Orchestration

Docker Enterprise Edition (EE) 2.0 has introduced integrated Kubernetes orchestration engine along with SWARM. Since Kubernetes is installed and configured as part of the of the upgrade to Docker EE 2.0 and Universal Control Plane (UCP) 3.x, it saves a lot of time which otherwise is needed to install and setup Kubernetes environment.

In this blog post, I'm discussing the upgrade process (not going to go through each step though. Because official Docker documentation is detail enough for this) and going to direct you to the right documentation and also discuss few issues that I encountered during the upgrade and how I resolved them.

Planning for Upgrade

1) Prerequisite check for hardware/software - Docker recommends at least 8 GB of physical memory available on UCP and Docker Trusted Registry (DTR) nodes and 4 GB for other worker nodes. See details hardware and software requirement here: https://docs.docker.com/ee/ucp/admin/install/system-requirements/

2) Firewall ports - since Kubernetes master and worker nodes will be part of the upgraded environment, additional ports required for Kubernetes need to open. Details on port used can be found here: https://docs.docker.com/ee/ucp/admin/install/system-requirements/#ports-used. I put together few lines of shell script to open firewall ports (uses firewall-cmd utility). Use/modify it as needed.

openFWPortsForDockerEE.sh

#!/bin/sh # openFWPortsForDockerEE.sh # Opens required ports for Docker EE 2.0/UCP 3.x # Ref: # https://docs.docker.com/ee/ucp/admin/install/system-requirements/#ports-used # https://docs.docker.com/datacenter/ucp/2.1/guides/admin/install/system-requirements/#network-requirements tcp_ports="179,443,80,2375,2376,2377,2380,4001,4443,4789,6443,6444,7001,7946,8080,10250,12376-12387" udp_ports="4789,7946" openFW() {    IFS=",";    for _port in $1; do       echo "Opening ${_port}/$2";       sudo firewall-cmd --permanent --zone=public --add-port=${_port}/$2;    done    IFS=" "; } openFW "${tcp_ports}" tcp; openFW "${udp_ports}" udp; # Recycle firewall sudo firewall-cmd --reload;

Backup Docker EE

You need to backup Docker Swarm, UCP, and DTR . Please follow this document (https://docs.docker.com/ee/backup/) for backup.

Upgrade Docker Engine

Very well documented step by step process can be found here: https://docs.docker.com/ee/upgrade/#upgrade-docker-engine

Upgrade UCP

UCP can be upgraded from UCP Web user interface (Web UI) or  command line interface (CLI). Both options are documented here: https://docs.docker.com/ee/ucp/admin/install/upgrade/#use-the-cli-to-perform-an-upgrade.

Note: If all possible try to use CLI instead of Web UI. I had upgraded my personal DEV environment using CLI and did not encounter any issue, however, one of my colleagues initially tried to use Web UI and  had issue. The upgrade process went forever, and failed.

Note: If you have less than 4 GB of memory, you'll get warning during the upgrade. It may complete successfully (as you see below) or may fail. So, it is best practice to fulfil the minimum requirement whenever possible. Below is output from my UCP 3.0 upgrade:

$> sudo docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:3.0.0 upgrade --interactive INFO[0000] Your engine version 17.06.2-ee-10, build 66261a0 (3.10.0-514.el7.x86_64) is compatible FATA[0000] Your system does not have enough memory. UCP suggests a minimum of 4.00 GB, but you only have 2.92 GB. You may have unexpected errors. You may proceed by specifying the '--force-minimums' fla g, but you may experience scale and performance problems as a result [osboxes@centosddcucp scripts]$ sudo docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:3.0.0 upgrade --interactive --force-minimums INFO[0000] Your engine version 17.06.2-ee-10, build 66261a0 (3.10.0-514.el7.x86_64) is compatible WARN[0000] Your system does not have enough memory. UCP suggests a minimum of 4.00 GB, but you only have 2.92 GB. You may have unexpected errors. WARN[0002] Your system uses devicemapper. We can not accurately detect available storage space. Please make sure you have at least 3.00 GB available in /var/lib/docker INFO[0006] Upgrade the UCP 3.0.0 installation on this cluster to 3.0.0 for UCP ID: nufs9fb696bs6rm4kxaauewly INFO[0006] Once this operation completes, all nodes in this cluster will be upgraded. Do you want proceed with the upgrade? (y/n): y INFO[0017] Pulling required images... (this may take a while) INFO[0017] Pulling docker/ucp-interlock:3.0.0 INFO[0048] Pulling docker/ucp-compose:3.0.0 INFO[0130] Pulling docker/ucp-dsinfo:3.0.0 INFO[0183] Pulling docker/ucp-interlock-extension:3.0.0 WARN[0000] Your system does not have enough memory. UCP suggests a minimum of 4.00 GB, but you only have 2.92 GB. You may have unexpected errors. WARN[0002] Your system uses devicemapper. We can not accurately detect available storage space. Please make sure you have at least 3.00 GB available in /var/lib/docker INFO[0007] Checking for version compatibility INFO[0007] Updating configuration for Interlock service INFO[0038] Updating configuration for existing UCP service INFO[0141] Waiting for cluster to finish upgrading INFO[0146] Success! Please log in to the UCP console to verify your system.

Note: You may also find your upgrade to UCP 3.x process getting stuck while updating ucp-kv, just like we had in one of our environments. The symptom and resolution are documented here: https://success.docker.com/article/upgrade-to-ucp-3-gets-stuck-updating-ucp-kv

After the Upgrade

If you run 'docker ps' after upgrade on UCP host, all UCP related processes (like docker/ucp-*) should be of version '3.x', if you notice any of those processes still in version '2.x', meaning upgrade is not quite successful. You can also run 'docker version' and make sure the output shows 'ucp/3.x'

If your upgrade is successful, after the upgrade, you are going to notice few things right way, some of them are listed below:

1) UCP Web UI looks different now. You are going to see Kubernetes and related resources standing out as the first class citizen.

2) You may also notice that your application is not accessible any more even though corresponding service(s) may seem to be running (specifically, if you used HTTP Routing Mesh (HRM) before the upgrade). We encountered an issue (related to HRM) in our DEV environment. Before the upgrade, we had something like this configuration (fragment from  our yaml file):

version: "3.1" services:    testsvc:       ...       ...       ports:          - "9080"          - "9443"       deploy:          ...          ...          labels:             - "com.docker.ucp.mesh.http.9080=external_route=http://testsvc.devdte.com:8080,internal_port=9080"             - "com.docker.ucp.mesh.http.9443=external_route=sni://testsvc.devdte.com:8443,internal_port=9443" ... ...

As shown above, internal port 9080 is mapped to external port 8080 (http) and internal port 9443 is mapped to external port 8443 (https) and 'testsvc.devdte.com' is configured as a host. And our routing mesh setting looked like as shown below:

Before the upgrade, the above configuration allowed us to access the service as shown below:

• http://testsvc.devdte.com:8080/xxx
  or
• https://testsvc.devdte.com:/8443/xxx

However, after the upgrade, we could access the application only on port 8443. If you encounter similar issue, refer to Layer 7 routing upgrade for more details.


3) Another interesting issue we encountered after the upgrade was related to HTTP header parameter being rejected. One of our applications relied on HTTP header parameter and the parameter had a underscore '_' (something like 'user_name'). After the upgrade, suddenly, application started responding with HTTP status code 502. After investigation, we found out that the Nginx - that's a part of Layer 7 routing solution, was silently rejecting this parameter because it had underscore '_'. Refer to my blog How to override Kubernetes Ingress-Nginx-Controller and Docker UCP Layer 7 Routing Configuration for details.

4) Lastly, if you are planning to use Kubernetes orchestration and 'kubectl' utility to connect to Kubernetes master, you need to download your client certificates bundle again. env.sh/env.cmd has been updated to set Kubernetes cluster, context and credentials configuration so that 'kubectl' command can securely establish connection to Kubernetes master and be able to communicate. Refer to CLI based access and Install the Kubernetes CLI for more details. Once you have installed 'kubectl' and downloaded and extracted client certificates bundle, test connectivity to Kubernetes master as follows:

# Change directory to the folder where you extracted you client certificates bundle # and run following command to set kubernetes context, credentials and cluster configuration $> eval "$(<env.sh)" Cluster "ucp_ddcucphost:6443_ppoudel" set. User "ucp_ddcucphost:6443_ppoudel" set. Context "ucp_ddcucphost:6443_ppoudel" created. # Confirm the connection to UCP. You should see something like this: $> kubectl config current-context ucp_ddcucphost:6443_ppoudel # Inspect Kubernetes resources $> kubectl get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 443/TCP 6d

How to override Kubernetes Ingress-Nginx-Controller and Docker UCP Layer 7 Routing Configuration

One of our dockerized applications mysteriously stopped working after we upgraded to Docker Enterprise Edition (EE) 2.0/Universal Control Plane (UCP) 3.x. After investigation, we found out that the Nginx that is being used as part of Docker Layer 7 routing solution was silently dropping  HTTP header parameter (refer to Missing (disappearing) HTTP Headers) which had underscore '_' (something like 'user_name') in it and our application required the value from that HTTP header parameter in order to function correctly. Note: our name based virtual hosting relied in Docker Layer 7 routing solution.
Later on, as part of migration to Kubernetes from Docker SWARM, we again encountered this issue as we were using Kubernetes' Ingress-Nginx-Controller.
In this post, I'm going to show how to resolve this issue whether it is with Docker UCP Layer 7 routing or Kubernetes' Ingress-Nginx-Controller.

Overriding Kubernetes' Ingress-Nginx-Controller configuration

Create a configMap as shown below. In this example, I'm overriding the 'underscores_in_headers' Nginx configuration to 'on' from default 'off'. Refer to this post to see what parameters are allowed in configMap.

ingress-nginx-config.yml

apiVersion: v1 kind: ConfigMap data:    enable-underscores-in-headers: "true" metadata:    name: nginx-configuration    namespace: ingress-nginx    labels:       app: ingress-nginx

The key here is:
data:
   enable-underscores-in-headers: "true"

If you have existing configMap object 'nginx-configuration', then you can edit and update the parameter's value that you want to override. If configMap object does not exist, then you can create it using 'kubectl' as shown below, however, make sure you are referring this configMap object in your controller's container spec.

#edit $> kubectl edit configMap/nginx-configuration -n ingress-nginx # It opens the configuration into your editor, you can update any configuration and save. Saving the yaml will update the resource in the API server. # Create $> kubectl create -f ingress-nginx-config.yml -n ingress-nginx

In order to verify whether the configuration of ingress-nginx-controller has been updated, you can do the following:

1. Find the ingress-nginx-controller pod using following command;
   

   $>kubectl get pods -n ingress-nginx

2. See nginx.conf file and make sure the parameter you are overriding has been updated. In this case we are looking underscores_in_headers value updated from 'off' to 'on'
   

   $> kubectl exec nginx-ingress-controller-68db848949-ncvj7 -n ingress-nginx cat /etc/nginx/nginx.conf | grep underscores_in_headers underscores_in_headers on;

Overriding/customizing Docker Layer 7 routing solution configuration 

The following steps you can using Docker CLI. Make sure, secure connection has been established from where you are running Docker CLI to UCP. You can do it using Client Certificate Bundle.

# export current ucp-interlock configuration to CURRENT_CONFIG_NAME variable $> CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' ucp-interlock) # Write information to config.toml file $> docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME > config.toml # Update config.toml as below. In this case we are overriding the value of nginx # configuration 'underscores_in_headers' from 'off' to 'on' by changing ucp-interlock service # configuration 'UnderscoresInHeaders' value from 'false' to 'true' # Create updated config object $> docker config create UPDATED_CONFIG_NAME config.toml # Verify the object created: $> docker config ls ID NAME CREATED UPDATED 061xu64qyotlbtrdz9l5e1s0h UPDATED_CONFIG_NAME 6 seconds ago 6 seconds ago # Update the ucp-interlock service to start using the new configuration: $> docker service update \ --config-rm $CURRENT_CONFIG_NAME \ --config-add source=$UPDATED_CONFIG_NAME,target=/config.toml \ ucp-interlock # Wait for a minute, make sure interlock service started successfully. Look the timestamp $> docker ps | grep interlock # Rollback (if necessary) $> docker service update --update-failure-action rollback ucp-interlock

Note: the above steps can be used to update/override any other Layer 7 routing configuration. Refer to Layer 7 routing configuration reference to find out all other configurable properties.

Note: Everytime you restart (disable/enable) the Layer 7 routing solution from UCP UI, it starts with default configuration, so you have to perform above steps again to override the configuration.