How to override Kubernetes Ingress-Nginx-Controller and Docker UCP Layer 7 Routing Configuration

One of our dockerized applications mysteriously stopped working after we upgraded to Docker Enterprise Edition (EE) 2.0/Universal Control Plane (UCP) 3.x. After investigation, we found out that the Nginx that is being used as part of Docker Layer 7 routing solution was silently dropping  HTTP header parameter (refer to Missing (disappearing) HTTP Headers) which had underscore '_' (something like 'user_name') in it and our application required the value from that HTTP header parameter in order to function correctly. Note: our name based virtual hosting relied in Docker Layer 7 routing solution.
Later on, as part of migration to Kubernetes from Docker SWARM, we again encountered this issue as we were using Kubernetes' Ingress-Nginx-Controller.
In this post, I'm going to show how to resolve this issue whether it is with Docker UCP Layer 7 routing or Kubernetes' Ingress-Nginx-Controller.

Overriding Kubernetes' Ingress-Nginx-Controller configuration

Create a configMap as shown below. In this example, I'm overriding the 'underscores_in_headers' Nginx configuration to 'on' from default 'off'. Refer to this post to see what parameters are allowed in configMap.

ingress-nginx-config.yml

apiVersion: v1 kind: ConfigMap data:    enable-underscores-in-headers: "true" metadata:    name: nginx-configuration    namespace: ingress-nginx    labels:       app: ingress-nginx

The key here is:
data:
   enable-underscores-in-headers: "true"

If you have existing configMap object 'nginx-configuration', then you can edit and update the parameter's value that you want to override. If configMap object does not exist, then you can create it using 'kubectl' as shown below, however, make sure you are referring this configMap object in your controller's container spec.

#edit $> kubectl edit configMap/nginx-configuration -n ingress-nginx # It opens the configuration into your editor, you can update any configuration and save. Saving the yaml will update the resource in the API server. # Create $> kubectl create -f ingress-nginx-config.yml -n ingress-nginx

In order to verify whether the configuration of ingress-nginx-controller has been updated, you can do the following:

1. Find the ingress-nginx-controller pod using following command;
   

   $>kubectl get pods -n ingress-nginx

2. See nginx.conf file and make sure the parameter you are overriding has been updated. In this case we are looking underscores_in_headers value updated from 'off' to 'on'
   

   $> kubectl exec nginx-ingress-controller-68db848949-ncvj7 -n ingress-nginx cat /etc/nginx/nginx.conf | grep underscores_in_headers underscores_in_headers on;

Overriding/customizing Docker Layer 7 routing solution configuration 

The following steps you can using Docker CLI. Make sure, secure connection has been established from where you are running Docker CLI to UCP. You can do it using Client Certificate Bundle.

# export current ucp-interlock configuration to CURRENT_CONFIG_NAME variable $> CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' ucp-interlock) # Write information to config.toml file $> docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME > config.toml # Update config.toml as below. In this case we are overriding the value of nginx # configuration 'underscores_in_headers' from 'off' to 'on' by changing ucp-interlock service # configuration 'UnderscoresInHeaders' value from 'false' to 'true' # Create updated config object $> docker config create UPDATED_CONFIG_NAME config.toml # Verify the object created: $> docker config ls ID NAME CREATED UPDATED 061xu64qyotlbtrdz9l5e1s0h UPDATED_CONFIG_NAME 6 seconds ago 6 seconds ago # Update the ucp-interlock service to start using the new configuration: $> docker service update \ --config-rm $CURRENT_CONFIG_NAME \ --config-add source=$UPDATED_CONFIG_NAME,target=/config.toml \ ucp-interlock # Wait for a minute, make sure interlock service started successfully. Look the timestamp $> docker ps | grep interlock # Rollback (if necessary) $> docker service update --update-failure-action rollback ucp-interlock

Note: the above steps can be used to update/override any other Layer 7 routing configuration. Refer to Layer 7 routing configuration reference to find out all other configurable properties.

Note: Everytime you restart (disable/enable) the Layer 7 routing solution from UCP UI, it starts with default configuration, so you have to perform above steps again to override the configuration.