How to Configure PostgreSQL with SSL/TLS support on Kubernetes

SSL is disabled in the default Postgresql configuration and I had to struggle a little bit while making Postgresql on Kubernetes with SSL/TLS support to work. After few research and trials, I was able to resolve the issues and here I'm sharing what I have done to make it work for me.

High Level Steps:

1. Customize postgresql.conf  (to add/edit SSL/TLS configuration) and create configMap object. This way, we don't need to rebuild the Postgres to apply custom postgresql.conf , because ConfigMap allows us you to decouple configuration artifacts from image content. 
2. Create secret type objects for server.key, server.crt, root.crt, ca.crt, and password file.
3. Define and use NFS type PersistentVolume (PV) and PersistentVolumeClaim (PVC)
4. Use securityContext to resolve permission issues.
5. Use '-c config_file=<config-volume-location>/postgresql.conf' to override the default postgresql.conf

Note: all files used in this post can be cloned/downloaded from GitHub https://github.com/pppoudel/postgresql-with-ssl-on-kubernetes.git

Let's get started

In the example, I'm using namespace called 'shared-services' and service account called 'shared-svc-accnt'. You can create your own namespace and service account or use the 'default'. In anyways, I have listed here necessary steps and yaml files can be downloaded from github.

Create namespace and service account

# Create namespace shared-services    $> kubectl create -f shared-services-ns.yml # Create Service Account shared-svc-accnt    $> kubectl create -f shared-svc-accnt.yml # Create a grant for service account shared-svc-accnt. Do this step as per your platform.

Create configMap object

I have put both postgresql.conf and pg_hba.conf under config directory. I have updated postgresql.conf as follows:

ssl = on ssl_cert_file = '/etc/postgresql-secrets-vol/server.crt' ssl_key_file = '/etc/postgresql-secrets-vol/server.key'

Note: the location '/etc/postgresql-config-vol' needs to be mounted while defining 'volumeMounts', which we will discuss later in the post.

Those three above listed are the main configuration items that need to have proper values in order to force Postgresql to support SSL/TLS. If you are using CA signed certificate, you also need to provide value for 'ssl_ca_file' and optionally 'ssl_crl_file'. Read Secure TCP/IP Connections with SSL for more details.
You also need to update the pg_hba.conf (HBA stands for host-based authentication) as necessary. pg_hba.conf is used to manage connection type, control access using a client IP address range, a database name, a user name, and the authentication method etc.

# Sample entries in pg_hba.conf # Trust local connection - no password required. local    all             all                                     trust # Only secured remote connection from given IP-Range accepted and password are encoded using MD5 #hostssl  all             all             < Cluster IP-Range >/< Prefix length >         md5 hostssl  all             all             10.96.0.0/16         md5

$> ls -l config/ -rw-------. 1 osboxes osboxes  4535 Sep 22 17:33 pg_hba.conf -rw-------. 1 osboxes osboxes 22781 Sep 23 03:03 postgresql.conf # Create configMap object $> kubectl create configmap postgresql-config --from-file=config/ -n shared-services configmap "postgresql-config" created # Review created object $> kubectl describe configMap/postgresql-config -n shared-services Name:         postgresql-config Namespace:    shared-services ...

Create secrets

I've created server.key and self signed certificate using OpenSSL. you can either do the same or have CA signed certificates. Here, we are not going to use the client certificate. Read section 18.9.3. Creating Certificates if you need help in creating certificates.

# Create MD5 hashed password to be used with postgresql $> POSTGRES_USER=postgres $> POSTGRES_PASSWORD=myp3qlpwD $> echo "md5$(echo -n $POSTGRES_PASSWORD$POSTGRES_USER | md5sum | cut -d ' ' -f1)" > secrets/postgresql-pwd.txt # Here are all files under secrets directory $> ls -la secrets/ -rw-rw-r--. 1 osboxes osboxes  13 Sep 22 23:42 postgresql-pwd.txt -rw-rw-r--. 1 osboxes osboxes 891 Sep 22 16:51 root.crt -rw-rw-r--. 1 osboxes osboxes 891 Sep 22 16:49 server.crt -r--------. 1 osboxes osboxes 887 Sep 22 16:43 server.key # Create secret postgresql-secrets $> kubectl create secret generic postgresql-secrets --from-file=secrets/ -n shared-services secret "postgresql-secrets" created # Verify $> kubectl describe secrets/postgresql-secrets -n shared-services Name:         postgresql-secrets Namespace:    shared-services Labels:        Annotations:   Type:  Opaque Data ==== server.key:          887 bytes postgresql-pwd.txt:  13 bytes root.crt:            891 bytes server.crt:          891 bytes

Note: As seen above, I have created MD5 hash using "md5<password>:<userid>". The reason, I added string "md5" in front of hashed string is that when Postgres sees "md5" as a prefix, it recognizes that the string is already hashed and does not try to hash again and stores as it is.

Create PersistentVolume (PV) and PersistentVolumeClaim (PVC) 

Let's go ahead and create PV and PVC. We will use 'Retain' as persistentVolumeReclaimPolicy, so that data can be retained even when Postgresql pod is destroyed and recreated.
Sample PV yaml file:

## shared-nfs-pv-postgresql.yml apiVersion: v1 kind: PersistentVolume metadata:   name: shared-nfs-pv-postgresql   namespace: shared-services spec:   capacity:     storage: 5Gi   accessModes:   - ReadWriteMany   nfs:     path: /var/postgresql/     server: 192.168.56.101   persistentVolumeReclaimPolicy: Retain

Sample PVC yaml file:

## shared-nfs-pvc-postgresql.yml apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: shared-nfs-pvc-postgresql   namespace: shared-services spec:   accessModes:     - ReadWriteMany   resources:     requests:       storage: 5Gi

PV and PVC creation and verification steps:

# Create persistentvolume $> kubectl create -f yaml/shared-nfs-pv-postgresql.yml persistentvolume "shared-nfs-pv-postgresql" created # Create persistentvolumeclaim $> kubectl create -f yaml/shared-nfs-pvc-postgresql.yml persistentvolumeclaim "shared-nfs-pvc-postgresql" created # Verify and make sure status of persistentvolumeclaim/shared-nfs-pvc-postgresql is Bound $> kubectl get pv,pvc -n shared-services NAME                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                       STORAGECLASS   REASON    AGE persistentvolume/shared-nfs-pv-postgresql   5Gi        RWX            Retain           Bound     shared-services/shared-nfs-pvc-postgresql                            32s NAME                                              STATUS    VOLUME                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE persistentvolumeclaim/shared-nfs-pvc-postgresql   Bound     shared-nfs-pv-postgresql   5Gi        RWX                           20s

Create deployment manifest file

Here is the one, I have put together. You can customize it further per your need.

--- # Service definition apiVersion: v1 kind: Service metadata:   name: sysg-postgres-svc   namespace: shared-services spec:   type: ClusterIP   ports:     - port: 5432       targetPort: 5432       protocol: TCP       name: tcp-5432   selector:       app: sysg-postgres-app --- # Deployment definition apiVersion: apps/v1beta2 kind: Deployment metadata:   name: sysg-postgres-dpl   namespace: shared-services spec:   selector:     matchLabels:       app: sysg-postgres-app   replicas: 1   template:     metadata:       labels:         app: sysg-postgres-app     spec:       serviceAccountName: shared-svc-accnt       securityContext:         runAsUser: 70         supplementalGroups: [999,1000]         fsGroup: 70       volumes:         - name: shared-nfs-pv-postgresql           persistentVolumeClaim:             claimName: shared-nfs-pvc-postgresql         - name: secret-vol           secret:             secretName: postgresql-secrets             defaultMode: 0640         - name: config-vol           configMap:             name: postgresql-config       containers:       - name: sysg-postgres-cnt         image: postgres:10.5-alpine         imagePullPolicy: IfNotPresent         args:           - -c           - hba_file=/etc/postgresql-config-vol/pg_hba.conf           - -c           - config_file=/etc/postgresql-config-vol/postgresql.conf         env:           - name: POSTGRES_USER             value: postgres           - name: PGUSER             value: postgres           - name: POSTGRES_DB             value: mmdb           - name: PGDATA             value: /var/lib/postgresql/data/pgdata           - name: POSTGRES_PASSWORD_FILE             value: /etc/postgresql-secrets-vol/postgresql-pwd.txt         ports:          - containerPort: 5432         volumeMounts:           - name: config-vol             mountPath: /etc/postgresql-config-vol           - mountPath: /var/lib/postgresql/data/pgdata             name: shared-nfs-pv-postgresql           - name: secret-vol             mountPath: /etc/postgresql-secrets-vol       nodeSelector:         kubernetes.io/hostname: centosddcwrk01

Deploy the Postgresql

Below steps show the creation of service and deployment as well as step to make sure that the Postgres is running with SSL enabled mode.

# Deploy $> kubectl apply -f yaml/postgres-deploy.yml service "sysg-postgres-svc" created deployment.apps "sysg-postgres-dpl" created # Verify $> kubectl get pods,svc -n shared-services NAME                                     READY     STATUS    RESTARTS   AGE pod/sysg-postgres-dpl-596754d5d4-mc8fm   1/1       Running   0          1h NAME                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE service/sysg-postgres-svc   ClusterIP   10.96.90.30           5432/TCP   1h # sh into the postgresql pod: $> kubectl exec -it sysg-postgres-dpl-596754d5d4-mc8fm /bin/sh -n shared-services / $ # Launch psql / $ psql -U postgres psql (10.5) Type "help" for help. # Verify SSL is enabled postgres=# SHOW ssl;  ssl -----  on (1 row) # Check the stored password. It should match the hashed value of "<password><user>" with "md5" prepended. postgres=#  select usename,passwd from pg_catalog.pg_shadow;  usename  |               passwd ----------+-------------------------------------  postgres | md5db59316e90b1afb5334a331081618af6 # Connect remotely. You need to provide password. $> kubectl exec -it sysg-postgres-dpl-596754d5d4-mc8fm -n shared-services -- psql "sslmode=require host=10.96.90.30 port=5432 dbname=mmdb" --username=postgres Password for user postgres: psql (10.5) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type "help" for help. (1 row)

Few key points

1) Using customized configuration file:
As you have seen above,  I have created configMap object postgresql-config and used it using option '-c config_file=/etc/postgresql-config-vol/postgresql.conf'. ConfigMap object postgresql-config is mapped to path '/etc/postgresql-config-vol' in volumeMounts definition.

containers: - name: sysg-postgres-cnt   imagePullPolicy: IfNotPresent   args:     - -c     - hba_file=/etc/postgresql-config-vol/pg_hba.conf     - -c     - config_file=/etc/postgresql-config-vol/postgresql.conf

volumeMounts:   - name: config-vol     mountPath: /etc/postgresql-config-vol

2) Creating environment variable from secrets:

env:   - name: POSTGRES_PASSWORD_FILE     value: /etc/postgresql-secrets-vol/postgresql-pwd.txt

And the secret is mapped to path /etc/postgresql-secrets-vol

volumeMounts:   - name: secret-vol     mountPath: /etc/postgresql-secrets-vol

3) PGDATA environment variable: 
The default value is '/var/lib/postgresql/data'. However, Postgres recommends "... if the data volume you're using is a fs mountpoint (like with GCE persistent disks), Postgres initdb recommends a subdirectory (for example /var/lib/postgresql/data/pgdata ) be created to contain the data.". Refer to https://hub.docker.com/_/postgres/

Here we assign /var/lib/postgresql/data/pgdata:

env:   - name: PGDATA     value: /var/lib/postgresql/data/pgdata

Troubleshooting

1) Make sure server.key, server.crt, and root.crt all have appropriate permissions that is 0400 (if owned by postgres process owner) or 0640 (if owned by root). If proper permissions is not applied, Postgresql will not start. and in log, you will see following FATAL message.

2018-09-22 18:26:22.391 UTC [1] FATAL:  private key file "/etc/postgresql-secrets-vol/server.key" has group or world access 2018-09-22 18:26:22.391 UTC [1] DETAIL: File must have permissions u=rw (0600) or less if owned by the database user, or permissions u=rw,g=r (0640) or less if owned by root. 2018-09-22 18:26:22.391 UTC [1] LOG:  database system is shut down

In order to apply proper permission in file level, you can use 'defaultMode', I'm using defaultMode: 0644 as shown below (fragment from postgres-deploy.yml)

- name: secret-vol   secret:     secretName: postgresql-secrets     defaultMode: 0640

2) Make sure to have right ownership - whether the files/directories are related to secret volume, config volume or persistence storage volume. Below is the error related PV path:

initdb: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted

In order to resolve the above issue, you need to use Kubernetes' provided securityContext options like 'runAsUser', 'fsGroup', 'supplementalGroups' and/or capabilities. SecurityContext can be defined in both pod level and container level. In my case, I've defined it in pod level as shown below (fragment from postgres-deploy.yml)

securityContext:   runAsUser: < specify your run as user >   fsGroup: < specify group >   supplementalGroups: [< comma delimited list of supplementalGroups >]

Read Configure a Security Context for a Pod or Container chapter from official Kubernetes site. I've also given some troubleshooting tips while using NFS type Persistent Volume and Claim in my previous blog How to Create, Troubleshoot and Use NFS type Persistent Storage Volume in Kubernetes

Below, I'm showing file permission per my configuration. Files are owned by root:postgres.

# Get running Kubernetes pod $> kubectl get pods -n shared-services NAME                                 READY     STATUS    RESTARTS   AGE sysg-postgres-dpl-596754d5d4-mc8fm   1/1       Running   0          12s # sh to running Kubernetes pod $> kubectl exec -it sysg-postgres-dpl-596754d5d4-mc8fm /bin/sh -n shared-services # Explore files / $ cd /etc/postgresql-secrets-vol/..data/ /etc/postgresql-secrets-vol/..2018_09_22_19_24_52.289379695 $ ls -la drwxr-sr-x    2 root     postgres       120 Sep 23 01:19 . drwxrwsrwt    3 root     postgres       160 Sep 23 01:19 .. -rw-r-----    1 root     postgres        13 Sep 23 01:19 postgresql-pwd.txt -rw-r-----    1 root     postgres       891 Sep 23 01:19 root.crt -rw-r-----    1 root     postgres       891 Sep 23 01:19 server.crt -rw-r-----    1 root     postgres       887 Sep 23 01:19 server.key

3) psql: FATAL:  no pg_hba.conf entry for host "10.0.2.15", user "postgres" ... This FATAL message usually appears when you are trying to establish connection to Postgres, but the way you are trying to  authenticate is not defined in pg_hba.conf. Either the source IP (from where the connection originates is out of range, security option is not supported. Check your pg_hba.conf file and make sure right entry has been added.

[Optional]  Creating custom Postgres Docker image with customized postgresql.conf

If you prefer to create custom Docker image with custom postgresql.conf rather than creating configMap and using '-c config_file' option, you can do so. Here is how:

Create Dockerfile:

FROM postgres:10.5-alpine COPY config/postgresql.conf /tmp/postgresql.conf COPY scripts/_updateConfig.sh /docker-entrypoint-initdb.d/ RUN chmod 755 /docker-entrypoint-initdb.d/_updateConfig.sh && chmod 644 /tmp/postgresql.conf

My custom postgresql.conf is located under config directory locally. It will be copied to /tmp when Docker image is created. _updateConfig.sh is located under scripts directory locally and copied to /docker-entrypoint-initdb.d/ in build time.

Create script file _updateConfig.sh as shown below. It assumes that default PGDATA value '/var/lib/postgresql/data' is being used.

#!/usr/bin/env bash cat /tmp/postgresql.conf > /var/lib/postgresql/data/postgresql.conf

Important: we can not directly copy the custom postgresql.conf into $PGDATA directory in build time because that directory does not exist yet.

Build the image:

Directory and files shown below are local:

$> ls -la postgresql/ drwxrwxr-x.  2 osboxes osboxes 4096 Sep 23 13:01 config -rwxr-xr--.  1 osboxes osboxes  227 Sep 22 19:14 Dockerfile drwxrwxr-x.  2 osboxes osboxes 4096 Sep 22 18:39 scripts drwxrwxr-x.  2 osboxes osboxes 4096 Sep 22 23:42 secrets drwxrwxr-x.  2 osboxes osboxes 4096 Sep 23 12:11 yaml $>cd postgresql # docker build -t <image tag> . # In my case I am using osboxes/postgres:10.5-sysg as image name and tag. $> docker build -t osboxes/postgres:10.5-sysg .

If you use custom Docker image built this way, you don't need to define configMap to use custom postgresql.conf.

How to Create, Troubleshoot and Use NFS type Persistent Storage Volume in Kubernetes

Whether you need to simply persists the data or share data among pods, one of the options is to use Network File System (NFS) type Persistent Volumes (PV).
However, you may encounter multiple issues and a lot of times error message(s) you see in the pod's log not detailed enough or even misleading. In this blog post, I'm going to show you step by step process (with real example) of creating PV, Persistence Volume Claims (PVC) and use them in a pod. We'll also discuss the possible issues and how to resolve them.

Prerequisites for this exercise:

1. Make sure you have working Kubernetes cluster where you can create resources as needed. 
2. Make sure you have a working Network File System (NFS) server and is accessible from all Kubernetes nodes in the Kubernetes cluster.

Process steps:

1) Allow Kubernetes pod/container to use NFS

1.1) Check, if selinux is enabled on your Kubernetes cluster nodes/hosts (where Kubernetes pod(s) will be created). If it is enabled, we need to make sure it lets container/pod to access remote NFS share.

$> sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28

1.2) If it is enabled, find out the value of 'virt_use_nfs'. You can use either 'getsebool' or 'semanage' utilities as shown below:

$> getsebool virt_use_nfs virt_use_nfs --> off or $> sudo semanage boolean -l | grep virt_use_nfs virt_use_nfs (off , off) Allow virt to use nfs

1.3) If value of 'virt_use_nfs' is 'off', make sure to enable it; otherwise, any attempt by Kubernetes pod to access NFS share may be denied and you may get '403 Forbidden error' from your application.You can use 'setsebool' tool to set value as '1' or 'on'

$> sudo setsebool -P virt_use_nfs 1 $> sudo semanage boolean -l | grep virt_use_nfs virt_use_nfs (on , on) Allow virt to use nfs

Note: -P option is to set the value permanently.

2) Create NFS share on NFS server

2.1) create  a directory on NFS server. My NFS server's IP is 192.168.56.101. Here I'm creating directory '/var/rabbitmq' on NFS server as a NFS share and assigning the ownership to 'osboxes:osboxes'. We'll discuss the ownership of the share and it's relationship to pod/container security context little later in the post.

# Create directory to be shared. sudo mkdir -p /var/rabbitmq # Change the ownership $> sudo chown osboxes:osboxes /var/rabbitmq

Important: The right ownership of the NFS share is crucial.

2.2) Add NFS share in /etc/exports file. Below, I'm adding all of my kubernetes nodes. Pods running on 192.168.56.101-103 will be able to access the NFS share. 'root_squash' option "squashes" the power of the remote root user to the lowest local user, preventing unauthorized alterations.

/var/rabbitmq/ 192.168.56.101(rw,sync,root_squash) /var/rabbitmq/ 192.168.56.102(rw,sync,root_squash) /var/rabbitmq/ 192.168.56.103(rw,sync,root_squash)

2.3) Export the NFS share.

sudo exportfs -a

3) Provisioning of PV and PVC

Let's create a PersistentVolume (PV), PersistentVolumeClaim (PVC) for RabbitMQ.
Note: it's important that the PVC and pod that uses it to be in the same namespace. You can create them all in default namespace. However, here I'm going to create a dedicated namespace for this purpose.

3.1) Create a new namespace or use existing one or default namespace.
Below yaml file (shared-services-ns.yml) defines a namespace object called 'shared-services':

apiVersion: v1 kind: Namespace metadata:    name: shared-services

To create the “shared-services” namespace, run the following command:

# Create a new namespace: $> kubectl create -f shared-services-ns.yml namespace "shared-services" created # Verify namespace is created successfully $> kubectl get namespaces shared-services NAME              STATUS    AGE shared-services   Active    36s

3.2) Create a new service account or use existing one or default:
If a service account is not set in the pod definition, the pod uses the default service account for the namespace. Here we are defining a new service account called 'shared-svc-accnt'. File: svcAccnt.yml

apiVersion: v1 kind: ServiceAccount metadata:    name: shared-svc-accnt    namespace: shared-services

To create a new service account 'shared-svc-accnt', run the following command:

# Create service account $> kubectl create -f svcAccnt.yml serviceaccount "shared-svc-accnt" created # Verify service account $> kubectl describe serviceaccount shared-svc-accnt -n shared-services Name:                shared-svc-accnt Namespace:           shared-services Labels:               Annotations:          Image pull secrets:   Mountable secrets:   shared-svc-accnt-token-mgk9w Tokens:              shared-svc-accnt-token-mgk9w Events:

3.3) Assign role/permission to service account:
Once, service account is created, make sure to provide necessary access permission to service account in the given namespace. Based on your Kubernetes platform, you may do it differently. Since, my Kubernetes is part of Docker Enterprise Edition (EE), I do it through Docker Universal Control Plane (UCP) as described in https://docs.docker.com/ee/ucp/authorization/grant-permissions/#kubernetes-grants. I'll assign 'restricted control' role to my service account 'shared-svc-accnt' in namespace 'shared-services'. If you are using MiniKube or other platform, you may want to refer to generic Kuberentes documents for RBAC and service account permission. Basically, you need to basically create the cluster role(s) and bind it to the service account. Here are some links to corresponding documentation. See https://v1-7.docs.kubernetes.io/docs/admin/authorization/rbac/#service-account-permissions and https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole

3.4) Define PV object in a yaml file (rabbitmq-nfs-pv.yml):

apiVersion: v1 kind: PersistentVolume metadata:   name: rabbitmq-nfs-pv   namespace: shared-services spec:   capacity:     storage: 5Gi   accessModes:   - ReadWriteMany   nfs:     path: /var/rabbitmq/     server: 192.168.56.101   persistentVolumeReclaimPolicy: Retain

Note: currently a PVcan have “Retain”, “Recycle”, or “Delete” reclaim policies. For dynamically provisioned PV, the default reclaim policy is “Delete”. Kubernetes supports following access modes:

• ReadWriteOnce – the volume can be mounted as read-write by a single node
• ReadOnlyMany – the volume can be mounted read-only by many nodes
• ReadWriteMany – the volume can be mounted as read-write by many nodes

To create a new PV 'rabbitmq-nfs-pv', run the following command:

# Create PV $> kubectl create -f rabbitmq-nfs-pv.yml persistentvolume "rabbitmq-nfs-pv" created # Verify PV $> kubectl describe pv rabbitmq-nfs-pv Name:            rabbitmq-nfs-pv Labels:           Annotations:      Finalizers:      [] StorageClass: Status:          Available Claim: Reclaim Policy:  Retain Access Modes:    RWX Capacity:        5Gi Node Affinity:    Message: Source:     Type:      NFS (an NFS mount that lasts the lifetime of a pod)     Server:    192.168.56.101     Path:      /var/rabbitmq/     ReadOnly:  false Events:

3.5) Define PVC object in a yaml file ( rabbitmq-nfs-pvc.yml):

apiVersion: v1 kind: PersistentVolumeClaim metadata:   name: rabbitmq-nfs-pvc   namespace: shared-services spec:   accessModes:     - ReadWriteMany   resources:     requests:       storage: 5Gi

Note: make sure to create PVC in the same namespace as your pod(s) that use it.

To create a new PVC 'rabbitmq-nfs-pvc', run the following command:

# Create PVC $> kubectl create -f rabbitmq-nfs-pvc.yml persistentvolumeclaim "rabbitmq-nfs-pvc" created # Verify PVC $> kubectl describe pvc rabbitmq-nfs-pvc -n shared-services Name:          rabbitmq-nfs-pvc Namespace:     shared-services StorageClass: Status:        Bound Volume:        rabbitmq-nfs-pv Labels:         Annotations:   pv.kubernetes.io/bind-completed=yes                pv.kubernetes.io/bound-by-controller=yes Finalizers:    [] Capacity:      5Gi Access Modes:  RWX Events:

Important: see the status above. It's "Bound" and it's bound to volume "rabbitqm-nfs-pv" that we created in previous step. If your PVC is not able to bind with PV, then it's a problem. It could be problem in defining the PV and PVC. Make sure your PV and PVC are of same storage class (if you are using one. For details refer to https://kubernetes.io/docs/concepts/storage/storage-classes/), and PV can fully satisfy the specification defined in PVC.


3.7) Now let's put together a simple yaml file that defines service and deployment objects for RabbitMQ (rabbitmq-nfs-pv-poc-depl.yml):

apiVersion: v1 kind: Service metadata:   name: rabbitmq-nfs-poc-svc   namespace: shared-services   labels:     app: rabbitmq-nfs-poc-svc spec:   type: NodePort   ports:   - name: http     port: 15672     targetPort: 15672   - name: amqp     protocol: TCP     port: 5672     targetPort: 5672   selector:     app: rabbitmq-app --- apiVersion: apps/v1beta2 # for versions prior to 1.9.0 kind: Deployment metadata:   name: rabbitmq-depl   namespace: shared-services spec:   selector:     matchLabels:       app: rabbitmq-app   replicas: 1   template:     metadata:       labels:         app: rabbitmq-app     spec:       serviceAccountName: shared-svc-accnt       securityContext:         runAsUser: 1000         supplementalGroups: [1000,65534]       containers:       - name: rabbitmq-cnt         image: rabbitmq         imagePullPolicy: IfNotPresent         #privileged: false         #securityContext:           #runAsUser: 1000         ports:         - containerPort: 15672           name: http-port           protocol: TCP         - containerPort: 5672           name: amqp           protocol: TCP         volumeMounts:           # 'name' must match the volume name below.           - name: rabbitmq-mnt             # Where to mount the volume.             mountPath: "/var/lib/rabbitmq/"       volumes:       - name: rabbitmq-mnt         persistentVolumeClaim:           claimName: rabbitmq-nfs-pvc

Note:
As seen in the rabbitmq-nfs-pv-poc-depl.yml above, I'm defining the security context in the pod level as:

securityContext:   runAsUser: 1000   supplementalGroups: [1000,65534]

Here runAsUser's value '1000' and supplementalGroups' value '1000' belong to user 'osboxes' and group 'osboxes'. gid '65534' belongs to group 'nfsnobody'.

$> id osboxes uid=1000(osboxes) gid=1000(osboxes) groups=1000(osboxes),10(wheel),983(docker) $> id nfsnobody uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)

My NFS share '/var/rabbitmq' is owned by 'osboxes:osboxes', so I'm specifying those values that belong to osboxes in the securityContext.

Security context can be defined both on pod level as well as container level. Security context defined in the pod level is applied to all containers in the pod. https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ has details about configuring security context for pod or container.


Following command creates rabbitmq deployment and service:

# Create objects  $> kubectl create -f rabbitmq-nfs-pv-poc-depl.yml service "rabbitmq-nfs-poc-svc" created deployment.apps "rabbitmq-depl" created # Get pods $> kubectl get pods -n shared-services NAME                            READY     STATUS    RESTARTS   AGE rabbitmq-depl-775496b9b-d85l7   1/1       Running   0          7s

Let's check the rabbitmq processes inside the container and files under '/var/rabbitmq' share on NFS server.

# Check process inside the container $> kubectl exec -it rabbitmq-depl-775496b9b-d85l7 /bin/bash -n shared-services $> ps -ef UID        PID  PPID  C STIME TTY          TIME CMD 1000         1     0  0 12:38 ?        00:00:00 /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server 1000       162     1  0 12:38 ?        00:00:00 /usr/lib/erlang/erts-9.3.3.2/bin/epmd -daemon 1000       321     1  5 12:38 ?        00:00:03 /usr/lib/erlang/erts-9.3.3.2/bin/beam.smp -W w - # Connect to NFS server  $> ssh osboxes@192.168.56.101 Last login: Sun Aug 26 14:48:19 2018 from centosddcclnt Make sure rabbitmq successfully created the file and review the file ownership $> cd /var/rabbitmq $> ls -la total 28 drwxr-xr-x.  5 osboxes osboxes   4096 Aug 26 13:40 . drwxr-xr-x. 25 root    root      4096 Aug 26 13:34 .. -rw-------.  1 osboxes nfsnobody   40 Aug 26 13:40 .bash_history drwxr-xr-x.  3 osboxes nfsnobody 4096 Aug 26 13:38 config -r--------.  1 osboxes nfsnobody   20 Aug 26 01:00 .erlang.cookie drwxr-xr-x.  4 osboxes nfsnobody 4096 Aug 26 13:38 mnesia drwxr-xr-x.  2 osboxes nfsnobody 4096 Aug 26 13:38 schema

4) Possible issues & troubleshooting

4.1) Pod remain in pending state and pod description shows 'mount failed: exit status 32' as shown below:

$> kubectl describe pod rabbitmq-shared-app -n shared-services Name:         rabbitmq-shared-app Namespace:    shared-services Node:         centosddcwrk01/192.168.56.103 Start Time:   Thu, 16 Aug 2018 17:03:19 +0100 Labels:       name=rabbitmq-shared-app Annotations:   Status:       Pending IP:   ...   ... ... Events:   Type     Reason                 Age   From                   Message   ----     ------                 ----  ----                   -------   ...   Warning  FailedMount            50s   kubelet, centosddcucp  MountVolume.SetUp failed for volume .... : mount failed: exit status 32

If you try to run the mount manually from inside the container, you may see following:

$> kubectl exec -it rabbitmq-depl-bd9689c8-7md48 /bin/bash -n shared-services root@rabbitmq-depl-bd9689c8-7md48:/# pwd / root@rabbitmq-depl-bd9689c8-7md48:/# mount -t nfs 192.168.56.101:/var/rabbitmq /tmp/test mount: wrong fs type, bad option, bad superblock on 192.168.56.101:/var/rabbitmq,        missing codepage or helper program, or other error        (for several filesystems (e.g. nfs, cifs) you might        need a /sbin/mount. helper program)        In some cases useful info is found in syslog - try        dmesg | tail or so.

In this case, review the '/etc/exports' file on NFS server.  This file controls which file systems are exported to remote hosts and specifies options. If your Kubernetes host/node is not listed
in this file with appropriate option(s), a pod running on that node will not be able to mount. Make sure to run the command 'sudo exportfs -a' once you have updated the /etc/exports. You can also try to manually mount from your host (instead of from within the container) in order to test if that host/node is authorized to mount. Refer to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/deployment_guide/s1-nfs-server-config-exports for details.


4.2) Pod fails to instantiate and you see 'chown: changing ownership of '/var/lib/rabbitmq': Operation not permitted' error in the log as shown below:

$> kubectl create -f rabbitmq-nfs-pv-poc-depl.yml service "rabbitmq-nfs-poc-svc" created deployment.apps "rabbitmq-depl" created $> kubectl get pods -n shared-services NAME                             READY     STATUS             RESTARTS   AGE rabbitmq-depl-5fff645d95-429vd   0/1       CrashLoopBackOff   1          14s $> kubectl logs rabbitmq-depl-5fff645d95-429vd -n shared-services chown: changing ownership of '/var/lib/rabbitmq': Operation not permitted

This means that the pod is able to mount successfully, however, it's not able to change the ownership of file/directory. The easiest way to resolve this issue is to have a common user that owns NFS share on NFS server and runAsUser of Kubernetes pod. For example, for this demo, I have used 'osboxes' user which owns the NFS share and also use this user's uid '1000' in the pod level security context.

$> ls -lZ /var/rabbitmq drwxr-xr-x. osboxes nfsnobody system_u:object_r:var_t:s0       ... $> id osboxes uid=1000(osboxes) gid=1000(osboxes) groups=1000(osboxes),10(wheel),983(docker)

In reality, it may not be that easy. You may not have access to remote NFS server or system administrator of NFS server is not willing to change the ownership of NFS share on NFS server. In this case (as a work-around), you can use 'root' as runAsUser like below in the container level:

securityContext:   runAsUser: 0

However, for this to work properly, the /etc/exports file on NFS server should not squash (use 'no_root_squash') the root. It should look something like this:

/var/rabbitmq/ 192.168.56.103(rw,sync,no_root_squash)

'no_root_squash' has it's own security consequences. See details here https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/deployment_guide/s1-nfs-server-config-exports

In summary, in order to grant pod's access to PVs you need to take considerations of:

• Finding the group ID and/or user ID assigned to the actual storage (on NFS server)
• SELinux considerations,
• Also making sure that the IDs allowed to access physical storage match the requirements of the particular pod.

The Group IDs, the user ID, and SELinux values can be defined in the pod's SecurityContext section. User IDs can also be defined to each container. So, in short you can use the following user, group and options to control and find the right combination:

• supplementalGroups
• fsGroup
• runAsUser
• seLinuxOptions

Hope, it helps you a little bit!

Note: yaml files used in this post can be downloaded from Github location: https://github.com/pppoudel/kube-pv-pvc-demo

Setting up CommandLine Environment for IBM® Bluemix® Container Service

Last week, I had gone through couple of steps to create an account (free - trial for 30 days) with IBM® Bluemix®, deploy my IBM Container (Docker based) and access my IBM Container running on Bluemix using Bluemix/CloudFoundry command line tools setup locally on my laptop. I have done it to prepare myself and also it's a kind of POC for upcoming project work. I've decided to share these steps so that other people in the same situation can benefit from it. Below are steps:

1. Make sure you have your IBM Container deployed and running on IBM Bluemix. If you don't have one follow the below sub steps:
• Create a free account with IBM Bluemix (https://console.ng.bluemix.net/)
• Once the account is created, you can create an IBM Container. See below quick steps:
• From left hand menu, click on "Containers"
• Then click on "Create Containers" link and follow the instruction.
  Note: you can select the container type from the available list or upload your own compatible container image. Below screen shot shows the container that I created:
  

  

  Running container

2. Now it's time to setup the command line tools on your local desktop. I have used the Fedora v24.x running as a virtual machine. 
• Download Bluemix_CLI_0.4.6_amd64.tar.gz from http://clis.ng.bluemix.net/ui/home.html and extract it to some directory:
  

  $>tar -xvzf ~/Downloads/Bluemix_CLI_0.4.6_amd64.tar.gz

• Among others files, you'll see  'install_bluemix_cli' executable file under /Bluemix_CLI
  

  $>sudo ./install_bluemix_cli

• Once it's installed, download CloudFoundry tool:
  

  $>sudo wget -O /etc/yum.repos.d/cloudfoundry-cli.repo https://packages.cloudfoundry.org/fedora/cloudfoundry-cli.repo

• Install CloudFoundry CLI:
  

  $>sudo yum install cf-cli ... Installed: cf-cli.x86_64 0:6.26.0-1

• Check the version:
  

  $>cf -v cf version 6.23.1+a70deb3.2017-01-13

• Install the IBM Bluemix Container Service plug-in (cf ic) to use the native Docker CLI. More details about it can be found here.
  

  $>cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64

• Verify the plugins:
  

  $>cf plugins Listing Installed Plugins... OK Plugin Name Version Command Name Command Help IBM-Containers 0.8.964 ic IBM Containers plug-in

3. It's time to login to CloudFoundry and run you container command to manage your container.
   
   • Login to Bluemix/CloudFoundry:
     

     $>cf login -a https://api.ng.bluemix.net Email> purna.poudel@gmail.com Password> Authenticating... OK

   • Login to Container:
     

     $> cf ic login Deleting old configuration file... Retrieving client certificates for IBM Containers... Storing client certificates in /home/osboxes/.ice/certs/... Storing client certificates in /home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846... OK The client certificates were retrieved. Checking local Docker configuration... OK Authenticating with the IBM Containers registry host registry.ng.bluemix.net... OK You are authenticated with the IBM Containers registry. ...

4. It's time to manage your Container(s) from your desktop using command line
   
   • Let's check our running Container process(es)
     

     $> cf ic ps  CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d476a406-ba9 registry.ng.bluemix.net/ibmliberty:webProfile7 "" 2 days ago Running 169.46.21.44:9080->9080/tcp, 169.46.21.44:9443->9443/tcp sysgLibertyCont

   • Let's inspect the running Container
     

     $> cf ic inspect d476a406-ba9 [ { "BluemixApp": null, "BluemixServices": null, "Config": { "AttachStderr": false, "AttachStdin": false, "AttachStdout": false, "Cmd": [], "Dns": "", "Env": [ "logging_password=", "space_id=7b9e7846-0ec8-41da-83e6-209a02e1b14a", "logstash_target=logmet.opvis.bluemix.net:9091", "metrics_target=logmet.opvis.bluemix.net:9095" ], "Hostname": "instance-002eacfa", "Image": "registry.ng.bluemix.net/ibmliberty:webProfile7", "ImageArchitecture": "amd64", "Labels": { "doc.url": "/docs/images/docker_image_ibmliberty/ibmliberty_starter.html" }, "Memory": 256, "MemorySwap": "", ....

List of IBM Bluemix Container Service plug-in (cf ic) commands for managing containers are available at https://console.ng.bluemix.net/docs/containers/container_cli_reference_cfic.html


Looks like you're really into Docker, see my other related blog posts below:

• Using Docker Secrets with IBM WebSphere Liberty Profile Application Server
• Make your container deployment portable
• Experience sharing - Dock Datacenter
• Setting up CLI environment for IBM Bluemix (this post)
• Quick start with IBM Datapower Gateway Docker Edition