SSL is disabled in the default Postgresql configuration and I had to struggle a little bit while making Postgresql on Kubernetes with SSL/TLS support to work. After few research and trials, I was able to resolve the issues and here I'm sharing what I have done to make it work for me.
High Level Steps:
- Customize postgresql.conf (to add/edit SSL/TLS configuration) and create configMap object. This way, we don't need to rebuild the Postgres to apply custom postgresql.conf , because ConfigMap allows us you to decouple configuration artifacts from image content.
- Create secret type objects for server.key, server.crt, root.crt, ca.crt, and password file.
- Define and use NFS type PersistentVolume (PV) and PersistentVolumeClaim (PVC)
- Use securityContext to resolve permission issues.
- Use '-c config_file=<config-volume-location>/postgresql.conf' to override the default postgresql.conf
Note: all files used in this post can be cloned/downloaded from GitHub
https://github.com/pppoudel/postgresql-with-ssl-on-kubernetes.git
Let's get started
In the example, I'm using namespace called 'shared-services' and service account called 'shared-svc-accnt'. You can create your own namespace and service account or use the 'default'. In anyways, I have listed here necessary steps and yaml files can be downloaded from github.
Create namespace and service account
Create configMap object
I have put both postgresql.conf and pg_hba.conf under config directory. I have updated postgresql.conf as follows:
Note: the location '/etc/postgresql-config-vol' needs to be mounted while defining 'volumeMounts', which we will discuss later in the post.
Those three above listed are the main configuration items that need to have proper values in order to force Postgresql to support SSL/TLS. If you are using CA signed certificate, you also need to provide value for 'ssl_ca_file' and optionally 'ssl_crl_file'. Read
Secure TCP/IP Connections with SSL for more details.
You also need to update the pg_hba.conf (HBA stands for host-based authentication) as necessary. pg_hba.conf is used to manage connection type, control access using a client IP address range, a database name, a user name, and the authentication method etc.
Create secrets
I've created server.key and self signed certificate using OpenSSL. you can either do the same or have CA signed certificates. Here, we are not going to use the client certificate. Read section
18.9.3. Creating Certificates if you need help in creating certificates.
Note: As seen above, I have created MD5 hash using "md5<password>:<userid>". The reason, I added string "md5" in front of hashed string is that when Postgres sees "md5" as a prefix, it recognizes that the string is already hashed and does not try to hash again and stores as it is.
Create PersistentVolume (PV) and PersistentVolumeClaim (PVC)
Let's go ahead and create PV and PVC. We will use 'Retain' as persistentVolumeReclaimPolicy, so that data can be retained even when Postgresql pod is destroyed and recreated.
Sample PV yaml file:
Sample PVC yaml file:
PV and PVC creation and verification steps:
Create deployment manifest file
Here is the one, I have put together. You can customize it further per your need.
Deploy the Postgresql
Below steps show the creation of service and deployment as well as step to make sure that the Postgres is running with SSL enabled mode.
Few key points
1) Using customized configuration file:
As you have seen above, I have created configMap object postgresql-config and used it using option '-c config_file=/etc/postgresql-config-vol/postgresql.conf'. ConfigMap object postgresql-config is mapped to path '/etc/postgresql-config-vol' in volumeMounts definition.
2) Creating environment variable from secrets:
And the secret is mapped to path /etc/postgresql-secrets-vol
3) PGDATA environment variable:
The default value is '/var/lib/postgresql/data'. However, Postgres recommends "...
if the data volume you're using is a fs mountpoint (like with GCE persistent disks), Postgres initdb recommends a subdirectory (for example /var/lib/postgresql/data/pgdata ) be created to contain the data.". Refer to
https://hub.docker.com/_/postgres/
Here we assign /var/lib/postgresql/data/pgdata:
Troubleshooting
1) Make sure server.key, server.crt, and root.crt all have appropriate permissions that is 0400 (if owned by postgres process owner) or 0640 (if owned by root). If proper permissions is not applied, Postgresql will not start. and in log, you will see following FATAL message.
In order to apply proper permission in file level, you can use 'defaultMode', I'm using defaultMode: 0644 as shown below (fragment from postgres-deploy.yml)
2) Make sure to have right ownership - whether the files/directories are related to secret volume, config volume or persistence storage volume. Below is the error related PV path:
In order to resolve the above issue, you need to use Kubernetes' provided securityContext options like 'runAsUser', 'fsGroup', 'supplementalGroups' and/or capabilities. SecurityContext can be defined in both pod level and container level. In my case, I've defined it in pod level as shown below (fragment from postgres-deploy.yml)
Read
Configure a Security Context for a Pod or Container chapter from official Kubernetes site. I've also given some troubleshooting tips while using NFS type Persistent Volume and Claim in my previous blog
How to Create, Troubleshoot and Use NFS type Persistent Storage Volume in Kubernetes
Below, I'm showing file permission per my configuration. Files are owned by root:postgres.
3)
psql: FATAL: no pg_hba.conf entry for host "10.0.2.15", user "postgres" ... This FATAL message usually appears when you are trying to establish connection to Postgres, but the way you are trying to authenticate is not defined in pg_hba.conf. Either the source IP (from where the connection originates is out of range, security option is not supported. Check your pg_hba.conf file and make sure right entry has been added.
[Optional] Creating custom Postgres Docker image with customized postgresql.conf
If you prefer to create custom Docker image with custom postgresql.conf rather than creating configMap and using '-c config_file' option, you can do so. Here is how:
Create Dockerfile:
My custom postgresql.conf is located under config directory locally. It will be copied to /tmp when Docker image is created. _updateConfig.sh is located under scripts directory locally and copied to /docker-entrypoint-initdb.d/ in build time.
Create script file _updateConfig.sh as shown below. It assumes that default PGDATA value '/var/lib/postgresql/data' is being used.
Important: we can not directly copy the custom postgresql.conf into $PGDATA directory in build time because that directory does not exist yet.
Build the image:
Directory and files shown below are local:
If you use custom Docker image built this way, you don't need to define configMap to use custom postgresql.conf.
