SysGenius-e-Log: Cloud

Showing posts with label Cloud. Show all posts

Making Your Container Deployment Portable

This post is a follow-up and extension of my previous post "Setting up CommandLine Environment for IBM® Bluemix® Container Service".
In this post, I'm further exploring the way of working with containers whether it's locally deployed native Docker or container created with IBM® Bluemix® Container Service. I'm going to show few basic scripting ideas, so that the same docker-compose.yml and other related files can be used no matter whether you are dealing with locally deployed native Docker container(s) or IBM container(s).
One step ahead, here we will be working with multiple containers and employing Docker Compose. I have used basic steps for this exercise from Bluemix tutorial (https://console.ng.bluemix.net/docs/containers/container_compose_intro.html#container_compose_config) and added few steps and logic to do basic automation and make it portable, so that it can be executed the same way independent of environment.

Pre-requisites for this exercise:

(native) Docker installed and running locally(may be on your laptop/desktop)
CommandLine environment setup for IBM® Bluemix® Container Service. See previous post "Setting up CommandLine Environment for IBM® Bluemix® Container Service".
Docker Compose version 1.6.0 or later installed on your laptop/desktop. See installation instruction here
lets-chat and mongo images are available in your local and Bluemix private registry.

As part of this exercise, we will be putting together 'docker-compose.yml' with replaceable variable(s), '.env' file for environment variables with default values, property file 'depl.properties' for environment specific properties, and script file 'autoDeploy.sh' with basic logic that can be executed to manage both native Docker as well as IBM Bluemix containers. We will be creating and linking following two containers.

lets-chat (basic chat application)
mongo (database to store data)

At the end, we'll also look into few possible issues that you may encounter.

Let's start with creating docker-compose.yml. Compose simplifies the definition and execution of multi-container Docker applications. See Docker Compose documentation for details.
Below is our simple docker-compose.yml which defines two containers <<lets-chat>> and <<lc-mongo>>. As you see a variable has been assigned as a value to 'image' attribute. It is in order to make it portable between native Docker and container to be deployed on IBM Bluemix as the image registry will be different. You can assign variable this way to any attribute as it's value which will be replaced by value of corresponding environment variable.


     lets-chat:

   image: ${LETS_CHAT_IMAGE}

   container_name: lets-chat

   ports:

      - "8080:8080"

   links:

      - lc-mongo:mongo

lc-mongo:

   image: ${MONGODB_IMAGE}

   container_name: lc-mongo

   expose:

      - "27017"

Now, let's see, where we can define environment variables. Docker supports either defining it through command shell as 'export VAR=VALUE' or defining them in '.env' file (Note: If you are deploying your service using 'docker stack deploy --compose-file docker-compose.yml <service-name>' instead of 'docker-compose up ...' values in the docker-compose.yml may not be replaced by corresponding environment values defined in .env file. See https://github.com/moby/moby/issues/29133). Environment variable defined through 'export VAR=VALUE' takes precedence. See more detail on variable substitution and declaring default environment variables in file.

Below is our '.env' file:


     # COMPOSE_HTTP_TIMEOUT default value is 60 seconds.

                                        COMPOSE_HTTP_TIMEOUT=120

                                        MONGODB_IMAGE=mongo

                                        LETS_CHAT_IMAGE=lets-chat

Usually, it is a best practice to define default variables with 'DEV/Development' environment specific values in '.env' file and have mechanism to override those values for higher environment(s). It helps to boost developers' productivity. In order to follow the above mentioned principle, I've defined my local native Docker container specific environment variables in my '.env' file and will have separate property file to define environment variables and their values for other environments (Bluemix in my case for this post).
Below is my property file 'depl.properties' which defines property and their Bluemix specific values:


     # Define property as _VARIABLE_NAME=VALUE where prefix will identify the environment like 'bluemix', 'native' etc.

# Note: variable with default value can be placed directly into '.env' file.

bluemix_API_ENDPOINT=https://api.ng.bluemix.net

bluemix_DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443

bluemix_DOCKER_CERT_PATH=/home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846-0ec8-41da-83e6-209a02e1b14a

bluemix_DOCKER_TLS_VERIFY=1

bluemix_REGISTRY=registry.ng.bluemix.net

bluemix_NAMESPACE=sysg

bluemix_ORG_NAME=porg

bluemix_SPACE_NAME=ptest

# reference property without '<prefix>_' as script sets environment variable without <prefix>_. See autoDeploy.sh

bluemix_MONGODB_IMAGE=${REGISTRY}/${NAMESPACE}/mongo

bluemix_LETS_CHAT_IMAGE=${REGISTRY}/${NAMESPACE}/lets-chat

Now, we need to have a script (logic) that can set appropriate environment variables based on the target environment.
Below is sample (autoDeploy.sh) script:


     #!/bin/sh

# Author: Purna Poudel

# Created on: 23 February, 2017



# project directory

pDir='.'

#property file

propFile=depl.properties


function usage {
   printf "Usage: $0 \n";
   printf "Options:\n";
   printf "%10s-t|--conttype:-u|--username:-p|--password\n";

}


OPTS=$(getopt -o t:u:p: -l conttype:,username:,password:, -- "$0" "$@");

if [ $? != 0 ]; then
   echo "Unrecognised command line option encountered!";
   usage;
   exit 1;

fi

eval set -- "$OPTS";


while true; do
   case "$1" in
      -t|--conttype)
         conttypearg="$1";
         conttype=$2;
         shift 2;;
      -u|--username)
         usernamearg="$1";
         username=$2;
         shift 2;;
      -p|--password)
         passwordarg="$1";
         password=$2;
         shift 2;;
    *)
         shift;
         break;;
   esac

done

if [[ $conttype == "" ]]; then
   echo "Valid non empty value for '--conttype' or '-t' is required."
   usage;
   exit 1

fi


# Reads each line if 'prefix' matches the supplied value of $conttype.

# Also excludes all commented (starting with #) lines, all empty lines and not related properties.



for _line in `cat "${pDir}/${propFile}" |  egrep '(^'$conttype'|^all)' |grep -v -e'#' | grep -v -e'^$'`; do
   echo "Reading line: $_line from source file: ${pDir}/${propFile}";
   # Assign property name to variable '_key'
   # Also remove the prefix, which is supposed to be the identifier for particular environment

   # in depl.properties file.
   # the final 'xargs' removes the leading and trailing blank spaces.
   _key=$(echo $_line | awk 'BEGIN {FS="="}{print $1}' | awk 'BEGIN {FS=OFS="_"}{print substr($0, index($0,$2))}' | xargs);
   # Assign property value to variable '_value'
   _value=`eval echo $_line | cut -d '=' -f2`;
   # Also declare shell variable and export to use as environment variable,
   declare $_key=$(echo $_value | xargs);
   echo "Setting environment variable: ${_key}=${!_key}";
   export ${_key}=${!_key};

done


if [[ $conttype == "bluemix" ]]; then
   # First log into CloudFoundry
   # cf login [-a API_URL] [-u USERNAME] [-p PASSWORD] [-o ORG] [-s SPACE]
   cf login -a ${API_ENDPOINT} -u ${username} -p ${password} -o ${ORG_NAME} -s ${SPACE_NAME};
   retSts=$?;
   if [ $retSts -ne 0 ]; then
      echo "Login to CloudFoundry failed with return code: "$retSts;
      exit $retSts;
   fi
   # then log into the IBM Container
   cf ic login
   retSts=$?;
   if [ $retSts -ne 0 ]; then
      echo "Login to IBM Container failed with return code: $retSts;"
      exit $retSts;
   fi

fi

# Stop and remove if container are running.

docker-compose ps | grep "Up";

retSts=$?;

if [ $retSts -eq 0 ]; then
   echo "Stopping existing docker-compose container...";
   docker-compose stop;
   sleep 5;

fi

docker-compose ps -q | grep "[a-z0-9]"

retSts=$?;

if [ $retSts -eq 0 ]; then
   echo "Removing existing docker-compose container...";
   docker-compose rm -f;
   sleep 5;

fi

# execute docker-compose

docker-compose up -d;

sleep 20;

# Make sure container built and running

docker-compose ps;

Now, it's time to test the logic above.

First, let's execute the script locally against native Docker.


     $> ./autoDeploy.sh -t native

lc-mongo    /entrypoint.sh mongod            Up      27017/tcp

lets-chat   /bin/sh -c (sleep 60; npm  ...   Up      5222/tcp, 0.0.0.0:8080->8080/tcp

Stopping existing docker-compose container...

Stopping lets-chat ... done

Stopping lc-mongo ... done

4afc9bc67f80fe0876fa2e5ce42af4616dbc64444c1c58128d0e63bf6007b55f

48beb1bb7423e103bfcdd4fc0ea8aa5e1ae766fcea70cf14a58df87a66e43f59

Removing existing docker-compose container...

Going to remove lets-chat, lc-mongo

Removing lets-chat ... done

Removing lc-mongo ... done

Creating lc-mongo

Creating lets-chat

  Name                 Command               State                Ports

-------------------------------------------------------------------------------------

lc-mongo    /entrypoint.sh mongod            Up      27017/tcp

lets-chat   /bin/sh -c (sleep 60; npm  ...   Up      5222/tcp, 0.0.0.0:8080->8080/tcp

As per script execution logic, it first identifies if any container instance of 'lc-mongo' and 'lets-chat', if so, it stops and removes the existing container then creates new one from existing images and starts and checks if they are running successfully. Since '-t native' option passed through command line, it didn't set any environment variable, but Docker Compose used the default environment variables defined in '.env' file.

It is time to test the same against IBM Bluemix Container Service. See below:


     $> ./autoDeploy.sh -t bluemix -u abc.def@xyz.com -p xxxxxxxxx

Reading line: bluemix_API_ENDPOINT=https://api.ng.bluemix.net from source file: ./depl.properties

Setting environment variable: API_ENDPOINT=https://api.ng.bluemix.net

Reading line: bluemix_DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443 from source file: ./depl.properties

Setting environment variable: DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443

Reading line: bluemix_DOCKER_CERT_PATH=/home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846-0ec8-41da-83e6-209a02e1b14a from source file: ./depl.properties

Setting environment variable: DOCKER_CERT_PATH=/home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846-0ec8-41da-83e6-209a02e1b14a

Reading line: bluemix_DOCKER_TLS_VERIFY=1 from source file: ./depl.properties

Setting environment variable: DOCKER_TLS_VERIFY=1

Reading line: bluemix_REGISTRY=registry.ng.bluemix.net from source file: ./depl.properties

Setting environment variable: REGISTRY=registry.ng.bluemix.net

Reading line: bluemix_NAMESPACE=sysg from source file: ./depl.properties

Setting environment variable: NAMESPACE=sysg

Reading line: bluemix_ORG_NAME=porg from source file: ./depl.properties

Setting environment variable: ORG_NAME=porg

Reading line: bluemix_SPACE_NAME=ptest from source file: ./depl.properties

Setting environment variable: SPACE_NAME=ptest

Reading line: bluemix_MONGODB_IMAGE=${REGISTRY}/${NAMESPACE}/mongo from source file: ./depl.properties

Setting environment variable: MONGODB_IMAGE=registry.ng.bluemix.net/sysg/mongo

Reading line: bluemix_LETS_CHAT_IMAGE=${REGISTRY}/${NAMESPACE}/lets-chat from source file: ./depl.properties

Setting environment variable: LETS_CHAT_IMAGE=registry.ng.bluemix.net/sysg/lets-chat

API endpoint: https://api.ng.bluemix.net

Authenticating...

OK



Targeted org porg



Targeted space ptest







API endpoint:   https://api.ng.bluemix.net (API version: 2.54.0)

User:           purna.poudel@gmail.com

Org:            porg

Space:          ptest

Deleting old configuration file...

Retrieving client certificates for IBM Containers...

Storing client certificates in /home/osboxes/.ice/certs/...



Storing client certificates in /home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846-0ec8-41da-83e6-209a02e1b14a...



OK

The client certificates were retrieved.



Checking local Docker configuration...

OK



Authenticating with the IBM Containers registry host registry.ng.bluemix.net...

OK

You are authenticated with the IBM Containers registry.

Your organization's private Bluemix registry: registry.ng.bluemix.net/sysg



You can choose from two ways to use the Docker CLI with IBM Containers:





Option 1: This option allows you to use 'cf ic' for managing containers on IBM Containers while still using the Docker CLI directly to manage your local Docker host.

        Use this Cloud Foundry IBM Containers plug-in without affecting the local Docker environment:





        Example Usage:

        cf ic ps

        cf ic images



Option 2: Use the Docker CLI directly. In this shell, override the local Docker environment by setting these variables to connect to IBM Containers. Copy and paste the following commands:

        Note: Only some Docker commands are supported with this option. Run cf ic help to see which commands are supported.

        export DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443

        export DOCKER_CERT_PATH=/home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846-0ec8-41da-83e6-209a02e1b14a

        export DOCKER_TLS_VERIFY=1



        Example Usage:

        docker ps

        docker images



lc-mongo              Up xxx.xx.0.xx:27017->27017/tcp

lets-chat             Up xxx.xx.0.xx:8080->8080/tcp

Stopping existing docker-compose container...

Stopping lets-chat ... done

Stopping lc-mongo ... done

ea11eda5-9ebc-45df-beb0-80f2ba8c44e7

1996dc00-d4a6-4ecf-9309-62c986781b88

Removing existing docker-compose container...

Going to remove lets-chat, c-mongo

Removing lets-chat ... done

Removing lc-mongo ... done

Creating lc-mongo

Creating lets-chat

  Name     Command   State              Ports

---------------------------------------------------------

lc-mongo              Up xxx.xx.0.xx:27017->27017/tcp

lets-chat             Up xxx.xx.0.xx:8080->8080/tcp

As you have noticed, we passed options '-t bluemix -u abc.def@xyz.com -p xxxxxxxxx' while executing the autoDeploy.sh. This enforced script to read properties from 'depl.properties' file and set corresponding environment variables specific for Bluemix. Everything else including docker-compose.yml and .env file remain unchanged.

Note: IPs, username and password masked.
In terms of defining properties specific to any environment, in this post, I'm just showing the case for two environments - local native Docker and IBM Bluemix Container Service environment, however,
if you have more environments, you can
define corresponding properties with appropriate prefix, for example:
dev_NAMSPACE=
tst_NAMESPACE=
qa_NAMESPACE=
prd_NAMESPCE=
And while running the build pass relevant container type option like '-t|--conttype dev|tst|qa|prd' then the script should set environment variable appropriately.

Note: You may need to update the logic in the autoDeploy.sh as per your requirement.

There are few other important aspect to remember while trying to make your code/script portable among native Docker and IBM Bluemix Container Services. Few of them are listed below:

Currently IBM Bluemix Container Service only supports Docker Compose version 1 of the docker-compose.yml file format. Refer https://docs.docker.com/compose/compose-file/compose-file-v1/ for detail.
IBM Bluemix Container Service may not support all Docker or Docker Compose commands or it has other commands that are not found in native Docker. Meaning in certain situation, you may still need to use the 'cf ic' commands instead of native docker command to perform task specific to IBM Bluemix Container Service. See the Supported Docker commands for IBM Bluemix Container Service plug-in (cf ic). The best way to find what native Docker commands are supported within IBM Bluemix or what 'cf ic' commands are available, just run the 'cf ic --help' and you'll see the list. The commands with '(Docker)' at the end are supported Docker commands.

Finally, let's talk about the possible issue(s) that you may encounter.


     Error response from daemon: 

400 The plain HTTP request was sent to HTTPS port

400 Bad Request

The plain HTTP request was sent to HTTPS port

nginx

/tmp/_MEI3n6jq4/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html

The above mentioned error was encountered while sending build context to IBM Bluemix Container Service. It was because the 'DOCKER_TLS_VERIFY' was set with empty value. You may encounter this error in any case when you are trying to establish secure connection, but any one of the following environment variables is not set correctly:
DOCKER_HOST
DOCKER_CERT_PATH
DOCKER_TLS_VERIFY

2)


     ERROR: for lets-chat  HTTPSConnectionPool(host='containers-api.ng.bluemix.net', port=8443): Read timed out. (read timeout=60)

ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.

If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

You may encounter the above mentioned error while executing 'docker-compse up' when request times out. The default read timeout is 60 sec. You can override this value by either defining it in '.env' file or as environment variable. e.g. 'export COMPOSE_HTTP_TIMEOUT=120'. Refer to https://docs.docker.com/compose/reference/envvars/ for all available environment variables.

That's it for this post. Try and let me know. You can find/get/download all files from GitLab here: https://gitlab.com/pppoudel/public_shared/tree/master/container_autodeploy

Looks like you're really into Docker, see my other related blog posts below:

Setting up CommandLine Environment for IBM® Bluemix® Container Service

Last week, I had gone through couple of steps to create an account (free - trial for 30 days) with IBM® Bluemix®, deploy my IBM Container (Docker based) and access my IBM Container running on Bluemix using Bluemix/CloudFoundry command line tools setup locally on my laptop. I have done it to prepare myself and also it's a kind of POC for upcoming project work. I've decided to share these steps so that other people in the same situation can benefit from it. Below are steps:

Make sure you have your IBM Container deployed and running on IBM Bluemix. If you don't have one follow the below sub steps:

Create a free account with IBM Bluemix (https://console.ng.bluemix.net/)
Once the account is created, you can create an IBM Container. See below quick steps:

From left hand menu, click on "Containers"
Then click on "Create Containers" link and follow the instruction.
Note: you can select the container type from the available list or upload your own compatible container image. Below screen shot shows the container that I created:

Running container

Now it's time to setup the command line tools on your local desktop. I have used the Fedora v24.x running as a virtual machine.

Download Bluemix_CLI_0.4.6_amd64.tar.gz from http://clis.ng.bluemix.net/ui/home.html and extract it to some directory:

$>tar -xvzf ~/Downloads/Bluemix_CLI_0.4.6_amd64.tar.gz
Among others files, you'll see 'install_bluemix_cli' executable file under /Bluemix_CLI

$>sudo ./install_bluemix_cli
Once it's installed, download CloudFoundry tool:

$>sudo wget -O /etc/yum.repos.d/cloudfoundry-cli.repo https://packages.cloudfoundry.org/fedora/cloudfoundry-cli.repo
Install CloudFoundry CLI:

$>sudo yum install cf-cli ... Installed: cf-cli.x86_64 0:6.26.0-1
Check the version:

$>cf -v cf version 6.23.1+a70deb3.2017-01-13
Install the IBM Bluemix Container Service plug-in (cf ic) to use the native Docker CLI. More details about it can be found here.

$>cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64

Verify the plugins:


$>cf plugins

Listing Installed Plugins...

OK



Plugin Name      Version   Command Name   Command Help

IBM-Containers   0.8.964   ic             IBM Containers plug-in

It's time to login to CloudFoundry and run you container command to manage your container.

Login to Bluemix/CloudFoundry:

$>cf login -a https://api.ng.bluemix.net Email> purna.poudel@gmail.com Password> Authenticating... OK


$> cf ic login

Deleting old configuration file...

Retrieving client certificates for IBM Containers...

Storing client certificates in /home/osboxes/.ice/certs/...



Storing client certificates in /home/osboxes/.ice/certs/containers-api.ng.bluemix.net/7b9e7846...



OK

The client certificates were retrieved.



Checking local Docker configuration...

OK



Authenticating with the IBM Containers registry host registry.ng.bluemix.net...

OK

You are authenticated with the IBM Containers registry.

...

It's time to manage your Container(s) from your desktop using command line

Let's check our running Container process(es)


$> cf ic ps

CONTAINER ID        IMAGE                                            COMMAND             CREATED             STATUS              PORTS                                                      NAMES

d476a406-ba9        registry.ng.bluemix.net/ibmliberty:webProfile7   ""                  2 days ago          Running             169.46.21.44:9080->9080/tcp, 169.46.21.44:9443->9443/tcp   sysgLibertyCont

Let's inspect the running Container


$> cf ic inspect d476a406-ba9

[

    {

        "BluemixApp": null,

        "BluemixServices": null,

        "Config": {

            "AttachStderr": false,

            "AttachStdin": false,

            "AttachStdout": false,

            "Cmd": [],

            "Dns": "",

            "Env": [

                "logging_password=",

                "space_id=7b9e7846-0ec8-41da-83e6-209a02e1b14a",

                "logstash_target=logmet.opvis.bluemix.net:9091",

                "metrics_target=logmet.opvis.bluemix.net:9095"

            ],

            "Hostname": "instance-002eacfa",

            "Image": "registry.ng.bluemix.net/ibmliberty:webProfile7",

            "ImageArchitecture": "amd64",

            "Labels": {

                "doc.url": "/docs/images/docker_image_ibmliberty/ibmliberty_starter.html"

            },

            "Memory": 256,

            "MemorySwap": "",

            ....

List of IBM Bluemix Container Service plug-in (cf ic) commands for managing containers are available at https://console.ng.bluemix.net/docs/containers/container_cli_reference_cfic.html

Looks like you're really into Docker, see my other related blog posts below:

Pages

Making Your Container Deployment Portable

Setting up CommandLine Environment for IBM® Bluemix® Container Service