SysGenius-e-Log: January 2017

Quick start with IBM Datapower Gateway for Docker

Lately, IBM has done great job by providing development version of Datapower in different flavors. It's really useful for doing POC stuff as well as testing application in development environment. Today, I had a chance to play with Docker edition of Datapower Gateway. Within few minutes, I was able to pull the Datapower Docker image from the Docker hub, create the Datapower Docker container, run, and play with Datapower. IBM Datapower Gateway for Docker web page (https://hub.docker.com/r/ibmcom/datapower/) has good information to start with. However, this image contains unconfigured Datapower Gateway, and you'll not be able to access either Datapower Web Management Console or establish SSH connection to Datapower even after the container is running. For that either you have to access the Datapower in interactive mode and enable the admin-state of Web Management and SSH, add configuration through Docker build or use (externalised) configuration file during run. Below we'll discuss two (out of three) options.
1) [optional] Create 'config' and 'local' directories on your host machine's file system from where you are going to execute the docker run command later in the step #2. For example, I have created following directory structure under: /opt/docker_dev/datapowerbuild/

 $ ls -rl datapowerbuild

              total 8

              drwxrwxr-x. 2 osboxes osboxes 4096 Jan 31 20:19 local

              drwxrwxr-x. 2 osboxes osboxes 4096 Jan 31 20:19 config

We are going to put docker configuration file(s) in those directories. Docker is able to access these external configuration. It is very powerful concept and explained as [data] volume in Docker documentation. Read details here.
2) [optional] Create auto-startup.cfg configuration file and put under 'config' directory created in step #1, so that Docker Web Management and SSH admin state is enabled when Docker container runs.

top; co

                

                ssh

                

                web-mgmt

                  admin enabled

                  port 9090

                exit

Above script is taken from https://github.com/ibm-datapower/datapower-tutorials/blob/master/using-datapower-in-ibm-container-service/src/auto-startup.cfg
3) Execute docker run command. It's assumed that you already have Docker installed, configured and running on your host machine. If not follow Docker installation steps here:

Following command is based on IBM's instruction available at https://hub.docker.com/r/ibmcom/datapower/

 $ docker run -it \ 

              -v $PWD/config:/drouter/config \

              -v $PWD/local:/drouter/local \

              -e DATAPOWER_ACCEPT_LICENSE=true \

              -e DATAPOWER_INTERACTIVE=true \

              -p 9090:9090 \

              -p 2222:22 \

              ibmcom/datapower

Note: make sure your machine is connected to internet otherwise it will not be able to pull the Datapower Docker image from Docker hub.
Some of the last line you'll see before logon prompt are:

 20170201T014408.003Z
              [0x00350014][mgmt][notice] ssh(SSH Service): tid(111): Operational
              state up

              20170201T014408.007Z [0x8100003b][mgmt][notice] domain(default):
              Domain configured successfully.

              20170201T014408.073Z [0x00350014][mgmt][notice]
              web-mgmt(WebNGUI-Settings): tid(303): Operational state up

4) logon to console using default userid 'admin' and password 'admin'.
5) Launch browser and type 'https://<host machine ip>:9090' to access Web Management console.

6) Create ssh connection to your host on port 2222 to access Datapower.
Note: If you have not performed optional steps #1 and #2, your web-mgmt and 'ssh' connection will not be available. Perform the optional steps #7.

7) [optional] Connect to Datapower console interactively and turn the Global Configuration mode.

 configure terminal

              web-mgmt

              admin-state enabled

              exit

              ssh

              exit

See the screen shots below for better clarity.

Enable web-mgmt admin-state

Enable ssh admin-state

Showing the status

Looks like you're really into Docker, see my other related blog posts below:

Scripting Ideas

Important: This page will be continually updated as I find new work-around or ideas while working on scripts (any script - shell, windows, or others).

Running dos2unix in batch mode:

One of my teammates today seemed pretty frustrated while trying to run 'dos2unix' command in batch mode. His script (see below) was almost doing the thing, however instead of updating the file, the content were displayed on screen (stdout).

His script (with issue) that sent output to stdout:

find . -type f -name "*.sh" |
              xargs -i dos2unix {};

Here is the corrected script, which correctly updates each file under current directory by converting end of line from Windows format to Unix format.

find . -type f -name "*.sh" |
              xargs -i dos2unix {} {};

As you have noticed, the only thing missing was the last set of '{}', which basically tells dos2unix to use the same filename for output as per input. Below is example using 'exec' instead of 'xargs' to achieve the same.

find . -type f -name ".sh" -exec
              dos2unix {} {} \;

Command reference links: find, xargs,dos2unix

Using variable in SED:

 file="myfile.txt";
              

              replaceme="iamnew"; 

              sed 's/iamold/'"${replaceMe}"'/g' < $file > $file".new";

 file="myfile.txt"; 

              replaceme="iamnew";
              

              sed "s/iamold/${replaceme}/g" < $file > $file".new";

Note: in above example, any occurrence of 'iamold' in 'myfile.txt' will be replace by 'iamnew' and written in 'myfile.txt.new'. Important thing here is the variable $replaceme should be in double quote. Below variant does not work. The variable '$replaceme' will not be expanded.

 file="myfile.txt"; 

              replaceme="iamnew";
              

              sed 's/iamold/${replaceme}/g' < $file > $file".new";

Command reference links: sed

Finding which process owns/listens on which port

Here, I'm finding which process/process ID is listening on port 9080. Here is how I can find out.
Note: the following has been tested on CentOS Linux.

1) Using 'netstat -lnp'

 

$> netstat -lnp | grep 9080

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java





# using sudo:

$> sudo netstat -lnp | grep 9080

tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java





# Or find all ports in use by certain process/PID

$> sudo netstat -lnp | grep java

tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java

tcp6       0      0 :::10010                :::*                    LISTEN      3840/java

tcp6       0      0 :::9443                 :::*                    LISTEN      3840/java

tcp6       0      0 127.0.0.1:57576         :::*                    LISTEN      3840/java



#by PID

$> sudo netstat -lnp | grep 3840

tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java

tcp6       0      0 :::10010                :::*                    LISTEN      3840/java

tcp6       0      0 :::9443                 :::*                    LISTEN      3840/java

tcp6       0      0 127.0.0.1:57576         :::*                    LISTEN      3840/java

2) Using 'lsof -i :<port>'

 

$> lsof -i :9080

COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

java    3840 osboxes  339u  IPv6  40626      0t0  TCP *:glrpc (LISTEN)

3) Using ss -ntlp

 

$>  ss -ntlp | grep 9080

LISTEN     0      128         :::9080                    :::*                   users:(("java",pid=3840,fd=339))

Retrieving Certificate and Updating kestore file.

Following file show example of retrieving Google certificate from www.google.com and adding it to local key.jks file. script file: retrieveAndUpdateCert.sh

 
     #! /bin/bash

# Remote host to retrieve certificate from

RHOST=www.google.com

# Remote port

RPORT=443

# key store file path

KS_FILEPATH=/opt/secrets/key.jks

# Certificate Alias

CERT_ALIAS=googlecert



# Retrieve the certificate and put in temporary file '/tmp/cert.crt' in this case.

# Refer to https://www.openssl.org/docs/man1.0.2/apps/openssl.html for openssl command details.

true | openssl s_client -connect ${RHOST}:${RPORT} 2>/dev/null | openssl x509 -in /dev/stdin > /tmp/cert.crt

# Install certificate using keytool

# keytool comes with Java.

# Refer to https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html for keytool command details.

keytool -import -file /tmp/cert.crt -alias ${CERT_ALIAS} -keystore ${KS_FILEPATH} -storepass $1

# View certs in the keystore:

keytool -list -v -keystore ${KS_FILEPATH} -storepass $1

Run file as:

 
     
           $> ./retrieveAndUpdateCert.sh <Your keystore password>

AWK numerical processing tricks

1. If you have number with 1000 separator (,) like 84,959, AWK fails to process the number correctly unless you remove the separator (,) from input. for example:

 
     $> echo "84,959|34,600" | awk 'BEGIN{FS=OFS="|";}{print $1/1000,$2/1000}'

     0.084|0.034

As seen from the above result, AWK only took the input values prefixed by comma. Fix is simple, just remove the "," from input value. The following line gives the correct result:

 
     $> echo "84,959|34,600" | awk 'BEGIN{FS=OFS="|";}{gsub(",","",$1);gsub(",","",$2); print $1/1000,$2/1000}'

     84.959|34.6

2. If you get some weird result while doing AWK numeric comparison, make sure the value is presented as number not string literal. For example:

  
     $> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2);if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}'

     99 is greater than 100

As seen from above, the result is not correct/expected. It is because, the value of num above is '99 ', i.e. there is a space character after 99, and AWK processes this as string comparison. Simple fix is to multiply the value by 1 or add 0 before doing numeric comparison.

 
     $> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2)*1;if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}'

99 is less than 100

 
$> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2)+0;if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}'

99 is less than 100

AWK printing from specific column/field to the end

In the following example, matrix.csv (comma delimited file) data is piped to awk which processes one row at a time (excluding first header row), first column is a time in milliseconds, so it converts into displayable date and prints, but rest of the columns (starting from 2nd column) require no processing, so it prints as it is.

 
       cat matrix.csv | awk 'BEGIN{FS=OFS=","}{if(NR > 1) {print strftime("%c", ($1 + 500)/1000), substr($0, index($0,$2))}}'

Using comma as a delimiter in for loop

By default 'for loop' expects input delimited by space (or tab or newline) character. However, if you need to use ',' (comma), one of the easiest way is to override Internal Field Separator (IFS) value. However, make sure to set it back to the original value. See the script below, it opens a set of firewall ports delimited by comma ','. Before the for loop, we set IFS="," and after the for loop, we set value back to space " ".

 
#!/bin/sh

tcp_ports="179,443,80,2375,2376,2377,2380,4001,4443,4789,6443,6444,7001,7946,8080,10250,12376-12387"

udp_ports="4789,7946"



openFW() {

  IFS=",";

 for _port in $1; do

  echo "Opening ${_port}/$2";

  sudo firewall-cmd --permanent --zone=public --add-port=${_port}/$2;

 done

 IFS=" ";

}



openFW "${tcp_ports}" tcp;

openFW "${udp_ports}" udp;



# Recycle firewall

sudo firewall-cmd --reload

Pages