SysGenius-e-Log: Scripting Ideas

Important: This page will be continually updated as I find new work-around or ideas while working on scripts (any script - shell, windows, or others).

Running dos2unix in batch mode:

One of my teammates today seemed pretty frustrated while trying to run 'dos2unix' command in batch mode. His script (see below) was almost doing the thing, however instead of updating the file, the content were displayed on screen (stdout).

His script (with issue) that sent output to stdout:

find . -type f -name "*.sh" |
              xargs -i dos2unix {};

Here is the corrected script, which correctly updates each file under current directory by converting end of line from Windows format to Unix format.

find . -type f -name "*.sh" |
              xargs -i dos2unix {} {};

As you have noticed, the only thing missing was the last set of '{}', which basically tells dos2unix to use the same filename for output as per input. Below is example using 'exec' instead of 'xargs' to achieve the same.

find . -type f -name ".sh" -exec
              dos2unix {} {} \;

Command reference links: find, xargs,dos2unix

Using variable in SED:

 file="myfile.txt";
              

              replaceme="iamnew"; 

              sed 's/iamold/'"${replaceMe}"'/g' < $file > $file".new";

 file="myfile.txt"; 

              replaceme="iamnew";
              

              sed "s/iamold/${replaceme}/g" < $file > $file".new";

Note: in above example, any occurrence of 'iamold' in 'myfile.txt' will be replace by 'iamnew' and written in 'myfile.txt.new'. Important thing here is the variable $replaceme should be in double quote. Below variant does not work. The variable '$replaceme' will not be expanded.

 file="myfile.txt"; 

              replaceme="iamnew";
              

              sed 's/iamold/${replaceme}/g' < $file > $file".new";

Command reference links: sed

Finding which process owns/listens on which port

Here, I'm finding which process/process ID is listening on port 9080. Here is how I can find out.
Note: the following has been tested on CentOS Linux.

1) Using 'netstat -lnp'

 

$> netstat -lnp | grep 9080

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java





# using sudo:

$> sudo netstat -lnp | grep 9080

tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java





# Or find all ports in use by certain process/PID

$> sudo netstat -lnp | grep java

tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java

tcp6       0      0 :::10010                :::*                    LISTEN      3840/java

tcp6       0      0 :::9443                 :::*                    LISTEN      3840/java

tcp6       0      0 127.0.0.1:57576         :::*                    LISTEN      3840/java



#by PID

$> sudo netstat -lnp | grep 3840

tcp6       0      0 :::9080                 :::*                    LISTEN      3840/java

tcp6       0      0 :::10010                :::*                    LISTEN      3840/java

tcp6       0      0 :::9443                 :::*                    LISTEN      3840/java

tcp6       0      0 127.0.0.1:57576         :::*                    LISTEN      3840/java

2) Using 'lsof -i :<port>'

 

$> lsof -i :9080

COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

java    3840 osboxes  339u  IPv6  40626      0t0  TCP *:glrpc (LISTEN)

3) Using ss -ntlp

 

$>  ss -ntlp | grep 9080

LISTEN     0      128         :::9080                    :::*                   users:(("java",pid=3840,fd=339))

Retrieving Certificate and Updating kestore file.

Following file show example of retrieving Google certificate from www.google.com and adding it to local key.jks file. script file: retrieveAndUpdateCert.sh

 
     #! /bin/bash

# Remote host to retrieve certificate from

RHOST=www.google.com

# Remote port

RPORT=443

# key store file path

KS_FILEPATH=/opt/secrets/key.jks

# Certificate Alias

CERT_ALIAS=googlecert



# Retrieve the certificate and put in temporary file '/tmp/cert.crt' in this case.

# Refer to https://www.openssl.org/docs/man1.0.2/apps/openssl.html for openssl command details.

true | openssl s_client -connect ${RHOST}:${RPORT} 2>/dev/null | openssl x509 -in /dev/stdin > /tmp/cert.crt

# Install certificate using keytool

# keytool comes with Java.

# Refer to https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html for keytool command details.

keytool -import -file /tmp/cert.crt -alias ${CERT_ALIAS} -keystore ${KS_FILEPATH} -storepass $1

# View certs in the keystore:

keytool -list -v -keystore ${KS_FILEPATH} -storepass $1

Run file as:

 
     
           $> ./retrieveAndUpdateCert.sh <Your keystore password>

AWK numerical processing tricks

1. If you have number with 1000 separator (,) like 84,959, AWK fails to process the number correctly unless you remove the separator (,) from input. for example:

 
     $> echo "84,959|34,600" | awk 'BEGIN{FS=OFS="|";}{print $1/1000,$2/1000}'

     0.084|0.034

As seen from the above result, AWK only took the input values prefixed by comma. Fix is simple, just remove the "," from input value. The following line gives the correct result:

 
     $> echo "84,959|34,600" | awk 'BEGIN{FS=OFS="|";}{gsub(",","",$1);gsub(",","",$2); print $1/1000,$2/1000}'

     84.959|34.6

2. If you get some weird result while doing AWK numeric comparison, make sure the value is presented as number not string literal. For example:

  
     $> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2);if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}'

     99 is greater than 100

As seen from above, the result is not correct/expected. It is because, the value of num above is '99 ', i.e. there is a space character after 99, and AWK processes this as string comparison. Simple fix is to multiply the value by 1 or add 0 before doing numeric comparison.

 
     $> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2)*1;if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}'

99 is less than 100

 
$> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2)+0;if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}'

99 is less than 100

AWK printing from specific column/field to the end

In the following example, matrix.csv (comma delimited file) data is piped to awk which processes one row at a time (excluding first header row), first column is a time in milliseconds, so it converts into displayable date and prints, but rest of the columns (starting from 2nd column) require no processing, so it prints as it is.

 
       cat matrix.csv | awk 'BEGIN{FS=OFS=","}{if(NR > 1) {print strftime("%c", ($1 + 500)/1000), substr($0, index($0,$2))}}'

Using comma as a delimiter in for loop

By default 'for loop' expects input delimited by space (or tab or newline) character. However, if you need to use ',' (comma), one of the easiest way is to override Internal Field Separator (IFS) value. However, make sure to set it back to the original value. See the script below, it opens a set of firewall ports delimited by comma ','. Before the for loop, we set IFS="," and after the for loop, we set value back to space " ".

 
#!/bin/sh

tcp_ports="179,443,80,2375,2376,2377,2380,4001,4443,4789,6443,6444,7001,7946,8080,10250,12376-12387"

udp_ports="4789,7946"



openFW() {

  IFS=",";

 for _port in $1; do

  echo "Opening ${_port}/$2";

  sudo firewall-cmd --permanent --zone=public --add-port=${_port}/$2;

 done

 IFS=" ";

}



openFW "${tcp_ports}" tcp;

openFW "${udp_ports}" udp;



# Recycle firewall

sudo firewall-cmd --reload

SysGenius-e-Log

Pages

Scripting Ideas