Scripting Ideas

Important: This page will be continually updated as I find new work-around or ideas while working on scripts (any script - shell, windows, or others).

Running dos2unix in batch mode:

One of my teammates today seemed pretty frustrated while trying to run 'dos2unix' command in batch mode. His script (see below) was almost doing the thing, however instead of updating the file, the content were displayed on screen (stdout).

His script (with issue) that sent output to stdout:

find . -type f -name "*.sh" | xargs -i dos2unix {};

Here is the corrected script, which correctly updates each file under current directory by converting end of line from Windows format to Unix format.

find . -type f -name "*.sh" | xargs -i dos2unix {} {};

As you have noticed, the only thing missing was the last set of '{}', which basically tells dos2unix to use the same filename for output as per input. Below is example using 'exec' instead of 'xargs' to achieve the same.

find . -type f -name ".sh" -exec dos2unix {} {} \;

Command reference links: find, xargs,dos2unix

Using variable in SED:

file="myfile.txt"; replaceme="iamnew"; sed 's/iamold/'"${replaceMe}"'/g' < $file > $file".new";

OR

file="myfile.txt"; replaceme="iamnew"; sed "s/iamold/${replaceme}/g" < $file > $file".new";

Note: in above example, any occurrence of 'iamold' in 'myfile.txt' will be replace by 'iamnew' and written in 'myfile.txt.new'. Important thing here is the variable $replaceme should be in double quote. Below variant does not work. The variable '$replaceme' will not be expanded.

file="myfile.txt"; replaceme="iamnew"; sed 's/iamold/${replaceme}/g' < $file > $file".new";

Command reference links: sed

Finding which process owns/listens on which port

Here, I'm finding which process/process ID is listening on port 9080. Here is how I can find out.
Note: the following has been tested on CentOS Linux.

1) Using 'netstat -lnp'

$> netstat -lnp | grep 9080 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp6 0 0 :::9080 :::* LISTEN 3840/java # using sudo: $> sudo netstat -lnp | grep 9080 tcp6 0 0 :::9080 :::* LISTEN 3840/java # Or find all ports in use by certain process/PID $> sudo netstat -lnp | grep java tcp6 0 0 :::9080 :::* LISTEN 3840/java tcp6 0 0 :::10010 :::* LISTEN 3840/java tcp6 0 0 :::9443 :::* LISTEN 3840/java tcp6 0 0 127.0.0.1:57576 :::* LISTEN 3840/java #by PID $> sudo netstat -lnp | grep 3840 tcp6 0 0 :::9080 :::* LISTEN 3840/java tcp6 0 0 :::10010 :::* LISTEN 3840/java tcp6 0 0 :::9443 :::* LISTEN 3840/java tcp6 0 0 127.0.0.1:57576 :::* LISTEN 3840/java

2) Using 'lsof -i :<port>'

$> lsof -i :9080 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 3840 osboxes 339u IPv6 40626 0t0 TCP *:glrpc (LISTEN)

3) Using ss -ntlp

$> ss -ntlp | grep 9080 LISTEN 0 128 :::9080 :::* users:(("java",pid=3840,fd=339))

Retrieving Certificate and Updating kestore file.

Following file show example of retrieving Google certificate from www.google.com and adding it to local key.jks file. script file: retrieveAndUpdateCert.sh

#! /bin/bash # Remote host to retrieve certificate from RHOST=www.google.com # Remote port RPORT=443 # key store file path KS_FILEPATH=/opt/secrets/key.jks # Certificate Alias CERT_ALIAS=googlecert # Retrieve the certificate and put in temporary file '/tmp/cert.crt' in this case. # Refer to https://www.openssl.org/docs/man1.0.2/apps/openssl.html for openssl command details. true | openssl s_client -connect ${RHOST}:${RPORT} 2>/dev/null | openssl x509 -in /dev/stdin > /tmp/cert.crt # Install certificate using keytool # keytool comes with Java. # Refer to https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html for keytool command details. keytool -import -file /tmp/cert.crt -alias ${CERT_ALIAS} -keystore ${KS_FILEPATH} -storepass $1 # View certs in the keystore: keytool -list -v -keystore ${KS_FILEPATH} -storepass $1

Run file as:

$> ./retrieveAndUpdateCert.sh <Your keystore password>

AWK numerical processing tricks

1. If you have number with 1000 separator (,) like 84,959, AWK fails to process the number correctly unless you remove the separator (,) from input. for example:

$> echo "84,959|34,600" | awk 'BEGIN{FS=OFS="|";}{print $1/1000,$2/1000}' 0.084|0.034

As seen from the above result, AWK only took the input values prefixed by comma. Fix is simple, just remove the "," from input value. The following line gives the correct result:

$> echo "84,959|34,600" | awk 'BEGIN{FS=OFS="|";}{gsub(",","",$1);gsub(",","",$2); print $1/1000,$2/1000}'
84.959|34.6


2. If you get some weird result while doing AWK numeric comparison, make sure the value is presented as number not string literal. For example:

$> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2);if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}' 99 is greater than 100

As seen from above, the result is not correct/expected. It is because, the value of num above is '99 ', i.e. there is a space character after 99, and AWK processes this as string comparison. Simple fix is to multiply the value by 1 or add 0 before doing numeric comparison.

$> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2)*1;if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}' 99 is less than 100

or

$> echo "Is 99 ( ninety nine) higher than 100?" | awk 'BEGIN{FS="(";}{num=substr($1,4,2)+0;if(num >= 100){ print num" is greater than 100"}else{print num" is less than 100"}}' 99 is less than 100

AWK printing from specific column/field to the end

In the following example, matrix.csv (comma delimited file) data is piped to awk which processes one row at a time (excluding first header row), first column is a time in milliseconds, so it converts into displayable date and prints, but rest of the columns (starting from 2nd column) require no processing, so it prints as it is.

cat matrix.csv | awk 'BEGIN{FS=OFS=","}{if(NR > 1) {print strftime("%c", ($1 + 500)/1000), substr($0, index($0,$2))}}'

Using comma as a delimiter in for loop

By default 'for loop' expects input delimited by space (or tab or newline) character. However, if you need to use ',' (comma), one of the easiest way is to override Internal Field Separator (IFS) value. However, make sure to set it back to the original value. See the script below, it opens a set of firewall ports delimited by comma ','. Before the for loop, we set IFS="," and after the for loop, we set value back to space " ".

#!/bin/sh tcp_ports="179,443,80,2375,2376,2377,2380,4001,4443,4789,6443,6444,7001,7946,8080,10250,12376-12387" udp_ports="4789,7946" openFW() {   IFS=",";  for _port in $1; do   echo "Opening ${_port}/$2";   sudo firewall-cmd --permanent --zone=public --add-port=${_port}/$2;  done  IFS=" "; } openFW "${tcp_ports}" tcp; openFW "${udp_ports}" udp; # Recycle firewall sudo firewall-cmd --reload

Reset Oracle Directory Manager's Password

Have you ever had situation that you needed to execute a command for  Oracle Directory Server which required root/Directory Manager's password and the password you had just did not work? I encountered one today and had to scramble to find a solution to reset it. 'pwdhash' tool that comes with Oracle Directory Server rescued me. Here is what I did:

1. Before resetting the password, you may want to try few of your guesses. Here is how you do it. Get the actual root/Directory Manager's password from dse.ldif file. It's with attribute 'nsslapd-rootpw:' something like: nsslapd-rootpw: {SSHA256}WYChc/pNA34fD8RKo//ReBCsGstkz0Ux54gfsMaruXhMP89tAnMtd
2. Then compare each of your guess with the encrypted password from dse.ldif using 'pwdhash'. It has option to compare '-c'. Below is how you do it. If password matches, you'll get message "password ok." otherwise "password does not match." is displayed.
   

   ./pwdhash -D <instance-location> -c "<encrypted-password>" <your-guess-password> # Actual example from my ODS instance $>cd /opt/ods/dsee7/bin $>./pwdhash -D /opt/ods/dsee7/instances/dsInst2 -c "{SSHA256}WYChc/pNA34fD8RKo//ReBCsGstkz0Ux54gfsMaruXhMP89tAnMtd" myPassw0rd ./pwdhash: password does not match.

3. If none of your guess matches then it's time to reset the password hard way. Here is how to do it:
   

   # Stop your Oracle Directory Instance $>cd /opt/ods/dsee7/bin $>./dsadm stop /opt/ods/dsee7/instances/dsInst2 Directory Server instance '/opt/ods/dsee7/instances/dsInst2' stopped # Generate the encrypted password $>./pwdhash -D /opt/ods/dsee7/instances/dsInst2 -s SSHA256 myPassw0rd {SSHA256}qOjAyposbx1LzM/LB4vk1ZKS2yNs2Oh0yDjo66GIjnMpIVMJMhi6fw==

4. Take the generated encrypted password from step #3 and replace the value of attribute 'nsslapd-rootpw:' in dse.ldif file and save it.
5. Restart the Oracle Directory Instance.
   

   # Start your Oracle Directory Instance $>cd /opt/ods/dsee7/bin $>./dsadm start /opt/ods/dsee7/instances/dsInst2 Directory Server instance '/opt/ods/dsee7/instances/dsInst2' started: pid=2982

That's it, password reset is done in hard way!!!

However in future, if you just want to change the root/Directory Manager's password, you can use the 'dsconf' command with 'set-server-prop' option. Below is more detail:

# Put new password in a temporary file. $>echo "_0d3mG4_" > /tmp/odspwd.txt # Now run the 'dsconf' command. You need to provide current password for Directory Manager when it prompts $>./dsconf set-server-prop -h localhost -p 1489 root-pwd-file:/tmp/odspwd.txt Enter "cn=Directory Manager" password:

Experience Sharing - TOGAF® Part I and Part II Certification in Two Weeks through Self Study

I passed the combined TOGAF® 9.1 Part I  and Part II exams on May 24^th. I did it through self study and wanted to share my experience so that it would be useful for others. Timeline wise, I spent around 2 weeks - started studying on May 11^th by purchasing official TOGAF® self-study package from the Open Group and gave exams on May 24^th here in Toronto with satisfactory results - i.e. I passed Part I with 87% and Part II with 85%.
In terms of preparation almost 60% of my time, I spent on foundation (Part I) study as I realized early that in order to really crack the Part II exam, I had to have the good knowledge of TOGAF® foundation. Having said so, however, I did not wait until the end to prepare for the Part II exam. After few days of Part I study, I kind of accommodated Part II as well, after every iteration of part I, which actually helped me to better understand the TOGAF® concept as Part II is based on knowledge, TOGAF® way of thinking, analysing and also some related experience as well. Below some details:

Registration:

I registered for TOGAF® 9 Combined Part I and II (Exam Number: OG0093; Exam details: http://certification.opengroup.org/examinations/TOGAF/TOGAF9-combined) through Prometric on May 11.
Note: the combined exam costs  495 USD. Also, note that total cost for combined exams is little bit less than giving them separately. Here is Prometric site for registration: https://www.prometric.com/en-us/clients/opengroup/Pages/landing.aspx

Preparation:

• Purchased TOGAF® 9 CERTIFICATION SELF-STUDY PACK, 3RD EDITION (SKU: B097; cost 59.90 USD) PDF version from the Open Group web site.
• Started studying with Pocket guide (G117p.pdf) included in the self-study package. It provided general understanding of TOGAF®. If you start with official TOGAF® 9.1 documentation or detailed study guide, you may easily get lost or be bored. So, my suggestion is "start light" and for that purpose pocket guide seems to be the best.
• Reviewed the Reference Cards included in the self-study package. Reference cards really helped me to understand the TOGAF® concept and methodology visually. Once I reviewed all 4 reference cards, I created my own versions of reference cards to make sure that I understood them well. Note: There are 4 reference cards N111, N112, N113, and N114 included in the self-study package.
• Once I completed the Pocket book and Reference cards, I did first iteration of Part I practice test consisting 40 questions and remembered doing fairly well. I reviewed the explanation for those questions which I answered incorrectly. That's when I also started looking into the official TOGAF® 9.1 documentation and reviewed the materials in detail related to each question.
• After doing Part I practice test and also taking notes from official TOGAF® 9.1 documentation, I tried the first 8 questions from Part II practice test. Again reviewed all answers and referred back to either official TOGAF® 9.1 documentation or study guides included in the package.
• Following YouTube tutorials also helped me to shape my TOGAF® concept:
• https://www.youtube.com/playlist?list=PL2FF5FCBA1AF9B8FA (10 parts tutorial). Very well explained in detail.
• https://www.youtube.com/playlist?list=PLM9BDLm3rDjf3sU_Uq_hmSltM7jj1dII0 (9 parts tutorial). Also very well explained in high level.
• Before doing the 2^nd set of 40 questions included in the self-study package for Part I test question, I did some tests from public sites. especially 3 set of 40 questions from http://theopenarch.com/81-tests/72-TOGAF®-9-exam-tests.html . I found, these questions relatively harder than the questions included in the official self-study package. I remembered doing not very well, so I had to refer back to official document, and study guide and go deep and detail and take notes.
• On/Around the 6^th day, I felt somehow comfortable with the overall TOGAF® document structure, and had clear high level TOGAF® concept so I decided to proceed with the Part I and Part II  preparations kind of in parallel, i.e. at any point in time I could be doing Part I or Part II tests and referring back to TOGAF® 9.1 documentation for detail.
•  Self-Study package contains bonus questions for both Part I and Part II exams. I also did tests from the following sites:

• http://theopenarch.com/81-tests/72-TOGAF®-9-exam-tests.html (has both Part I and Part II test questions)
• https://setandbma.wordpress.com/2011/05/16/TOGAF-preparation-aid-for-part-2/ helpful questions to understand the TOGAF® artifacts and viewpoints.
• http://wenku.baidu.com/view/c9d8c7b0f524ccbff12184d2.html for Part II test questions.

• I did two iterations of practice tests and reviewed the answers. I did detail study from official TOGAF® 9.1 document for any questions that I answered wrong during each iteration. I did 3^rd iteration only to those questions that were marked wrong in the 2^nd iteration.
• My study style:
• Each day I started with the review of previous day's learning and any important notes.
• On the last day before the test, I reviewed all my personal notes.

In summary (based on my experience),  Part I exam, requires well versed TOGAF® knowledge, so, make sure you know all the TOGAF® terminologies, and methodologies. TOGAF® Part II, however in addition to TOGAF® knowledge, also requires very TOGAF® way of thinking and analysing each scenario.By the way, scenarios in the Part II exams are fairly long and it is easy to loose the track, so, make sure to note (in memory or on provided pad) what' the ask and concerns are. Most of the choices for each scenario  are equally long and pretty close to each other (in terms of meaning), so try to think in TOGAF® way to select the right one and avoid the distractor. Even though, Part II is an open book exam, avoid using attached reference document in the beginning. If you are not confident with your selection, mark it and move forward. Once you answer all your questions, then you can go back to marked questions and open the PDF document provided as reference to fine tune your answer. This way, you'll save time and also won't leave behind any question unanswered because of lack of time. 

That's it. All the best if any one of you are taking TOGAF® exam any time soon!

SMF Explained with Real Examples

     As part of an Automated Environment Management initiative, recently, I had to work with Solaris Service Management Facility (SMF) extensively. In order to make the implementation right, I had to look several different documentations, blogs, consult with colleagues, learn through error and trial methods. The main intention of this blog is - kind of 'give back to community', what I have learned.
Let's get started. Since, you are visiting this page, I assume, you are already familiar with the SMF concept and looking to resolve specific issue or digging some specific information in SMF domain. If you are new to SMF, you can learn the basic of SMF from Oracle site here.

Why SMF?
      In Unix world, traditionally we used to work with init script as a start-up mechanism. SMF may have been evolved from legacy init, but it is much more flexible, robust, and very well crafted. SMF helps to make environment management easier, uniform and align with security practices as well. 
     In this posts, I'll explain SMF through a working example. First, I have to warn you that some of the steps here possibly overdone to show the concept in detail or underdone to minimize the size of the blog.
     Anyway, here is what we are going to do. We have an environment as described below:

• Solaris 11 (64 bit VirtualBox guest VM running on 64 bit Windows 10 Host) and Oracle Directory Server 11g installed under /opt/ods/dsee7

And as part of this learning process, I'll show you the following:

1. Create no-login user 'dirservd', this user will own the Oracle Directory Server (ODS) instances to be created.
2. Create 'odsadm' user that will operate/manage SMF services for ODS instances to be created.
3. Create two Oracle Directory Server (ODS) instances 'odsInst1' and 'odsInst2', both listening on privileged ports (ports ranging from 1 to 1023)
4. Define SMF service by customizing 'svcbundle' generated service manifest file and adding required method credential privileges for 'dirservd' user and required SMF service management authorization for 'odsadm' user.
5. Define script file to be invoked by service method execution.
6. Show how to add and assign authorization.
7. Import service artifacts to create SMF service for ODS instances.

As we go through, I'll also highlight little bit on following topics:

1. Using product specific tool to create service vs. using standard SMF tool - any benefit? - provide specific example for ODS. 
2. Generating SMF manifest file vs re-using available template and modifying for particular use.
3. Solaris 10 vs Solaris 11 from SMF perspective - few notes related to examples.
4. Parameterizing the SMF manifest file.
5. Debugging SMF service.

Create user accounts and groups to be used

• Group 'dirservd' and user 'dirservd' will be created and for security reason, user account 'dirservd' will be converted to 'no-login'. 'dirservd' will be a daemon/application user.
• User 'odsadm' will be created as a member of default group 'staff'. 'odsadm' will be service operator/administrator's user account.

# Execute the following command as root $> groupadd dirservd $> useradd -c "Directory Server reserved UID" -d / -g dirservd dirservd # Check the password status of dirservd $> passwd -s dirservd dirservd  UP # In Solaris 11, if you look the password status of newly created user, you'll see 'UP', # which basically means not activated. # Next, we'll turn this account as 'no-login' account, meaning, this account can not be used to login into the system. $> passwd -N dirservd   passwd: password information changed for dirservd $> passwd -s dirservd dirservd  NL # Create user 'odsadm' useradd -c "ODS Admin" -m -s /bin/bash -d /export/home/odsadm -g staff odsadm # Assign password for 'odsadm' $> passwd odsadm New Password: Re-enter new Password: passwd: password successfully changed for odsadm

Note: If you like to know more about 'no-login' account, I suggest you to look Glenn Brunnet's blog, "Managing Non-Login and Locked Solaris Accounts"

Create Oracle Directory Server (ODS) Instances

As stated above, for this scenario, we will create two instances both listening on privileged ports (ports ranging from 1 to 1023). As you know by default only root user can start process listening on privileged ports.

• 1st instance will listen on default ports: 389 (non-secure) and 636 (secure)
• 2nd instance will listen on: 489 (non-secure) and 736 (secure)

Notes:

1. make sure above mentioned ports are not being used by another processes. You can check using netstat command: netstat -na and grep the above mentioned ports.
2. Now, in order to create instance(s) that listen on privileged ports, you need to execute 'dsadm' command as root user.

# Execute the following command as root $> cd /opt/ods/dsee7/bin # Create instance odsInst1 $> ./dsadm create -u dirservd -g dirservd -p 389 -P 636 /opt/ods/dsee7/instances/odsInst1 Choose the Directory Manager password: Confirm the Directory Manager password: Use command 'dsadm start '/opt/ods/dsee7/instances/odsInst1'' to start the instance # Create instance odsInst2 $> ./dsadm create -u dirservd -g dirservd -p 489 -P 736 /opt/ods/dsee7/instances/odsInst2 Choose the Directory Manager password: Confirm the Directory Manager password: Use command 'dsadm start '/opt/ods/dsee7/instances/odsInst2'' to start the instance

As seen above, two ODS instances have been created. Now, it's time to create SMF service. You can actually use ODS command 'dsadm' to create SMF service, but it's not flexible enough. When I say not flexible enough, I mean this command does not give you following options:

• no option to customize service name,
• no option to specify service model like 'transient' or 'daemon' etc.
• no option to specify execution 'privileges' in the service definition level
• no option to specify service management authorization in the service definition level.
• you can't view the manifest file.

Here, anyway, I quickly show you how to use 'dsadm' command to create SMF service, but we'll delete the service thus created and use generic SMF tool to recreate.

# Execute the following command as root $> cd /opt/ods/dsee7/bin # Creating SMF service for odsInst1 using dsadm command $> ./dsadm enable-service -T SMF /opt/ods/dsee7/instances/odsInst1 Registering '/opt/ods/dsee7/instances/odsInst1' as 'ds7-opt-ods-dsee7-instances-odsInst1' in SMF ... Instance /opt/ods/dsee7/instances/odsInst1 registered in SMF Use comamnd 'dsadm start '/opt/ods/dsee7/instances/odsInst1'' to activate the service # Let's activate the service $> ./dsadm start '/opt/ods/dsee7/instances/odsInst1' Directory Server instance '/opt/ods/dsee7/instances/odsInst1' started: pid=1826 # Now let's take a look of service definition $> svcs -l ds7-opt-ods-dsee7-instances-odsInst1 fmri         svc:/application/sun/ds7_-opt-ods-dsee7:ds7-opt-ods-dsee7-instances-odsInst1 name         Directory Server restarter    svc:/system/svc/restarter:default manifest     /opt/ods/dsee7/instances/dsInst1/tmp/smf.manifest dependency   require_all/none svc:/system/filesystem/local:default (online) dependency   require_all/none svc:/network/initial:default (online) # It's time to delete the service, let's first stop the service $> svcadm disable ds7-opt-ods-dsee7-instances-odsInst1 # Unregister the service using dsadm command $> ./dsadm disable-service -T SMF /opt/ods/dsee7/instances/odsInst1 Instance /opt/ods/dsee7/instances/odsInst1 is not registered in SMF #However, if you check the service, it is still listed in SMF service database. $> svcs -l ds7-opt-ods-dsee7-instances-odsInst1 fmri         svc:/application/sun/ds7_-opt-ods-dsee7:ds7-opt-ods-dsee7-instances-odsInst1 name         Directory Server enabled      false state        disabled next_state   none state_time   Tue May 10 07:00:15 2016 restarter    svc:/system/svc/restarter:default manifest     /opt/ods/dsee7/instances/dsInst1/tmp/smf.manifest dependency   require_all/none svc:/system/filesystem/local:default (online) dependency   require_all/none svc:/network/initial:default (online) # So, in order to permanently delete it from SMF database, use following command: # svccfg delete <full service name> $> svccfg delete svc:/application/sun/ds7_-opt-ods-dsee7:ds7-opt-ods-dsee7-instances-odsInst1

We explored 'dsadm' tool little bit, now let's move forward and explore generic SMF ways.

Create SMF Manifest file.

Service manifest file is one of the most important component of Service Management Framework. It defines basic service information as well as service dependencies, required privileges, and start, stop command along with other required information. Now the question is how to put together a SMF manifest file. There are few options here:

1. Copy one of the service manifest files for other service and customize it or create from scratch manually. You can find these files under /lib/svc/manifest or /var/svc/manifest in your Solaris server.
2. Generate one using 'svcbundle' and customize it.

If you go with option #1, there are a lot of documentations from Oracle or others available on the web. I found Oracle White Paper 'How to Create an Oracle® Solaris Service Management Facility Manifest' very informative.
However, as part of this exercise, we look into option #2. We'll use 'svcbundle' to generate our minimal SMF manifest file and customize it for our purpose. I found Glynn Foster's Oracle Technet article on using svcbundle very handy. However, the best way to learn how to use svcbundle is to run command with 'help' option and play with it:

$> svcbundle help Usage: svcbundle [-i | -o output_file] -s name=value ... [help [name]] ....

I also suggest you to look the XML schme file (service_bundle.dtd.1) that dictates the structure of each SMF Manifest file. It is available under '/usr/share/lib/xml/dtd/' in your Solaris box.

Here we will generate a simple SMF manifest file for a single ODS instance and modify it for our multi-instances service environment:

$> svcbundle -o /tmp/odstemp.xml -s bundle-type=manifest -s enabled=false \ -s service-name=application/ods \ -s service-property=ods:ODS_HOME:astring:/opt/ods/dsee7 \ -s service-property=ods:ODS_INSTANCES_HOME:astring:/opt/ods/dsee7/instances \ -s instance-name=odsInst1 \ -s instance-property=odsInst:EXEC_OPTIONS:astring:"" \ -s model=transient -s start-method="/lib/svc/method/manage_ods start" \ -s stop-method="/lib/svc/method/manage_ods stop" \ -s refresh-method="/lib/svc/method/manage_ods refresh"

Above, we are using '-o <outputfile>' option instead of '-i' install option, so that we can modify the generated file before installing it. As you can see, start-method, stop-method, and refresh-method, refers to custom script file '/lib/svc/method/manage_ods'. I'll show you how to put together that file below. We have also defined two common application properties at service level (service-property), which are common to both instances:
ODS_HOME
ODS_INSTANCES_HOME
And one property at instance level (instance-property), which can be unique to each instance.
EXEC_OPTIONS
Note: It is just to show how instance specific parameter can be used. In this example, we are assigning empty string for EXEC_OPTION
Here is the generated file:

<?xml version="1.0" ?> <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'> <!--     Manifest created by svcbundle (2016-May-08 21:06:50-0400) --> <service_bundle type="manifest" name="application/ods">     <service version="1" type="service" name="application/ods">         <!--             The following dependency keeps us from starting until the             multi-user milestone is reached.         -->         <dependency restart_on="none" type="service"             name="multi_user_dependency" grouping="require_all">             <service_fmri value="svc:/milestone/multi-user"/>         </dependency>         <exec_method timeout_seconds="60" type="method" name="start"             exec="/lib/svc/method/manage_ods start"/>         <exec_method timeout_seconds="60" type="method" name="stop"             exec="/lib/svc/method/manage_ods stop"/>         <exec_method timeout_seconds="60" type="method" name="refresh"             exec="/lib/svc/method/manage_ods refresh"/>         <property_group type="framework" name="startd">             <propval type="astring" name="duration" value="transient"/>         </property_group>         <property_group type="application" name="ods">             <propval type="astring" name="ODS_HOME" value="/opt/ods/dsee7"/>             <propval type="astring" name="ODS_INSTANCES_HOME"                 value="/opt/ods/dsee7/instances"/>         </property_group>         <instance enabled="false" name="odsInst1">             <property_group type="application" name="odsInst">                 <propval type="astring" name="EXEC_OPTIONS" value=""/>             </property_group>         </instance>         <template>             <common_name>                 <loctext xml:lang="C">                     <!--                         Replace this comment with a short name for the                         service.                     -->                 </loctext>             </common_name>             <description>                 <loctext xml:lang="C">                     <!--                         Replace this comment with a brief description of                         the service                     -->                 </loctext>             </description>         </template>     </service> </service_bundle>

Below is customized SMF Manifest file for our purpose. See the comments within the file for more information.

<?xml version="1.0"?> <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">   <!--   Service Manifest Template for Oracle Directory Service. --> <service_bundle type='manifest' name='application/ods'>     <service version='1' type='service' name='application/ods'>         <!--           The following dependency keeps us from starting until the           multi-user milestone is reached.         -->         <dependency restart_on="none" type="service"             name="multi_user_dependency" grouping="require_all">             <service_fmri value="svc:/milestone/multi-user"/>         </dependency>         <!--           Wait for network interfaces to be initialized.         -->         <dependency name='network' grouping='require_all' restart_on='none' type='service'>             <service_fmri value='svc:/milestone/network:default'/>         </dependency>         <!--             Wait for all local filesystems to be mounted.         -->         <dependency name='filesystem-local' grouping='require_all' restart_on='none' type='service'>             <service_fmri value='svc:/system/filesystem/local:default'/>         </dependency>                 <!-- execution credentials provided to non-root user 'oracleds'. This user has just enough privileges              to own the process even if it is listening on privilege ports.         -->         <exec_method timeout_seconds='60' type='method' name='start' exec='/lib/svc/method/manage_ods start'/>         <exec_method timeout_seconds='60' type='method' name='stop' exec='/lib/svc/method/manage_ods stop'/>         <exec_method timeout_seconds='120' type='method' name='refresh' exec='/lib/svc/method/manage_ods refresh'/>                 <property_group type='framework' name='startd'>             <!--             The duration property value transient ensures to execute the start method once and does not             execute it again if the method exits with $SMF_EXIT_OK. The svc.startd daemon does not try             to restart the script after its first execution.             -->             <propval type='astring' name='duration' value='transient'/>             <!-- sub-process core dumps shouldn't restart session -->             <propval type='astring' name='ignore_error' value='core,signal'/>         </property_group>         <!-- authorization: Any user assigned with the following authorizations will be able to manage service.              In our case 'odsadm' user will be assigned following two authorization.         -->         <property_group type='framework' name='general'>            <propval type='astring' name='action_authorization' value='solaris.smf.manage.ods'/>            <propval type='astring' name='value_authorization' value='solaris.smf.value.ods'/>         </property_group>         <!-- Define here all application specific parameters that are common to all instances             and to be passed to 'manage_ods' method -->         <property_group type='application' name='ods'>             <propval type='astring' name='ODS_HOME' value='/opt/ods/dsee7'/>             <propval type='astring' name='ODS_INSTANCES_HOME' value='/opt/ods/dsee7/instances'/>                                    </property_group>                    <!--           Define ODS instances         -->         <!--             odsInst1         -->         <instance enabled='false' name='odsInst1'>             <method_context>                 <!-- On Solaris systems, in order for a non-root user to use privileged ports,                      it needs net_privaddr privilege. The net_privaddr privilege allows a process to                      bind to a privileged port number (1-1023). -->                 <method_credential user='dirservd' group='dirservd'                 privileges='basic,proc_owner,net_privaddr,file_dac_read,file_dac_write,file_dac_search'/>             </method_context>             <!-- Define instance specific parameters (if any) to be passed to 'manage_ods' method. Example: -->             <property_group type='application' name='odsInst'>                 <propval type='astring' name='EXEC_OPTIONS' value=''/>                            </property_group>         </instance>                 <!--             odsInst2         -->         <instance enabled='false' name='odsInst2'>             <method_context>                 <method_credential user='dirservd' group='dirservd'                 privileges='basic,proc_owner,net_privaddr,file_dac_read,file_dac_write,file_dac_search'/>             </method_context>             <!-- Define instance specific parameters (if any) to be passed to 'manage_ods' method. Example: -->             <property_group type='application' name='odsInst'>                 <propval type='astring' name='EXEC_OPTIONS' value=''/>                            </property_group>         </instance>         <stability value='Evolving'/>         <template>             <common_name>                 <loctext xml:lang='C'>Oracle Directory Server</loctext>             </common_name>             <description>                 <loctext xml:lang="C">                     Refer to Oracle Directory Server Enterprise Edition 11g Release 1 Administration Guide                 </loctext>                            </description>         </template>            </service> </service_bundle>

In customized file above, added fragments of code is in green and modified fragment is in blue. Relevant comment has also been added so that it is easy to understand. As you can see new instance definition for 'odsInst2' has also been added along with privileges for 'dirservd' user and authorizations 'solaris.smf.manage.ods' and 'solaris.smf.value.ods' for service operator/administrator to administer the service. These two authorization will be assigned to 'odsadm' user.

Create custom script file to be invoked by service executable methods

  Below is script source for 'manage_ods' file. As shown, script retrieves parameter values supplied from SMF manifest for the service and instance and defines start, stop and refresh methods.

#!/sbin/sh # Common service method for ODS Administration. # . /lib/svc/share/smf_include.sh getsmfpropval(){     val=$(svcprop -p $1 $SMF_FMRI)     if [[ ! -z "$val" && -n "$val" && "$val" != '""' ]]; then         echo $val     fi } if [ -z $SMF_FMRI ]; then     echo "Service Management framework variables  not initialized."     exit $SMF_EXIT_ERR fi # Service instance name INST_NAME=${SMF_FMRI##*:} # Retrieve property values set in the service manifest file. ODS_HOME=$(getsmfpropval ods/ODS_HOME) ODS_INSTANCES_HOME=$(getsmfpropval ods/ODS_INSTANCES_HOME) EXEC_OPTIONS=$(getsmfpropval odsInst/EXEC_OPTIONS) INST_HOME=$ODS_INSTANCES_HOME/$INST_NAME if [[ -z $ODS_HOME || ! -d $ODS_HOME ]]; then     echo "ods/ODS_HOME property either not set, value empty or directory '$ODS_HOME' does not exist."     exit $SMF_EXIT_ERR_CONFIG fi if [[ -z $ODS_INSTANCES_HOME || ! -d $ODS_INSTANCES_HOME ]]; then     echo "ods/ODS_INSTANCES_HOME property either not set, empty or directory '$ODS_INSTANCES_HOME' does not exist."     exit $SMF_EXIT_ERR_CONFIG fi case "$1" in     'start')         action="start"         ;;     'refresh')         action="restart"         ;;     'stop')         action="stop"         ;;     *)         echo "Usage: $0 {start|stop|refresh}"         exit $SMF_EXIT_ERR_CONFIG         ;; esac if [[ ! -z $EXEC_OPTIONS && "$EXEC_OPTIONS" != "" ]]; then     ODS_CMD="$ODS_HOME/bin/dsadm ${EXEC_OPTIONS} ${action} ${INST_HOME}" else     ODS_CMD="$ODS_HOME/bin/dsadm ${action} ${INST_HOME}" fi echo "##### $SMF_FMRI ${action}ing ODS instance ${INST_NAME} with command: ${ODS_CMD} #####" ${ODS_CMD} 2>&1 if [ $? -ne 0 ]; then     echo "$SMF_FMRI failed to start ODS instance: ${INST_NAME}"     exit $SMF_EXIT_ERR_FATAL fi exit $SMF_EXIT_OK

Pink colored lines in the script above show how parameters (as property name and value) set in the SMF manifest file are retrieved here in the script and utilized, making it usable from one environment to another and one instances to multiple.
At this point we have everything we need in order to create SMF service. Let's upload/stage our SMF manifest file 'ods.xml' and script file 'manage_ods' into Solaris server somewhere in the /tmp directory. Below are some important steps, you need to carry out in order to create SMF service successfully:

1) Validate the SMF Manifest file as shown below and make sure no error.

$> svccfg validate /tmp/ods.xml

2) [Optional ] Make sure there is no existing SMF service with the same name. Below command queries the SMF service database. Here is an example:

svcs -a | grep ".*application/ods/odsInst.*" # If existing service with the same name is found and you decide to delete, use the following commands # 1. Stop the given service: # svcadm disable service-name svcadm disable odsInst1 svcadm disable odsInst2 # 2. if SMF manifest file exists, then remove the manifest file. # Usual locations for application specific Manifest files: /lib/svc/manifest/site or /var/svc/manifest/site rm -f /lib/svc/manifest/site/ods.xml # if manifest file does not exist, then delete the service using command below: # svccfg delete <full-service-name> svccfg delete svc:/application/ods:odsInst1 svccfg delete svc:/application/ods:odsInst2 # 3. Restart manifest-import service svcadm restart manifest-import

3) Copy files to target location

# Depending upon whether it is Solaris 10 or Solaris 11; the default location is: # Manifest file: /var/svc/manifest/site (Solaris 10); /lib/svc/manifest/site (Solaris 11) # Script file: /lib/svc/method # copy manifest to /lib[var]/svc/manifest/site. Example below is for Solaris 11: $> cp /tmp/ods.xml /lib/svc/manifest/site $> chown root:sys /lib/svc/manifest/site/ods.xml $> chmod 444 /lib/svc/manifest/site/ods.xml # Copy script file: $> cp /tmp/manage_ods /lib/svc/method    $> chown root:bin /lib/svc/method/manage_ods    $> chmod 555 /lib/svc/method/manage_ods

4) Add required authorization(s) defined in the SMF manifest file into /etc/security/auth_attr

# for Solaris 10: $> echo "solaris.smf.manage.ods:::Allows to manage smf service for ods::" >> /etc/security/auth_attr $> echo "solaris.smf.value.ods:::Allows to change smf service for ods::" >> /etc/security/auth_attr # for Solaris 11: $> auths add -t "Allows to manage smf service for ods" solaris.smf.manage.ods    $> auths add -t "Allows to change smf service for ods" solaris.smf.value.ods

5) Assign authorizations to service operator/administrator account.

In our case we'll assign these authorizations to 'odsadm' user.
Note: Before modifying the authorization for a given account, you may want to see what currently assigned values are and take a note just in case if you need to backout.

# As root run the following command and take a note of existing auths just in case if you mess things # auths <user> # or just run 'auths' command if you are logged in as a particular user. # Once existing auths are noted run following command # For Solaris 11: $> usermod -A +solaris.smf.manage.ods,solaris.smf.value.ods odsadm # For Solaris 10: $> usermod -A <existing authorizations>,solaris.smf.manage.ods,solaris.smf.value.ods odsadm

Note: the usermod command basically adds/updates /etc/user_attr file. If you are an advanced user, you may directly update /etc/user_attr file.

6) Register the service into SMF

# only for Solaris 10, import manifest $> svccfg -v import /var/svc/manifest/site/ods.xml # For both Solaris 10 and Solaris 11, restart the manifest-import service $> svcadm restart manifest-import

7) Verify the service

# Checking svc:/application/ods:odsInst1 $> svcs -l odsInst1 fmri         svc:/application/ods:odsInst1 name         Oracle Directory Server enabled      false state        disabled next_state   none state_time   Mon May 09 17:23:45 2016 restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/site/ods.xml dependency   require_all/none svc:/milestone/multi-user (online) dependency   require_all/none svc:/milestone/network:default (online) dependency   require_all/none svc:/system/filesystem/local:default (online) # Checking svc:/application/ods:odsInst2 $> svcs -l odsInst2 fmri         svc:/application/ods:odsInst2 name         Oracle Directory Server enabled      false state        disabled next_state   none state_time   Mon May 09 17:23:45 2016 restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/site/ods.xml dependency   require_all/none svc:/milestone/multi-user (online) dependency   require_all/none svc:/milestone/network:default (online) dependency   require_all/none svc:/system/filesystem/local:default (online)

Note: As you can see the currently service is in disabled state, it's because in the manifest file we defined initial state to be 'enabled=false' If you need service to be started automatically on system reboot make sure to have 'enabled=true'.

8) Now let's start the service as 'odsadm' user and make sure it is started

# change to 'odsadm' user $> su - odsadm # start the odsInst1 $> svcadm enable odsInst1 # Check the status $> svcs -l odsInst1 fmri         svc:/application/ods:odsInst1 name         Oracle Directory Server enabled      true state        online next_state   none state_time   Mon May 09 19:18:36 2016 logfile      /var/svc/log/application-ods:odsInst1.log restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/site/ods.xml dependency   require_all/none svc:/milestone/multi-user (online) dependency   require_all/none svc:/milestone/network:default (online) dependency   require_all/none svc:/system/filesystem/local:default (online) # start the odsInst2 $> svcadm enable odsInst2 # Check the status $> svcs -l odsInst2 fmri         svc:/application/ods:odsInst2 name         Oracle Directory Server enabled      true state        online next_state   none state_time   Mon May 09 19:23:27 2016 logfile      /var/svc/log/application-ods:odsInst2.log restarter    svc:/system/svc/restarter:default manifest     /lib/svc/manifest/site/ods.xml dependency   require_all/none svc:/milestone/multi-user (online) dependency   require_all/none svc:/milestone/network:default (online) dependency   require_all/none svc:/system/filesystem/local:default (online)

9) Troubleshooting (if any)

• Look the service log. You can find the location of service log by running command 'svcs -l <service-name>'
• Use svcs debugging using 'svcs -xv' which gives all services that are having issue(s) or 'svcs -xv <service-name>' for particular service.
• A lot of time it could be privilege related. You can debug it using 'ppriv' command. See ppriv details.

This concludes our SMF walkthrough. Hope, you find it useful. Now, you go ahead and use SMF in your own environment. Happy SMFing!

Few Tips on Secure vs. Non-Secure WAS Web Server Plug-in Connection

In this blog, I am focusing only on one aspect of WebSphere Application Server (WAS) Web Server plug-in to Application Server connection i.e. secure vs. non-secure, what has changed in recent versions of WAS and any tips and tricks.
By default, when plug-in configuration file (plugin-cfg.xml) is generated, it creates configuration for both secure (HTTPS) and non-secure (HTTP) transport channels for the communication to
the application server(s). Below is fragment of plugin-cfg.xml:

... <ServerCluster CloneSeparatorChange="false" ... >       <Server ConnectTimeout="0" ExtendedHandshake="false" ...> <!-- non-secure (HTTP) transport channel -->              <Transport Hostname="MyHost" Port="9080" Protocol="http"/> <!-- secure (HTTPS) transport channel --> <Transport Hostname="MyHost" Port="9443" Protocol="https">                    <property Name="keyring" Value="c:\IBM\Plugins\plugin-key.kdb"/>                    <Property Name="stashfile" Value="c:\IBM\Plugins\plugin-key.sth"/>               </Transport>        </Server>   </ServerCluster> ...

When you have two transport channels defined, a lot of time it creates confusion because people may not know which communication channel is actually going to be used or whether they really need both channels in their configuration?
When you have both channels defined (just like shown in the example above), if the incoming traffic is secure (HTTPS), it automatically chooses the secure channel (MyHost:9443 in example) to create connection to the back-end application server, but if the incoming traffic is non-secure (HTTP), it by default chooses non-secure (HTTP) (MyHost:9080 in example).

• What if reverse case scenario? 
• Or only one type of back-end connection is defined/available and opposite type of incoming traffic is encountered? 

Looks like there are some recent changes and gotcha here.

Incoming secure (HTTPS) traffic and and only non-secure (HTTP) transport channel defined/available:

• version 8.5.5.0 and later:

 In version 8.5.5.0 or latter, in this particular case, plug-in won't create any connection to the    application server, because it interprets this situation as a security risk and request fails. If plugin trace is enabled, you'll see something like below in the plugin.log:

[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereFindTransport: Nosecure transports available
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find a transport
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ESI: getResponse: failed to get response: rc = 4
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - DEBUG: ESI: esiHandleRequest: failed to get response
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - DEBUG: ESI: esiRequestUrlStackDestroy
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereHandleRequest: Failed to handle request

However, you can use the plug-in custom property UseInsecure=true in the plugin-cfg.xml file; In this case plugin will use non-secure (HTTP) transport channel to establish connection to the application server despite the secure incoming request. You can add custom property in two wasys:
1. You can directly modify the plugin-cfg.xml as follows:

<Config ASDisableNagle="false" AcceptAllContent="true" AppServerPortPreference="HostHeader" ... UseInsecure="true"> ... </Config>

2. Or add this property through WebSphere Administration Console (Servers > Web Servers > Web_server_name > Plug-in properties > Custom properties page) and regenerate the plugin-cfg.xml.

Once the UseInsecure=true custom property becomes effective, the above mentioned scenario can create connection successfully. Below are some relevant lines from plugin.log (trace enabled)

[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - TRACE: ws_common: websphereFindTransport: Finding the transport for server 021313E0
websphereFindTransport: Setting the transport(case 3): OND2C00981304.cihs.ad.gov.on.ca on 080
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereExecute: Executing the transaction with the app server reqInfo is e671f78 useExistingStream=0, client->stream=00000000
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: Getting the stream to the app server (keepalive 28)
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: calling blocking connect
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: Setting socket to non-block for ServerIOTimeout over HTTP
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: socket 3564 connected to OND2C00981304.cihs.ad.gov.on.ca:9080 timeout=900
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: lib_stream: openStream: Opening the  stream soc=3564

• Previous versions:

By default, in previous versions of WAS plug-in for web server, if the web server plug-in received secure (HTTPS) request but could not create secure connection to the application server (either secure transport channel not defined or secure connection could not be established),
it would create a non-secure (HTTP) connection (if one is defined and available). If HTTP transport not defined, then no connection would be created.
So, the behaviour in older version of WAS that defaulted HTTPS to HTTP when secure connection was not available, was a real problem from security prospective. Read more about the problem as documented here:
http://www-01.ibm.com/support/docview.wss?uid=swg1PM85452.

As mentioned above, in WAS version 8.5.5.0 and later, it has been fixed and it only defaults from HTTPS to HTTP if it is explicitly configured to default when HTTPS connection can not be
established. There still seems to be some logging issue in version 8.5.5 with UseInsecure="true" and fix is available in fix pack 8.5.5.2. Read detail here: http://www-01.ibm.com/support/docview.wss?uid=swg1PM96173

Incoming HTTP traffic and only secure (HTTPS) transport channel to back end configured/available:

   In this case, there should be no issue as long as keyring that contains the certificate of back-end application server and stash file properly configured for the transport channel.

Note: In order to minimize confusion, if you are sure only secure (HTTPS) connection should be allowed for your implementation, you can simply comment out the non-secure transport channel configuration in the plugin-cfg.xml or vice-versa.

Hope these tips help to eradicate (if any) confusion you to might have related to plug-in to application server connection from security perspective.

Read more about the Web server plug-in connections in IBM Knowledge Center here: http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cwsv_plugin_connections.html?lang=en.