SMF Explained with Real Examples

     As part of an Automated Environment Management initiative, recently, I had to work with Solaris Service Management Facility (SMF) extensively. In order to make the implementation right, I had to look several different documentations, blogs, consult with colleagues, learn through error and trial methods. The main intention of this blog is - kind of 'give back to community', what I have learned.
Let's get started. Since, you are visiting this page, I assume, you are already familiar with the SMF concept and looking to resolve specific issue or digging some specific information in SMF domain. If you are new to SMF, you can learn the basic of SMF from Oracle site here.

Why SMF?
      In Unix world, traditionally we used to work with init script as a start-up mechanism. SMF may have been evolved from legacy init, but it is much more flexible, robust, and very well crafted. SMF helps to make environment management easier, uniform and align with security practices as well. 
     In this posts, I'll explain SMF through a working example. First, I have to warn you that some of the steps here possibly overdone to show the concept in detail or underdone to minimize the size of the blog.
     Anyway, here is what we are going to do. We have an environment as described below:
  • Solaris 11 (64 bit VirtualBox guest VM running on 64 bit Windows 10 Host) and Oracle Directory Server 11g installed under /opt/ods/dsee7
And as part of this learning process, I'll show you the following:
  1. Create no-login user 'dirservd', this user will own the Oracle Directory Server (ODS) instances to be created.
  2. Create 'odsadm' user that will operate/manage SMF services for ODS instances to be created.
  3. Create two Oracle Directory Server (ODS) instances 'odsInst1' and 'odsInst2', both listening on privileged ports (ports ranging from 1 to 1023)
  4. Define SMF service by customizing 'svcbundle' generated service manifest file and adding required method credential privileges for 'dirservd' user and required SMF service management authorization for 'odsadm' user.
  5. Define script file to be invoked by service method execution.
  6. Show how to add and assign authorization.
  7. Import service artifacts to create SMF service for ODS instances.
As we go through, I'll also highlight little bit on following topics:
  1. Using product specific tool to create service vs. using standard SMF tool - any benefit? - provide specific example for ODS. 
  2. Generating SMF manifest file vs re-using available template and modifying for particular use.
  3. Solaris 10 vs Solaris 11 from SMF perspective - few notes related to examples.
  4. Parameterizing the SMF manifest file.
  5. Debugging SMF service.

Create user accounts and groups to be used

  • Group 'dirservd' and user 'dirservd' will be created and for security reason, user account 'dirservd' will be converted to 'no-login'. 'dirservd' will be a daemon/application user.
  • User 'odsadm' will be created as a member of default group 'staff'. 'odsadm' will be service operator/administrator's user account.
# Execute the following command as root
$> groupadd dirservd
$> useradd -c "Directory Server reserved UID" -d / -g dirservd dirservd
# Check the password status of dirservd
$> passwd -s dirservd
dirservd  UP
# In Solaris 11, if you look the password status of newly created user, you'll see 'UP',
# which basically means not activated.
# Next, we'll turn this account as 'no-login' account, meaning, this account can not be used to login into the system.

$> passwd -N dirservd
  passwd: password information changed for dirservd
$> passwd -s dirservd
dirservd  NL
# Create user 'odsadm'
useradd -c "ODS Admin" -m -s /bin/bash -d /export/home/odsadm -g staff odsadm
# Assign password for 'odsadm'
$> passwd odsadm
New Password:
Re-enter new Password:
passwd: password successfully changed for odsadm


Note: If you like to know more about 'no-login' account, I suggest you to look Glenn Brunnet's blog, "Managing Non-Login and Locked Solaris Accounts"

Create Oracle Directory Server (ODS) Instances

As stated above, for this scenario, we will create two instances both listening on privileged ports (ports ranging from 1 to 1023). As you know by default only root user can start process listening on privileged ports.
  • 1st instance will listen on default ports: 389 (non-secure) and 636 (secure)
  • 2nd instance will listen on: 489 (non-secure) and 736 (secure)
Notes:
  1. make sure above mentioned ports are not being used by another processes. You can check using netstat command: netstat -na and grep the above mentioned ports.
  2. Now, in order to create instance(s) that listen on privileged ports, you need to execute 'dsadm' command as root user.
# Execute the following command as root
$> cd
/opt/ods/dsee7/bin
# Create instance
odsInst1
$> ./dsadm create -u dirservd -g dirservd -p 389 -P 636 /opt/ods/dsee7/instances/odsInst1
Choose the Directory Manager password:
Confirm the Directory Manager password:
Use command 'dsadm start '/opt/ods/dsee7/instances/odsInst1'' to start the instance
# Create instance odsInst2
$> ./dsadm create -u dirservd -g dirservd -p 489 -P 736 /opt/ods/dsee7/instances/odsInst2
Choose the Directory Manager password:
Confirm the Directory Manager password:
Use command 'dsadm start '/opt/ods/dsee7/instances/odsInst2'' to start the instance

As seen above, two ODS instances have been created. Now, it's time to create SMF service. You can actually use ODS command 'dsadm' to create SMF service, but it's not flexible enough. When I say not flexible enough, I mean this command does not give you following options:
  • no option to customize service name,
  • no option to specify service model like 'transient' or 'daemon' etc.
  • no option to specify execution 'privileges' in the service definition level
  • no option to specify service management authorization in the service definition level.
  • you can't view the manifest file.
Here, anyway, I quickly show you how to use 'dsadm' command to create SMF service, but we'll delete the service thus created and use generic SMF tool to recreate.
# Execute the following command as root
$> cd /opt/ods/dsee7/bin
# Creating SMF service for odsInst1 using dsadm command
$> ./dsadm enable-service -T SMF /opt/ods/dsee7/instances/odsInst1
Registering '/opt/ods/dsee7/instances/odsInst1' as 'ds7-opt-ods-dsee7-instances-odsInst1' in SMF ...
Instance /opt/ods/dsee7/instances/odsInst1 registered in SMF
Use comamnd 'dsadm start '/opt/ods/dsee7/instances/odsInst1'' to activate the service
# Let's activate the service
$> ./dsadm start '/opt/ods/dsee7/instances/odsInst1'
Directory Server instance '/opt/ods/dsee7/instances/odsInst1' started: pid=1826
# Now let's take a look of service definition
$> svcs -l ds7-opt-ods-dsee7-instances-odsInst1
fmri         svc:/application/sun/ds7_-opt-ods-dsee7:ds7-opt-ods-dsee7-instances-odsInst1
name         Directory Server
restarter    svc:/system/svc/restarter:default
manifest     /opt/ods/dsee7/instances/dsInst1/tmp/smf.manifest
dependency   require_all/none svc:/system/filesystem/local:default (online)
dependency   require_all/none svc:/network/initial:default (online)
# It's time to delete the service, let's first stop the service
$> svcadm disable ds7-opt-ods-dsee7-instances-odsInst1
# U
nregister the service using dsadm command
$> ./dsadm disable-service -T SMF /opt/ods/dsee7/instances/odsInst1
Instance /opt/ods/dsee7/instances/odsInst1 is not registered in SMF
#However, if you check the service, it is still listed in SMF service database.
$> svcs -l ds7-opt-ods-dsee7-instances-odsInst1
fmri         svc:/application/sun/ds7_-opt-ods-dsee7:ds7-opt-ods-dsee7-instances-odsInst1
name         Directory Server
enabled      false
state        disabled
next_state   none
state_time   Tue May 10 07:00:15 2016
restarter    svc:/system/svc/restarter:default
manifest     /opt/ods/dsee7/instances/dsInst1/tmp/smf.manifest
dependency   require_all/none svc:/system/filesystem/local:default (online)
dependency   require_all/none svc:/network/initial:default (online)
# So, in order to permanently delete it from SMF database, use following command:
# svccfg delete <full service name>
$> svccfg delete svc:/application/sun/ds7_-opt-ods-dsee7:ds7-opt-ods-dsee7-instances-odsInst1

We explored 'dsadm' tool little bit, now let's move forward and explore generic SMF ways.

Create SMF Manifest file.

Service manifest file is one of the most important component of Service Management Framework. It defines basic service information as well as service dependencies, required privileges, and start, stop command along with other required information. Now the question is how to put together a SMF manifest file. There are few options here:
  1. Copy one of the service manifest files for other service and customize it or create from scratch manually. You can find these files under /lib/svc/manifest or /var/svc/manifest in your Solaris server.
  2. Generate one using 'svcbundle' and customize it.
If you go with option #1, there are a lot of documentations from Oracle or others available on the web. I found Oracle White Paper 'How to Create an Oracle® Solaris Service Management Facility Manifest' very informative.
However, as part of this exercise, we look into option #2. We'll use 'svcbundle' to generate our minimal SMF manifest file and customize it for our purpose. I found Glynn Foster's Oracle Technet article on using svcbundle very handy. However, the best way to learn how to use svcbundle is to run command with 'help' option and play with it:
$> svcbundle help
Usage: svcbundle [-i | -o output_file] -s name=value ... [help [name]]
....

I also suggest you to look the XML schme file (service_bundle.dtd.1) that dictates the structure of each SMF Manifest file. It is available under '/usr/share/lib/xml/dtd/' in your Solaris box.

Here we will generate a simple SMF manifest file for a single ODS instance and modify it for our multi-instances service environment:
$> svcbundle -o /tmp/odstemp.xml -s bundle-type=manifest -s enabled=false \
-s service-name=application/ods \
-s service-property=ods:ODS_HOME:astring:/opt/ods/dsee7 \
-s service-property=ods:ODS_INSTANCES_HOME:astring:/opt/ods/dsee7/instances \
-s instance-name=odsInst1 \
-s instance-property=odsInst:EXEC_OPTIONS:astring:"" \
-s model=transient -s start-method="/lib/svc/method/manage_ods start" \
-s stop-method="/lib/svc/method/manage_ods stop" \
-s refresh-method="/lib/svc/method/manage_ods refresh"
Above, we are using '-o <outputfile>' option instead of '-i' install option, so that we can modify the generated file before installing it. As you can see, start-method, stop-method, and refresh-method, refers to custom script file '/lib/svc/method/manage_ods'. I'll show you how to put together that file below. We have also defined two common application properties at service level (service-property), which are common to both instances:
ODS_HOME
ODS_INSTANCES_HOME
And one property at instance level (instance-property), which can be unique to each instance.
EXEC_OPTIONS
Note: It is just to show how instance specific parameter can be used. In this example, we are assigning empty string for EXEC_OPTION
Here is the generated file:

<?xml version="1.0" ?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
    Manifest created by svcbundle (2016-May-08 21:06:50-0400)
-->
<service_bundle type="manifest" name="application/ods">
    <service version="1" type="service" name="application/ods">
        <!--
            The following dependency keeps us from starting until the
            multi-user milestone is reached.
        -->
        <dependency restart_on="none" type="service"
            name="multi_user_dependency" grouping="require_all">
            <service_fmri value="svc:/milestone/multi-user"/>
        </dependency>
        <exec_method timeout_seconds="60" type="method" name="start"
            exec="/lib/svc/method/manage_ods start"/>
        <exec_method timeout_seconds="60" type="method" name="stop"
            exec="/lib/svc/method/manage_ods stop"/>
        <exec_method timeout_seconds="60" type="method" name="refresh"
            exec="/lib/svc/method/manage_ods refresh"/>
        <property_group type="framework" name="startd">
            <propval type="astring" name="duration" value="transient"/>
        </property_group>
        <property_group type="application" name="ods">
            <propval type="astring" name="ODS_HOME" value="/opt/ods/dsee7"/>
            <propval type="astring" name="ODS_INSTANCES_HOME"
                value="/opt/ods/dsee7/instances"/>
        </property_group>
        <instance enabled="false" name="odsInst1">
            <property_group type="application" name="odsInst">
                <propval type="astring" name="EXEC_OPTIONS" value=""/>
            </property_group>
        </instance>
        <template>
            <common_name>
                <loctext xml:lang="C">
                    <!--
                        Replace this comment with a short name for the
                        service.
                    -->
                </loctext>
            </common_name>
            <description>
                <loctext xml:lang="C">
                    <!--
                        Replace this comment with a brief description of
                        the service
                    -->
                </loctext>
            </description>
        </template>
    </service>
</service_bundle>


Below is customized SMF Manifest file for our purpose. See the comments within the file for more information.
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">  
<!--
  Service Manifest Template for Oracle Directory Service.
-->

<service_bundle type='manifest' name='
application/ods'>
    <service version='1'
type='service' name='application/ods'>
        <!--
          The following dependency keeps us from starting until the
          multi-user milestone is reached.
        -->
        <dependency restart_on="none" type="service"
            name="multi_user_dependency" grouping="require_all">
            <service_fmri value="svc:/milestone/multi-user"/>
        </dependency>
        <!--
          Wait for network interfaces to be initialized.
        -->
        <dependency name='network' grouping='require_all' restart_on='none' type='service'>
            <service_fmri value='svc:/milestone/network:default'/>
        </dependency>
        <!--
            Wait for all local filesystems to be mounted.
        -->
        <dependency name='filesystem-local' grouping='require_all' restart_on='none' type='service'>
            <service_fmri value='svc:/system/filesystem/local:default'/>
        </dependency>

       
        <!-- execution credentials provided to non-root user 'oracleds'. This user has just enough privileges
             to own the process even if it is listening on privilege ports.
        -->
        <exec_method
timeout_seconds='60' type='method' name='start' exec='/lib/svc/method/manage_ods start'/>
        <exec_method
timeout_seconds='60' type='method' name='stop' exec='/lib/svc/method/manage_ods stop'/>
        <exec_method
timeout_seconds='120' type='method' name='refresh' exec='/lib/svc/method/manage_ods refresh'/>
       
        <property_group type='framework' name='startd'>
            <!--
            The duration property value transient ensures to execute the start method once and does not
            execute it again if the method exits with $SMF_EXIT_OK. The svc.startd daemon does not try
            to restart the script after its first execution.
            -->
            <propval type='astring' name='duration' value='transient'/>
           
<!-- sub-process core dumps shouldn't restart session -->
            <propval type='astring' name='ignore_error' value='core,signal'/>

        </property_group>
        <!-- authorization: Any user assigned with the following authorizations will be able to manage service.
             In our case 'odsadm' user will be assigned following two authorization.
        -->
        <property_group type='framework' name='general'>
           <propval type='astring' name='action_authorization' value='solaris.smf.manage.ods'/>
           <propval type='astring' name='value_authorization' value='solaris.smf.value.ods'/>
        </property_group>

        <!-- Define here all application specific parameters that are common to all instances
            and to be passed to 'manage_ods' method -->
        <property_group
type='application' name='ods'>
            <propval type='astring'
name='ODS_HOME' value='/opt/ods/dsee7'/>
            <propval
type='astring' name='ODS_INSTANCES_HOME' value='/opt/ods/dsee7/instances'/>                           
        </property_group>           
        <!--
          Define ODS instances
        -->
        <!--
            odsInst1
        -->
        <instance
enabled='false' name='odsInst1'>
            <method_context>
                <!-- On Solaris systems, in order for a non-root user to use privileged ports,
                     it needs net_privaddr privilege. The net_privaddr privilege allows a process to
                     bind to a privileged port number (1-1023). -->
                <method_credential user='dirservd' group='dirservd'
                privileges='basic,proc_owner,net_privaddr,file_dac_read,file_dac_write,file_dac_search'/>
            </method_context>

            <!-- Define instance specific parameters (if any) to be passed to 'manage_ods' method. Example: -->
            <property_group
type='application' name='odsInst'>
                <propval
type='astring' name='EXEC_OPTIONS' value=''/>               
            </property_group>
        </instance>
       
        <!--
            odsInst2
        -->
        <instance
enabled='false' name='odsInst2'>
            <method_context>
                <method_credential user='dirservd' group='dirservd'
                privileges='basic,proc_owner,net_privaddr,file_dac_read,file_dac_write,file_dac_search'/>
            </method_context>
            <!-- Define instance specific parameters (if any) to be passed to 'manage_ods' method. Example: -->
            <property_group type='application' name='odsInst'>
                <propval
type='astring' name='EXEC_OPTIONS' value=''/>               
            </property_group>
        </instance>

        <stability value='Evolving'/>
        <template>
            <common_name>
                <loctext xml:lang='C'>Oracle Directory Server</loctext>
            </common_name>
            <description>
                <loctext xml:lang="C">
                    Refer to Oracle Directory Server Enterprise Edition 11g Release 1 Administration Guide
                </loctext>               
            </description>
        </template>       
    </service>
</service_bundle>

In customized file above, added fragments of code is in green and modified fragment is in blue. Relevant comment has also been added so that it is easy to understand. As you can see new instance definition for 'odsInst2' has also been added along with privileges for 'dirservd' user and authorizations 'solaris.smf.manage.ods' and 'solaris.smf.value.ods' for service operator/administrator to administer the service. These two authorization will be assigned to 'odsadm' user.

Create custom script file to be invoked by service executable methods

  Below is script source for 'manage_ods' file. As shown, script retrieves parameter values supplied from SMF manifest for the service and instance and defines start, stop and refresh methods.
#!/sbin/sh
# Common service method for ODS Administration.
#

. /lib/svc/share/smf_include.sh

getsmfpropval(){
    val=$(svcprop -p $1 $SMF_FMRI)
    if [[ ! -z "$val" && -n "$val" && "$val" != '""' ]]; then
        echo $val
    fi
}

if [ -z $SMF_FMRI ]; then
    echo "Service Management framework variables  not initialized."
    exit $SMF_EXIT_ERR
fi

# Service instance name
INST_NAME=${SMF_FMRI##*:}
# Retrieve property values set in the service manifest file.
ODS_HOME=$(getsmfpropval ods/ODS_HOME)
ODS_INSTANCES_HOME=$(getsmfpropval ods/ODS_INSTANCES_HOME)
EXEC_OPTIONS=$(getsmfpropval odsInst/EXEC_OPTIONS)

INST_HOME=$ODS_INSTANCES_HOME/$INST_NAME

if [[ -z $ODS_HOME || ! -d $ODS_HOME ]]; then
    echo "ods/ODS_HOME property either not set, value empty or directory '$ODS_HOME' does not exist."
    exit $SMF_EXIT_ERR_CONFIG
fi

if [[ -z $ODS_INSTANCES_HOME || ! -d $ODS_INSTANCES_HOME ]]; then
    echo "ods/ODS_INSTANCES_HOME property either not set, empty or directory '$ODS_INSTANCES_HOME' does not exist."
    exit $SMF_EXIT_ERR_CONFIG
fi

case "$1" in
    'start')
        action="start"
        ;;
    'refresh')
        action="restart"
        ;;
    'stop')
        action="stop"
        ;;
    *)
        echo "Usage: $0 {start|stop|refresh}"
        exit $SMF_EXIT_ERR_CONFIG
        ;;
esac

if [[ ! -z $EXEC_OPTIONS && "$EXEC_OPTIONS" != "" ]]; then
    ODS_CMD="$ODS_HOME/bin/dsadm ${EXEC_OPTIONS} ${action} ${INST_HOME}"
else
    ODS_CMD="$ODS_HOME/bin/dsadm ${action} ${INST_HOME}"
fi

echo "##### $SMF_FMRI ${action}ing ODS instance ${INST_NAME} with command: ${ODS_CMD} #####"

${ODS_CMD} 2>&1

if [ $? -ne 0 ]; then
    echo "$SMF_FMRI failed to start ODS instance: ${INST_NAME}"
    exit $SMF_EXIT_ERR_FATAL
fi
exit $SMF_EXIT_OK

Pink colored lines in the script above show how parameters (as property name and value) set in the SMF manifest file are retrieved here in the script and utilized, making it usable from one environment to another and one instances to multiple.
At this point we have everything we need in order to create SMF service. Let's upload/stage our SMF manifest file 'ods.xml' and script file 'manage_ods' into Solaris server somewhere in the /tmp directory. Below are some important steps, you need to carry out in order to create SMF service successfully:

1) Validate the SMF Manifest file as shown below and make sure no error.
$> svccfg validate /tmp/ods.xml

2) [Optional ] Make sure there is no existing SMF service with the same name. Below command queries the SMF service database. Here is an example:
svcs -a | grep ".*application/ods/odsInst.*"
# If existing service with the same name is found and you decide to delete, use the following commands
# 1. Stop the given service:
# svcadm disable service-name
svcadm disable odsInst1
svcadm disable odsInst2
# 2. if SMF manifest file exists, then remove the manifest file.
# Usual locations for application specific Manifest files: /lib/svc/manifest/site or /var/svc/manifest/site
rm -f
/lib/svc/manifest/site/ods.xml
# if manifest file does not exist, then delete the service using command below:
# svccfg delete <full-service-name>
svccfg delete
svc:/application/ods:odsInst1
svccfg delete
svc:/application/ods:odsInst2
# 3. Restart manifest-import service
svcadm restart manifest-import


3) Copy files to target location

# Depending upon whether it is Solaris 10 or Solaris 11; the default location is:
# Manifest file: 
/var/svc/manifest/site (Solaris 10); /lib/svc/manifest/site (Solaris 11)
# Script file: /lib/svc/method
# copy manifest to /lib[var]/svc/manifest/site. Example below is for Solaris 11:
$> cp /tmp/ods.xml /lib/svc/manifest/site
$> chown root:sys
/lib/svc/manifest/site/ods.xml
$> chmod 444
/lib/svc/manifest/site/ods.xml
# Copy script file:
$> cp /tmp/manage_ods
/lib/svc/method   
$> chown root:bin
/lib/svc/method/manage_ods   
$> chmod 555
/lib/svc/method/manage_ods


4) Add required authorization(s) defined in the SMF manifest file into /etc/security/auth_attr

# for Solaris 10:
$> echo "solaris.smf.manage.ods:::
Allows to manage smf service for ods::" >> /etc/security/auth_attr
$> echo "solaris.smf.value.ods:::Allows to change smf service for ods::" >> /etc/security/auth_attr
# for Solaris 11:
$> auths add -t "
Allows to manage smf service for ods" solaris.smf.manage.ods   
$> auths add -t "
Allows to change smf service for ods" solaris.smf.value.ods

5) Assign authorizations to service operator/administrator account.

In our case we'll assign these authorizations to 'odsadm' user.
Note: Before modifying the authorization for a given account, you may want to see what currently assigned values are and take a note just in case if you need to backout.

# As root run the following command and take a note of existing auths just in case if you mess things

# auths <user>
# or just run 'auths' command if you are logged in as a particular user.
# Once existing auths are noted run following command
# For Solaris 11:
$> usermod -A +
solaris.smf.manage.ods,solaris.smf.value.ods odsadm
# For Solaris 10:
$> usermod -A <existing authorizations>,solaris.smf.manage.ods,solaris.smf.value.ods odsadm


Note: the usermod command basically adds/updates /etc/user_attr file. If you are an advanced user, you may directly update /etc/user_attr file.

6) Register the service into SMF

# only for Solaris 10, import manifest
$> svccfg -v import /var/svc/manifest/site/ods.xml
# For both Solaris 10 and Solaris 11, restart the manifest-import service
$> svcadm restart manifest-import


7) Verify the service

# Checking svc:/application/ods:odsInst1
$> svcs -l odsInst1
fmri         svc:/application/ods:odsInst1
name         Oracle Directory Server
enabled      false
state        disabled

next_state   none
state_time   Mon May 09 17:23:45 2016
restarter    svc:/system/svc/restarter:default
manifest     /lib/svc/manifest/site/ods.xml
dependency   require_all/none svc:/milestone/multi-user (online)
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)


# Checking svc:/application/ods:odsInst2
$> svcs -l odsInst2
fmri         svc:/application/ods:odsInst2
name         Oracle Directory Server
enabled      false
state        disabled

next_state   none
state_time   Mon May 09 17:23:45 2016
restarter    svc:/system/svc/restarter:default
manifest     /lib/svc/manifest/site/ods.xml
dependency   require_all/none svc:/milestone/multi-user (online)
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)

Note: As you can see the currently service is in disabled state, it's because in the manifest file we defined initial state to be 'enabled=false' If you need service to be started automatically on system reboot make sure to have 'enabled=true'.

8) Now let's start the service as 'odsadm' user and make sure it is started

# change to 'odsadm' user
$> su - odsadm
# start the odsInst1
$> svcadm enable odsInst1
# Check the status
$> svcs -l odsInst1
fmri         svc:/application/ods:odsInst1
name         Oracle Directory Server
enabled      true
state        online

next_state   none
state_time   Mon May 09 19:18:36 2016
logfile      /var/svc/log/application-ods:odsInst1.log
restarter    svc:/system/svc/restarter:default
manifest     /lib/svc/manifest/site/ods.xml
dependency   require_all/none svc:/milestone/multi-user (online)
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)

# start the odsInst2
$> svcadm enable odsInst2
# Check the status
$> svcs -l odsInst2
fmri         svc:/application/ods:odsInst2
name         Oracle Directory Server
enabled      true
state        online

next_state   none
state_time   Mon May 09 19:23:27 2016
logfile      /var/svc/log/application-ods:odsInst2.log
restarter    svc:/system/svc/restarter:default
manifest     /lib/svc/manifest/site/ods.xml
dependency   require_all/none svc:/milestone/multi-user (online)
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)

9) Troubleshooting (if any)

  • Look the service log. You can find the location of service log by running command 'svcs -l <service-name>'
  • Use svcs debugging using 'svcs -xv' which gives all services that are having issue(s) or 'svcs -xv <service-name>' for particular service.
  • A lot of time it could be privilege related. You can debug it using 'ppriv' command. See ppriv details.
This concludes our SMF walkthrough. Hope, you find it useful. Now, you go ahead and use SMF in your own environment. Happy SMFing!

Few Tips on Secure vs. Non-Secure WAS Web Server Plug-in Connection

In this blog, I am focusing only on one aspect of WebSphere Application Server (WAS) Web Server plug-in to Application Server connection i.e. secure vs. non-secure, what has changed in recent versions of WAS and any tips and tricks.
By default, when plug-in configuration file (plugin-cfg.xml) is generated, it creates configuration for both secure (HTTPS) and non-secure (HTTP) transport channels for the communication to
the application server(s). Below is fragment of plugin-cfg.xml:
...
<ServerCluster CloneSeparatorChange="false" ... >
      <Server ConnectTimeout="0" ExtendedHandshake="false" ...>
<!-- non-secure (HTTP) transport channel -->
             <Transport Hostname="MyHost" Port="9080" Protocol="http"/>
<!-- secure (HTTPS) transport channel -->
<Transport Hostname="MyHost" Port="9443" Protocol="https">
                   <p
roperty Name="keyring" Value="c:\IBM\Plugins\plugin-key.kdb"/>
                   <Property Name="stashfile" Value="c:\IBM\Plugins\plugin-key.sth"/>
              </Transport>

       </Server>
  </ServerCluster>
...

When you have two transport channels defined, a lot of time it creates confusion because people may not know which communication channel is actually going to be used or whether they really need both channels in their configuration?
When you have both channels defined (just like shown in the example above), if the incoming traffic is secure (HTTPS), it automatically chooses the secure channel (MyHost:9443 in example) to create connection to the back-end application server, but if the incoming traffic is non-secure (HTTP), it by default chooses non-secure (HTTP) (MyHost:9080 in example).

  • What if reverse case scenario? 
  • Or only one type of back-end connection is defined/available and opposite type of incoming traffic is encountered? 

Looks like there are some recent changes and gotcha here.


Incoming secure (HTTPS) traffic and and only non-secure (HTTP) transport channel defined/available:

  • version 8.5.5.0 and later:


 In version 8.5.5.0 or latter, in this particular case, plug-in won't create any connection to the    application server, because it interprets this situation as a security risk and request fails. If plugin trace is enabled, you'll see something like below in the plugin.log:

[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereFindTransport: Nosecure transports available
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find a transport
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ESI: getResponse: failed to get response: rc = 4
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - DEBUG: ESI: esiHandleRequest: failed to get response
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - DEBUG: ESI: esiRequestUrlStackDestroy
[Thu Nov 26 15:36:52 2015] 00003578 00004774 - ERROR: ws_common: websphereHandleRequest: Failed to handle request

However, you can use the plug-in custom property UseInsecure=true in the plugin-cfg.xml file; In this case plugin will use non-secure (HTTP) transport channel to establish connection to the application server despite the secure incoming request. You can add custom property in two wasys:
1. You can directly modify the plugin-cfg.xml as follows:
<Config ASDisableNagle="false" AcceptAllContent="true" AppServerPortPreference="HostHeader" ... UseInsecure="true">
...
</Config>
2. Or add this property through WebSphere Administration Console (Servers > Web Servers > Web_server_name > Plug-in properties > Custom properties page) and regenerate the plugin-cfg.xml.

Once the UseInsecure=true custom property becomes effective, the above mentioned scenario can create connection successfully. Below are some relevant lines from plugin.log (trace enabled)

[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - TRACE: ws_common: websphereFindTransport: Finding the transport for server 021313E0
websphereFindTransport: Setting the transport(case 3): OND2C00981304.cihs.ad.gov.on.ca on 080
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereExecute: Executing the transaction with the app server reqInfo is e671f78 useExistingStream=0, client->stream=00000000
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: Getting the stream to the app server (keepalive 28)
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: calling blocking connect
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: Setting socket to non-block for ServerIOTimeout over HTTP
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: ws_common: websphereGetStream: socket 3564 connected to OND2C00981304.cihs.ad.gov.on.ca:9080 timeout=900
[Thu Nov 26 15:46:33 2015] 0000379c 000014b8 - DEBUG: lib_stream: openStream: Opening the  stream soc=3564

  • Previous versions:


By default, in previous versions of WAS plug-in for web server, if the web server plug-in received secure (HTTPS) request but could not create secure connection to the application server (either secure transport channel not defined or secure connection could not be established),
it would create a non-secure (HTTP) connection (if one is defined and available). If HTTP transport not defined, then no connection would be created.
So, the behaviour in older version of WAS that defaulted HTTPS to HTTP when secure connection was not available, was a real problem from security prospective. Read more about the problem as documented here:
http://www-01.ibm.com/support/docview.wss?uid=swg1PM85452.

As mentioned above, in WAS version 8.5.5.0 and later, it has been fixed and it only defaults from HTTPS to HTTP if it is explicitly configured to default when HTTPS connection can not be
established. There still seems to be some logging issue in version 8.5.5 with UseInsecure="true" and fix is available in fix pack 8.5.5.2. Read detail here: http://www-01.ibm.com/support/docview.wss?uid=swg1PM96173

Incoming HTTP traffic and only secure (HTTPS) transport channel to back end configured/available:

   In this case, there should be no issue as long as keyring that contains the certificate of back-end application server and stash file properly configured for the transport channel.

Note: In order to minimize confusion, if you are sure only secure (HTTPS) connection should be allowed for your implementation, you can simply comment out the non-secure transport channel configuration in the plugin-cfg.xml or vice-versa.

Hope these tips help to eradicate (if any) confusion you to might have related to plug-in to application server connection from security perspective.

Read more about the Web server plug-in connections in IBM Knowledge Center here: http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cwsv_plugin_connections.html?lang=en.

Accelerated Cryptography using On-Chip Instructions : from Java Application Perspective

     In this blog, I am exploring accelerated crypto processing options using on-chip instructions. I am looking these options from applications (specifically Java) perspective. Let me start with one of my favourite application servers - WebSphere Application Server (WAS). One of the new features in IBM WebSphere Application Server version 8.5.0.1 lets you
take advantage of the Intel Advanced Encryption Standard (AES) New Instruction (AES-NI) set built-in (on-chip) within Intel Westmere and successor family of processors when dealing with AES cryptography. As per Intel, if exploited correctly, AES-NI not only boosts performance of cryptographic operations but also has security advantages. It may help to eliminate timing and cache based attacks. Since AES is currently one of the most popular block ciphers, wide range of applications are able to take benefit from these built-in instructions. Enabling this feature for WAS is easy. You need to define system property com.ibm.crypto.provider.doAESInHardware
and assign value true. You can do it by defining it under Generic JVM arguments setting through WebSphere Administration Console. However, here are few pre-requisites in order for it to work:

  • Java version: IBM SDK version 7 SR 3 or higher.
  • WAS version: 8.5.0.1 or higher
  • JCE provider: IBM JCE provider.
  •    Note: IBM PKCS11 provider does not use the Intel AES-NI instructions.
       Note: Update JCE provider in java.security file located under $JAVA_HOME/jre/lib/security
  • Processor: Intel Westmere and successor family of processors
In order to verify whether the underlying processor supports the AES-NI instructions or not, you can use the following system property to generate appropriate JCE tracing:
com.ibm.crypto.provider.AESNITrace=true
Basically, setting com.ibm.crypto.provider.doAESInHardware=true is no harm. if it is set and supported by underlying Intel processor, IBM JCE provider attempts to use AES-NI otherwise
it uses software module for cryptographic operations. Refer to IBM Knowledge Center for more information. For details on AES-NI, refer to
artcle Intel® Advanced Encryption Standard (Intel® AES) Instructions Set - Rev 3.01 by Shay Gueron (Intel) at https://software.intel.com/en-us/articles/intel-advanced-encryption-standard-aes-instructions-set

     If your platform is not WAS, but let's say, WebLogic on Solaris, you're well covered there as well. Starting from version 10 update 8, Solaris supports AES-NI using Solaris Cryptographic Framework (SCF). Java applications that use SunPKCS11 JCE provider will benefit AES acceleration for encryption/decryption through Intel AES-NI on Solaris. Very detail information about Java cryptography using AES-NI on Solaris can be found in Ramesh Nagappan's web log here. If you are looking Intel AES-NI support on Solaris in general, see Dan Anderson's blog Intel AES-NI Optimization on Solaris. Obviously, AES-NI support on Solaris is available only for Solaris x86 64-bit, running on a Intel microprocessor that supports the AES-NI instruction set, what about similar feature on Solaris powered by Sun/Oracle T series of processors? Guess what? Sun/Oracle SPARC processors are actually the leader in supporting hardware-accelerated crypto processing. Even though all T series chips supported some level of it, starting with T4, crypto became the part of the core instruction set accessible via non-privileged instructions. It is now one of the basic services offered by CPU. Very interesting blog about Java EE Application
Servers, SPARC T4, Solaris Containers, and Resource Pools by Jeff Taylor can be found here. If you are interested in utilizing SPARC on-chip crypto processor for applications hosted on WebSphere Application Server, this OTN white paper (http://www.oracle.com/technetwork/server-storage/sun-sparc-enterprise/documentation/ibm-websphere-sparc-t5-2332327.pdf) gives a lot of information. Read the section, Impact of SPARC T5 Hardware Encryption for Secure Traffic. Specifically, in this section it talks about how to use Oracle Solaris Kernel module called Kernel SSL proxy (KSSL), which can be used for offloading operations such as SSL/TLS. KSSL processes SSL traffic via Oracle SCF in the Kernel and thus improves the performance. White paper also clearly shows the performance comparison between on-chip and software crypto modules.

      This is just a small attempt to put together few options available in terms of on-chip accelerated crypto processing for Java applications. Obviously, there are number of other solutions in the market not covered here. If you are researching for suitable cryptographic solution for your next project, you may start by reviewing Validated FIPS 140-1 and FIPS 140-2 Cryptographic Modules list, maintained by National Institute of Standards and Technology (NIST) here.
      As seen from different tests available from individual tester or from vendor, on-chip crypto accelerator really performs well in comparison to software module if implemented correctly. So, if your platform supports, consider this option for your next project and get both performance and security benefits.

Possible Issues caused by CA Wily Introscope APM BRTM feature

After going live with CA APM on week-ends, right next Monday morning when site opened for business, help desk started getting string of complains. Either both or one of the following were reported:
1) Users experiencing spinning wheel when clicking submit on their web page.
2) Some users were losing their sessions and were prompted to re-login to their web application.

Initially, we were completely lost as we did not know what's causing this. Odd thing, not all users were experiencing the difficulties. And also server resources (CPU, memory) usage were normal. Performance was also normal for those users who did not experience the above mentioned issues.   After analyzing the web server access log, we were able to put some pattern based on the affected users' User Agent information. Some how, all the affected users were using Internet Explorer (IE) browsers.  IE 9/Trident 5.0, IE 10/Trident 6.0.  Even though more than 60% of our users were using Firefox (different versions), none of them reported any such issues, only IE users were affected. With this information in hand, we asked QA to test with different browser and they were able to reproduce the issues with IE and not with Firefox.

     Next step was to identify what was causing the issues. Again, looking into the Access log, we were able to identify that the request URL coming from those users just before they were kicked out had query parameter 'WilyCmd=cmdMetrics' like '/abc/def.jsp?WilyCmd=cmdMetrics'.
We also noticed that number of http(s) requests were considerably higher than usual. Basically almost each request url has duplicate request with the above mentioned query parameter appended. This information allowed us to pin-point the newly installed monitoring tool's BRTM Business Transaction Monitoring" feature (which injects Java Script into response header so that it's executed on user's browser to make another request with query parameter 'WilyCmd=cmdMetrics' appended), as primary suspect.  In our case, we would normally see in average 450,000 requests/day, but with this feature in place we recorded 750,000 requests in average per day.
You can read more about BRTM feature here:
https://wiki.ca.com/display/APMDEVOPS98/APM+for+Browser+Response+Time+Monitor

In the above mentioned link, you can also see a short statement about IE, it states, "... The CA BRTM JavaScript is loaded asynchronously so it does not block the loading and execution of any application JavaScript files and other components. However, Internet Explorer (6 through 9) does not wait on asynchronous loads before it generates the load event. This limitation can affect metric generation for Internet Explorer."

From our own investigation, we concluded that BRTM feature caused the above mentioned issues when using certain versions of IE browser. As a work-around, we had to turn off this feature until it is fixed by the vendor. Disabling BRTM feature is easy, you need to assign 'false' to 'introscope.agent.brtm.enabled' property in 'IntroscopeAgent.profile' file and restart the application server.

Lesson learned: until the issue is fixed, make sure you test your web application using all supported browsers before enabling this feature into your production environment.

WebSphere Application Server (WAS) High Performance Extensible Logging (HPEL) - some tips and tricks

As we know, WebSphere Application Server generally supports two types of logging mechanisms - Basic and High Performance Extensible Logging (HPEL).  If your application(s) logs with high frequency, there is a real possibility that system will be benefited (performance wise), if you enable HPEL. On the performance benefit of HPEL, IBM documentation says,
"HPEL has been designed and tested to significantly outperform the existing basic log and trace facility. One result is that the application server can  run with trace enabled while causing less impact to performance than tracing the same components using basic logging. Another result is that applications that frequently write to the logs might run faster with HPEL ..."
Reference: https://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/ctrb_HPELOverview.html

I have recently worked on a project to change logging from Basic to HPEL and based on my experience, I'm listing few tips and tricks below.


1. Enabling HPEL:

If you have just few App Servers in your server farm, then you can do this task manually. However, if you need to enable this for dozens of App Servers then, it's better to automate the task using script. WAS InfoCenter provides some fragments of scripts here and there, but nothing that you can take and execute. Below is a simple Jython script that I put together. It enables or disables HPEL based on passed argument to all servers within the given WAS Cell. You can also exclude specific server(s) as needed. Below script excludes 'nodeagent' and 'dmgr'.

enableHPEL.py
import sys
global AdminConfig
global AdminControl

action = sys.argv[0]

if action == 'true':
enableHPEL='true'
enableRAS='false'
else:
enableHPEL='false'
enableRAS='true'
#endElse


#Get Cell, nodes, and servers
cellName = AdminControl.getCell( )
cellID = AdminConfig.getid("/Cell:"+cellName+"/" )
nodes = AdminConfig.list("Node", cellID ).split()

for nodeID in nodes:
nodeName = AdminConfig.showAttribute(nodeID, "name")
servers = AdminConfig.list("Server", nodeID ).split()
for serverID in servers:
srvName = AdminConfig.showAttribute(serverID, "name" )
print "/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
if srvName == 'nodeagent' or srvName == 'dmgr':
continue
#endIf
HPELService = AdminConfig.getid("/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName+"/HighPerformanceExtensibleLogging:/");
RASLogService = AdminConfig.getid("/Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName+"/RASLoggingService:/");
if enableHPEL == 'true':
print "Enabling HPELService for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
AdminConfig.modify(HPELService, "[[enable true]]")
AdminConfig.modify(RASLogService, "[[enable false]]")
#endIf
if enableRAS == 'true':
print "Enabling RASLogService for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
AdminConfig.modify(HPELService, "[[enable false]]")
AdminConfig.modify(RASLogService, "[[enable true]]")
#endIf
print "Saving configuration for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
AdminConfig.save()
print "Synchronizing configuration for /Cell:"+cellName+"/Node:"+nodeName+"/Server:"+srvName
Sync1 = AdminControl.completeObjectName('type=NodeSync,process=nodeagent,node='+nodeName+',*')
synced = AdminControl.invoke(Sync1, 'sync')
print "Synchronization completed: "+synced
#endFor
#endFor

Above script is for full WAS profile, for WAS Liberty profile you have to do it manually for one WAS liberty server and may copy the configuration to other Liberty servers. You can find details here: https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_confHPEL.html?cp=SSEQTP_8.5.5
Or use Eclipse WAS Liberty tool as outlined here: https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/t_edit_server_config.html?cp=SSEQTP_8.5.5

If you need to customize HPEL setting for full WAS profile further (like changing size of log repository, location of repository, buffering etc.)  using wsadmin scripting, you can follow the following link for Full WAS profile:
http://www-01.ibm.com/support/knowledgecenter/SSD28V_8.5.5/com.ibm.websphere.base.doc/ae/ttrb_confHPELwsadmin.html
Similar options for WAS Liberty profile can be found here:
https://www-01.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_confHPEL.html?cp=SSEQTP_8.5.5

You can execute the above mentioned script as follows using wsadmin.sh[bat].
Enable HPEL:
wsadmin -username <username> -password <password> -lang jython -f ~/enableHPEL.py true

Disable HPEL:
wsadmin -username <username> -password <password> -lang jython -f ~/enableHPEL.py false

Note: provided script changes the configuration using wsadmin's AdminConfig object, which requires App Server to be restarted in order to take effect. You can use AdminControl object to do the same in runtime and that takes effect immediately.


2. Usability:

One of the major concerns with HPEL I found was usability issue. Once HPEL is enabled, the System logs and traces are written in binary, and people can not just open the log files using any text editor as they used to do it. There is an option to write plain text log also, but by enabling that option we loose certain performance benefit of HPEL. Even though IBM provides pretty good tools ('logViewer.sh[bat]' with full WAS profile, and 'binaryLog' with WAS Liberty profile) either to convert the binary logs into text format or just monitor the logs in real time, it's not easy to convince people to change their way of doing things. It takes time. One catch though, IBM's tools require WAS installed in the machine where tools are supposed to run. Not everybody who needs to look logs has WAS installed and your log repository server where you copy your binary log repositories for processing also may not have WAS installed. So, alternative is to convert logs on the App Server, where its generated  (may not be a good practice as it uses CPU cycles of production server)  and provide text logs to whoever need to review them.
I thought that it would be really convenient if we can have a tool to process these binary logs without WAS. And indeed, with little customization, you can use IBM's tool without WAS, which I am going to explain here:

2.1. Preparing custom 'logViewer' tool so that it can be run independent of WAS:

2.1.A) Create following directory structure under /tmp and copy corresponding file from your WAS installation:

/tmp/hpelviewer
bin
logViewer.sh (copy from WAS_INSTALLED_DIR/bin directory)
logViewer.bat (copy from WAS_INSTALLED_DIR/bin directory)
lib
bootstrap.jar (copy from WAS_INSTALLED_DIR/lib directory)
com.ibm.hpel.logging.jar  (copy from WAS_INSTALLED_DIR/plugins directory)
startup.jar  (copy from WAS_INSTALLED_DIR/lib directory)
properties
WsHeader  (copy from WAS_INSTALLED_DIR/properties directory)
WsLevels.properties  (copy from WAS_INSTALLED_DIR/properties directory)


2.1.B) Once the files are copied, open the logViewer.sh from /tmp/hpelviewer/bin and change as follow (actual script lines are in blue):
#Set JAVA_HOME at the top:
JAVA_HOME=/cygdrive/c/java/jdk1.7.0_45
#leave following line as it is:
binDir=`dirname "$0"`
#Comment the following line:
#. "$binDir/setupCmdLine.sh"
#Instead set the WAS_HOME:
setWASHome()
{
    CUR_DIR=$(pwd)
WAS_DIR="${binDir}"/..
cd "${WAS_DIR}"
WAS_HOME=$(pwd)
cd "${CUR_DIR}"  
}
setWASHome

#Leave following lines as it is:
if [ -f ${JAVA_HOME}/bin/java ]; then
    JAVA_EXE="${JAVA_HOME}/bin/java"
else
    JAVA_EXE="${JAVA_HOME}/jre/bin/java"
fi


#comment the following lines as we need to make them compatible to Unix and Cygwin:
#WAS_HEADER="-Dlogviewer.custom.header=${WAS_HOME}/properties/WsHeader"
#WAS_LEVELS="-Dlogviewer.custom.levels=${WAS_HOME}/properties/WsLevels.properties"
#Instead add the following lines:
PATH=.:"$JAVA_HOME"/bin/:"$JAVA_HOME"/jre/bin:$PATH
WAS_CLASSPATH="$WAS_HOME/properties:$WAS_HOME/lib/bootstrap.jar:$WAS_HOME/lib/com.ibm.hpel.logging.jar"
WAS_HEADER_PROP="${WAS_HOME}/properties/WsHeader"
WAS_LEVELS_PROP="${WAS_HOME}/properties/WsLevels.properties"
WAS_LOGGING="-Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Djava.util.logging.configureByServer=true"
# to be used as default log repository path
LOG_ROOT="$WAS_HOME/logs"

##
## Determine the platform and convert path accordingly
##
case $OSTYPE in
  cygwin)
    uname=CYGWIN_NT
        ;;
  *)
    uname=`uname`
esac

case $uname in
  CYGWIN_*)
WAS_HEADER_PROP=$(cygpath -w "$WAS_HEADER_PROP")
WAS_LEVELS_PROP=$(cygpath -w "$WAS_LEVELS_PROP")
LOG_ROOT=$(cygpath -w "$LOG_ROOT")
WAS_HOME=$(cygpath -w "$WAS_HOME")
        WAS_CLASSPATH=$(cygpath -pw "$WAS_CLASSPATH")
        PATH=$(cygpath -pw "$PATH")
        ;;
  *)
;;
esac
OSGI_INSTALL="-Dosgi.install.area=$WAS_HOME"
WAS_HEADER="-Dlogviewer.custom.header=$WAS_HEADER_PROP"
WAS_LEVELS="-Dlogviewer.custom.levels=$WAS_LEVELS_PROP"

#Finally, modify the Java execution command and options as follow:

${JAVA_EXE} \
-Dwas.install.root="$WAS_HOME" \
-Duser.install.root="$WAS_HOME" \
-Dlog.repository.root="$LOG_ROOT" \
$WAS_LOGGING \
$WAS_HEADER \
$WAS_LEVELS \
-classpath $WAS_CLASSPATH  com.ibm.ws.bootstrap.WSLauncher \
com.ibm.ws.logging.hpel.viewer.LogViewer "$@"

2.1.C) Save the changes.

Note: In the above script 'cygpath' has been used to make it compatible with Cygwin. for more information about 'cygpath' see http://www.cygwin.com/cygwin-ug-net/cygpath.html

2.2. Preparing custom 'binaryLog' tool (for Liberty profile) so that it can be run independent of WAS:

2.2.A) Create following directory structure under /tmp and copy corresponding file from your WAS installation:
/tmp/hpelviewer
bin
binaryLog
binaryLog.bat
tools
ws-binarylogviewer.jar
lib
com.ibm.ws.kernel.cmdline_1.0.10.jar
com.ibm.ws.logging.hpel_1.0.10.jar

2.2.B) customization of binaryLog is easier as IBM has written it in in such a way that it can be run under Cygwin as well as other Unix like platform. Just add the following two lines (actual lines to be added are in blue) in the beginning of the script.
WLP_INSTALL_DIR=..
#optionally set JAVA_HOME
JAVA_HOME=/cygdrive/c/java/jdk1.8.0

2.2.C) Save the file.

Note: follow similar tricks if you need to modify corresponding BAT files for Windows.
Once your customization is complete, just create a zip file (for example hpelviewer.zip) making hpelviewer as root directory of the zip,  and give it your users.
They can just extract the zip and and execute as follow. Only thing they need is Java installed on their machine.
Here is an exmple:
cd <hpelviewer_extract_dir>/hpelviewer/bin
./logViewer.sh -repositoryDir <repositoryDir>
Or:
cd <hpelviewer_extract_dir>/hpelviewer/bin
./binaryLog view  {serverName | repositoryPath}

Note: Since the property files, jar files and script files for WAS full profile and WAS Liberty profile are unique, you can actually create a single zip file by combining files for both. Here is how the file structure for combined  tool would look like:

hpelviewer
bin
logViewer.sh
logViewer.bat
binaryLog
binaryLog.bat
tools
ws-binarylogviewer.jar
lib
bootstrap.jar
com.ibm.hpel.logging.jar
startup.jar
com.ibm.ws.kernel.cmdline_1.0.10.jar
com.ibm.ws.logging.hpel_1.0.10.jar
properties
WsHeader
WsLevels.properties

Create a zip and give it to your users, so that they can use it to view binary logs generated by both WAS full profile or Liberty profile.

Happy HPEL !!!

Updating Property Files using Shell Script

Recently, one of my colleagues asked me some hints on how to read updated properties from one property file, search the same property in the target property file (files) and update/replace the corresponding property value (s) in the target property file (s) if property key matches. He wanted to do this operation using just the regular shell script, so No ANT or Java or Perl. I've put together a simple script file for him and decided to publish it here so that other people with similar needs can be benefited as I did not find any other similar posting.


Let's say, here is how the source property file looks like:

--------- sFile.properties --------
# Updated property values
connection.username=mssqluser
connection.password=*********
connection.hostname=thishost.domain
connection.time.format=yyyy-MM-dd HH:mm:ss

Let's say, here is how one of the target property files looks like:

--------- tFile.properties --------
# Connection properties
connection.username=xyz
connection.password=abc456789
connection.hostname=localhost

connection.time.format =yyyy-MMM-dd HH:mm:ss 

 

Here is how the shell script file look like:

---------- propertyUpdater.sh ------

#!/bin/bash
# Source input property file that provides new/updated property values.
sFile='/cygdrive/c/temp/Bscripts/sFile.properties'

# Target directory where multiple property files whose property value(s) need to be updated/replaced.
tDir='/cygdrive/c/temp/Bscripts/props'

# Reads each line and assigns value to variable _line, excluding all commented (starting with #) lines and all empty lines
for _line in `cat "$sFile" | grep -v -e'#' | grep -v -e'^$'`; do
    echo "Reading line: $_line from source file: $sFile"
    _key=`echo $_line | cut -d '=' -f1`
    _value=`echo $_line | cut -d '=' -f2`
    echo "Retrieved property key: $_key with value: $_value"
    # Comment following 'for' loop if you are using 'tFile' variable.
    for _file in `find $tDir -type f -print | grep ".properties$"`; do
       echo "Updating target property file: $tDir/$_file"
       sed -i "s/^$_key=.*/$_line/g" "$tDir/$_file"
       # for those properties which have space between key and '=' sign.
       sed -i "s/^$_key[ \t]=.*/$_line/g" "$tDir/$_file"
    done   
done


--------- tFile.properties after the update -----

# Connection properties
connection.username=mssqluser
connection.password=*********
connection.hostname=thishost.domain
connection.time.format=yyyy-MM-dd HH:mm:ss

 

Hope, it helps!