msgbartop
Tips and Tricks site for advanced HP-UX Engineers
msgbarbottom

01 Oct 07 Welcome to hpux.ws

This site was created for two reasons. It was meant to document procedures, scripts and processes that I felt were important to myself and the greater HP-UX community.

This project is owned by ISN Corporation, a Subchapter S Corporation based in Chicago, Illinois, USA.

If you want to contribute to the site, let me know. hpuxadmin in gtalk hpuxconsulting in yahoo messenger

Regards,

Steven “Shmuel” Protter: Rosh Tzurim, Israel

Tags: , , ,

15 Jan 16 Automated setboot check and correction

When you use drd to patch and update systems offline to reduce downtime there is an unintended impact: setboot issues.

Using HP best practices after you boot the new image the setboot -a (alternate) and -p (primary) settings are often the same.

Below is an audit and correction script that helps you track the issue and limit manual intervention and the human error it can introduce:
myserv0:root > cat 349_bootconf
#!/bin/ksh
#########################################################################
# default_umask

#HPUX_SCRIPTS=/opt/depots/scripts/system_build/HPUX
#COMMON=/opt/depots/scripts/system_build/COMMON
# Load common environment
. /var/adm/bin/.scriptenv

#
# The point here is there should be an a primary boot disk
# and an alternate boot disk and they need to be different
#
pboot=$(/usr/sbin/setboot | grep ^Primary | awk ‘{ print $NF }’ | awk -F\/ ‘{print $NF}’ |
awk -F\) ‘{print $1}’);
aboot=$(/usr/sbin/setboot | grep ^Alternate |awk ‘{ print $NF }’|awk -F\/ ‘{print $NF}’ |
awk -F\) ‘{print $1}’);

if [ “$aboot” = “$pboot” ]
then
echo “NOTICE – ${hn} The primary boot disk ${pboot} is the same as the alternate boot disk ${aboot}”
else
echo “pass – The primary boot disk ${pboot} is the different than the alternate boot disk ${aboot}”

fi

if [ “$1” = “-y” ];then
echo “This may need to be remediated manually.”

#
# attempt to figure this out in an automated fashion
#
#
# Determine what the boot dg is.
# Try to use DRD configuration to determine the alt. boot disk and set it.
> /tmp/drdstatus.tfile.txt
/opt/drd/bin/drd status -x logfile=/tmp/drdstatus.tfile.txt
CLONE_DISK=$(awk ‘/Clone Disk: /{ print $NF}’ /tmp/drdstatus.tfile.txt | awk -F\/ ‘{ print $4 }’ | awk -F\) ‘{ print $1 }’);
echo “Clone disk is ${CLONE_DISK}”
setboot -a /dev/rdisk/${CLONE_DISK}
fi

echo “#### end report $0 ${sn} ####”

There is an audit script:

myserver0:root > /var/adm/bin/audit/349_bootconf
Executing HP-UX specific environment parameters…
NOTICE – The primary boot disk disk1972 is the same as the alternate boot disk disk1972
#### end report /var/adm/bin/audit/349_bootconf aappch0 ####

mysys03:root > setboot
Primary bootpath : 2/0/2/1/0/4/1.0x50060e80166f4202.0x4001000000000000 (/dev/rdisk/disk2490)
HA Alternate bootpath :
Alternate bootpath : 2/0/2/1/0/4/0.0x50060e80166f4212.0x4001000000000000 (/dev/rdisk/disk2490)

Autoboot is ON (enabled)
Hyperthreading : ON
: ON (next boot)

This is wrong but is a known issue that results from my patch methodology

First step to fixing is to confirm current booted details and drd details

mysys03:root > lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/disk/disk2490_p2 — Boot Disk
Boot: lvol1 on: /dev/disk/disk2490_p2
Root: lvol3 on: /dev/disk/disk2490_p2
Swap: lvol2 on: /dev/disk/disk2490_p2
Dump: lvol2 on: /dev/disk/disk2490_p2, 0

lvlnboot: Volume group not activated.
Cannot display volume group “/dev/vgAP1”.
lvlnboot: Volume group not activated.
Cannot display volume group “/dev/vgsapAP1”.
mysys03:root > cat /var/adm/bin/drd_data
DISK1=/dev/disk/disk2490
DISK2=/dev/disk/disk1951
mysys03:root > drd status

======= 01/14/16 14:15:17 PST BEGIN Displaying DRD Clone Image Information
(user=root) (jobid=mysys03)

* Clone Disk: /dev/disk/disk1951
* Clone EFI Partition: AUTO file present, Boot loader present
* Clone Rehost Status: SYSINFO.TXT not present
* Clone Creation Date: 01/07/16 15:00:27 PST
* Last Sync Date: None
* Clone Mirror Disk: None
* Mirror EFI Partition: None
* Original Disk: /dev/disk/disk2490
* Original EFI Partition: AUTO file present, Boot loader present
* Original Rehost Status: SYSINFO.TXT not present
* Booted Disk: Original Disk (/dev/disk/disk2490)
* Activated Disk: Original Disk (/dev/disk/disk2490)

======= 01/14/16 14:15:40 PST END Displaying DRD Clone Image Information
succeeded. (user=root) (jobid=mysys03)
Fix is currently manual

mysys03:root > setboot -a /dev/rdisk/disk1951
Alternate boot path set to 2/0/2/1/0/4/0.0x50060e80166f4212.0x4000000000000000 (/dev/rdisk/disk1951)
mysys03:root > setboot
Primary bootpath : 2/0/2/1/0/4/1.0x50060e80166f4202.0x4001000000000000 (/dev/rdisk/disk2490)
HA Alternate bootpath :
Alternate bootpath : 2/0/2/1/0/4/0.0x50060e80166f4212.0x4000000000000000 (/dev/rdisk/disk1951)

Autoboot is ON (enabled)
Hyperthreading : ON
: ON (next boot)

Possible automated fix (needs to be verified manually first use).

mysys00:root > ./349_bootconf -y
Executing HP-UX specific environment parameters…
NOTICE – The primary boot disk disk1972 is the same as the alternate boot disk disk1972
This may need to be remediated manually.

======= 01/14/16 14:43:22 PST BEGIN Displaying DRD Clone Image Information
(user=root) (jobid=aappch0)

* Clone Disk: /dev/disk/disk2236
* Clone EFI Partition: AUTO file present, Boot loader present
* Clone Rehost Status: SYSINFO.TXT not present
* Clone Creation Date: 01/14/16 14:00:36 PST
* Last Sync Date: None
* Clone Mirror Disk: None
* Mirror EFI Partition: None
* Original Disk: /dev/disk/disk1972
* Original EFI Partition: AUTO file present, Boot loader present
* Original Rehost Status: SYSINFO.TXT not present
* Booted Disk: Original Disk (/dev/disk/disk1972)
* Activated Disk: Original Disk (/dev/disk/disk1972)

======= 01/14/16 14:43:45 PST END Displaying DRD Clone Image Information
succeeded. (user=root) (jobid=aappch0)

Clone disk is disk2236
Alternate boot path set to 3/0/4/0/0/0/0/4/0/0/1.0x50060e80166f4273.0x4001000000000000 (/dev/rdisk/disk2236)
#### end report ./349_bootconf aappch0 ####
aappch0:root > ./349_bootconf
Executing HP-UX specific environment parameters…
Pass – The primary boot disk disk1972 is the different than the alternate boot disk disk2236
#### end report ./349_bootconf aappch0 ####
aappch0:root > setboot
Primary bootpath : 3/0/6/0/0/0/0/4/0/0/0.0x50060e80166f4213.0x4000000000000000 (/dev/rdisk/disk1972)
HA Alternate bootpath :
Alternate bootpath : 3/0/4/0/0/0/0/4/0/0/1.0x50060e80166f4273.0x4001000000000000 (/dev/rdisk/disk2236)

Autoboot is ON (enabled)
Hyperthreading : ON
: ON (next boot)

Tags: , ,

28 Oct 15 Keeping track of san disks

HP-UX does not make it easy to keep track of SAN presented disks. HBA switch ports are in short supply in many data centers. It is important for performance and reliability to be able to account for how many disks are presented to what HBA WWN ports.

This articles outlines a generic method of doing so. It is better than fcmsutil output but is based only on tools provided with the OS (with 1 small exception).

To make sharing easier, I will provide links to scripts. It is up to you to perform due diligence. The scripts write no data and do not change your system. They are provided without warranty under US Law by ISN Corporation.

I recommend against cutting and pasting scripts from this web page, errors are introduced. They are based on korn shell and do not work with bash shell. They probably work on POSIX shell but were not tested. They are specific to HP-UX B.11.31 but can if you wish be adapted to older versions of the OS.

Links to scripts are at the bottom of the post which is quite long.

Script names accurately describe their functionality
fcdisplaydev.ksh us a utility script designed to provide fcmsutil output to the other two scripts.

myserv1:root > cat pathcount_byhbaport.ksh
#!/bin/ksh
#
# Disk inventory by wwn
#
# 11.31 agile only
#
# Whole system version

## build an arry to hold wwn info
#
ap=0
zerod=0
idevice=$1

ls /dev/fc*| while read -r dv
do
/opt/fcms/bin/fcmsutil $dv | awk ‘/N_Port Port World Wide Name/{ print $(NF) }’ | while read -r wwpn
do
### echo ” array count ${ap} ..”
wwnarray[${ap}]=${wwpn}
wwncount[${ap}]=${zerod}
(( ap = ap + 1 ))
done
done
##echo ${#wwnarray[*]}
##echo ${#wwncount[*]}

calc_path()
{
### function to click counter of disks to port wwn
## set -x
## echo “calc_path $1 >>>”
fwwn=$1
fp=0
while [ ${fp} -le ${ap} ]
do
wwnport=${wwnarray[$fp]}
wwnportc=${wwncount[$fp]}
if [ “${fwwn}” = “${wwnport}” ]
then
##echo “updating wwn count ${fwwn} ..”
(( wwnportc = wwnportc + 1 ))
wwncount[${fp}]=${wwnportc}
fi
(( fp = fp + 1 ))
done
##set +x
}

if [ ! -z “$idevice” ]
then
dv=”/dev/rdisk/${idevice}”
scsimgr -p lun_map -D ${dv} | awk -F: ‘{ print $3 }’ | awk -F. ‘{ print $1 }’ | while read -r hp
do
##/var/adm/bin/fcdisplaydev.ksh ${hp}
wwnfound=$(/var/adm/bin/fcdisplaydev.ksh ${hp});
calc_path ${wwnfound}
done
else
ioscan -NfnCdisk | grep rdisk | grep -v p | awk ‘{ print $(NF) }’ | while read -r dv
do
### echo “cheking hba path disk … ${dv} ”
### scsimgr -p lun_map -D ${dv}
### scsimgr -p lun_map -D ${dv} | awk -F. ‘{ print $2 }’
scsimgr -p lun_map -D ${dv} | awk -F: ‘{ print $3 }’ | awk -F. ‘{ print $1 }’ | while read -r hp
do
##/var/adm/bin/fcdisplaydev.ksh ${hp}
wwnfound=$(/var/adm/bin/fcdisplaydev.ksh ${hp});
calc_path ${wwnfound}
done

done
fi
fp=0
echo “===========================================”
echo “= World wide port name: count ”
if [ ! -z “$idevice” ]
then
echo “= Individual device /dev/rdisk/${idevice} ”
fi
echo “===========================================”
while [ ${fp} -lt ${ap} ]
do
dv1=${wwnarray[$fp]}
dv2=${wwncount[$fp]}
echo “| ${dv1} : ${dv2} |”

(( fp = fp + 1 ))
done
echo “===========================================”

myserv1:root > cat fcdisplaydev.ksh
#!/bin/ksh
hwp=$1
ls /dev/fc*| while read -r dv
do
foundfc=$(/opt/fcms/bin/fcmsutil $dv | awk ‘/Hardware Path is/{ print $(NF) }’ | grep ${hwp} |wc -l);
if [ ${foundfc} -eq 1 ]
then
# echo “$dv to be checked.”
/opt/fcms/bin/fcmsutil $dv |awk ‘/N_Port Port World Wide Name/{ print $(NF)}’
fi
done

cat pathforalldisks.ksh
#!/bin/ksh

xpinfo -i > /tmp/xpinfo.txt

ioscan -NfnCdisk|grep rdisk | awk ‘{ print $NF }’ | grep -v _p | awk -F\/ ‘{ print $NF}’ | while read -r dsk
do
/var/adm/bin/pathcount_byhbaport.ksh $dsk
ldev=$(grep “${dsk} ” /tmp/xpinfo.txt | awk ‘{ print $6 }’ );
echo “LDEV: ${ldev}”

done

Script output. Modified to protect the security of the test systems.

fcmsutil

fcmsutil /dev/fcd0

Vendor ID is = 0x1077
Device ID is = 0x2422
PCI Sub-system Vendor ID is = 0x103C
PCI Sub-system ID is = 0x12DF
PCI Mode = PCI-X 133 MHz
ISP Code version = 5.6.5
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x018100
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500143800117ef3d
N_Port Port World Wide Name = 0x500143800117ef3c
Switch Port World Wide Name = 0x20810027f8a27cd4
Switch Node World Wide Name = 0x10000027f8a27cd4
N_Port Symbolic Port Name = myserv0_fcd0
N_Port Symbolic Node Name = myserv0_HP-UX_B.11.31
Driver state = ONLINE
Hardware Path is = 0/2/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
TYPE = PFC
NPIV Supported = YES
Driver Version = @(#) fcd B.11.31.1403 Dec 4 2013

There is a slight error, if you have a fix please share
./pathcount_byhbaport.ksh
calc_path[14]: wwnportc = wwnportc + 1 : bad number
===========================================
= World wide port name: count
===========================================
| 0x500143800117ef3c : 24 |
| 0x500143800117ef3e : 0 |
| 0x500143800117ef40 : 24 |
| 0x500143800117ef42 : 0 |
===========================================

LDEV info is specific ti Hitachi VSP xpinfo utility. You will have to adapt that code to other storage providers.

./pathforalldisks.ksh
calc_path[14]: wwnportc = wwnportc + 1 : bad number
===========================================
= World wide port name: count
= Individual device /dev/rdisk/disk172
===========================================
| 0x500143800117ef3c : 1 |
| 0x500143800117ef3e : 0 |
| 0x500143800117ef40 : 1 |
| 0x500143800117ef42 : 0 |
===========================================
LDEV: 03:f3

Link to http://www.hpux.ws/scripts/pathcount_byhbaport.ksh

Link to http://www.hpux.ws/scripts/pathforalldisks.ksh

Link to fcdisplaydev.ksh

All scripts are provided with no warranty. Use them at your own risk.

Tags: , , ,

13 Oct 15 debugging startup problems: trick of the day

Got an unfindable syntax error in startup?

We have the trick for you courtesy of JRF of the ITRC forums

/sbin/rc[102]: syntax error at line 188 : `”‘ unmatched
/sbin/rc1.d/S090hpvmguest[286]: /usr/bin/getconf: not found
/sbin/rc1.d/S090hpvmguest[286]: /usr/bin/grep: not found
/sbin/rc1.d/S105vparinit[63]: /usr/sbin/kcpath: not found

sh -vx /etc/rc.config 2>&1|more

+ fname=netconf
+ [ -f /etc/rc.config.d/netconf -a netconf != core -a netconf = netconf ]
+ . /etc/rc.config.d/netconf
/etc/rc.config[13]: Syntax error at line 188 : `”‘ is not matched.

Problem startup file in this case is netconf

Tags: , , ,

24 Jul 15 Single user vg00 reconfiguration for a san boot hp-ux system

San boot system.

HBA has to be replaced.

Then you have to boot single user mode to re-establish all your paths.

Procedure authored by my colleague Mahesh Koduru

Before you start make sure you have a current map file hosted on root filesystem.

“Reboot the server and follow these steps :
Interrupt the boot and boot the system in maintenance mode.

fs0:\EFI\HPUX> hpux

HPUX> boot -lm –lq vmunix This will bring the System into Maintance mod

#vgdisplay vg00 The VG should be in deactivated Mod

#ll /dev/*/group Collect the Group file

#vgexport -p -s -m vg00.map /dev/vg00 Keep the Map file in present directory , ie root

# ll vg00.map
#vgexport -v /dev/vg00

#mkdir -m 755 /dev/vg00
#mknod /dev/vg00/group c 64 0x030000 ##This is an example your major/minor number may vary
#vgimport -s -N -m vg00.map /dev/vg00 ## The -N is B.11.31 only to convert to agile storage
#vgchange -a y vg00
#mount -a Mount only Root filesystems

#setboot Check and correct setboot issues

#lvlnboot -v vg00 Check and correct lvlnboot issues , lvrmboot command can be used if needed

#/usr/sbin/lvlnboot -v
# lvlnboot -r /dev/vg00 Execute for fixing Boot Labels
# lvlnboot -r /dev/vg00/lvol3
# lvlnboot -b /dev/vg00/lvol1
# lvlnboot -s /dev/vg00/lvol2
# lvlnboot -d /dev/vg00/lvol2
# lvlnboot -v
# lvlnboot -R

Note: Comment the swap in fstab and then issue below command.
# shutdown -ry 0

23 Jul 15 HP-UX patch depot tutorial

Patch Depot tutorial

So you downloaded your QPK after doing swainv analysis of what patches your HP-UX server fleet needs. You have a bundle, but you do not want the default name bundle. No problem use the create script to customize:

The following command sets the bundle name (-b ) and the tilte of the bundle ( -t )
./create_depot_hpux.11.31 -b 201510HPUXPATCHMYCOMPANY -t MYCOMPANYFALL2015
… some output

A directory depot is created
cd depot

Check the bundle list. The names should be meaningful
swlist –l bundle –s $PWD
# Initializing…
# Contacting target “myhost”…
#
# Target: myhost:/Depots/tmp/depot
#

201510HPUXPATCHMYCOMPANY B.2015.07.23 MYCOMPANYFALL2015
QPKAPPS B.11.31.1503.411a Applications Patches for HP-UX 11i v3, March 2015
QPKBASE B.11.31.1503.411a Base Quality Pack Bundle for HP-UX 11i v3, March 2015

Next step copy them to a single install point:
swcopy -x enforce_dependencies=FALSE -x reinstall=TRUE -x write_remote_files=TRUE -d -s $PWD \* @ /Depots/B.11.31/2015FY_second

… some output deleted …

PHSS_43882.HVSD-KRN,r=1.0,a=HP-UX_B.11.31_IA,v=HP,fr=1.0,fa=HP-UX_B.11.31_IA
PHSS_43883.HSSN-KRN,r=1.0,a=HP-UX_B.11.31_IA,v=HP,fr=1.0,fa=HP-UX_B.11.31_IA
PHSS_43884.IGSSN-KRN,r=1.0,a=HP-UX_B.11.31_IA,v=HP,fr=1.0,fa=HP-UX_B.11.31_IA
PHSS_43886.GVSD-KRN,r=1.0,a=HP-UX_B.11.31_IA,v=HP,fr=1.0,fa=HP-UX_B.11.31_IA
* Selection succeeded.

* Beginning Analysis and Execution
* Session selections have been saved in the file
“/root/.sw/sessions/swcopy.last”.
WARNING: “myhost:/Depots/B.11.31/2015FY_second”: The software
dependencies for 4 products or filesets cannot be resolved.
* The execution phase succeeded for
“myhost:/Depots/B.11.31/2015FY_second”.
* Analysis and Execution succeeded.

NOTE: More information may be found in the agent logfile using the
command “swjob -a log myhost-2347 @
myhost:/Depots/B.11.31/2015FY_second”.

======= 07/23/15 14:49:03 PDT END swcopy SESSION (non-interactive)
(jobid=myhost-2347)

Next check the destination depot
swlist –l depot –s /Depots/B.11.31/2015FY_second

# Initializing…
# Contacting target “myhost”…
#
# Target: myhost:/Depots/B.11.31/2015FY_second
#

201510HPUXPATCHMYCOMPANY B.2015.07.23 MYCOMPANYFALL2015
QPKAPPS B.11.31.1503.411a Applications Patches for HP-UX 11i v3, March 2015
QPKBASE B.11.31.1503.411a Base Quality Pack Bundle for HP-UX 11i v3, March 2015

Then a problem comes up and you need to add a new patch to your already build depot, say PHSS_44116.depot a fix to issues starting hpvm guests.
swcopy -d -s $PWD/PHSS_44116.depot \* @ /Depots/B.11.31/2015FY_second

======= 07/23/15 14:58:12 PDT BEGIN swcopy SESSION (non-interactive)
(jobid=myhost-2348)

* Session started for user “root@myhost”.

* Beginning Selection
* Target connection succeeded for
“myhost:/Depots/B.11.31/2015FY_second”.
* Source: /Depots/B.11.31/PHSS_44116.depot
* Targets: myhost:/Depots/B.11.31/2015FY_second
* Software selections:
PHSS_44116.HPVM-CORE,r=1.0,a=HP-UX_B.11.31_IA,v=HP,fr=1.0,fa=HP-UX_B.11.31_IA
PHSS_44116.VIRT-PROVIDER,r=1.0,a=HP-UX_B.11.31_IA,v=HP,fr=1.0,fa=HP-UX_B.11.31_IA
* Selection succeeded.

* Beginning Analysis and Execution
* Session selections have been saved in the file
“/root/.sw/sessions/swcopy.last”.
* The analysis phase succeeded for
“myhost:/Depots/B.11.31/2015FY_second”.
* The execution phase succeeded for
“myhost:/Depots/B.11.31/2015FY_second”.
* Analysis and Execution succeeded.

NOTE: More information may be found in the agent logfile using the
command “swjob -a log myhost-2348 @
myhost:/Depots/B.11.31/2015FY_second”.

======= 07/23/15 14:58:14 PDT END swcopy SESSION (non-interactive)
(jobid=myhost-2348)

This method can be used to deliver any software built in depot format.
Let’s check the patch became a part of the depot:
myhost:root > swlist -s /Depots/B.11.31/2015FY_second
# Initializing…
# Contacting target “myhost”…
#
# Target: myhost:/Depots/B.11.31/2015FY_second
#

#
# Bundle(s):
#

201510HPUXPATCHMYCOMPANY B.2015.07.23 MYCOMPANYFALL2015
QPKAPPS B.11.31.1503.411a Applications Patches for HP-UX 11i v3, March 2015
QPKBASE B.11.31.1503.411a Base Quality Pack Bundle for HP-UX 11i v3, March 2015
#
# Product(s) not contained in a Bundle:
#

PHSS_44116 1.0 HPVM B.06.30 CORE PATCH
myhost:root >

Tags: , , ,

30 Apr 15 Quick guide to configuring secondary hp-ux swap

I always have to look at Google for this.

To make this site more comprehensive, here it is.

disk5 is designated as swap disk, size 128 GB. Server has 64 GB of RAM.

pvcreate /dev/rdisk/disk5
vgcreate -p 10 -l 10 /dev/vgswap /dev/disk/disk5

vgdisplay /dev/vgswap
vgdisplay -v /dev/vgswap
lvcreate -l 32767 -n lv_swap /dev/vgswap
swapinfo -tam
swapon -p 2 /dev/vgswap/lv_swap
vi /etc/fstab
/dev/vgswap/lv_swap … swap pri=2 0 0

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 24576 0 24576 0% 0 – 1 /dev/vg00/lvol2
dev 131068 0 131064 0% 0 – 2 /dev/vgswap/lv_swap
reserve – 290 -290
memory 62330 7566 54764 12%
total 217974 7856 210114 4% – 0 –

A reboot test is recommended where possible.

Tags: , , , ,

20 Apr 15 Serviceguard cluster creation:tip of the day

The following command conducts a cleanup and preparation step for any nodes named in a serviceguard cluster.

It is something I should have known, but learned recently:

cmpreparecl -n sgnode0 -n sgnode1

It does all the steps that would normally be done by hand.

Tags: , ,

03 Mar 15 Making sure MWA is running properly

What follows is a health check script that checks the installation status of HP Operations Agent and the run status of the two mwa daemons that measure performance.

When run with the -y parameter the script will attempt to correct installed status of HP Operations Agent.

If you want the script, please email me via the sites response form. Cutting and pasting from this site can be done, but may be a very frustrating endeavor.

I have added commentary to the script, which may introduce run errors if screen scraped.

myserva:root > cat 247_mwarun
#!/bin/ksh
############################################################################
# make sure scopeux is running, if not run if not installed install.ed

# Load common environment
. /var/adm/bin/.scriptenv
echo “. Checking for mwa software installed and running on ${hn}.”

is=myserva
if [ “${hn}” = “myserva” ]; then is=”myservb”;fi

ps -ef >/tmp/plist.txt

srun=$(awk ‘/scopeux/{print $NF}’ /tmp/plist.txt | wc -l);
mrun=$(awk ‘/midaemon/{print $NF}’ /tmp/plist.txt | wc -l);
swlist -l bundle TC097EA > /tmp/swlist.txt
mwainst=$(awk ‘/TC097EA/{ print $NF}’ /tmp/swlist.txt| wc -l);

#echo “scopeux procs running: $srun mwa installed: $mwainst”
if [ “$1” = “-y” ];then
CHANGES=1
fi

if [ ${srun} -eq 0 ] || [ ${mrun} -eq 0 ] ;then
if (($CHANGES));then
if [ ${mwainst} -ne 1 ]
then
### depot server location is in variable ${is}. This is an ignite depot server.
swinstall -x mount_all_filesystems=false -s ${is}:/Depots/B.11.31/2014midyear_depot TC097EA
rc=$?
echo “mwa TC097EA install succeeded checking sd on ${hn}…”
swlist -l bundle TC097EA > /tmp/swlist.txt
mwainst=$(awk ‘/TC097EA/{ print $NF}’ /tmp/swlist.txt| wc -l);
if [ ${mwainst} -eq 1 ];then echo ” pass – mwa NOW installed.” ;fi
optstat=$(/var/adm/bin/bdfmegs “/opt ” |awk ‘!/File-System/{print $5}’);
echo “${hn} /opt is ${optstat} full remediate if above 85% …”
else
mwa start all
fi
else
echo ” NOTICE – mwa not installed or scopeux/midaemon is not running on ${hn} .(-y will fix).”
fi
else
echo ” pass – mwa installed. scopeux/midaemon is running on ${hn}.”
fi
optstat=$(/var/adm/bin/bdfmegs “/opt ” |awk ‘!/File-System/{print $5}’);
echo “${hn} /opt is ${optstat} full remediate if above 85% …”
rm -f /tmp/plist.txt
rm -f /tmp/swlist.txt
echo “#### end report $0 ${sn} ####”

Script depends on Bill Hassell’s bdfmegs script. bdf can be made to work.
Typical output is:

myserv0:root > ./247_mwarun
Executing HP-UX specific environment parameters…
. Checking for mwa software installed and running on myserv0.
pass – mwa installed. scopeux/midaemon is running on mserv0.
myserv0 /opt is 68% full remediate if above 85% …
#### end report ./247_mwarun myserv0 ####
myserv0:root > mwa stop all

Shutting down Perf Agent collection software
Shutting down scopeux, pid(s) 28345
The Perf Agent collector, scopeux has been shut down successfully.
NOTE: The ARM registration daemon ttd will be left running.

OVOA is running. Not shutting down coda
myserv0:root > ./247_mwarun
Executing HP-UX specific environment parameters…
. Checking for mwa software installed and running on myserv0.
NOTICE – mwa not installed or scopeux/midaemon is not running on myserv0 .(-y will fix).
myserv0 /opt is 68% full remediate if above 85% …
#### end report ./247_mwarun myserv0 ####

Tags: ,

03 Mar 15 scopeux and midaemon don’t want to run

midaemon and scopeux combine to collect performance data on HP-UX.

They both need to be running to properly collect data.

These are part of a depot called measureware which is part of the base OS.

To see if it is installed:
swlist -l bundle TC097EA
myserv0:root > swlist -l bundle TC097EA
# Initializing…
# Contacting target “myserv0″…
#
# Target: myserv0:/
#

TC097EA 11.20.000 HP Operations Agent

If not installed, HP Operations Agent can be downloaded from HP if you have a software contract with HP.

It is also delivered as part of openview, which is a separately licensed product.

I recently implemented performance data collection on a fleet of 100+ servers where I work.

On three of the servers, the daemons refused to run normally.

The following error was recorded in the file /var/opt/perf/status.mi
Unable to find newly enabled CPU.
Please use -prealloc to allocate bufsets for all CPUs.

Here are the steps to implement.
mwa stop all
/opt/perf/bin/ovpa stop
/opt/perf/bin/pctl stop
perfstat

kill any processes gently identified as running in perfstat output.

Edit the file /etc/rc.config.d/ovpa
MIPARMS=”-prealloc=2 -pids 10000 -kths 10000 -smdvss 512M”
export MIPARMS

2 is the number of physical cpus in the box.
If present the file /var/opt/perf/datafiles/RUN should be deleted.


mwa start all
perfstat

Check back in 1 hour and one day that midaemon and scopeux are still running.
Check /var/opt/perf/datafiles for updated log files.

Tags: , , , ,

24 Feb 15 Server backplane layout for rx class servers

HP-UX hardware is a bit Byzantine in nature. The following script demystifies things a bit and allows us to work at remote data centers with accuracy on cable issues both fiber and copper.

If you want to use this script, use the contact form to reach out to me. Most people will find cutting and pasting from the website very frustrating.

I am also reaching out to the community, dare I say fan base for assistance. I am fairly certain that the fiber side accurately reports status of hba connections. I am fairly certain that the copper side needs to be improved.

This is tested on rx7640 and rx8640 systems. I am fairly confident that it will work on superdomes. I just do not happen to have any around to test on.

Script includes an environment plugin called .scriptenv . I will provide relevant code from it as some of what it does is customer specific and can not be released into the wild. If functionality is needed everywhere, I try to provide it via a centrally managed file. I provide functionality to both command prompts and scripting this way. It allows me to make the scripts generic and change one file with environment customizations.

Script is based on olrad -q output. It uses several deprecated utilities such as lanadmin and lanscan. Since I currently doubt HP-UX v4 is in the works, this should not be a problem.

The output then the script:

IP addresses and mac addresses have been changed to protect the innocent:

./backplane.layout.ksh
Executing HP-UX specific environment parameters…
m rx8640
ml ia64 hp server rx8640
——————————————————————–
Valid model found: rx8640
8-0-1-1 3/0/8/1
Nework info:
info lan3: 192.189.44.111 3/0/8/1/0/6/0 0x0025B3E86088 UP up(1) up(1)
info lan4: 192.101.0.155 3/0/8/1/0/6/1 0x0025B3E86089 UP up(1) up(1)
HBA info:
hba dev: /dev/fcd2
drviver state: ONLINE
fcms device: 3/0/8/1/0/4/0
WWN sys: 0x500143800542b4f9
WWP sys: 0x500143800542b4f8
Switch port: 0x20280027f8a26cd4
Switch node: 0x10000027f8a26cd4
hba dev: /dev/fcd3
drviver state: ONLINE
fcms device: 3/0/8/1/0/4/1
WWN sys: 0x500143800542b4fb
WWP sys: 0x500143800542b4fa
Switch port: 0x20a80027f8a09ff8
Switch node: 0x10000027f8a09ff8
——————————————————————–
8-0-1-7 3/0/2/1
Nework info:
info lan1: 192.8.0.95 3/0/2/1/0/6/0 0x0025B3E86066 UP up(1) up(1)
info lan2: 119.101.3.71 3/0/2/1/0/6/1 0x0025B3E86067 UP up(1) up(1)
HBA info:
hba dev: /dev/fcd0
drviver state: AWAITING_LINK_UP
fcms device: 3/0/2/1/0/4/0
WWN sys: 0x500143800542b4b5
WWP sys: 0x500143800542b4b4
Switch port: 0000000000000000
Switch node: 0000000000000000
hba dev: /dev/fcd1
drviver state: AWAITING_LINK_UP
fcms device: 3/0/2/1/0/4/1
WWN sys: 0x500143800542b4b7
WWP sys: 0x500143800542b4b6
Switch port: 0000000000000000
Switch node: 0000000000000000
——————————————————————–
8-0-1-5 3/0/6/0/0/0
Nework info:
ifconfig: no such interface
info lan7: 3/0/6/0/0/0/0/2/0/0/0 0x001E0B5186F0 UP up(1) up(1)
info lan8: 0.0.0.0 3/0/6/0/0/0/0/2/0/0/1 0x001E0B5186F1 UP up(1) up(1)
HBA info:
hba dev: /dev/fclp6
drviver state: ONLINE
fcms device: 3/0/6/0/0/0/0/4/0/0/0
WWN sys: 0x5001438001314ad5
WWP sys: 0x5001438001314ad4
Switch port: 0x20260027f8a09ff8
Switch node: 0x10000027f8a09ff8
hba dev: /dev/fclp7
drviver state: ONLINE
fcms device: 3/0/6/0/0/0/0/4/0/0/1
WWN sys: 0x5001438001314ad7
WWP sys: 0x5001438001314ad6
Switch port: 0x20950027f8a26cd4
Switch node: 0x10000027f8a26cd4
——————————————————————–
8-0-1-6 3/0/4/0/0/0
Nework info:
info lan5: 0.0.0.0 3/0/4/0/0/0/0/2/0/0/0 0x001E0B5186B4 UP up(1) up(1)
info lan6: 0.0.0.0 3/0/4/0/0/0/0/2/0/0/1 0x001E0B5186B5 UP up(1) up(1)
HBA info:
hba dev: /dev/fclp4
drviver state: ONLINE
fcms device: 3/0/4/0/0/0/0/4/0/0/0
WWN sys: 0x5001438001314a99
WWP sys: 0x5001438001314a98
Switch port: 0x200f0027f8a09ff8
Switch node: 0x10000027f8a09ff8
hba dev: /dev/fclp5
drviver state: ONLINE
fcms device: 3/0/4/0/0/0/0/4/0/0/1
WWN sys: 0x5001438001314a9b
WWP sys: 0x5001438001314a9a
Switch port: 0x20140027f8a09ff8
Switch node: 0x10000027f8a09ff8
——————————————————————–

Key output is this:
8-0-1-6
This is how the backplane is actually labeled.

cat backplane.layout.ksh
#!/usr/bin/ksh
#
# vgfiberswitch.ksh
# Steven “Shmuel” Protter steven.protter@hcl.com
#
. /var/adm/bin/.scriptenv
PATH=/opt/fcms/bin:${PATH}
####
# binary location
#
LS=/usr/sbin/lanscan
IS=/usr/sbin/ioscan
NW=/usr/sbin/nwmgr
IF=/usr/sbin/ifconfig
FU=/opt/fcms/bin/fcmsutil
LL=/usr/sbin/linkloop
LA=/usr/sbin/lanadmin

echo “m ${modinfo}”
echo “ml ${modinfol}”
echo “——————————————————————–”
if [ “${modinfo}” = “rx8640” ] ||[ “${modinfo}” = “rx7640” ]
then
echo “Valid model found: ${modinfo}”
else
echo “INVALID model found: ${modinfo}”
echo “Exiting return code 1 ………”
exit 1
fi
## determine number of multifunction cards to expect
ncards=$(olrad -n);
olrad -q| awk ‘/Yes/{ printf “%s %s\n”,$1,$2 }’ | while read -r slot hwp
do
echo “${slot} ${hwp}”
echo “Nework info:”
## network info
${LS} | grep “${hwp}” | awk ‘{ printf “%s %s %s %s %s\n”,$1,$2,$3,$4,$5 }’ | while read -r fwp mad nn st ln
do
### echo “hwp: ${fwp} mad: ${mad} netnum: ${nn} status ${st} lan:${ln}”
ipady=$(ifconfig ${ln} |awk ‘/inet/{print $2}’);
last1=$(lanadmin -g $nn |awk ‘/Administration Status/{ print $NF}’);
last2=$(lanadmin -g $nn |awk ‘/Operation Status/{ print $NF}’);
echo “info ${ln}: ${ipady} ${fwp} ${mad} ${st} ${last1} ${last2}”
done
echo “HBA info:”
ioscan -fnCfc -H ${hwp} | awk ‘/dev/{print $NF}’ |while read -r dv
do
nnwwn=$(${FU} ${dv}|awk ‘/N_Port Node World Wide Name/{print $NF}’);
npwwn=$(${FU} ${dv}|awk ‘/N_Port Port World Wide Name/{print $NF}’);
spwwn=$(${FU} ${dv}|awk ‘/Switch Port World Wide Name/{print $NF}’);
snwwn=$(${FU} ${dv}|awk ‘/Switch Node World Wide Name/{print $NF}’);
drst=$(${FU} ${dv}|awk ‘/Driver state/{print $NF}’);
fchw=$(${FU} ${dv}|awk ‘/Hardware Path is/{print $NF}’);
echo “hba dev: ${dv}”
echo “drviver state: ${drst}”
echo “fcms device: ${fchw}”
echo “WWN sys: ${nnwwn}”
echo “WWP sys: ${npwwn}”
echo “Switch port: ${spwwn}”
echo “Switch node: ${snwwn}”
done

echo “——————————————————————–”
done

relevant parts of .scriptenv

OSM=$(uname -s);
OSV=$(uname -r);

# determine the third octet
hn=$(hostname);

## /Depots/B.11.31/2014midyear_depot
## /Depots/B.11.23/2014midyear_depot

if [ “${OSM}” = “HP-UX” ]
then
echo “Executing HP-UX specific environment parameters…”
modinfo=$(model | awk ‘{print $NF}’);
modinfol=$(model | awk ‘{printf “%s %s %s %s\n”,$1,$2,$3,$4}’);
isvirt=$(model | awk ‘/Virtual/{print $NF}’ | wc -l);
fi

Script is provided without warranty.

Link to script..

Tags: , , ,