msgbartop
Tips and Tricks site for advanced HP-UX Engineers
msgbarbottom

01 Oct 07 Welcome to hpux.ws

This site was created for two reasons. It was meant to document procedures, scripts and processes that I felt were important to myself and the greater HP-UX community.

This project is owned by ISN Corporation, a Subchapter S Corporation based in Chicago, Illinois, USA.

If you want to contribute to the site, let me know. hpuxadmin in gtalk hpuxconsulting in yahoo messenger

Regards,

Steven “Shmuel” Protter: Rosh Tzurim, Israel

Tags: , , ,

30 Oct 14 EFI compatible ioscan: Tip of the day

Ever had a problem equating your disk device output for boot disks to what you see on the EFI prompt?

 

map -r

?????

# ioscan -fneCdisk

Class     I  H/W Path  Driver S/W State   H/W Type     Description
==================================================================
disk      0  0/2/1/0.0.0.0.0          sdisk   CLAIMED     DEVICE       HP      DG0300BALVP
/dev/dsk/c0t0d0     /dev/dsk/c0t0d0s3   /dev/rdsk/c0t0d0s2
/dev/dsk/c0t0d0s1   /dev/rdsk/c0t0d0    /dev/rdsk/c0t0d0s3
/dev/dsk/c0t0d0s2   /dev/rdsk/c0t0d0s1
Acpi(HWP0002,PNP0A03,200)/Pci(1|0)/Sas(Addr5000C5000A9C4FC9, Lun0)/HD(Part1,SigD061ACEA-862A-11DE-8000-D6217B60E588)/\EFI\HPUX\HPUX.EFI

Note: EFI is not agile aware you get nothing useful by trying ioscan -NfneCdisk

Tags: , , ,

29 Oct 14 Puzzle of the day. Strict logical volume prevents you from extending a filesystem

Thanks to Veerappan Dhandapani of HCL Technologies for bringing me this problem

You have 50 GB free in the volume group.

You try to extend the logical volume and you cant.

Basics:

1) bdf is the tool for measuring filesystem size not logical volume size

2)lvdisplay is the tool for measuring logical volume size.

Steps:

1) Extend the logical volume. That is why the previous two number points are important. In process after extending the logical volume size the lv size and file system size are not equal

2) Extend the file system.

root@mybox# vgdisplay -v vgdata | grep -i “Free PE”
Free PE 1784
Free PE 511
Free PE 0
Free PE 0
Free PE 0
Free PE 0
Free PE 1273
Free PE 0

root@mybox# lvdisplay /dev/vgdata/lv_prod_datastaging
— Logical volumes —
LV Name /dev/vgdata/lv_prod_datastaging
VG Name /dev/vgdata
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 245760
Current LE 7680
Allocated PE 7680
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict
IO Timeout (Seconds) default

root@mybox# lvextend -L 278528 /dev/vgdata/lv_prod_datastaging
lvextend: Not enough free physical extents available.

The problem here is the allocation policy.

lvchange -C n -s n /dev/vgdata/lv_prod_datastaging

See the man page for details. -C is for contiguous -s is for strict allocation.

We changed both to n(o).

Off camera we extended the logcal volume

lvdisplay /dev/vgdata/lv_prod_datastaging

— Logical volumes —
LV Name                     /dev/vgdata/lv_prod_datastaging
VG Name                     /dev/vgdata
LV Permission               read/write
LV Status                   available/syncd
Mirror copies               0
Consistency Recovery        MWC
Schedule                    parallel
LV Size (Mbytes)            302848
Current LE                  9464
Allocated PE                9464
Stripes                     0
Stripe Size (Kbytes)        0
Bad block                   on
Allocation                  non-strict
IO Timeout (Seconds)        default

We get the size from the logical volume size to feed into the Online JFS extend command so we do not have to do math.

fsadm -F vxfs -b 302848m /prod/datastaging

bdfmegs is Bill Hassell’s new and improved bdf

/var/adm/bin/bdfmegs /prod/datastaging
File-System                      Mbytes    Used   Avail %Used Mounted on
/dev/vgdata/lv_prod_datastaging   310.1g  215.7g   94.0g  70% /prod/datastaging

 

 

Tags: , , , , , , ,

27 Oct 14 EFI boot configuration rx7640 san boot vpmon example

EFI boot configuration

 

This example uses AUTO file to boot vpmon. This example happens to use HP-UX B.11.23 but the procedure is the same for B.11.31 though the disk paths will look different.

AUTO

boot /stand/vpmon

map –r

The bootable disks show Part1 on the disk list (highlighted). Local disks (which should not be used are SCSI, SAN disks show a WWN(also bold) output is truncated.

 

fs0   : Acpi(HWP0002,PNP0A03,100)/Pci(3|0)/Scsi(Pun6,Lun0)/HD(Part1,Sig80AE9EAE-9DB5-11DE-8000-D6217B60E588)

fs1   : Acpi(HWP0002,PNP0A03,100)/Pci(3|0)/Scsi(Pun6,Lun0)/HD(Part3,Sig80AE9F12-9DB5-11DE-8000-D6217B60E588)

fs2   : Acpi(HWP0002,PNP0A03,101)/Pci(1|0)/Pci(4|1)/Scsi(Pun6,Lun0)/HD(Part1,Sig2B540C3C-01A8-11DD-8002-D6217B60E588)

fs3   : Acpi(HWP0002,PNP0A03,0)/Pci(3|0)/Scsi(Pun6,Lun0)/HD(Part1,Sig89D238D4-9FB4-11E1-8002-D6217B60E588)

fs4   : Acpi(HWP0002,PNP0A03,0)/Pci(3|0)/Scsi(Pun6,Lun0)/HD(Part3,Sig89D23906-9FB4-11E1-8004-D6217B60E588)

fs5   : Acpi(HWP0002,PNP0A03,1)/Pci(1|0)/Pci(4|1)/Scsi(Pun5,Lun0)/HD(Part1,SigBE2E924E-A068-11E1-8002-D6217B60E588)

fs6   : Acpi(HWP0002,PNP0A03,1)/Pci(1|0)/Pci(4|1)/Scsi(Pun5,Lun0)/HD(Part3,SigBE2E928A-A068-11E1-8004-D6217B60E588)

fs7   : Acpi(HWP0002,PNP0A03,6)/Pci(1|0)/Pci(4|0)/Fibre(WWN50060E80166F4213,Lun4000000000000000)/HD(Part1,SigB790304C-43E0-11E3-8002-D6217B60E588)

fs8   : Acpi(HWP0002,PNP0A03,6)/Pci(1|0)/Pci(4|0)/Fibre(WWN50060E80166F4213,Lun4000000000000000)/HD(Part3,SigB7903088-43E0-11E3-8004-D6217B60E588)

fs9   : Acpi(HWP0002,PNP0A03,6)/Pci(1|0)/Pci(4|0)/Fibre(WWN50060E80166F4213,Lun4001000000000000)/HD(Part1,Sig1177FB50-5AD6-11E4-8002-D6217B60E588)

fsA   : Acpi(HWP0002,PNP0A03,6)/Pci(1|0)/Pci(4|0)/Fibre(WWN50060E80166F4213,Lun4001000000000000)/HD(Part3,Sig1177FB82-5AD6-11E4-8004-D6217B60E588)

fsB   : Acpi(HWP0002,PNP0A03,E)/Pci(1|0)/Pci(4|1)/Fibre(WWN50060E80166F4273,Lun4000000000000000)/HD(Part1,SigB790304C-43E0-11E3-8002-D6217B60E588)

fsC   : Acpi(HWP0002,PNP0A03,E)/Pci(1|0)/Pci(4|1)/Fibre(WWN50060E80166F4273,Lun4000000000000000)/HD(Part3,SigB7903088-43E0-11E3-8004-D6217B60E588)

fsD   : Acpi(HWP0002,PNP0A03,E)/Pci(1|0)/Pci(4|1)/Fibre(WWN50060E80166F4273,Lun4001000000000000)/HD(Part1,Sig1177FB50-5AD6-11E4-8002-D6217B60E588)

fsE   : Acpi(HWP0002,PNP0A03,E)/Pci(1|0)/Pci(4|1)/Fibre(WWN50060E80166F4273,Lun4001000000000000)/HD(Part3,Sig1177FB82-5AD6-11E4-8004-D6217B60E588)

 

To set a boot disk you need to fs#:

Then cd /EFI/HPUX and run the bcfg command

In this example, the boot disks for the vpar are fs7: and fsB:

 

fs7:

cd EFI

cd HPUX

fs7: \EFI\HPUX> bcfg boot add 1 hpux.efi “HPUX primary vpar boot”

fsB: \EFI\HPUX>  bcfg boot add 2 hpux.efi “HPUX alternate vpar boot”

 

Boot menu should look like this:

 

HPUX primary vpar boot

HPUX alternate vpar boot

LAN [Acpi(HWP0002,PNP0A03,101)/Pci(1|0)/Pci(6|1)/Mac(001F296EAD7

LAN [Acpi(HWP0002,PNP0A03,1)/Pci(1|0)/Pci(6|0)/Mac(001E0B83F25E)

LAN [Acpi(HWP0002,PNP0A03,1)/Pci(1|0)/Pci(6|1)/Mac(001E0B83F25F)

EFI Shell [Built-in]

Internal DVD (Upper) Cabinet 0 [Acpi(HWP0002,PNP0A03,100)/Pci(3|

Internal DVD (Lower) Cabinet 0 [Acpi(HWP0002,PNP0A03,0)/Pci(3|1)

LAN [Acpi(HWP0002,PNP0A03,101)/Pci(1|0)/Pci(6|0)/Mac(001F296EAD7

HPUX Vpar

Boot Option Maintenance Menu

 

Recommendation: test both your options.

 

AUTO file was edited on the OS as follows:

 efi_cp -d /dev/rdsk/c18t0d0s1 -u /EFI/HPUX/AUTO AUTO
<edit with vi>
 efi_cp -d /dev/rdsk/c18t0d0s1 AUTO /EFI/HPUX/AUTO

Repeat for both boot disks.

root@myvpmon0# lvlnboot -v
Current path “/dev/dsk/c20t0d0s2″ is an alternate link, skip.
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c18t0d0s2 (0/0/6/1/0/4/0.1.128.0.0.0.0) — Boot Disk
/dev/dsk/c20t0d0s2 (0/0/14/1/0/4/1.1.140.0.0.0.0)
Boot: lvol1     on:     /dev/dsk/c18t0d0s2
/dev/dsk/c20t0d0s2
Root: lvol3     on:     /dev/dsk/c18t0d0s2
/dev/dsk/c20t0d0s2
Swap: lvol2     on:     /dev/dsk/c18t0d0s2
/dev/dsk/c20t0d0s2
Dump: lvol2     on:     /dev/dsk/c18t0d0s2, 0

 

Tags: , , , , ,

22 Oct 14 fun vpar facts: see vpmon/monitor boot disk path.

When running vpmon/vpar virtualization, any changes made to the vpar database vpardb are automatically saved to:

/stand/vpdb

In fact you can boot vpmon of a disk, stand up the vpars, and reven remove the original vpmon system and boot disk and the database will maintain integrity on the remaining systems.

To sync/verify the vpardb

vparstatus -D /stand/vpdb

vparstatus -m

Console path:  No path as console is virtual
Monitor boot disk path:  1.0.4.0.0.0.0.4.0.0.0.1.128.0.0.0.0
Monitor boot filename:  /stand/vpmon
Database filename:  /stand/vpdb
Memory ranges used:  0×0/349069312 monitor
0x14ce6000/327680 firmware
0x14d36000/417792 monitor
0x14d9c000/925696 firmware
0x14e7e000/1417216 monitor
0x14fd8000/50495488 firmware
0×18000000/134213632 monitor
0x3ffec000/81920 firmware
0x79ffc000000/67108864 firmware
0x89ffc000000/67108864 firmware

will show you what hardware was actually used to boot the vpmon. You may find your monitor disk path no longer exists and your vpars are running just fine.

Tags: , , ,

15 Sep 14 Console information on an rx2660 (and probably others)

I found my console for an rx2660 ignite server offline. Was not able to ping it.

So I had to go a hunting on the Internet for a procedure to display the actual information from the running OS.

The IP addresses have been changed to protect the innocent.

/opt/propplus/bin/cprop -detail -c “Management Processor”

[Component]: Management Processor
[Table]: Management Processor
——————————————————-
****************************************************
[Hash ID]: Management Processor:c318dfa5x309885cd
< ClassName:HP_ManagementProcessor NameSpace:root/cimv2 >
[UniqueIdentifier]: 0.28.196.251.43.153
[ControllerType]: Unknown
[IPAddress]: 192.168.7.156
[URL]: https://192.168.7.156
[Dedicated]: Management
[CreationClassName]: HP_ManagementProcessor
[Name]: Management Processor
[EnabledState]: Enabled
[OperationalStatus]: OK
[FirmwareRevision]: F.002.025.000
[LEDColour]: 0
[LEDBlinkRate]: 0
[NICCondition]: 2
****************************************************

I found this output worthy of note as well:

 

hostname:root > /opt/propplus/bin/cprop -list
****************************************
STATUS | COMPONENT NAME
========================================
No status | Software Bundles
No status | System Summary
Normal | Cooling
Normal | Processors
Normal | FC HBA
No status | FRU Information
No status | Firmware Information
Normal | IOTree
Normal | Memory
No status | Mem Error
No status | Memory Utilization
Normal | Management Processor
Minor | Network Information
Minor | Power
No status | Process Information
No status | Software Products
Normal | Temperature
Normal | Voltage
No status | Boot Device Configuration
No status | Crash Dump Configuration
Unknown | RAIDSA HBA
Normal | SAS HBA
Unknown | SCSI HBA
Normal | Disk Drive
****************************************

Tags: , , ,

29 Aug 14 HPVM storage discovery & passthrough

This applies to hpvm, definitely version 4.00, probably all the way through version 6.2. HP-UX 11.31

 

You have a HPVM host named hpvm1 and it has a guest dguest1

 

Virtual Machine Name VM # Type OS Type State     #VCPUs #Devs #Nets Memory

==================== ===== ==== ======= ========= ====== ===== ===== =======

dguest0                 9 SH   HPUX   On (OS)        4   41     4   48 GB

dguest1                 8 SH   HPUX   On (OS)       4   41     4   48 GB

 

Storage team presents you the storage.

ioscan -Nfncdisk (or)

ioscan -fnCdisk

Disk turns out to be disk 5

hpvmmodify -p 8 -a disk:avio_stor::disk:/dev/rdisk/disk5

To pull back disk288 from a guest on:

rmsf -k -H <hardware path from ioscan -NfnCdisk>

hpvmmodify -p 8 -d disk:avio_stor::disk:/dev/rdisk/disk288

 

06 Aug 14 Custom naming your bi-annual HP-UX patch sets

Having a name associated with your bi-annual patch file makes it easier to inventory than the default BUNDLE

This is based on doing a QPK download which requires a support agreement. Output is from 11.23 it worked with 11.31 as well.

./create_depot_hpux.11.23 -b”201407HPUXPATCHMINE” -t 201407HPUXPATCHMINE

< .. lots of boring output >

# DEST -s the destination of the patch set.

cd depot

swcopy -x enforce_dependencies=false -s $PWD \* @ $DEST

< .. lots of boring output >

mygush0:root > swlist -l bundle -s $DEST
# Initializing…
# Contacting target “mygush0″…
#
# Target:  mygush0:/Depots/B.11.23/2014midyear_depot
#

201407HPUXPATCHMINE         B.2014.08.06   201407HPUXPATCHMINE
DNSUPGRADE                    C.9.3.2.13.0   BIND UPGRADE
FEATURE11i                    B.11.23.1009.083 Feature Enablement Patches for HP-UX 11i v2, September 2010
HPSIM-HP-UX                   C.07.03.00.00.03 HP Systems Insight Manager Server Bundle
HWEnable11i                   B.11.23.1012.085a Hardware Enablement Patches for HP-UX 11i v2, October 2010
JAVAOOB                       2.05.00        Java2 Out-of-box for HP-UX
NodeHostNameXpnd              B.11.23.01     Nodename, Hostname expansion enhancement
OpenSSL                       A.00.09.08za.002 Secure Network Communications Protocol
QPKAPPS                       B.11.23.1012.086a Applications Patches for HP-UX 11i v2, December 2010
QPKBASE                       B.11.23.1012.086a Base Quality Pack Bundle for HP-UX 11i v2, December 2010

More fun

Tags: , ,

20 Mar 14 Performance Measurement using measureware

Measureware Extract Documentation

 

 

 

Necessary Processes:

root@myserver:/tmp/fog> ps -ef | grep scopeux

    root  3246     1  0  Mar 19  ?         0:40 /opt/perf/bin/scopeux

List of possible reporting parameters:

/var/opt/perf/reptfile

 

Running datafiles live here:

root@myserver:/root> ll /var/opt/perf/datafiles

total 154528

-rw-r–r–   1 root       sys             31 Feb 21 08:50 RUN

-rw-r–r–   1 root       root           105 Sep 27  2012 agdb

-rw-r–r–   1 root       root             0 Sep 27  2012 agdb.lk

-rw-rw-rw-   1 root       root           168 Feb 21 18:28 classinfo.db

-rw-r–r–   1 root       root       4206652 Feb 21 18:20 logappl

-rw-r–r–   1 root       root       24054172 Feb 21 18:25 logdev

-rw-r–r–   1 root       root       6464936 Feb 21 18:25 logglob

-rw-r–r–   1 root       root        352232 Feb 21 10:55 logindx

-rw-r–r–   1 root       root            15 Sep 27  2012 logpcmd0

-rw-r–r–   1 root       root       32673802 Feb 21 18:28 logproc

-rw-r–r–   1 root       root       9740096 Feb 21 18:25 logtran

drwxr-xr-x   2 root       root            96 Sep 27  2012 lost+found

-rw-r–r–   1 root       root       1504540 Oct 31  2012 mikslp.db

 

Here is a typical template to generate data into a spreadsheet.

cat mwatemplate

REPORT “MWA Export on !SYSTEM_ID”

FORMAT ASCII

HEADINGS ON

SEPARATOR=”|”

SUMMARY=60

MISSING=0

DATA TYPE GLOBAL

YEAR

DATE

TIME

GBL_CPU_TOTAL_UTIL

GBL_CPU_SYS_MODE_UTIL

GBL_CPU_USER_MODE_UTIL

GBL_CPU_SYSCALL_UTIL

GBL_CPU_INTERRUPT_UTIL

GBL_PRI_QUEUE

GBL_CPU_CSWITCH_UTIL

GBL_SWAP_SPACE_UTIL

GBL_DISK_UTIL_PEAK

GBL_DISK_SUBSYSTEM_QUEUE

GBL_MEM_UTIL

GBL_MEM_CACHE_HIT_PCT

GBL_MEM_PAGEIN_RATE

GBL_MEM_PAGEOUT_RATE

GBL_MEM_SWAPIN_RATE

GBL_MEM_SWAPOUT_RATE

GBL_MEM_QUEUE

GBL_NET_PACKET_RATE

GBL_NET_OUTQUEUE

GBL_NETWORK_SUBSYSTEM_QUEUE

 

Here is a script to process the measureware output and generate a spreadsheet using the above template file:

 

#################### Begin Sample Extract Script ####################

#!/usr/bin/ksh

#

# Extract to spreadsheet midnight to 6 am

/opt/perf/bin/extract -xp -r /root/mwatemplate -g -b today 0:00 -e today 06:00 -f testfile.txt

####################  End Sample Extract Script  ####################

 

It is simple but effective. The command above looks at data between midnight and 6 am today. A look at the man page for extract will show you how to look at different data sets. There are an endless number of options. Choose template options based on the nature of the problem you are facing.

 

 

 

Tags: ,

06 Mar 14 Serviceguard complete cluster standup (Long article)

Serviceguard stand up Notes:

Important parts of the .ascii file.

CLUSTER_NAME prod_sgcluster

HOSTNAME_ADDRESS_FAMILY IPV4

FIRST_CLUSTER_LOCK_VG /dev/vgsgpackagename

## Is it safer to maybe use a dedicated SG Lock disk.

NODE_NAME sgclusternode01

NETWORK_INTERFACE lan900

HEARTBEAT_IP 192.168.208.209

NETWORK_INTERFACE lan901

HEARTBEAT_IP 10.43.128.209

# CLUSTER_LOCK_LUN

FIRST_CLUSTER_LOCK_PV /dev/disk/disk20

# Link Aggregate lan900 contains the following port(s): lan0,lan4

# Warning: There are no standby network interfaces for lan900.

# Link Aggregate lan901 contains the following port(s): lan8,lan12

# Warning: There are no standby network interfaces for lan901.

NODE_NAME sgclusternode02

NETWORK_INTERFACE lan900

HEARTBEAT_IP 192.168.209.98

NETWORK_INTERFACE lan901

HEARTBEAT_IP 10.43.128.210

# CLUSTER_LOCK_LUN

FIRST_CLUSTER_LOCK_PV /dev/disk/disk59

MEMBER_TIMEOUT 14000000

# Configuration/Reconfiguration Timing Parameters (microseconds).

AUTO_START_TIMEOUT 600000000

NETWORK_POLLING_INTERVAL 2000000

NETWORK_FAILURE_DETECTION INOUT

# NETWORK_AUTO_FAILBACK

# When set to YES a recovery of the primary LAN interface will cause failback

# from the standby LAN interface to the primary.

# When set to NO a recovery of the primary LAN interface will do nothing and

# the standby LAN interface will continue to be used until cmmodnet -e lanX

# is issued for the primary LAN interface.

NETWORK_AUTO_FAILBACK YES

SUBNET 192.168.208.0

IP_MONITOR ON

POLLING_TARGET 192.168.208.1

SUBNET 192.168.0.0

IP_MONITOR OFF

# Package Configuration Parameters.

# Enter the maximum number of packages which will be configured in the cluster.

# You can not add packages beyond this limit.

# This parameter is required.

MAX_CONFIGURED_PACKAGES 300

You will want to make sure both nodes of the cluster are running and the volume group that the lock disks belongs to is cluster aware.

NOTE: When standing up a one node cluster cmquerycl will remove lock disk configuration and a lock disk is not needed for such a configuration.

cmrunnode #on each node

vgchange –a n vgsgpackagename

vgchange –c y vgsgpackagename

cmquerycl -v -C prod_sgcluster.ascii -n sgclusternode01 -n sgclusternode02

cmcheckconf -v -C prod_sgcluster.ascii

cmapplyconf -v -C prod_sgcluster.ascii

Cluster should look roughly like this:

root@sgclusternode02:/root> cmviewcl

CLUSTER STATUS

prod_sgcluster up

NODE STATUS STATE

sgclusternode01 up running

sgclusternode02 up running

PACKAGE STATUS STATE AUTO_RUN NODE

root@sgclusternode02:/root>

Standing up your first package.

Scenario assumes you are standing up a package named sgpackagename.

mkdir –p /etc/cmcluster/packages/sgpackagename

cd /etc/cmcluster/packages/sgpackagename

Using your method of choice, copy in a package conference file and a task0 (external script to start Oracle. Choose your clone source carefully. Make sure you choose a working package to template. If you are standing up an SAP instance choose an SAP package to clone, sgpackagename is a standard database package and a better template candidate.

Permissions may matter:

oot@sgclusternode02:/etc/cmcluster/packages/sgpackagename> ll

total 176

-rw——- 1 root sys 71668 Dec 3 16:12 sgpackagename.conf

-r-x—— 1 root sys 11005 Dec 9 11:35 task0

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename>

Use the method of choice to modify your configuration files:

Here is what needs to be changed ( In bold ):

package_name sgpackagename

package_description “sgpackagename Instance”

package_type failover

node_name sgclusternode01

node_name sgclusternode02

ip_subnet 192.168.208.0

ip_address 192.168.209.48

Change the bold values below for each file system.

fs_name /dev/vgsgpackagename/oraarch

fs_server “”

fs_directory /data/oracle/sgpackagename/arch

fs_type “vxfs”

fs_mount_opt “-o ioerror=mwdisable,largefiles,delaylog,cio”

fs_umount_opt “”

fs_fsck_opt “”

The actual data may vary. The fs_mount_opt are important and may change. The parameter cio is necessary for Oracle to have asynchronous I/O.

external_script /etc/cmcluster/packages/sgpackagename/task0

Create the mount points and change permission to oracle:dba. Again there are many ways to do this, your mileage may vary. You can check return codes if you are particularly precise, echo $?

awk ‘/^fs_dir/{print $NF}’ sgpackagename.conf | while read –r mp

do

mkdir –p $mp

ll –d $mp

chown oracle:dba $mp

ll –d $mp

done

Your final output from the ll –d command should look this

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/arch

drwxr-xr-x 4 oracle dba 96 Dec 10 11:40 /data/oracle/sgpackagename/export

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/undo1

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/redo1

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/redo2

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/ctrl1

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/ctrl2

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/data1

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/index1

drwxr-xr-x 4 oracle dba 96 Dec 10 13:59 /apps/oracle/sgpackagename

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/system

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/temp1

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/users

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/tools

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/fra

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/diag

drwxr-xr-x 3 oracle dba 96 Dec 9 11:04 /data/oracle/sgpackagename/admin

edit the task0 file

function start_commmand

su oracle -c “/apps/oracle/general/sh/change_backup_mode.ksh sgpackagename cold end”

Note this section is different for SAP. Change the instance name.

function stop_command

#### Database startup script, normally created by dba#####

su oracle -c “/apps/oracle/general/sh/change_backup_mode.ksh sgpackagename cold begin”

Comment out (in bold) the start/stop code so first package run only stands up the filesystems:

case ${1} in

start)

# start_command $*

# exit_val=$?

;;

stop)

# stop_command $*

# exit_val=$?

;;

Detect your storage from storage system.

ioscan –fnNC disk

# detects the storage

For this example we detected the following agile devices:

disk59

disk60

disk61

disk62

disk63

disk64

disk65

disk66

get the EMC serial number of each of the disks and match it against the serial number list given to you by storage in the dashboard request.

This is a bit tedious so I created a simple script for the job:

dislev.ksh

#!/usr/bin/ksh

args=$#

if [ $args -ne 1 ];then echo “1 argument last part of disk ex disk60″;exit 1;fi

dv=$1

diskinfo /dev/rdisk/${dv}

ldev=$(/usr/bin/inq -nodots -sym_wwn | grep “$dv ” | grep -v rdsk |awk ‘{print $3}’ |awk ‘{print substr($0,length($0) – 3,length($0))}’);

echo “LDEV is: ${ldev}”

root@sgclusternode02:/root/shuffle> ./disldev.ksh disk59

SCSI describe of /dev/rdisk/disk59:

vendor: EMC

product id: SYMMETRIX

type: direct access

size: 154176000 Kbytes

bytes per sector: 512

LDEV is: 4F2F

The size is a reality check. You can also use inq utility to get the size. By using a second utility (diskinfo) this creates a natural double check. There are probably a dozen different right ways to complete this task.

My disk list is kept for verification purposes in a text file root@sgclusternode02:/root/shuffle> cat request.610.dlist.txt

610 is the request from the stand up rquest.

awk ‘{print $NF}’ request.610.dlist.txt | while read –r dvn

do

pvcreate $dvn

rc=$?

echo “RC from pvcreate is ${rc}”

done

Create the volume group using LVM 2.2

vgcreate –V <version number> -s <pe size> -S <Max size in TB> <volume group name> <disk list>

vgcreate -V 2.2 -s 32 -S 2t vgsgpackagename /dev/disk/disk59 …. /dev/disk/disk66

The next step is best scripted:

<command> -n name –L <size in MB> <volume group name>

lvcreate -n oraadmin -L 10000 vgsgpackagename

echo $?

newfs -F vxfs -o largefiles /dev/sgpackagename/roraadmin

echo $?

.. repeat for every file system defined in the storage dashboard request .. I have scripted this entire process based on dashboard input and will provide the script as an appendix.

Volume group must be exported in preview mode:

vgexport -v -s -p -m vgsgpackagename.map vgsgpackagename

Use scp to copy the map file to the failover node

On failover node:

vgimport -v -s -N -m vgsgpackagename.map vgsgpackagename

vgchange –a y vgsgpackagename

vgdisplay –v vgsgpackagename (Look for logical volumes disks being correct)

vgchange –a n vgsgpackagename

vgchange –c y vgsgpackagename (Must be successful check return code $? )

vgchange –c n vgsgpackagename

On primary node (either will work but being consistent is helpful).

vgchange –c y vgsgpackagename

cd /etc/cmcluster/packages/sgpackagename

cmcheckconf –P sgpackagename.conf

(When all cmcheckconf issues are resolved)

cmapplyconf –P sgpackagename.conf

cmrunpckg –v –n <node_hostname> sgpackagename

cmmodpkg –e sgpackagename

cmviewcl (Look for your package to be running)

Error logs live: /var/adm/cmcluster/log/<packagename>.log

Setting up asynchronous I/O

root@sat1hcmdb090:/root> getprivgrp

global privileges: CHOWN

dba: MLOCK

getprivgroup is a display command use command:

setprivgrp dba MLOCK

Standing up your first package:

Make sure you are in the directory: /etc/cmcluster/packages/sgpackagename

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmcheckconf -P sgpackagename.conf

cmcheckconf: Verification completed. No errors found.

Use the cmapplyconf command to apply the configuration.

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmapplyconf -P sgpackagename.conf

One or more of the specified packages are running. Any error in the

proposed configuration change could cause these packages to fail.

Ensure configuration changes have been tested before applying them.

Modify the package configuration ([y]/n)? y

Completed the cluster update

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmmodpkg -e sgpackagename

cmmodpkg: Completed successfully on all packages specified

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmviewcl -v

CLUSTER STATUS

prod_sgcluster up

NODE STATUS STATE

sgclusternode01 up running

Cluster_Lock_LVM:

VOLUME_GROUP PHYSICAL_VOLUME STATUS

/dev/vgsgpackagename /dev/disk/disk20 up

Network_Parameters:

INTERFACE STATUS PATH NAME

PRIMARY up LinkAgg0 lan900

PRIMARY up LinkAgg1 lan901

NODE STATUS STATE

sgclusternode02 up running

Cluster_Lock_LVM:

VOLUME_GROUP PHYSICAL_VOLUME STATUS

/dev/vgsgpackagename /dev/disk/disk59 up

Network_Parameters:

INTERFACE STATUS PATH NAME

PRIMARY up LinkAgg0 lan900

PRIMARY up LinkAgg1 lan901

PACKAGE STATUS STATE AUTO_RUN NODE

sgpackagename up running enabled sgclusternode02

Policy_Parameters:

POLICY_NAME CONFIGURED_VALUE

Failover configured_node

Failback manual

Node_Switching_Parameters:

NODE_TYPE STATUS SWITCHING NAME

Primary up enabled sgclusternode01

Alternate up enabled sgclusternode02 (current)

Other_Attributes:

ATTRIBUTE_NAME ATTRIBUTE_VALUE

Style modular

Priority no_priority

Check that your file systems are mounted (About a dozen different ways to do this):

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> bdf | grep sgpackagename

102400000 91742 95913999 0% /data/oracle/sgpackagename/arch

102400000 13178819 83644864 14% /data/oracle/sgpackagename/export

102400000 91742 95913999 0% /data/oracle/sgpackagename/undo1

10256384 20006 9596612 0% /data/oracle/sgpackagename/redo1

10256384 20006 9596612 0% /data/oracle/sgpackagename/redo2

10256384 20006 9596612 0% /data/oracle/sgpackagename/ctrl1

10256384 20006 9596612 0% /data/oracle/sgpackagename/ctrl2

307200000 141939 287866940 0% /data/oracle/sgpackagename/data1

204800000 116841 191890469 0% /data/oracle/sgpackagename/index1

20480000 3972498 15475861 20% /apps/oracle/sgpackagename

10256384 20006 9596612 0% /data/oracle/sgpackagename/system

102400000 91742 95913999 0% /data/oracle/sgpackagename/temp1

10256384 20006 9596612 0% /data/oracle/sgpackagename/users

10256384 20006 9596612 0% /data/oracle/sgpackagename/tools

102400000 91742 95913999 0% /data/oracle/sgpackagename/fra

10256384 20006 9596612 0% /data/oracle/sgpackagename/diag

10256384 20006 9596612 0% /data/oracle/sgpackagename/admin

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> mount | grep sgpackagename

/data/oracle/sgpackagename/arch on /dev/vgsgpackagename/oraarch ioerror=mwdisable,largefiles,delaylog,cio,dev=80000002 on Mon Dec 9 11:41:17 2013

/data/oracle/sgpackagename/export on /dev/vgsgpackagename/oraexport ioerror=mwdisable,largefiles,delaylog,cio,dev=80000009 on Mon Dec 9 11:41:17 2013

/data/oracle/sgpackagename/undo1 on /dev/vgsgpackagename/oraundo1 ioerror=mwdisable,largefiles,delaylog,cio,dev=8000000c on Mon Dec 9 11:41:17 2013

/data/oracle/sgpackagename/redo1 on /dev/vgsgpackagename/oraredo1 ioerror=mwdisable,largefiles,delaylog,cio,dev=80000005 on Mon Dec 9 11:41:17 2013

/data/oracle/sgpackagename/redo2 on /dev/vgsgpackagename/oraredo2 ioerror=mwdisable,largefiles,delaylog,cio,dev=80000004 on Mon Dec 9 11:41:17 2013

/data/oracle/sgpackagename/ctrl1 on /dev/vgsgpackagename/oractrl1 ioerror=mwdisable,largefiles,delaylog,cio,dev=80000006 on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/ctrl2 on /dev/vgsgpackagename/oractrl2 ioerror=mwdisable,largefiles,delaylog,cio,dev=80000007 on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/data1 on /dev/vgsgpackagename/oradata1 ioerror=mwdisable,largefiles,delaylog,cio,dev=80000003 on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/index1 on /dev/vgsgpackagename/oraindex1 ioerror=mwdisable,largefiles,delaylog,cio,dev=80000008 on Mon Dec 9 11:41:18 2013

/apps/oracle/sgpackagename on /dev/vgsgpackagename/orabin ioerror=mwdisable,largefiles,delaylog,nodatainlog,dev=80000001 on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/system on /dev/vgsgpackagename/orasystem ioerror=mwdisable,largefiles,delaylog,cio,dev=8000000a on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/temp1 on /dev/vgsgpackagename/oratemp1 ioerror=mwdisable,largefiles,delaylog,cio,dev=8000000b on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/users on /dev/vgsgpackagename/orausers ioerror=mwdisable,largefiles,delaylog,cio,dev=8000000d on Mon Dec 9 11:41:18 2013

/data/oracle/sgpackagename/tools on /dev/vgsgpackagename/oratools ioerror=mwdisable,largefiles,delaylog,cio,dev=8000000e on Mon Dec 9 11:41:19 2013

/data/oracle/sgpackagename/fra on /dev/vgsgpackagename/orafra ioerror=mwdisable,largefiles,delaylog,cio,dev=8000000f on Mon Dec 9 11:41:19 2013

/data/oracle/sgpackagename/diag on /dev/vgsgpackagename/oradiag ioerror=mwdisable,largefiles,delaylog,cio,dev=80000010 on Mon Dec 9 11:41:19 2013

/data/oracle/sgpackagename/admin on /dev/vgsgpackagename/oraadmin ioerror=mwdisable,largefiles,delaylog,cio,dev=80000011 on Mon Dec 9 11:41:19 2013

Conduct failover test (Pre-installation)

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmhaltpkg -v sgpackagename

Disabling automatic failover for failover packages to be halted.

Halting package sgpackagename

Successfully halted package sgpackagename

One or more packages or package instances have been halted.

The failover packages have AUTO_RUN disabled and no new instance can start automatically. To allow automatic start, enable AUTO_RUN via cmmodpkg -e <package_name>

cmhaltpkg: Completed successfully on all packages specified

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmviewcl

CLUSTER STATUS

prod_sgcluster up

NODE STATUS STATE

sgclusternode01 up running

sgclusternode02 up running

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE

sgpackagename down halted disabled unowned

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmrunpkg -v -n sgclusternode01 sgpackagename

Running package sgpackagename on node sgclusternode01

Successfully started package sgpackagename on node sgclusternode01

cmrunpkg: All specified packages are running

root@sgclusternode02:/etc/cmcluster/packages/sgpackagename> cmviewcl

CLUSTER STATUS

prod_sgcluster up

NODE STATUS STATE

sgclusternode01 up running

PACKAGE STATUS STATE AUTO_RUN NODE

sgpackagename up running disabled sgclusternode01

NODE STATUS STATE

sgclusternode02 up running

Re run your mount test on the node the package is now running on (see above).

You may wish to leave package failover set to disabled to avoid surprises.

Email your DBA give them the file system(s) to do installation. Flip the database dashboard request to “Application installation & configuration”

Notify your IA/PM via email.

Be prepared to run root.sh when your DBA needs it run.

Steps after DBA completion

Modify task 0 uncomment the 4 lines of code you commented.

case ${1} in

start)

start_command $*

exit_val=$?

;;

stop)

stop_command $*

exit_val=$?

Check your package changes:

cmcheckconf –P sgpackagename.conf

cmapplyconf –P sgpackagename.conf

Make sure backups are configured before the next step. Also have storage test DR backup of LUNS if applicable (Model and dev usually do not have DR).

Re-run the failover tests. Check log files:

/var/adm/cmcluster/log/sgpackagename.log

Check database run status.

ps –ef | grep sgpackagename

Lessons learned:

· DBA’s do not necessarily understand clustering. If they say the start/stop script works but you are getting errors in Serviceguard testing, ask them to check the return code (echo $?). Explain t the DBA that Serviceguard will not function correctly if it receives non-zero return codes.

· Try not to modify task0 to default to zero return codes. This defeats the purpose of Serviceguard checking return codes.

· Never run pvcreate –f (If you feel the need, deconstruct your volume group and use pvremve)

· Never run cmdisklock -f reset <disk device>, this destroys all data on the lock disk and generally there is data on the lock disk. Note: Maybe a better design would be a dedicated lock disk.

Appendix 1 Additional automation.

This is optional, but making processes repeatable is important so you might like this.

While validating the request on the storage dashboard, I noted (and slightly massaged) the data to make it machine (awk) readable. Scripts are local, I use them in /root/shuffle.

root@sgclusternode02:/root/shuffle> cat request.610.data.txt

sgclusternode01|NEW|/apps/oracle/sgpackagename | 20 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/arch | 100 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/data1 | 300 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/redo2 | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/redo1 | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/ctrl1 | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/ctrl2 | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/index1 | 200 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/export | 100 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/SYSTEM | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/temp1 | 100 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/undo1 | 100 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/users | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/tools | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/fra | 100 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/diag | 10 GB

sgclusternode01|NEW|/data/oracle/sgpackagename/admin | 10 GB

The following script reads the earlier request.{request number}.dlist.txt file to get its device list, reads the data.txt file and completely automates the VG build including the map export.

Script requires the instance name (not the vg name, it does that itself) and the request number from the dashboard. Use it or not, the choice is yours.

#!/usr/bin/ksh

args=$#

if [ $args -ne 2 ];then echo “2 arguments instance name and request ### required”;exit 1;fi

inn=$1

vgn=”vg${inn}”

req=$2

disklist=”request.${req}.dlist.txt”

actonreturn()

{

rcc=$1

if [ $rcc -ne 0 ];then echo “FAIL FAIL $0 FAILED”; exit 1;fi

}

devl=””

# pvcreate

awk ‘{ print $1 }’ ${disklist} | while read -r dv

do

devr=”/dev/rdisk/${dv}”

echo “pvcreate ${devr}”

pvcreate ${devr}

rc=$?

echo “pvcreate rc:${rc}”

actonreturn ${rc}

devl=”$devl /dev/disk/${dv}”

done

#vgcreate -V 2.2 -s 32 -S 1t <vg name> disks in /dev/disk/disk### format

echo “vgcreate -V 2.2 -s 32 -S 1t ${vgn} ${devl}”

vgcreate -V 2.2 -s 32 -S 2t ${vgn} ${devl}

rc=$?

echo “vgcreate rc:${rc}”

actonreturn ${rc}

infile=”request.${req}.data.txt”

if [ -f ${infile} ]

then

echo “- pass – file ${infile} found”

else

echo “- FAIL – file ${infile} MIA”

exit 1

fi

fstabstub=fstab.${vgn}

echo “##### New Oracle Instance vg ${vgn} ####” > ${fstabstub}

innu=$(echo ${inn} | awk ‘{print toupper($0)}’);

echo “Instance Name: ${inn} ${innu}”

awk ‘{print}’ ${infile} | while read -r DL

do

# echo “Data Line ${DL}”

mp=$(echo ${DL} | awk -F\| ‘{ print $3}’);

# echo “mount point is ${mp}”

mpszg=$(echo ${DL} | awk ‘{ print $3}’);

mpszm=”${mpszg}000″

# echo “mount point size in GB ${mpszg} in fake MB ${mpszm}”

mpname=$(echo ${mp} | awk -F\/ ‘{print $NF}’);

if [ "${mpname}" = "${innu}" ]

then

mpname=”orabin”

else

mpname=$(echo ${mpname} | awk ‘{print tolower($0)}’);

mpname=”ora${mpname}”

fi

mkdir -p ${mp}

chown oracle:dba ${mp}

# echo “mount point name is ${mpname}”

echo “lvcreate command is : lvcreate -n ${mpname} -L ${mpszm} ${vgn}”

#### uncomment for realrun ###

lvcreate -n ${mpname} -L ${mpszm} ${vgn}

rc=$?

actonreturn ${rc}

echo “newfs command is : newfs -F vxfs -o largefiles /dev/${vgn}/r${mpname}”

newfs -F vxfs -o largefiles /dev/${vgn}/r${mpname}

rc=$?

actonreturn ${rc}

done

vgexport -v -s -p -m ${vgn}.map ${vgn}

1.

26 Feb 14 scsimgr task: removing a LUN

scsimgr is the tool of choice for getting rid of storage that is in the HP-UX kernel and the underlying storage has been pulled by your storage administrator. In this case the lun has been marked as FAILED.

HP-UX 11.31, any Operating Environment, though these tasks go easier if the OE is more current.

scsimgr lun_map> /tmp/lun_map.out

 

        LUN PATH INFORMATION FOR LUN : /dev/rdisk/disk3436

 

Total number of LUN paths     = 2

World Wide Identifier(WWID)    = 0×60000970000192605240533039434139

 

LUN path : lunpath2275

Class                         = lunpath

Instance                      = 2275

Hardware path                 = 0/0/0/5/0/0/1.0x500009720851e1ad.0x446e000000000000

SCSI transport protocol       = fibre_channel

State                         = UNOPEN

Last Open or Close state      = FAILED

 

LUN path : lunpath2274

Class                         = lunpath

Instance                      = 2274

Hardware path                 = 0/0/0/8/0/0/1.0x500009720851e191.0x446e000000000000

SCSI transport protocol       = fibre_channel

State                         = UNOPEN

Last Open or Close state      = FAILED

 

scsimgr replace_wwid -D /dev/rdisk/disk3436

echo $?

Zero is good, non-zero probably means you are booting.