Some of the usual tasks for a system administrator is to recover crashed systems or root passwords. Such kind of tasks usually requires that the system is started in some recovery mode in order to have access to the OS filesystems. On Linux, having the OS filesystems mounted may turn out into a tricky task when LVM is used.

This post will show briefly, how to have access to the basic OS logical volumes.

Boot the System

Using the CD-ROM, boot the system in single-user mode (RedHat has the “rescue” mode).

Scan the Physical Volumes

To mount the filesystems, first we need to make the system aware of the Volume Groups. We can do that by scanning the physical volumes.

# lvm pvscan

Scan the Volume Groups

Next, scan the volume groups

# lvm vgscan

Scan the Logical Volumes

At this point, the Logical Volumes should already be available. Just in case, check for the logical volumes

# lvm lvscan

The logical volumes will appear with the status of inactive, don’t panic!!

Activate the Logical Volumes

The following command will make the Logical Volume active.

# lvm lvchange -ay <logical_volume>

Note: if you want to perform a batch activation, use:

# lvm lvscan | awk ‘{ print $2 }’ | xargs -i lvm lvchange -ay {}

 Check your volumes again with lvscan and you should see all of them activated.

 Now you can fsck or mount your filesystems to change the root password or whatever you want to do.

Suppose that you got a VG with several disks, let’s say… 46 LUNs and now you’re in need not only to add more disks but also add larger disks. Now, imagine that you cannot work with the VG factor attribute for playing with the MAX PPs per PV because it would actually decrease the number of disks you can add to the VG.

It may be time to convert your VG to a Big or Scalable type and get rid of these limits.

On this article I will show you some of the magics this undocumented command can do for us.

The task

First of all, let’s take a look to the vg_hellpit for some important information. Observe that it has a PP Size of 128Mb, the MAX PVs is 128 and the MAX PPs per PV is 1016, which allows me to add LUNs up to 127Gb.

root@hellhound::/> lsvg vg_hellpit
VOLUME GROUP:       vg_hellpit VG IDENTIFIER:  00c061fa00004c000000012a73949e63
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      36744 (4703232 megabytes)
MAX LVs:            512                      FREE PPs:       63 (8064 megabytes)
LVs:                6                        USED PPs:       36681 (4695168 megabytes)
OPEN LVs:           6                        QUORUM:         24 (Enabled)
TOTAL PVs:          46                       VG DESCRIPTORS: 46
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         46                       AUTO ON:        yes
MAX PPs per VG:     130048
MAX PPs per PV:     1016                     MAX PVs:        128
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable

Now let’s consider that the world changes and needs changes…and that now 127Gb is no longer considered a big LUN. Let’s think a little bigger… 300Gb. Your old VG does not support it anymore. Yes, you have a few choices, but along with the simple technical stuff, comes the complexity of the customer, a production environment and the need to increase that same filesystems. And now you’re f**** up, as usual.

In order to convert your volume group to big or scalable, you will need to have enough free PP on each disk of the Volume Group you’re converting. Let’s take a brief look to vg_hellpit (I will show only a few disks) and see what we have.

root@hellhound::/> lsvg -p vg_hellpit | egrep “hdisk(10|11|12|13|14)”
hdisk10           active            799         0           00..00..00..00..00
hdisk11           active            799         0           00..00..00..00..00
hdisk12           active            799         0           00..00..00..00..00
hdisk13           active            799         0           00..00..00..00..00
hdisk14           active            799         5           00..00..00..00..05

Oh shit, 4 of my PVs have no free PPs available. The first logical though would be to run a reorgvg and get things done. But wait, we are dealing with a huge VG, (only 4Tb of data) and that task could take hours to complete.

Hmm, what if we could migrate 1 single PP from hdisk10, hdisk11, hdisk12 and hdisk13 to hdisk14?? We would get the space we want! And now you ask, but how??

The lmigratepp command will migrate exactly 1 PP between 2 given PVs from the same VG. If you ever had the curiosity to check what reorgvg does when it is running, you have probably seen this program running. The syntax is sort of simple:

Usage: lmigratepp -g VGid -p SourcePVid -n SourcePPnumber -P DestinationPVid -N DestinationPPnumber

Using the command

With that syntax in mind, let’s get the information we need to use the command.

First of all, we need the VGid.

root@hellhound::/> lsvg vg_hellpit | grep IDENTIFIER
VOLUME GROUP:       vg_hellpit             VG IDENTIFIER:  00c061fa00004c000000012a73949e63

Now, get the PVid from the source disk, let’s take hdisk10 as the source disk and hdisk14 as the destination disk.

root@hellhound::/> lspv -L hdisk10 | grep IDENTIFIER
PV IDENTIFIER:      00c061faf380a2b5 VG IDENTIFIER     00c061fa00004c000000012a73949e63

root@hellhound::/> lspv -L hdisk14 | grep IDENTIFIER
PV IDENTIFIER:      00c061fa3bc03a3e VG IDENTIFIER     00c061fa00004c000000012a73949e63

You must have noticed that the VG IDENTIFIER is the same for both disks, obviously.

Now, let us choose one PP to be migrated from hdisk10 to hdisk14. Let make the PP number 1. For this specific situation, it could be any arbitrary PP as the chvg command would reorganize the PPs as needed during the conversion.

We’re almost there. It is time to find where to place that PP we are moving from hdisk10 on hdisk14. The lspv command should give you that information.

root@hellhound::/> lspv -p hdisk14 | grep free
640-644   free    inner edge

From the result the PPs at positions 640, 641, 642, 643 and 644 are free on hdisk14. Now, let’s put it all together…

VGID: 00c061fa00004c000000012a73949e63 (vg_hellpit)
Source PVID: 00c061faf380a2b5 (hdisk10)
Source PP: 0001
Destination PVID: 00c061fa3bc03a3e (hdisk14)
Destination PP: 640 (the first free PP taken from lspv)

Running the command

The command is quiet it will not return you any messages upon success.

root@hellhound::/> lmigratepp -g 00c061fa00004c000000012a73949e63 -p 00c061faf380a2b5 -n 0001 -P 00c061fa3bc03a3e -N 640

Let’s check the free PPs on hdisk14 again.

root@hellhound::/> lspv -p hdisk14 | grep free
641-644   free    inner edge

Nice!! It changed!! Further, let’s check the PV map…

root@hellhound::/> lsvg -p vg_data_file | egrep “hdisk(10|11|12|13|14)”
hdisk10           active            799         1           01..00..00..00..00
hdisk11           active            799         0           00..00..00..00..00
hdisk12           active            799         0           00..00..00..00..00
hdisk13           active            799         0           00..00..00..00..00
hdisk14           active            799         4           00..00..00..00..04

Wooohooo. Cool! We moved 1 PP from hdisk10 to hdisk14 and now hdisk10 has 1 free PP and hdisk14 has 4 free PPs, instead of the 5 initial PPs. Repeating the process for the other disks will give you this final result:

root@hellhound::/> lsvg -p vg_data_file | egrep “hdisk(10|11|12|13|14)”
hdisk10           active            799         1           01..00..00..00..00
hdisk11           active            799         1           01..00..00..00..00
hdisk12           active            799         1           01..00..00..00..00
hdisk13           active            799         1           01..00..00..00..00
hdisk14           active            799         1           00..00..00..00..01

Hot stuff! You can now work on converting the VG to scalable mode and satisfy your DBA fellows with plenty of disk space.

Note: this post is based on GPFS 3.1. The procedures described here may be different on newer versions. Make sure you check the documentation for updates.

Adding a new disk to an existing GPFS filesystem

Adding a disk to a GPFS filesystem is a simple task to accomplish but may be kind of tricky for those who are not used to this feature. On this tutorial I am assuming that the procedure for configuring or recognizing a new disk on the system is well known.

You must be aware that the maximum disk size supported by the filesystem depends on the size of the disks used during the creation of the filesystem.

The basic steps to accomplish the whole task are:

1. Configure the disk and set its PVID on all nodes where it must be available.
2. Create the description file for the new disks
3. Create the NSD (Network Shared Disk)
4. Add the disk to the desired filesystem

Step 1 – Create the disk descriptor file

To create a NSD, you must have a file describing how your disk is going to be used on GPFS.

You can use the command mmlsnsd to show the current filesystems and the disks associated to them. That helps if you are not aware of how the system is organized.

The description file format is consisted of one single line per disk using the following format:

DiskName:PrimaryServer:BackupServer:DiskUsage:FailureGroup:DesiredName:StoragePool

Example:

hdisk1:hostmgr1:hostmgr2:dataAndMetadata:4001:nsddisk1:system

Here, hdisk1 will be managed by the hosts hostmgr1 and hostmgr2, it will contain both data and metadata, will belong to the failure group 4001, its NSD worldwide name will be nsddisk1 and it will reside on the system storage pool.

The NSD name (DesiredName) is a unique disk identifier among the nodes on the cluster. You can think of it like a PVID.

Notice that, when you specify the PrimaryServer and BackupServer, you are instructing your entire cluster to do the I/O through these 2 nodes. If all of your nodes on the cluster have SAN-attached disks and they have all the same disks configured, you should ommit the PrimaryServer and BackupServer instructions to get a local I/O behavior.

Step 2 – Create the Network Shared Disk

This step will prepare the disks described on the description file to be used on the GPFS

mmcrnsd -F <description_file>

Note: The mmcrnsd command also modifies the description file so that it can be used for other operations. If you cat the file you will see that the original description will be commented out and a new definition will appear below that line.

The new description file can be used for tasks like adding the disk to a filesystem or deleting it from the filesystem.

Step 3 – Add the new NSD to a filesystem

Using the description file modified on Step 2, add the disk to the filesystem. Make sure that the description file only contains the NSD entries you want to add to the filesystem. The mmadddisk command will add all disks found on the description file to the specified filesystem.

mmadddisk <filesystem_name> -F <description_file>

Removing a disk from a GPFS filesystem

Removing a disk from a GPFS filesystem is a simple task. With a simple command, the GPFS will attempt to reallocate all data from the disk being removed to the remaining disks on the filesystem prior to the removal. It will also, optionally, rebalance the filesystem.

It is recommended that disk removal tasks are performed when system I/O is low. The reallocation of data will demand I/O from the server.

If you have the disk descriptor file with the disks you want to remove:

mmdeldisk <filesystem> -F <description_file>

If you do not have the file, you can specify the NSD name of the disk:

mmdeldisk <filesystem> <NSD>

References

• GPFS V3.1 Administration and Programming Reference

• GPFS V3.1 Concepts, Planning, and Installation Guide

#
# remove the rpm lock files
#
rm /var/lib/rpm/__db*

#
# then rebuild the RPM database
#
rpm –rebuilddb

During the upgrade process we often face problems due to failed hardware. Thus a good step to take prior to install a new LIC version is to check whether the system is ready to receive a new LIC version.

On the HMC CLI this is easily accomplished w/ the following command.

#
# Check whether the system is ready for a firmware operation.
#
updlic -o k -m <frame>

hscroot@hmc:~> updlic -o k -m myframe
Licensed Internal Code Readiness Check detected the following errors:
HSCF0071E Communication error to Secondary Flexible Service Processor on myframe:
No connection exists.

The example above shows that the secondary FSP needs maintenance. This means that an upgrade of the system firmware will not work until we have that problem fixed.

I started digging up for errors. Nothing found on the LPAR or the VIOS. Then I noticed that whenever we tried accessing anything on /var (read/write) the system would hang. Well, simple! I just needed to rung a filesystem check on /var. But wait… how?

Whenever the system comes up, /var is mounted on boot and some services start using it (i.e. the error daemon). On this specific filesystem, I tried fsck -n, umount -f and a remount, all without success.

Then I decided to for the following plan:

• Create a new empty filesystem
• Restore /var backups from TSM (only the basic strucutre would fit)
• Bring the server up in single user
• Check and fix the original filesystem.

Followed the plan and by the time I was running fsck on hd9var, I got the following results:

root@hellhound::/> fsck /dev/hd9var
fsck: 0506-040 Cannot find the Vfs value for file system /dev/hd9var.

Looking out for that message I found the command getlvcb which is undocumented but is found on IBM’s Redbook AIX Logical Volume Manager from A to Z: Troubleshooting and Commands.

#
# getlvcb reads information from the Logical Volume Control Block.
#
getlvcb -AT /dev/hd9var

This returned nothing to me.

root@hellhound::/> getlvcb -AT /dev/hd9var
root@hellhound::/>

The solution would be much simpler than I thought. I just added a new temporary reference to hd9var on /etc/filesystems and

#
# chfs /tempvar will update the Logical Volume Control Block (lvcb) information.
#
root@hellhound::/> chfs /tempvar

#
# fsck now knew what it is dealing with and would be able to repair the filesystem
#
root@hellhound::/> fsck -y /tempvar
The current volume is: /dev/hd9var
“/etc/filesystems” 237 lines, 3945 characters
*** Phase 1 – Initial inode scan
Directory inode 4 has a node with an invalid sorting table (FIXED)
Superblock marked dirty because repairs are about to be written.
*** Phase 2 – Process remaining directories
*** Phase 3 – Process remaining files
*** Phase 4 – Check and repair inode allocation map
*** Phase 5 – Check and repair block allocation map
Inodes not connected to the root directory
tree have been detected.  Will reconnect.
File system is clean.
Superblock is marked dirty (FIXED)
All observed inconsistencies have been repaired.

After a double check, mounts and umounts, fixed the /etc/filesystems back to the correct state (/var over /dev/hd9var) and the system would boot up gracefully in multi-user mode.

$ startsrc -s sendmail
0513-059 The sendmail Subsystem has been started. Subsystem PID is 241830.

Everything appeared to be fine. However ps would reveal that sendmail was not running.

After struggling a while, I checked the console logs and found the message:

         0 Fri Nov 12 13:17:00 EST 2010 Invalid operation mode: use the “-bd” flag when invoking sendmail via SRC

Then I wondered how to check for the starting flags…and I found it!

$ odmget -q subsysname=sendmail SRCsubsys
SRCsubsys:
    subsysname = “sendmail”
    synonym = “”
    cmdargs = “”
    path = “/usr/lib/sendmail”
    uid = 0
    auditid = 0
    standin = “/dev/console”
    standout = “/dev/console”
    standerr = “/dev/console”
    action = 2
    multi = 0
    contact = 3
    svrkey = 0
    svrmtype = 0
    priority = 20
    signorm = 0
    sigforce = 0
    display = 1
    waittime = 20
    grpname = “mail”

How to change it? Simple, chssys!!

$ chssys -s sendmail -a “-bd”

After that sendmail started gracefully.

$ startsrc -s sendmail
0513-059 The sendmail Subsystem has been started. Subsystem PID is 6201366.

$ lssrc -ls snmpd | grep -i community

#
# Test the community name found
#

# HACMP 5.4
$ /usr/es/sbin/cluster/utilities/clsnmpinfo -c <community> -m dump

$ svmon -P -O pgsz=on

#
# List paging space usage by user
#

$ svmon -U -O pgsz=on

Compiling a new version would not be a viable solution as we’re dealing with customer’s servers lacking the necessary tools for compiling a new version.

After some research I came up by finding a way to get that information with the svmon command.

#
# List processes and files open
#
svmon -P -O filename=on | awk ‘{if(/Pid/) { getline; print $0; }; if(NF == 1) { print $0 }; }’

Ouput:

——————————————————————————-
6029462 dsmc             44499     7983        0    42374
/usr/tivoli/tsm/client/ba/bin/dsmc
/var/dsm_sched.log
/var/dsm_error.log
/usr/tivoli/tsm/client/lang/en_US/dsmclientV3.cat
——————————————————————————-
4915356 IBM.CSMAgentR    18754     7992        0    18091
/usr/sbin/rsct/bin/vac8/IBM.CSMAgentRMd
/var/ct/3484560716/registry/local_tree/IBM,ManagementSe
/var/ct/3484560716/lck/mc/RMIBM.CSMAgentRM
/var/ct/3484560716/registry/local_tree/IBM,ManagementSe
/var/ct/IBM.CSMAgentRM.stderr