Solaris product registry

Things I have learnt about the product registry today.

1. Some products (eg Sun Studio 12, Sun Cluster) register themselves into this mysterious registry even though they are just a bundle of standard Solaris packages.
2. To inspect and administer packages in the registry, use the prodreg command.
3. The product registry database is a flat file - /var/sadm/install/productregistry
4. If the Sun Studio installer script takes forever to start up then killing any gconftool-2 processes on a headless box will greatly speed this up.
   

The old "Solaris du and df not matching" chestnut

Today we had a DBA come to us saying they had deleted a file but the filesystem space hadn't freed up. He'd killed the job that created the large (100GB+) file but the filesystem was still reporting as 100% full.

We tried pfiles against all the running processes and grepping for the filesystem in question. This took a while (over 800 processes) and didn't return any output.

Next stop was to look through a ps listing (grepping out anything obviously not very helpful) to see if anything jumped out. Needless to say this was not very productive.

fuser -cu against our filesystem gave only a handful of processes so we ran a pfiles against a likely looking one and it gave the following output;

root # pfiles 14648
14648: /usr/lib/ssh/sftp-server
Current rlimit: 256 file descriptors
0: S_IFSOCK mode:0666 dev:341,0 ino:3286 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX
1: S_IFSOCK mode:0666 dev:341,0 ino:3286 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX
2: S_IFSOCK mode:0666 dev:341,0 ino:8982 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX
3: S_IFSOCK mode:0666 dev:341,0 ino:3286 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX
4: S_IFSOCK mode:0666 dev:341,0 ino:3286 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(16384),SO_RCVBUF(5120)
sockname: AF_UNIX
5: S_IFREG mode:0644 dev:154,1008 ino:10873 uid:1200 gid:208 size:136971687936
O_RDONLY|O_LARGEFILE


The file opened by fd5 looks pretty big so we might have found the culprit. Another clue was that there was no filename listed for fd5. Next step, confirm the size of fd 5.
root # du -sh /proc/14648/fd/5
128G 5

OK, this is a likely candidate and once the sftp connection was stopped the space was indeed returned.


However, if we look at the dev: entry in the pfiles output it gives a clue as to what we should have grepped for in our original pfiles trawl. dev:154,1008 gives us major number of 154 and minor number 1008.

root # grep 154 /etc/name_to_major
vxio 154

Not very helpful. What if we look at this the other way round, what is the major/minor of our target filesystem?

root # ls -l /dev/vx/dsk/tempdg/db_raw
lrwxrwxrwx 1 root other 60 Apr 9 15:13 /dev/vx/dsk/tempdg/db_raw -> ../../../../devices/pseudo/vxio@0:tempdg,db_raw,1008,blk

root # ls -l /devices/pseudo/vxio@0:tempdg,db_raw,1008,blk
brw------- 1 root root 154, 1008 Mar 7 04:44 /devices/pseudo/vxio@0:tempdg,db_raw,1008,blk

So that double confirms what we knew and next time running pfiles and looking for processes with an open file against dev:major,minor but with no filename listed should do the trick.