Notes from the Ether

Simplified instructions for breaking & rebuilding raid devices (after the disk has been replaced physically) … assuming c0t0d0 was your busted disk.

###COPY YOUR DRIVE CONFIGURATION (TOC) from slice 2
prtvtoc /dev/rdsk/c0t1d0s2 | fmthard -s – /dev/rdsk/c0t0d0s2
###REBUILD YOUR RAID DATABASES IN SLICE 7
# metadb -a -f -c 3 c1t0d0s7

###DETACH THE SUBMIRROR
# metadetach -f d5 d15
d5: submirror d15 is detached

####CLEAR THE DEVICE CONFIGURATION FROM THAT SLICE
# metaclear d15
d15: Concat/Stripe is cleared

###INITIALIZE A NEW RAID DEVICE ON THE SLICE
# metainit d15 1 1 c1t0d0s5
d15: Concat/Stripe is setup

###ATTACH THE SLICE TO THE EXISTING RAID DEVICE TO RESYNC
bash-2.05# metattach d5 d15
d5: submirror d15 is attached

WASH RINSE REPEAT ON REMAINING SLICES

Thankfully, Netapp doesn’t require us to stick with one speed on our snapmirrors forever and ever.  We have the ability to use more/less bandwidth as appropriate, which is nice … particularly when we’re initializing snapmirror volumes that require a TON of data to be sent across the WAN.

The command is simple:

snapmirror throttle <bw in KB> filer:dest_volume

Substitute a 0 for “bw in KB” and you’ve opened up the pipe for complete consumption, or choose a number that’s suitable.  The Netapp docs say “KiloBytes” … so you’ll have to do your own math to find your number.  Happy transferring. 

Okay, I just wasted a ton of time on this … and it’s stupid!

 Our developers needed the CF8 server to begin development on a new project.  No problem!  Install the server on a 30-day trial and let it run until I can get caught up and purchase the license.  Done and done.  Everyone’s happy.

Fast forward a week, the license has arrived and I need to install it.  But where?  I go round and round in the admin looking for something in the left nav that will show me where to update the license.  Nothing.  Search Adobe.com.  Nothing.  Search Google.  Thank goodness for google, but this STILL took way too long.

Answer:  Top right corner, click on the blue “i” for ‘System Information.’  Inside that screen, it’s a simple text box that you update and you’re done.  Seriously, that’s it!

I’ve found this little legacy app on our network that runs on MSDE.  Of course, there’s no documentation on it, so to restor the app on a new server requires me to reset the sa login.

Here’s how:

from the command prompt, execute:

osql -E -Q “sp_password NULL, ‘newpassword’, ‘sa’” #### replace newpasswd with a real password.

 Thats it!  Then you can login and test your new password with : osql -U sa -P newpasswd.

That should get you in, then you can run exec sp_helpdb and see your databases.

Today I had to set up some SQL Log Shipping, but the source server had transaction logs from the past few months that were never cleaned out because SQL 2005 was running with a “Full” backup model (as opposed to “simple”, that never cleaned out the transaction log files after the backups ran.

 So, today, I had to clean out the logs.  I started by switching and running a simple backup.  This was easy enough in the “Options” of hte database, but also could’ve avoided this step by running the following command:

BACKUP LOG <DatabaseName> TO DISK = ‘<BackupFile>’ or in context: BACKUP LOG TestDB TO DISK=’C:\TestDB1.bak’

That backs up the logfiles, and allows me to truncate the data that’s been backed up.  Now, to shrink the logfile back down so that it doesn’t eat my entire server, I run the DBCC Shrinkfile command:

 DBCC SHRINKFILE (<FileName>, <TargetSize>)   The target size is in MB, so to shrink my logfile to 500 MB, I ran this:

DBCC SHRINKFILE (DB_logfile, 500)That did the trick.  Now my logfile is back down to 500 MB and I can start working on log shipping so that I have some redundancy.

So todays’ battle was installing linux (this particular install was Oracle Unbreakable) onto an IBM blade, that just didn’t want to load the X installer when running the installation in graphical mode.  For those reading who are saying “just install in text mode” .. yes, I get it .. but that’s not the point.  I went all the way thru text mode and the install crashed anyway … so I was on a mission with the video mode thing.

I tried installing with the vga=790 tag at the linux install prompt, but .. no go.  It would show certain screens, but not others.  I have a feeling it has to do with Linux misdiagnosing the type of Video cards in the system. 

I had to settle for the linux text install and then configuring the system post-install so that XWindows would work properly.  Here’s  the change I had to make:

1. Log in as root and using vi (or any other editor), edit the /etc/X11/xorg.conf file. Search for the following Driver line in the Device section:

   Driver “ati”

   Replace “ati” with “vesa” The final section looks similar to this:

   Section “Device”
   Identifier “Videocard0″
   Driver “vesa”
   VendorName “Videocard vendor”
   BoardName “ATI ES1000″
   EndSection

2. Turn off dpms. While still editing the xorg.conf file, search for the following dpms option line:

   Option “dpms”Change the option line to:

   Option “dpms” “off”

  Again,this was all done post-install, so this was all done in the OS.  After I was done with these changes, I edited my /etc/inittab file to boot to runlevel 5 (graphical level).  Running “init 6″ forced a reboot and from there, I was able to see my X install in the blade management center.

After a few months of running RHEL guests on VMWare, we came to the realization that they just don’t maintain their clocks very well.  A little research showed that this is a commong problem in VMWare.  It sounds like it’s attributed to the fact that the machines don’t have access to the “entire” processor … I’m not really sure how that works, but I’ll believe it until someone tells me otherwise.

 Regardless, NTP is necessary in this situation because we’ve got apps that rely on accurate time running on that server.  Enter VMware tools.

By mounting the VMWare tools CD .. (it’s not a real CD, just software in the mgmt suite), and installing it, we can synch the VMWare Guest OS to the ESX host that it’s running on.  (The ESX hosts are also automatically running NTP that synchs to VMWare’s own time-server cluster).

 After mounting the CD, I ran: 

vmware-config-tools.pl

This runs thru a real basic config script that configures the software.  After that, I ran:

vmware-toolbox &

This actually launches the the management tool in XWindows for configuration.  In there, I just clicked the “Synch NTP with ESX Host.” button and my problem was solved.

Okay, I don’t really like just ripping stuff off like this … but this was perfect and I didn’t want to lose track of it.

Brandon Hutchinson has some awesome stuff in his site … this was one of them:

I will configure the version of Sendmail (8.11.7) that was installed by default on a Solaris 8 system. Sendmail will be configured to use header sender and envelope sender address masquerading and a “smart host.”

On Solaris 9, substitute main.mc for main-v7sun.mc in the instructions.

1. Change to the directory containing the Sendmail configuration files.
cd /usr/lib/mail/cf

2. Make a copy of main-v7sun.mc as sendmail.mc, and make modifications to sendmail.mc.
cp main-v7sun.mc sendmail.mc

3. Configure sendmail.mc. In this example, we want to use the “smart host” smart_host.example.com and masquerade both the header sender and envelope sender addresses as example.com.

vi sendmail.mc

Insert the following entries before the MAILER lines:

define(`SMART_HOST’, `smart_host.example.com’)
MASQUERADE_AS(`example.com‘)
FEATURE(`masquerade_envelope’)

4. Build the sendmail.cf file from the sendmail.mc file.
# /usr/ccs/bin/m4 ../m4/cf.m4 sendmail.mc > sendmail.cf

5. Test the sendmail.cf file.
/usr/lib/sendmail -bt -C./sendmail.cf

Make sure that root is an “exposed user.” An exposed user is a user that will not be masqueraded. This is used when accounts, such as root, are not unique across systems.

> $=E
root

Test header sender and envelope sender masquerading:
> /tryflags HS
> /try esmtp user@host.example.com
Rcode = 0, addr = user@example.com

> /tryflags ES
> /try esmtp user@host.example.com
Rcode = 0, addr = user@example.com

6. Backup the existing sendmail.cf file.
cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.date

7. Install the new sendmail.cf file.
cp sendmail.cf /etc/mail/sendmail.cf

8. Sendmail the Sendmail process a SIGHUP to begin using the new configuration file.
kill -HUP `head -1 /var/run/sendmail.pid`

When I created this most recent batch of webservers, I got a little stingy with the drive space. I skimped on the “/” partition, and as you can see, I’m paying for it now:

[root@server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2.5G 2.3G 80M 97% /
/dev/mapper/VolGroup00-LogVol04
29G 9.1G 19G 33% /apps
/dev/sda1 99M 13M 82M 14% /boot
none 4.0G 0 4.0G 0% /dev/shm
/dev/mapper/VolGroup00-LogVol03
2.9G 37M 2.8G 2% /opt
/dev/mapper/VolGroup00-LogVol02
1008M 933M 25M 98% /var

Here’s how I was able to solve my problem.

1) Turn off the VM and expand the drive. (I’m adding 5 Gigs to “/” for a total of 7.5). I’m doing this thru the vmware VirtualCenter management software, but the documentation recommends the following command:

vmware-vdiskmanager -x 10GB /path/machineName.vmdk

2) After powering the VM back on, I now have to create a partition that will incorporate the new space:

[root@server ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 5874.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (5222-5874, default 5222):
Using default value 5222
Last cylinder or +size or +sizeM or +sizeK (5222-5874, default 5874):
Using default value 5874

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Command (m for help): p

Disk /dev/sda: 48.3 GB, 48318382080 bytes
255 heads, 63 sectors/track, 5874 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 5221 41833260 8e Linux LVM
/dev/sda3 5222 5874 5245222+ 8e Linux LVM

Command (m for help): q

I now have 3 partitions, including my most recent, partition 3 that has been formatted to use LVM. Now I must reboot my VM so that it will include this partition in the table at reboot.

3) Now I must create a physical volume so that I can access my partition with LVM.

[root@server ~]# pvcreate /dev/sda3
Physical volume “/dev/sda3″ successfully created

4) Following that, I need to extend my VolGroup into my new physical Volume:

[root@server ~]# vgextend VolGroup00 /dev/sda3
Volume group “VolGroup00″ successfully extended

5) With that complete, I need to extend the logical volume into the newly created space. I know that I planned to give myself 5 gigs, but I want to see how the system is really going to give me, so I’m going to run:

[root@server ~]# vgdisplay | grep ‘Free’
Free PE / Size 165 / 5.16 GB

So now I can be sure that I have 5 GB to work with. I can run lvextend to grow my logical volume across the new partition:

[root@server ~]# lvextend -L+5G /dev/VolGroup00/LogVol00
Extending logical volume LogVol00 to 7.47 GB
Logical volume LogVol00 successfully resized

With that completed, I now have the Logical Volume sitting at 7.47 Gigs. I now need to expand the file system to take advantage of the space that was created. This step is normally taken care of with the resize2fs command, but since I’m running RHEL 4, I can take advantage of ext2online command and don’t have to take the filesystem down. I like this idea much better:

[root@server ~]# ext2online /dev/VolGroup00/LogVol00
ext2online v1.1.18 – 2001/03/18 for EXT2FS 0.5b

That’s it! Now when I run my df -h, I have all the space I was needing:

[root@oasslcwpa01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
7.4G 2.2G 4.9G 31% /
/dev/mapper/VolGroup00-LogVol04
29G 7.3G 21G 27% /apps
/dev/sda1 99M 13M 82M 14% /boot
none 4.0G 0 4.0G 0% /dev/shm
/dev/mapper/VolGroup00-LogVol03
2.9G 192M 2.6G 7% /opt
/dev/mapper/VolGroup00-LogVol02
1008M 447M 511M 47% /var

Special thanks to swizzling.org, who’s tutorial I modified minimally for my own benefit here.