13 Jul 2009 @ 10:48 AM 

As part of the large VMware project I have been working on for the past month or so, I was tasked with setting up a weekly backup of all Virtual Machines. Now, this would typically be a simple task, using the VCB Framework and the integration module for my favorite backup solution (Go VERITAS *cough* i mean Symantec!). However, this ended up being a significantly more difficult task, due to our early adoption of ESX 4.0 and vSphere.

In order to use VCB and Symantec Backup Exec 11+ to backup Virtual Machines in fullVM mode, several components are needed.

1. An ESX host with a VCB license, or preferably, a vSphere or VI3 server.

2. A windows machine to run Backup Exec 11+, The VCB Framework (VCB Proxy), and the VCB Backup Exec integration module (included on the VI3 CD or vSphere CD, also available from vmware.com).

In my case, I am running a single vSphere ESX 4.0 host (called VM1), and I have a server called UTIL that runs vSphere Server, VCB, and Backup Exec 12.5 SP1.

My initial plan was to use the included backup integration module, so I tested the VCB framework using the following command:

"%VCBBASE%\vcbVmName" -h UTIL -u vmadmin -p password -s name:TESTBOX

-h is the vSphere Server host, -u is the user account with admin access to vSphere Server, -p is that user’s password, and -s is a search string, which can be many things, but I chose to use the VM Name

Output:

Found VM:
moref:vm-252
name:TESTBOX
uuid:4207f2bf-8b25-8c4e-a07d-28c839a21377
ipaddr:10.0.0.110

Official Syntax Listing:

VMware Consolidated Backup -- VM Locator Utility
Version 1.5.0.4948 (build-150805)
Copyright (C) 1998-2009 VMware, Inc.  All rights reserved.
Usage:
vcbVmName -h <url> -u <username> [-p <password>]
 -s <searchSpec> [-c <cachefile>] [<verbosity>]

 <url>   := <hostname>[":"<port>]
 <username>   := <string>
 <password>   := <string>
 <searchSpec> := <Type>":"<Qualifier>
 <Type>   := "Any"|"IpAddr"|"MoRef"|"Uuid"|"Name"|"Powerstate"
<Qualifier>   := <string>
<cachefile>   := <filename>
<verbosity>   := -L (0-6)

If no password is specified on the command line, you will be prompted for one.

Consult the VI3 Backup Guide for more information.

The successful output tells me that my VCB framework is communicating to my vSphere Server correctly.

The next thing I did was install the Symantec VCB integration package using its included install.bat. After following all the prompts, I successfully tested the pre/post commands that came with the integration package. Which led me to believe I was ready to begin backing up VMs! I created a Backup Exec job, following the guide within the VCB integration package, and ran my backup job, only for it to fail instantly.

After a long (5-week) service call to Symantec we were told that the Backup Exec integration module is incompatible with ESX 4.0 and there were no plans to write a new one. (oh early adoption woes!!!) Our only hope of ever using Backup Exec to backup VMs was to use the new (and $4000) VMware backup add-in. Needless to say, we didn’t budget for any additional backup software in our project plan, and as such, I had to find an alternate solution, which brings me to the meat of this post.

After several discussions with my IT director, we decided to write a script that uses the VCB framework to mount a VM, and backup its .vmdk files and .vmx files to an external hard drive. This process was fairly simple, and when it came to testing everything seemed to work. But it took nearly 10 hours to backup a 130GB Virtual Machine over the network. The VCB mounter has several options it can use to mount a VM. See the syntax listing below:

VMware Consolidated Backup -- Virtual Machine Mount Utility
Version 1.5.0.4948 (build-150805)
Copyright (C) 1998-2009 VMware, Inc.  All rights reserved.
Usage:
vcbMounter -h <url> -u <username> [-p <password>] <operation> [<mode>] [<verbosity>]
 <url> := <hostname>[":"<port>]
 <operation> := <mount_op> | <file_umount_op> | <auto_umount_op>
 <verbosity> := -L (0-6)
 <mode> := -m "san"|"nbd"|"nbdssl"|"hotadd"
 <mount_op> := -a <VM> <mountPoint> [<flavor>][<datastores>]
 <file_umount_op> := -d <VM> <SsId> <mountPoint>
 <auto_umount_op> := -U <mountpoint_dir>

 <VM> := <moref>|<uuid>|<ipaddr>|<...> [-c <cacheFile>]
 (See VM Backup guide for complete list of search
 criteria.)
 <mountPoint> := -r <dir>
 <flavor> := <fullvm_flavor>|<file_flavor>
 <datastores> := -C <datastore_catalog_file>
 <SsId> := -n "ssid:"<Sdk-MoRef>
 <fullvm_flavor> := -t "fullvm" [<export-flags>]
 <file_flavor> := -t "file"
 <export-flags> := -M (0|1) -F (0|1)

If no password is specified on the command line, you will be prompted for one.

The various modes are important as they significantly effect the speed at which a VM mount can be performed. SAN is the best possible solution. In this case, your VCB proxy has direct access to the SAN. In our case, we are using an iSCSI (Dell MD3000i) for shared storage, however, the device is currently directly attached to our ESX host (as we only have one). In short, this mode wouldn’t have worked for us using our current infrastructure. NBD is a network transfer of all the .vmdk files from the host to the VCB Proxy (usually via the vSphere Server), and this is the method that I chose for my initial attempt at a VCB FullVM backup. NBDSSL is the same as NBD, but encrypted over the network using SSL. Finally there is HOTADD which can only be used when running the VCB Proxy inside a Virtual Machine. HOTADD is a miracle of modern technology–it allows you to, while the VM is running simply add another handle to VM-A’s hard drive to VM-B, allowing for instant access to all files and folders stored on that drive. Now, our goal is a FullVM backup, so I called VMWare Support to get an idea on how this would work for us.

After a brief conversation with support, I was informed that HOTADD mode can be used within a VM, but that the disks will still be “copied” onto the VCB Proxy’s hard drive. I was also told that since the copy would simply be a disk->disk copy and not a disk->network->disk copy, we should experience a significantly faster backup time. So i gave it a shot, I converted my batch file script for NBD backups to work with HOTADD, I installed VCB into my TESTBOX VM, and cranked away. Everything looked like it was going to work, but the .vmdk files never appeared within the VM. So, back to support, we troubleshot for a few days. Turns out VCB will not work on Windows XP. At this point my frustration level is pretty high, I’ve been working for weeks on something I said would take hours.

So at this point I deploy a new VM, called VCB, from a template, patch it, join it to my AD domain, Install the VCB Framework, vSphere Client, run my test commands, and finally run my script! But alas I’ve hit another snag. Turns out HOTADD is only supported with VMs that have only virtual SCSI disks–I was testing on TESTBOX which is an XP VM and has “IDE” disks. No big deal. I re-write my script to take two parameters: VM Name and Backup Mode. Test again, and FINALLY we have a valid FullVM backup of a virtual machine.

In order to backup the VM’s successfully, we need to back them up to an external media. We have an external drive that we wanted to store the VCB backups on, it connects via SATA/ESATA; and was previously plugged into our Physical server: UTIL. So I removed the drive from the UTIL server and plugged it into our ESX host. On a side note here: whenever you attach a physical piece of hardware to a Virtual Machine, you limit that VM’s ability to migrate off that host. Which can affect High Availability and DRS So, after the drive was plugged in, I attempted to attach it via the vSphere Client, but for some reason the RDM creation radio button is greyed out.

RDM creation is greyed out, for an unknown reason.

I didn’t bother calling support on this one yet (though I need to), as I already know how to create an RDM via command line:

esxcfg-scsidevs -l

Output:

t10.ATA_____SAMSUNG_HD502HI_________________________S1VZJDWS462094______
 Device Type: Direct-Access
 Size: 476940 MB
 Display Name: Local ATA Disk (t10.ATA_____SAMSUNG_HD502HI_________________________S1VZJDWS462094______)
 Plugin: NMP
 Console Device: /dev/sda
 Devfs Path: /vmfs/devices/disks/t10.ATA_____SAMSUNG_HD502HI_________________________S1VZJDWS462094______
 Vendor: ATA       Model: SAMSUNG HD502HI   Revis: 1AG0
 SCSI Level: 5  Is Pseudo: false Status: on
 Is RDM Capable: false Is Removable: false
 Is Local: true
 Other Names:
 vml.01000000005331565a4a44575334363230393420202020202053414d53554e

The above output was truncated to focus on the disk I am creating a RDM for, but that command will list ever disk and its detailed information; and is new to ESX 4.0. So now that I have my drive identifiers, I can create an RDM like this:

vmkfstools -r /vmfs/devices/disks/t10.ATA_____SAMSUNG_HD502HI_________________________S1VZJDWS462094______ /vmfs/volumes/Shared_Storage/VCB/External.vmdk

OR

vmkfstools -r /vmfs/devices/disks/vml.01000000005331565a4a44575334363230393420202020202053414d53554e /vmfs/volumes/Shared_Storage/BCB/External.vmdk

Either way will accomplish the same thing, as any disk identifier can be used. In theory, i could just use “/dev/sda” as well. Regardless, I now have a .vmdk file which I have attached to my VM named VCB. I then updated my batch file to use the new drive instead of the VMFS storage, ran a few tests, created some scheduled tasks, and now, FINALLY… I have FullVM backups of my virtual machines!

This adventure has been one of the longest and most drawn out tasks I have ever had to complete. It was extremely frustrating, especially my experience with Symantec Support. However, I got the job done, and in the end it is a decent solution for VCB backups. In case of an emergency (Hurricane, Fire, etc) all we need to restore all of our virtual servers is a single external hard drive. By no means is this a viable replacement for our daily file-based tape backups. But in the event of a catastrophic failure, we can restore our .vmdk files instead of spending days re-configuring our entire server infrastructure. I have uploaded my VCB batch file for your using pleasure. Feel free to modify and redistribute it as you see fit, I only ask you keep my email and or a link to this blog within the comments.

I hope you learned as much as I did.

Daenks

Posted By: Daenks
Last Edit: 13 Jul 2009 @ 10:48 AM

EmailPermalinkComments (0)
Tags
 06 Jul 2009 @ 3:47 PM 

I have setup VMWare ESX to use iSCSI countless times, but today, during my Support call to VMWare I realized I had never done so via the command line, which was exactly what I needed to do. Luckily, there is a handy KB article on VMWare’s webpage.

Basically, this KB gives us step by step instructions on setting up the vSwitch and Port Group for iSCSI, but it misses a few key elements needed for a successful configuration:

  1. TCP 3260 outbound needs to be opened on the ESX host to allow communication.
  2. The Software iSCSI initiator needs to be enabled
  3. Send/Discovery Targets need to be added
  4. The HBA needs to be scanned for disks

1. To configure the firewall:

esxcfg-firewall -o 3260,tcp,out,iSCSI

2. To enable the Software iSCSI Initiator:

esxcfg-swiscsi -e

3. Send/Discovery Targets need to be added:

vmkiscsi-tool -D -a <SendTarget> vmhba33

4. The HBA needs to be scanned for disks:

esxcfg-rescan vmhba33
OR
esxcfg-swiscsi -s

In step 4, you may choose either command, esxcfg-rescan scans a specific HBA (in this case, the iSCSI HBA), and esxcfg-swiscsi rescans all Software iSCSI HBAs. It is also important to note that the HBA for iSCSI is not always 33; but unless you have more than 33 physical HBAs, you shouldn’t run into this problem.

Daenks

Posted By: Daenks
Last Edit: 10 Jul 2009 @ 01:34 PM

EmailPermalinkComments (0)
Tags
Tags: , ,
Categories: vmware
 06 Jul 2009 @ 3:06 PM 

I spent several hours on the phone today with VMWare support writing a kickstart config file for my ESX 4.0 host.

There is some good news and some bad news about KickStart in ESX 4.0

Good News:

  • After installing from CD, ESX creates a ks.cfg file for you in /root that contains all the settings you used during install, so if you don’t plan on doing any additional configuration, you don’t even need to write a ks.cfg file.
  • There are a plethora of additional options that have been added since 3.5
  • KickStart files can be pointed to after booting from CD, where as previously (3.5) you had to have a PXE server to even think about an automated installation.

Bad News:

  • KickStart files from 3.5 are completely incompatible with 4.0
  • Because of CLI changes, any %post commands have to be re-planned
  • You can no longer use VMWare WebServices to generate a ks.cfg file from your current configuration. (This has been replaced with Host Profiles, which comes with ESX Enterprise Plus and vSphere Server)

Overall, I had a decent experience with support, and we ironed out my issues pretty quickly.  I learned a great deal about the new command structure in 4.0, which is similar enough to 3.5 t get the hang of quickly.

My biggest caveat wasn’t the creation of the KickStart file, but learning how to configure iSCSI from the command line, which I will share with you in a separate post. If you’re interested in seeing my KickStart file, you may do so here.

Daenks

Posted By: Daenks
Last Edit: 06 Jul 2009 @ 03:19 PM

EmailPermalinkComments (0)
Tags
Tags: , , ,
Categories: vmware

 Last 50 Posts
Change Theme...
  • Users » 727
  • Posts/Pages » 8
  • Comments » 0
Change Theme...
  • VoidVoid « Default
  • LifeLife
  • EarthEarth
  • WindWind
  • WaterWater
  • FireFire
  • LightLight

About



    No Child Pages.