Greg Shields just wrote an article for RedmondMag that explains how you can install VMware ESX Server inside a virtual machine running on VMware Workstation 6.  While this certainly isn't going to be useful in a production environment, it does work for testing and learning, or in Greg's case, training others on how to install and setup the server class virtualization product without the need to buy expensive hardware.  He says, "With just a little tweaking, you can get Workstation 6.0 to fully support VMware ESX instances running inside VMware Workstation. A VM within a VM? Absolutely -- and depending on your desktop's configuration, it'll pack enough performance to help you develop your virtualization skills right there at your desktop."

He starts his explanation and steps with:

Beefy Workstations
When you're building a VM-within-a-VM environment, you need a powerful desktop host. After all, that host is running virtual machines that are also running virtual machines. They'll require lots of RAM and processor power.

You'll need a desktop with at least a dual core that also supports Intel's Virtualization Technology (VT) processor extensions or newer versions of AMD's equivalent AMD-V extensions. The virtualization extensions are a necessity because they help the processor handle the workload associated with virtualization more efficiently. Your system should also have at least 3GB of RAM to support all the virtual machines you plan to run.

In order to successfully complete your VMotion setup, you'll need at least four simultaneously running virtual machines. Two of those will be ESX servers, the third will serve as the Virtual Center management server, and you will use the fourth as an iSCSI target for shared storage between the hosts.

ESX Server can use SAN-based storage, NFS and iSCSI targets for its shared storage where VMotion-capable virtual machines are housed. iSCSI is an excellent low-cost medium for shared storage as there are numerous open source iSCSI tools available on the Internet.

OpenFiler, which is an easy to use Linux application, is a good choice. You can download it from www.openfiler.com. Once you've installed it within a Linux virtual machine, use OpenFiler's Web interface to create a disk partition, a volume group and a volume for the shared storage. Once you're done, you can also use the same interface to share that volume as an iSCSI target with your ESX servers.

Recursive VMs
Once you've completed the setup process, you can create a fully functional ESX virtual machine by following these steps (or find them online here):

• Create a new custom virtual machine. Set its hardware compatibility to "Workstation 5" and check the box marked "ESX Server Compatible."

• Select the Red Hat Linux operating system, then provide the machine a name and location.

• Give the virtual machine two virtual processors and around 1024MB of RAM. The quantity of RAM assigned to the virtual machine will depend on the RAM available in the host system.

• Select the bridged network option and choose an LSI Logic SCSI adapter for the virtual machine.

• Create a new virtual SCSI disk. Make sure it's large enough to host the ESX software and any other hosted virtual machines. It's usually a good idea to create these disks as pre-allocated to ensure the best performance.

• Once you've created the virtual machine, remove it from the VMware Workstation interface and the favorites list, and close down that Workstation instance. This ensures that the system isn't using the virtual machine's files we need to edit next.

• Open the new virtual machine's .VMX file in a text editor and make sure the following lines are present for each connected Ethernet adapter. If there are multiple cards configured for the ESX instance, there will be one copy of each of these lines for each adapter:

ethernet0.present = "TRUE"

    ethernet0.virtualDev = "e1000"

    ethernet0.connectionType = "bridged"

    ethernet0.addressType = "generated"

You'll need to add the following lines in the .VMX file for Intel-based CPUs:

monitor_control.restrict_backdoor = TRUE

    monitor_control.vt32 = TRUE

If the processor is AMD-based, replace the line monitor_control.vt32 = TRUE with monitor_control.enable_svm = TRUE.

Reconnect the virtual machine with the Workstation interface, make sure the ESX media is in the CD drive and connected into the virtual machine, and boot the virtual machine.

Final Steps
This will begin the process of installing the ESX operating system into the virtual machine. Once the installation and subsequent reboot is complete, you can tell whether the install was successful if the server boots to the typical ESX screen. Repeat the process for the second ESX server and create the Windows Server virtual machine for the Virtual Center server.

Once all components are configured, connect the ESX servers to the OpenFiler iSCSI data store and create your VM within a VM.

Read the entire article and find out more at RedmondMag, here.