Share Virtual Machines

I have reviewed several appliances in the IT Administration and Systems Infrastructure categories of the VMware Appliance Marketplace to identify the Top Ten and more appliances. Here are the steps I followed for selecting the appliances listed below:

• I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
• I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
• I also discarded several appliances rated 4.5 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.

Which virtual appliance do you use the most and why do you like it?

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

virtualization.info is reporting that a glimpse of the Cisco, VMWare, EMC strategy has emerged in a post on the personal blog of Chad Sakac, Sr. Director VMWare Strategic Alliance at EMC.

Upon reading Chad’s blog, my impression is that a partnership ecosystem seems to be emerging between Cisco, VMWare and EMC for supporting private clouds within an Enterprise and public clouds, a la EC2, through

• A deep integration with the VMWare Hypervisor or “VMWare’s cloud operating system” using standard API’s
• A broad integration to provide a cross-vendor management fabric that spans across management tools of the respective vendors and enables management of VM’s/Virtual Appliances, the underlying host servers, storage and the network.
• VM/Virtual appliance portability across the private and public clouds

This management layer will permit control over individual VM’s and groups of VM’s within this cloud and permit applications (virtual appliances) to be deployed using Just Enough OS’s (JeOS)

EMC’s internal goals (and perhaps VMWare and Cisco’s, too) seem to be

1) To Drive 100% virtualization

Requires A Virtualization Layer that can literally meet the scaling, performance and availability goals of any x86 workload.

EVERY EMC product is being turned into a Virtual Appliance.

Physical adaptability (i.e. increase/decrease CPU/Memory model) needs to extend into the Networking and Storage stacks. People will REALLY start to see “purpose built servers/network/storage” for VMware in 2009

2)  To drive API Integration

Streamline the integration of existing management tools and capabiities with VMWare’s management tools and capabilties.  

These are about making sure that virtual world is able to do everything the phyiscal world can do.   They make sure that the datacenter CAN be 100% virtualized

3)  To create infrastructure that understands and responds to “VM/Application objects”

the next phase is where things really get blended – where thin provisioning is integrated, where management tasks are integrated, where “VM object awareness” is added, and where networking policy portability really takes off.

vCenter is surely a critical new management point – so expect to see core management capability for EMC storage integrated into vCenter in the very near term. … We’ll leverage existing open APIs to create plug-in extension models. BUT at the same time – we will continue to integrate into the vCenter APIs for integrated views in management frameworks that are “home” to people other than VMware Administrators.

Epilogue

Done right – the Private Cloud and Public Cloud can share the applications transparently, and the “Public Cloud” infrastructure layers can “read the same bar-codes”  Clearly the infrastructure needs to be a bit different (management model, federation, multi-tenancy, scale and price points are all different), but they need to be linked.

This ain’t about consolidating servers (though includes that too!).   It **IS** about the next big transformation we all see coming in the IT space we deal in.   We’re gearing up, and as leaders in our respective spaces, focusing our resources, and driving towards a vision. 

If Cisco, VMWare and EMC indeed work together on this, they will be able to dominate this market for years to come. Very cool, Chad! Thanks for sharing your views.

Dell has set up a Virtual Appliance Portal , where it is offering pre-built, ready to deploy, virtual appliances so that customers can try them before buying them.

• Symantec Mail Security 8300 Series – Virtual Edition.
• Altor Networks Virtual Firewall
• Red Hat application stack including Red Hat Enterprise Linux® , JBoss® Application Server, JBoss Hibernate, a choice of open source databases, and ApacheTM Web Server.
• Exchange Server 2007 SP1
• Ubuntu 8.10 desktop and server images
• SUSE Enterprise Linux 10
• Identiphi SAFSolution Enterprise Solution
• Enterasys Secure Netwroks Dragon Enterprise Security Suite
• Double Take VHD test drive for data protection
• Proofpoint anti-spam virtual appliance
• Certeon aCelera for WAN acceleration
• Reflex virtual security appliance

There is a section half way down on that portal titled “Try Virtual, Buy Physical”. A cynical reader might say, “Hmmmm, that virtual appliance directory is just a gimmick for selling more hardware”.

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

The Microsoft Virtual Hard Disk (VHD) format is the common virtualization file format for

• Virtual PC (host: Windows XP or 2000, guest: multiple PC-based operating systems)
• Virtual Server (host: Windows 2003, Small Business Server or XP, guest: Windows 2003, XP, 2000 and NT4 SP6a)
• Windows Server 2008 with built-in hypervisor Hyper-V (host: Windows 2008 64-bit, 64-bit and 32-bit guests: Windows 2008, 2003, SuSe Enterprise Server 10, WIndows Vista, Windows XP)

Microsoft’s has introduced the Run IT on a Virtual Hard Disk program so that customers can try Microsoft’s and its partner’s products downloaded as a pre-configured virtual hard disk, i.e., virtual appliance, for free in their own environment , for 30, 60 or 90 days, without the need for dedicated servers or complex installations.

Windows Server 2008 VHD’s:

Pre-requisites: Microsoft Server 2008 Hyper-V must be installed on a physical machine. A 64-bit system with hardware-assisted virtualization enabled and data execution prevention (DEP) is required. You should have a clean install of x64 edition of Windows Server 2008 to be able to use the Hyper-V technology. Detailed pre-requisites are availabe here.

Full Install:

• Windows Server 2008 Standard Edition x86 (Full Install) VHD
• Windows Server 2008 Standard Edition x64 (Full install) VHD
• Windows Server 2008 Enterprise Edition x86 (Full Install) VHD
• Windows Server 2008 Enterprise Edition x64 (Full Install) VHD

 
Core Install:

• Windows Server 2008 Standard Edition x86 (Core Install) VHD
• Windows Server 2008 Standard Edition x64 (Core install) VHD
• Windows Server 2008 Enterprise Edition x86 (Core Install) VHD
• Windows Server 2008 Enterprise Edition x64 (Core Install) VHD

Application VHD’s:

Pre-requisites:: Microsoft Virtual PC or Virtual Server

None of Microsoft’s partners are offering pre-configured VHD’s on Microsoft’s site at the time of writing. I am wondering why this is so?

The traffic on our blog tells us that there is intense interest in the topic of shrinking virtual machines – vmdk’s vhd’s, vdi’s, to recover free space and improve performance. Our offer to release a software tool to let you view how much storage you could recover has brought several responses from people who are willing to try it.

I found the following response very interesting because it outlines the development usage of VM’s, in contrast with production scenarios that one usually reads about, within a very large multinational software development company whose products are used worldwide:

We capture the build environment and its dependencies together with the end product of the build, the released software, in a VM. During the lifecyle of a major product release, we end up with over 300 builds, i.e., over 300 VM’s. Some of these VM’s are shared with other development teams who have to integration test their builds with our own. Builds corresponding to the major external milestones, e.g., Beta-1, Beta-2, etc. are provided to our technology partners and even to end customers so that we can get early feedback on our product.

We store all these VM’s on a Windows file share and we preserve all VM’s throughout the release cycle and those corresponding to major internal integration and external Beta milestones for at least 6 months beyond the release date. Our goal is to be able to recreate an issue and verify that it is indeed fixed in a later release. So our storage needs are growing annually and its demands on our budget are rising in prominence. We would love to be able to shrink these images to the minimum essential size and leverage our investment in storage far more effectively.

Our Release Management team uses spreadsheets and file naming conventions for managing this store. However, at any given point in time it is very hard for us to match issues that are reported precisely to the release (VM) where it was introduced.

We do share these VM’s between teams but the sharing is achieved using file shares on the LAN. This is very limiting for us since we have development centers in the US (East and West Coast), China, India and sales engineers globally, but given the network bandwidth demands, we don’t even attempt large transfers. Given the size of these images, we support http downloads from an intranet site for teams that are not on our LAN.

In summary, this development team is facing three problems:

1. VM compaction to recover free space and reduce impact on their budget
2. VM tagging to identify them by key attributes
3. VM network transfers take very long to complete

Thank you for sharing this kind of input, we intend to help you resolve such issues.

How to replicate Hyper-V VHDs for DR?
The author is looking for a block-level transfer tool like rsync on Windows. A respondent has suggested using Volume Shadow Copy Services (VSS). However, VSS needs space for shadow copy data and this becomes an issue if you have large VM’s to transfer and are short of space.

scp is a secure transfer tool like rsync that is used for performing remote copies of files, including vhd mages across the LAN or the Internet.

Mircosoft recommends that customers can implement a backup or a disater recovery solution within their WAN using the File Replication Services (FRS2) of the Distributed File System (DFS) of Windows 2003 server R2, or later. This solution will perform well over the WAN, if WAN acceleration is in use. However, if WAN acceleration is not in use, then they should enable the Remote Differential Compression (RDC) protocol available in Windows 2003 Server R2, which Optimizes File Replication over Limited-Bandwidth Networks

Read about the use of rsync for vmdk, vhd backup and disaster recovery.

I use ftp to transfer large VM image of my code to a remote development team based in India and rsync for copying and backing up code, configuration and data from ec2. I researched the web for best practices that have evolved for speeding up large VM transfers. It seems there are none today, unless you are transferring VM’s on your company’s WAN and they are using WAN accleration to improve the transfer rate. However, I have found two models for using rsync with vmdk’s and  vhd’s. Here’s a sample of use cases:

Cloud-centric usage

rsync is used for copying and backing up code, configuration and data from cloud-based services like Amazon ec2.

• Here is an excellent guide to Backup with rsync and ec2
• rsync, s3 and backups
• Migrate to aws/ec2

Traditional usage

rsync is used for backing up large VM’s to a remote store or for disaster recovery

• rsync to offsite locations
• rsync virtual machine backups
• Staging a disaster recovery VM

Read about Backup, Disaster Recovery for Windows VM’s