Share Virtual Machines

Our readers are primarily asking questions like:

• How can I free disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
• Where can I find the best virtual appliances/ Top 10 virtual appliances?
• How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
• Who are the competitors for ec2?

An analysis of the search terms shows interesting clusters:

If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.

Thank you. I appreciate your interest in this blog.

What should you do when you run out of virtual disk space and have to expand it? Here is an iinsightful article that describes the gymanstics.

[reposted from rosensharma.wordpress.com]

I stole “I shrink, therefore I am” from my wife’s good friend Arun Verma, who is incredibly creative, and makes some of the best lamps ever. He also does websites and ads if you are interested.

I have a macbook and use vmware fusion to run a windows XP VM. I keep all my data on a hosted folder on the mac’s operating system. So the VM is basically programs and user settings. In addition I have several images which I work with: Red Hat Enterprise, Ubuntu, Win 2K3 etc. Not atypical of someone who either develops or tinkers with technology.

My problem is that out of a 120GB hard disk, I am upto 100GB, and a whopping 60GB of that is virtual images. I have about 8. So I wanted to see if I could compress the virtual images in some fashion. I decided to run a small test of how much dedup would buy me over gzip

w2k3.vhd: Original size: 1.6GB
w2k3.vhd.gz: 712 MB

Further Analysis of the image showed that there were
14K Zero Filled Blocks, and
About 40K blocks occurred more than once

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

Total VM’s on VMWare Virtual Appliance Marketplace : 2560
Number of distinct download sites providing these appliances : 674

The following table lists the compression formats used for 408 appliances we have examined:

Amongst the compression formats, zip is the most popular one (42%), with 7z the next in sequence (23%) for Windows; gz is the most popular one for Linux (18%)

We intend to publish more statistics over the next few days. Watch this space.

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

Top Ten Posts for Q4 2008

1. Defragment Ubuntu, Fedora, ext3, ext4
2. How to convert a VMWare VMDK to a Microsoft, Xen VHD
3. VirtualBox – setup, share, shrink, convert
4. How does shrink with vmware disk manager work?
5. Virtual Machine Disk Image Compression
6. Compressing Virtual Images
7. rsync vm, vhd for backup, disaster recovery, ec2
8. How to launch first AMI on Amazon EC2?
9. Steve Herrod’s Top 10 Predictions for Virtulaization for 2009
10. scp, VSS for Windows VHD backup, disaster recovery

Top Ten Search Terms for Q4 2008

• ubuntu ext4
• vhd on usb
• convert vmdk to vhd
• ext4 defrag
• amazon ec2 “virtual server” vhd
• ext3 defragmentation
• rsync vhd
• convert vdi to vhd
• shrink vmdk esxi
• compress vmdk
• compress virtual image

The traffic on our blog tells us that there is intense interest in the topic of shrinking virtual machines – vmdk’s vhd’s, vdi’s, to recover free space and improve performance. Our offer to release a software tool to let you view how much storage you could recover has brought several responses from people who are willing to try it.

I found the following response very interesting because it outlines the development usage of VM’s, in contrast with production scenarios that one usually reads about, within a very large multinational software development company whose products are used worldwide:

We capture the build environment and its dependencies together with the end product of the build, the released software, in a VM. During the lifecyle of a major product release, we end up with over 300 builds, i.e., over 300 VM’s. Some of these VM’s are shared with other development teams who have to integration test their builds with our own. Builds corresponding to the major external milestones, e.g., Beta-1, Beta-2, etc. are provided to our technology partners and even to end customers so that we can get early feedback on our product.

We store all these VM’s on a Windows file share and we preserve all VM’s throughout the release cycle and those corresponding to major internal integration and external Beta milestones for at least 6 months beyond the release date. Our goal is to be able to recreate an issue and verify that it is indeed fixed in a later release. So our storage needs are growing annually and its demands on our budget are rising in prominence. We would love to be able to shrink these images to the minimum essential size and leverage our investment in storage far more effectively.

Our Release Management team uses spreadsheets and file naming conventions for managing this store. However, at any given point in time it is very hard for us to match issues that are reported precisely to the release (VM) where it was introduced.

We do share these VM’s between teams but the sharing is achieved using file shares on the LAN. This is very limiting for us since we have development centers in the US (East and West Coast), China, India and sales engineers globally, but given the network bandwidth demands, we don’t even attempt large transfers. Given the size of these images, we support http downloads from an intranet site for teams that are not on our LAN.

In summary, this development team is facing three problems:

1. VM compaction to recover free space and reduce impact on their budget
2. VM tagging to identify them by key attributes
3. VM network transfers take very long to complete

Thank you for sharing this kind of input, we intend to help you resolve such issues.

Shrinking a VM today is a time consuming process: zero out the free blocks, defrag, use vmware’s tools to shrink it partition by parition. I wanted a tool which would tell me after doing all that how much would it help. This turned out to be a complicated process as you have to find the free clusters in NTFS and find out whether they are actually allocated as in sparse VMDK files all blocks may not be allocated. I ran the tool on some VMs see below: Windows 2003, Windows NT & a couple of XP VMs.

fcp -f c:\work\vhd\w2k3.vhd
NTFS Free Sectors 594632
Free Sectors Allocated in Virtual Image 294496
Maximum Possible Saving by Shrinking 143 MB

fcp -f “c:\work\vm\w1\Windows XP Professional.vmdk”
NTFS Free Sectors 1049160
Free Sectors Allocated in Virtual Image 883144
Maximum Possible Saving by Shrinking 431 MB

fcp -f “c:\work\vhd\Windows XP Hard Disk.vhd”
NTFS Free Sectors 30124176
Free Sectors Allocated in Virtual Image 847013
Maximum Possible Saving by Shrinking 413 MB

fcp -f “c:\work\vm\wnta\wnta.vmdk”
NTFS Free Sectors 27724512
Free Sectors Allocated in Virtual Image 71656
Maximum Possible Saving by Shrinking 34 MB

I am thinking of releasing the tool, it only supports vmdk & vhd files today with NTFS file system. If you would like to get an early release, would be thrilled to share let me know.