shareVM- Share insights about using VM's

Our readers are primarily asking questions like:

• How can I free up disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
• Where can I find the best virtual appliances/ Top 10 virtual appliances?
• How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
• Who are the competitors for ec2?

An analysis of the search terms shows interesting clusters:

If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.

Thank you. I appreciate your interest in this blog.

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

Now you can download and run vmsi to determine whether it is worth shrinking that VM on your disk. We are releasing vmsi for both the Windows NTFS and Ubuntu platforms. Please read the Release Notes to understand the current limitations.

Let us know how it works, please send email to  support at sharevm dot com

Top Ten Posts for Q4 2008

1. Defragment Ubuntu, Fedora, ext3, ext4
2. How to convert a VMWare VMDK to a Microsoft, Xen VHD
3. VirtualBox – setup, share, shrink, convert
4. How does shrink with vmware disk manager work?
5. Virtual Machine Disk Image Compression
6. Compressing Virtual Images
7. rsync vm, vhd for backup, disaster recovery, ec2
8. How to launch first AMI on Amazon EC2?
9. Steve Herrod’s Top 10 Predictions for Virtulaization for 2009
10. scp, VSS for Windows VHD backup, disaster recovery

Top Ten Search Terms for Q4 2008

• ubuntu ext4
• vhd on usb
• convert vmdk to vhd
• ext4 defrag
• amazon ec2 “virtual server” vhd
• ext3 defragmentation
• rsync vhd
• convert vdi to vhd
• shrink vmdk esxi
• compress vmdk
• compress virtual image

From informal surveys conducted with ex-colleagues and developers in other companies, and the inquiries we are receiving, the picture that has emerged in my mind is that shrinking VM’s is a need for

• Developers who can store a limited number of VMs on their personal workstations/laptops.
• Developers and particularly System Administrators who are concerned about the performance of their app within the VM at run time
• Release Engineers/System Administratrators who have to transfer VM images from a staging/build/test machine to one or more production servers in a remote data center, or in the cloud, and are interested in reducing their size, and hence the elapsed time for the file transfer .

Release Engineers/System Administrators dealing with production VMWare ESX deployments seem to be storing VM’s on storage that supports de-duplication, primarily from

• Netapp (Vaughan Stewart’s article) (I found this reference on Yellow Bricks) and
• EMC

Development teams in large companies do use NetApp filers to provide shared NFS-mounted storage for development teams. However, I am not sure whether these filers support de-duplication, and if they do, whether it is enabled. If de-duplication is indeed enabled, then these shared filesystems typically act as passive storage for VM’s created by developers/build masters and the overall demand for storage reduces considerably.

However, there are development teams that also use commodity storage for file shares. If they are Linux file shares, then the ext3 file system seems to be the industry-standard; otherwise, they are Windows NTFS file shares. Both these file systems are also found on developer desktops – they don’t support de-duplication and have the exact same defragmentation and VM shrinking challenges that the developer faces on his personal workstation – except that the pain for managing file shares is felt by the IT admin/operator responsible for its well being. (It is worth reading this informative article about dealing with VM snapshots and linnked clones on Windows NTFS).

The major pain for managing several VM’s stored on commodity storage on a workstation/laptop seems to be felt by the developer since he can only offload older VM’s on a USB drive or a shared filesystem for “archival”, and has to keep the current one’s that he is testing on the local disk. We are hoping this blog is providing pointers to the “how to”  for the developers, release engineers and system adminsitrators.

The traffic on our blog tells us that there is intense interest in the topic of shrinking virtual machines – vmdk’s vhd’s, vdi’s, to recover free space and improve performance. Our offer to release a software tool to let you view how much storage you could recover has brought several responses from people who are willing to try it.

I found the following response very interesting because it outlines the development usage of VM’s, in contrast with production scenarios that one usually reads about, within a very large multinational software development company whose products are used worldwide:

We capture the build environment and its dependencies together with the end product of the build, the released software, in a VM. During the lifecyle of a major product release, we end up with over 300 builds, i.e., over 300 VM’s. Some of these VM’s are shared with other development teams who have to integration test their builds with our own. Builds corresponding to the major external milestones, e.g., Beta-1, Beta-2, etc. are provided to our technology partners and even to end customers so that we can get early feedback on our product.

We store all these VM’s on a Windows file share and we preserve all VM’s throughout the release cycle and those corresponding to major internal integration and external Beta milestones for at least 6 months beyond the release date. Our goal is to be able to recreate an issue and verify that it is indeed fixed in a later release. So our storage needs are growing annually and its demands on our budget are rising in prominence. We would love to be able to shrink these images to the minimum essential size and leverage our investment in storage far more effectively.

Our Release Management team uses spreadsheets and file naming conventions for managing this store. However, at any given point in time it is very hard for us to match issues that are reported precisely to the release (VM) where it was introduced.

We do share these VM’s between teams but the sharing is achieved using file shares on the LAN. This is very limiting for us since we have development centers in the US (East and West Coast), China, India and sales engineers globally, but given the network bandwidth demands, we don’t even attempt large transfers. Given the size of these images, we support http downloads from an intranet site for teams that are not on our LAN.

In summary, this development team is facing three problems:

1. VM compaction to recover free space and reduce impact on their budget
2. VM tagging to identify them by key attributes
3. VM network transfers take very long to complete

Thank you for sharing this kind of input, we intend to help you resolve such issues.

Shrinking a VM today is a time consuming process: zero out the free blocks, defrag, use vmware’s tools to shrink it partition by parition. I wanted a tool which would tell me after doing all that how much would it help. This turned out to be a complicated process as you have to find the free clusters in NTFS and find out whether they are actually allocated as in sparse VMDK files all blocks may not be allocated. I ran the tool on some VMs see below: Windows 2003, Windows NT & a couple of XP VMs.

fcp -f c:\work\vhd\w2k3.vhd
NTFS Free Sectors 594632
Free Sectors Allocated in Virtual Image 294496
Maximum Possible Saving by Shrinking 143 MB

fcp -f “c:\work\vm\w1\Windows XP Professional.vmdk”
NTFS Free Sectors 1049160
Free Sectors Allocated in Virtual Image 883144
Maximum Possible Saving by Shrinking 431 MB

fcp -f “c:\work\vhd\Windows XP Hard Disk.vhd”
NTFS Free Sectors 30124176
Free Sectors Allocated in Virtual Image 847013
Maximum Possible Saving by Shrinking 413 MB

fcp -f “c:\work\vm\wnta\wnta.vmdk”
NTFS Free Sectors 27724512
Free Sectors Allocated in Virtual Image 71656
Maximum Possible Saving by Shrinking 34 MB

I am thinking of releasing the tool, it only supports vmdk and vhd files today with NTFS file system. If you would like to get an early release, would be thrilled to share let me know.

Why use VirtualBox?

• VMware Workstation vs. Sun XVM VirtualBox
• Switching from VMWare to VirtualBox – A Primer

 How to setup?

• Using VirtualBox to run Ubuntu

How to share?

• Share your VirtualBox VM across different OS’s
• Windows Vista guest and Ubuntu host

How to shrink images?

• VirtualBox – Shrink your VDI images
• Shrinking VDI file with NTFS partition

How to convert?

• from VirtualBox .vdi to VMWare  .vmdk formats

Esoteric use

• Run IE on non-Windows OS