Share Virtual Machines

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

Have you ever wondered how much space will you be able to recover by shrinking a VM?

Now you can download and run vmsi to determine whether it is worth shrinking that VM on your disk. We are releasing vmsi for both the Windows NTFS and Ubuntu platforms. Please read the Release Notes to understand the current limitations.

Let us know how it works, please send email to  support at sharevm dot com

Top Ten Posts for Q4 2008

1. Defragment Ubuntu, Fedora, ext3, ext4
2. How to convert a VMWare VMDK to a Microsoft, Xen VHD
3. VirtualBox – setup, share, shrink, convert
4. How does shrink with vmware disk manager work?
5. Virtual Machine Disk Image Compression
6. Compressing Virtual Images
7. rsync vm, vhd for backup, disaster recovery, ec2
8. How to launch first AMI on Amazon EC2?
9. Steve Herrod’s Top 10 Predictions for Virtulaization for 2009
10. scp, VSS for Windows VHD backup, disaster recovery

Top Ten Search Terms for Q4 2008

• ubuntu ext4
• vhd on usb
• convert vmdk to vhd
• ext4 defrag
• amazon ec2 “virtual server” vhd
• ext3 defragmentation
• rsync vhd
• convert vdi to vhd
• shrink vmdk esxi
• compress vmdk
• compress virtual image

From informal surveys conducted with ex-colleagues and developers in otherr companies, and the inquiries we are receiving, the picture that has emerged in my mind is that shrinking VM’s is a need for

• Developers who can store a limited number of VMs on their personal workstations/laptops.
• Developers and particularly System Administrators who are concerned about the performance of their app within the VM at run time
• Release Engineers/System Administratrators who have to transfer VM images from a staging/build/test machine to one or more production servers in a remote data center, or in the cloud, and are interested in reducing their size, and hence the elapsed time for the file transfer .

Release Engineers/System Administrators dealing with production VMWare ESX deployments seem to be storing VM’s on storage that supports de-duplication, primarily from

• Netapp (Vaughan Stewart’s article) (I found this reference on Yellow Bricks) and
• EMC

Development teams in large companies do use NetApp filers to provide shared NFS-mounted storage for development teams. However, I am not sure whether these filers support de-duplication, and if they do, whether it is enabled. If de-duplication is indeed enabled, then these shared filesystems typically act as passive storage for VM’s created by developers/build masters and the overall demand for storage reduces considerably.

However, there are development teams that also use commodity storage for file shares. If they are Linux file shares, then the ext3 file system seems to be the industry-standard; otherwise, they are Windows NTFS file shares. Both these file systems are also found on developer desktops – they don’t support de-duplication and have the exact same defragmentation and VM shrinking challenges that the developer faces on his personal workstation – except that the pain for managing file shares is felt by the IT admin/operator responsible for its well being. (It is worth reading this informative article about dealing with VM snapshots and linnked clones on Windows NTFS).

The major pain for managing several VM’s stored on commodity storage on a workstation/laptop seems to be felt by the developer since he can only offload older VM’s on a USB drive or a shared filesystem for “archival”, and has to keep the current one’s that he is testing on the local disk. We are hoping this blog is providing pointers to the “how to”  for the developers, release engineers and system adminsitrators.

The traffic on our blog tells us that there is intense interest in the topic of shrinking virtual machines – vmdk’s vhd’s, vdi’s, to recover free space and improve performance. Our offer to release a software tool to let you view how much storage you could recover has brought several responses from people who are willing to try it.

I found the following response very interesting because it outlines the development usage of VM’s, in contrast with production scenarios that one usually reads about, within a very large multinational software development company whose products are used worldwide:

We capture the build environment and its dependencies together with the end product of the build, the released software, in a VM. During the lifecyle of a major product release, we end up with over 300 builds, i.e., over 300 VM’s. Some of these VM’s are shared with other development teams who have to integration test their builds with our own. Builds corresponding to the major external milestones, e.g., Beta-1, Beta-2, etc. are provided to our technology partners and even to end customers so that we can get early feedback on our product.

We store all these VM’s on a Windows file share and we preserve all VM’s throughout the release cycle and those corresponding to major internal integration and external Beta milestones for at least 6 months beyond the release date. Our goal is to be able to recreate an issue and verify that it is indeed fixed in a later release. So our storage needs are growing annually and its demands on our budget are rising in prominence. We would love to be able to shrink these images to the minimum essential size and leverage our investment in storage far more effectively.

Our Release Management team uses spreadsheets and file naming conventions for managing this store. However, at any given point in time it is very hard for us to match issues that are reported precisely to the release (VM) where it was introduced.

We do share these VM’s between teams but the sharing is achieved using file shares on the LAN. This is very limiting for us since we have development centers in the US (East and West Coast), China, India and sales engineers globally, but given the network bandwidth demands, we don’t even attempt large transfers. Given the size of these images, we support http downloads from an intranet site for teams that are not on our LAN.

In summary, this development team is facing three problems:

1. VM compaction to recover free space and reduce impact on their budget
2. VM tagging to identify them by key attributes
3. VM network transfers take very long to complete

Thank you for sharing this kind of input, we intend to help you resolve such issues.

Shrinking a VM today is a time consuming process: zero out the free blocks, defrag, use vmware’s tools to shrink it partition by parition. I wanted a tool which would tell me after doing all that how much would it help. This turned out to be a complicated process as you have to find the free clusters in NTFS and find out whether they are actually allocated as in sparse VMDK files all blocks may not be allocated. I ran the tool on some VMs see below: Windows 2003, Windows NT & a couple of XP VMs.

fcp -f c:\work\vhd\w2k3.vhd
NTFS Free Sectors 594632
Free Sectors Allocated in Virtual Image 294496
Maximum Possible Saving by Shrinking 143 MB

fcp -f “c:\work\vm\w1\Windows XP Professional.vmdk”
NTFS Free Sectors 1049160
Free Sectors Allocated in Virtual Image 883144
Maximum Possible Saving by Shrinking 431 MB

fcp -f “c:\work\vhd\Windows XP Hard Disk.vhd”
NTFS Free Sectors 30124176
Free Sectors Allocated in Virtual Image 847013
Maximum Possible Saving by Shrinking 413 MB

fcp -f “c:\work\vm\wnta\wnta.vmdk”
NTFS Free Sectors 27724512
Free Sectors Allocated in Virtual Image 71656
Maximum Possible Saving by Shrinking 34 MB

I am thinking of releasing the tool, it only supports vmdk & vhd files today with NTFS file system. If you would like to get an early release, would be thrilled to share let me know.

I read Steve Herrod’s (CTO, VMWare) Top 10 predictions for virtualization in 2009 and have summarized them below:

1. Virtual Desktop Initiative (VDI): Companies will invest in virtualizing desktops primarily to drive down the total cost of ownership.
2. Virtualization-aware storage “that offer native array support for common storage operations on virtual machines such as replication and migration; thin provisioning and de-duplication capabilities to optimize storage usage – which is particularly important for the desktop use case; and virtual machine-based storage (virtual storage arrays) solutions.”
3. High-end applications will leverage virtualization support in CPU’s to gain significant performance improvements and be able to run in VM’s.
4. Virtualization-aware cloud services: Standardization of OS and application images in virtual machines  enable cloud services to import and export industry-standard virtual machines and provide additional compute capacity on demand. “On the user level, it enables virtual desktops to follow users as they travel. On the enterprise level, it enables workloads to be automatically redistributed to meet capacity needs and take advantage of eco-friendly locations where electricity can be tapped at much lower costs“.
5. Virtualization-aware networking:  For example, VMWare and Cisco are collaborating on integrating VMware Virtual Desktop Infrastructure (VDI) solutions with Cisco Application Delivery Networking solutions to improve the performance of virtual desktops delivered across wide-area networks (WANs). In addition, network vendors are also optimizing for virtualization network  traffic, remote display protocols; network management tools will see through the virtualization layer to monitor and manage at the virtual machine -level, and vendors  will begin shipping software-based network switches.
6. Virtualization in smart phones: Ultra-thin hypervisors will provide a common platform abstraction across devices from several manufacturers so that the same software stack works across all phones enabling a faster time to market with a much lower product development cost. End users will so be able to run multiple profiles, e.g., one for personal use and one for work use – on the same phone
7. Virtualization-Focused Security Solutions: Today there is a dearth of practical solutions that can run within a VM without impacting the performance significantly. A new generation of virtualization-aware security solutions will emerge.
8. Management tools with increased focus on the virtual datacenter: BMC, CA, HP and IBM have announced products, standard APIs and integration technologies that facilitate the integration of management functions into virtualization platforms to enable end-to-end management processes spanning heterogeneous datacenter environments, a wide variety of application stacks, and physical and virtual use cases.
9. Green datacenters: Virtualization is driving server consolidation and also incorporates power management features to quiesce VM’s when they are not in use and bring them online on demand. This continues to lead to reduction in demands for electrical power within the datacenter.
10. Cloud services providers are using virtualization as the core infrastructure upon which their services are offered.

I found it particularly striking that the standardization of the execution environment provided by a VM is driving large business opportunities in three disparate segments, namely, VDI, virtualization-aware cloud services and virtualization in smart phones. The predictions form an interesting list because they echo some of the themes that are driving your readership of this blog, namely, developer experiences with using vm’s, how to use a cloud service like ec2, shrinking vm’s, sharing vm’s, etc

Why use VirtualBox?

• VMware Workstation vs. Sun XVM VirtualBox
• Switching from VMWare to VirtualBox – A Primer

 How to setup?

• Using VirtualBox to run Ubuntu

How to share?

• Share your VirtualBox VM across different OS’s
• Windows Vista guest and Ubuntu host

How to shrink images?

• VirtualBox – Shrink your VDI images
• Shrinking VDI file with NTFS partition

How to convert?

• from VirtualBox .vdi to VMWare  .vmdk formats

Esoteric use

• Run IE on non-Windows OS