Share Virtual Machines

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

Most of us who have copied large files over the Internet (or even within the company’s network for that matter) have been surprised by the amount of time it takes. We dont have a mapping in our mind that can relate the network bandwidth in Mbps with the size of the file to be transferred (GB). I found a nifty file transfer time calculator on the Internet and computed the effective transfer rate for several different bandwidth options by subtracting the signal overhead and an average 5% Layer 4 overhead from the rated capacity. The table below shows the expected transfer times for different network bandwidths, your mileage will vary with the Internet/internal network traffic at your site.

The timings for T1 correlate very closely with my experience of transferring VM’s between my machine and ec2 when I am the sole user. It takes me over half a day to transfer a 10Gb VM and this is really wearing me down.

The chronic problem is how to reduce the time taken to copy VM’s to a remote datacanter, either one owned by your company or a cloud provider, over your company’s private network or on the Internet. Neitwork bandwidth will always remain a gating issue for most of us who are developers.

I use ftp to transfer large VM image of my code to a remote development team based in India and rsync for copying and backing up code, configuration and data from ec2. I researched the web for best practices that have evolved for speeding up large VM transfers. It seems there are none today, unless you are transferring VM’s on your company’s WAN and they are using WAN accleration to improve the transfer rate. However, I have found two models for using rsync with vmdk’s and  vhd’s. Here’s a sample of use cases:

Cloud-centric usage

rsync is used for copying and backing up code, configuration and data from cloud-based services like Amazon ec2.

• Here is an excellent guide to Backup with rsync and ec2
• rsync, s3 and backups
• Migrate to aws/ec2

Traditional usage

rsync is used for backing up large VM’s to a remote store or for disaster recovery

• rsync to offsite locations
• rsync virtual machine backups
• Staging a disaster recovery VM

Read about Backup, Disaster Recovery for Windows VM’s

I got an Amazon EC2 account and wanted to upload image from my laptop to Amazon. First few attempts took way too long (creating the AMI was another saga). So I thought there has to be a better way to do this. After searching for a couple of hours, here are all the links I found. Finally I just went to amazon and redid all the work starting from a pre-registered public AMI. 

Solutions

• Dutch University uses Bit Torrent to Update Workstations
• How to transfer .vmdk files over the network
  • WinSCP & Veeam FastSCP

Research

• Tools & Techniques for Managing Virtual Machine Images [Brief Summary]
• Using Bit Torrent for transferring virtual machine images

Patents

• Deployment & Scaling of Virtual Environments

Experience with running out of storage

• Need to run lots of Virtual Machines on a laptop – but need more storage?

Background about sparse files, Windows NTFS and VHD

• NTFS Compression, sparse files and operations on them
• UNIX sparse files
• Be Careful with VHDs and Windows XP Compressed Folders
• Why do VHD’s grow?
• What is a VHD file, what is a disk image?

VHD Pre-Compactor & Compactor

When you create a new VM you can pre-allocate all the storage in one shot or let the storage grow on demand. In the latter case, the VM uses a sparse file

• Compact your Microsoft Virtual PC 2004 VHD Files
• Compressing Microsoft Virtual Server 2005 VHD files
• Virtualization with Microsoft Virtual Server 2005 by Roger Dittner, David Rule, Jr., Ken Majors
• How to compress Microsoft Virtual PC 2007 hard disks
• Compressing Microsoft Virtual PC 2007 Images
• Compressing Microsoft Windows Server 2008 Hyper-V disk images

VMDK Wipe & Shrink

• Wipe & Shrink
• Enable compression on individual .vmdk files
• Issues

3rd Party (Not from MSFT & VMW)

• Raxco Software
  • PerfectDisk for VmWare
  • for ESX
• Parallels Compressor
  • Desktop Version
  • Server Version

Rsync: for transferring files between machines

• Rsync Virtual Machine Backups
• Virtualize a server with minimum downtime

Research

• IZO: Applications of Large-Window Compressions to Virtual Machine Management (Mark et al. IBM Almaden, USENIX, Nov 2008)
• CZIP [ppt][paper]
• redundancy elimination within large collections of files
• Data Domain SISL
• Hyperfactor (IBM Data Depulication)
• Network Applicance