shareVM- Share insights about using VM's

Wikipedia describes defragmentation as

a process that reduces the amount of fragmentation in file systems. It does this by physically organizing the contents of the disk to store the pieces of each file close together and contiguously. It also attempts to create larger regions of free space using compaction to impede the return of fragmentation.

Generically, the defragmentation of a Windows guest within a virtual disk running on a Windows host (Windows on Windows) requires a three-step process:

1. Defragment the guest
2. Defragment the virtual disk
3. Defragment the host

On a Linux host or guest, the ext3 and ext4 file systems are more resilient to defragmentation.

Windows on Windows

You should perform the following steps whether you are using a Microsoft VHD, VirtualBox VDI or VMware VMDK virtual disk,

1. On a Windows guest OS, run the Windows Disk Defragmenter to defragment the files within the volumes stored inside the virtual disk.
2. Next, power down the virtual machine and defragment the virtual disk using contig. Defragmenting the virtual disk simply reorganizes the blocks so that used blocks move towards lower-numbered sectors and unused blocks move towards higher-numbered sectors.
3. Run the Windows Disk Defragmenter to achieve an overall defragmentation of all files on the host including the virtual disk.

VMware VMDK specific

The following steps can be used generically for VMware VMDK, for Windows on WIndows or any other suppoted platforms. vmware-vdiskmanger:is a standalone tool for defragmenting a growable VMware Workstation, VMware Fusion or VMware Server, vmdk when it is offline. Note that you cannot defragment:

• Preallocated virtual disks
• Physical hard drives
• Virtual disks that are associated with snapshots.

The recommended steps for defragmenting a vmdk are:

1. On a Windows guest OS, run the Windows Disk Defragmenter to defragment the files within the volumes stored inside the VMDK.
2. Next, power down the virtual machine and defragment the vmdk using the command vmware-vdiskmanager -d myVirtualDisk.vmdk. Defragmenting the vmdk simply reorganizes the blocks so that used blocks move towards lower-numbered sectors and unused blocks move towards higher-numbered sectors.
3. If the host OS is also Windows, run the Windows Disk Defragmenter to achieve an overall defragmentation of all files on the host including the VMDK.

Our readers are primarily asking questions like:

• How can I free up disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
• Where can I find the best virtual appliances/ Top 10 virtual appliances?
• How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
• Who are the competitors for ec2?

An analysis of the search terms shows interesting clusters:

If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.

Thank you. I appreciate your interest in this blog.

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

One of my earliest posts on this blog was about installing and configuring Ubuntu 8.10 within a VM, and my feeling that I did springboard into the deep end of a frigid pool on a frosty winter day. I am delighted with the experience of installing and using wubi- the ubuntu installer for windows, on my Windows XP laptop.

User Experience

wubi is an innovative approach towards introducing Windows users to Ubuntu Linux. It preserves the user experience of installing a Windows application using a standard installer and uninstalling it later from Control Panel >  Add/Remove Programs seamlessly.

• It provides an installation wizard implemented as a  standard Windows executable (.exe), double click on it and Ubuntu gets installed; users don’t have to deal with ISO images
• wubi installs ubuntu on the desktop as a NTFS file and uses the ext3 filesystem for its contents within (escalating file defragmentation needs, many of our readers visit this blog prescisly for this topic)
• NTFS-3G (Linux NTFS) driver with write support
• Grub4Dos as a boot loader – every time the laptop reboots, I am presented with a choice of whether I want to start Windows XP or Ubuntu
• Ubuntu appears as a program in the Control Panel > Add/Remove Programs.
• Uninstall removes all the artifacts cleanly

Pros and Cons:

• The Ubuntu GUI looks stunning , however, it is sluggish compared with the response time I get on XP (Harvey Su, remember him?, has a ghoulish imagination and he has by now trained all of us to view the background pattern as a giant lion that  has a void in its skull and a  skeleton dangling from its open jaws. Definitely PG-17 material)
• Ubuntu found the printers on our LAN (this blew me away), it even found the PostScript profile (.ppd file?) for our HP LaserJet automatically, however, I when I tried to install it, the installation failed with no diagnostics.
• It could not find the Postscript profile  for another Canon printer/copier/scanner, however, unlike Windows, it did not lead me to a website where I could download it from.
• It found a Broadcom wireless driver for my Dell laptop, I was able to install and activate it, however, some interaction with our Active Directory authentication prevented me from getting access to our secure wireless network. Once again, there was no diagnostic to indicate what went wrong.
• I would be glad to send the wubi developers log files except that I don’t know what to send to whom, however, it would be nice to have a utility that gathers all the relevant diagnostics and beams it up to “mother ship” over http.
• Update: It would be cool if I could share a folder between XP and Ubuntu so that I could install the ISO’s I had downloaded earlier on  XP.

In conclusion

There is an excellent how-to guide prepared by parthodeep for your reference.

Kudos to Agostino Russo and team for an outstanding job.

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

I ran into this issue a couple of days ago on my windows and unix host machines running vmware workstation: namely the vm was responding really slowly. On windows XP (and beyond as I understand) there is defrag utility (accessories->system tools). I ran it with the analyze option

XP Defrag Utility

It produced the following report:

XP Defrag Report

Which showed that my VM had 263 fragments!!!

My next thought was whether I could find a script to run this periodically …. I could only find scripts to run defrag from the command line which produces output (below) but no information on specific files. I am in search of something like filefrag for ext* which can tell me fragment info just for one file.

C:\>defrag -a -v c:\
Windows Disk Defragmenter
Copyright (c) 2001 Microsoft Corp. and Executive Software International, Inc.

Analysis Report

Volume size = 74.46 GB
Cluster size = 4 KB
Used space = 70.85 GB
Free space = 3.61 GB
Percent free space = 4 %

Volume fragmentation
Total fragmentation = 22 %
File fragmentation = 42 %
Free space fragmentation = 3 %

File fragmentation
Total files = 362,428
Average file size = 305 KB
Total fragmented files = 11,723
Total excess fragments = 36,590
Average fragments per file = 1.10

Pagefile fragmentation
Pagefile size = 2.00 GB
Total fragments = 1

Folder fragmentation
Total folders = 83,631
Fragmented folders = 2,269
Excess folder fragments = 7,870

Master File Table (MFT) fragmentation
Total MFT size = 437 MB
MFT record count = 446,433
Percent MFT in use = 99
Total MFT fragments = 3

You should defragment this volume.

Top Ten Posts for Q4 2008

1. Defragment Ubuntu, Fedora, ext3, ext4
2. How to convert a VMWare VMDK to a Microsoft, Xen VHD
3. VirtualBox – setup, share, shrink, convert
4. How does shrink with vmware disk manager work?
5. Virtual Machine Disk Image Compression
6. Compressing Virtual Images
7. rsync vm, vhd for backup, disaster recovery, ec2
8. How to launch first AMI on Amazon EC2?
9. Steve Herrod’s Top 10 Predictions for Virtulaization for 2009
10. scp, VSS for Windows VHD backup, disaster recovery

Top Ten Search Terms for Q4 2008

• ubuntu ext4
• vhd on usb
• convert vmdk to vhd
• ext4 defrag
• amazon ec2 “virtual server” vhd
• ext3 defragmentation
• rsync vhd
• convert vdi to vhd
• shrink vmdk esxi
• compress vmdk
• compress virtual image

Defragmentation is not necessary on Ubuntu, Fedora, or other Linux implementations when you use the ext3 file system. This is because it is designed to reduce fragmentation, it tries to allocate a free block nearest to others blocks for the same file.

Wikipedia says there are user space tools for defragmenting ext3, however, “A true defragmentation tool does not exist for ext3“.”

There are userspace defragmentation tools like Shake and defrag. Shake works by allocating space for the whole file bolt upright and hoping that it will make the newly allocated file less fragmented. It also tries to write files used at the same time next to each other. Defrag works by copying each file over itself. However they only work if the filesystem is reasonably empty. A true defragmentation tool does not exist for ext3.

How to find out if ext3 file system is fragmented?

• Overview: Fragmentation and Unix file systems
• Measuring fragmentation on ext3 in Linux
• Determining file fragmentation on ext3 file systems
• Measuring ext3 fragmentation – Tools and Techniques

ext4 : The next generation of the ext3 filesystem

• ext4 supports online defragmentation
• How to get ext4 for Ubuntu and Fedora
• Tools: ext4 defragemntation with e4defrag

Reiserfs

Wikipedia says

There are no programs to specifically defragment a ReiserFS file system, although tools have been written to automatically copy the contents of fragmented files hoping that more contiguous blocks of free space can be found. However, Reiser4 will have a repacker that optimizes file fragmentation

• Ubuntu forums: resierfs and defrag and fsck