Share Virtual Machines

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

I run vmware workstation 6.1 on my laptop and use a ubuntu VM in it. The vmdk is setup to use sparse files with max extent 2GB (vmdk file described below). I started with the ubuntu server which is very minimal and then installed ubuntu-xdesktop on it. The xdesktop package installed close to a 1000 packages and expanded the sparse disk a lot. Here is a result of running windows defrag (start–>programs–>accessories–>system tools–>disk defragmenter)

Fragments  File Size       Most fragmented files

202             1.56 GB         workvmu1u1-s001.vmdk

So it has 202 fragments. 1.56GB/202 ~ 84MB.

The logical allocation units can be

- grain: of the grainsize = 128 sectors or 64KB.

- grain table: Pointers to 512 grains are stored in a grain table. thus the grain table covers about 32Mb. Looks like it would allocate 2 graintable?

It is possible it allocates a grain-table worth of data at one time (although may not be used for the same grain table) and the underlying OS as it got more fragmented could have split single allocations into more than one fragments.

Complete Fragmentation Report

Volume (C:)
Volume size                                = 74.46 GB
Cluster size                               = 4 KB
Used space                                 = 60.42 GB
Free space                                 = 14.04 GB
Percent free space                         = 18 %

Volume fragmentation
Total fragmentation                        = 19 %
File fragmentation                         = 38 %
Free space fragmentation                   = 0 %

File fragmentation
Total files                                = 332,864
Average file size                          = 293 KB
Total fragmented files                     = 11,312
Total excess fragments                     = 57,361
Average fragments per file                 = 1.17

Pagefile fragmentation
Pagefile size                              = 2.00 GB
Total fragments                            = 1

Folder fragmentation
Total folders                              = 78,930
Fragmented folders                         = 2,309
Excess folder fragments                    = 8,623

Master File Table (MFT) fragmentation
Total MFT size                             = 416 MB
MFT record count                           = 412,420
Percent MFT in use                         = 96 %
Total MFT fragments                        = 3

——————————————————————————–
Fragments       File Size       Most fragmented files
3,542           555 MB          System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP360A0064665.exe
1,791           214 MB          workdownload7.0-RELEASE-i386-livefs.iso
1,768           695 MB          workdownload7.0-RELEASE-i386-disc2.iso
1,022           509 MB          workdownload7.0-RELEASE-i386-disc1.iso
849             8 MB            Documents and SettingsrosenLocal SettingsTempdd_NET_Framework20_Setup02DE.txt
790             352 MB          workdownload7.0-RELEASE-i386-disc3.iso
470             237 MB          workdownload7.0-RELEASE-i386-docs.iso
440             27 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP344A0062479.reg
347             22 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP377A0070304.old
329             21 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP343A0062133.reg
291             1 KB            WINDOWSsystem32configsystem.LOG
285             3 MB            Program FilesMicrosoft SQL Server100Setup BootstrapLogVSExpress_9.0sql_engine_core_shared_Cpu32_1.log
263             16 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP344A0062478.reg
252             16 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP343A0062132.reg
247             15 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP359A0064513.exe
218             14 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP359A0064400.rbf
202             1.56 GB         workvmu1u1-s001.vmdk
182             11 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP384A0071645.dll
182             13 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP349A0063506.rbf
179             20 MB           WINDOWSpfirewall.log
174             11 MB           System Volume Information_restore{46DE8921-1D39-44D2-A9E9-64119261F211}RP343A0061844.rbf

u1.vmdk descriptor file

# Disk DescriptorFile
version=1
encoding=”windows-1252″
CID=d23aac87
parentCID=ffffffff
createType=”twoGbMaxExtentSparse”

# Extent description
RW 4192256 SPARSE “u1-s001.vmdk”
RW 4192256 SPARSE “u1-s002.vmdk”
RW 4192256 SPARSE “u1-s003.vmdk”
RW 4192256 SPARSE “u1-s004.vmdk”
RW 8192 SPARSE “u1-s005.vmdk”

# The Disk Data Base
#DDB

ddb.virtualHWVersion = “7″
ddb.uuid = “60 00 C2 98 55 91 80 6d-aa c9 e9 78 bd b7 51 f4″
ddb.geometry.cylinders = “1044″
ddb.geometry.heads = “255″
ddb.geometry.sectors = “63″
ddb.adapterType = “lsilogic”
ddb.toolsVersion = “7427″

I did some analysis of the layout of a VMDK file. The graph below shows this analysis: x-axis is the distance in the vmdk file for two logically adjacent blocks(sectors). For for example is block 33 is at offset 3467 and  block 34 is at offset 3468 then the distance in the vmdk file is 3468-3467=1.  The Y-axis is log to the base 2 of the frequency of the distance.

The graph shows that most logically adjacent blocks have adjacent offsets in the vmdk file. This file had a grain size of 128, so you would expect that only 2/128 sectors would have non-adjacent neighbours (1/64). That is roughly the case. The rest of the distances are clustered around multiples of 128 (as one would expect?). I am not sure if this has to do with NTFS allocating blocks in some sequence or VMW workstation expanding the disk in a particular way.

The vmdk was a windows vmdk 4GB disk with windows XP. Very little use after installing the base OS. Installed a couple of packages like Flex Builder.

Another interesting plot would be to plot the actual distance on the physical disk on the host. Thats next.