fatvm – Share how to skinny "fat" VM's

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

 Ubuntu JeOS

• How to develop virtual appliances using Ubuntu JeOS

VMWare Studio

• Studio: A Free Virtual Appliance Authoring Tool With Robust Management Features
• Videos: Learn how to build a VMWare virtual appliance
• Create your own Linux appliances

Parallels

• How to build a Parallel’s virtual appliances

rPath rBuilder

• rBuilder Appliance creator
• Best practices for building virtual appliances

Rightscale

RightImages provide ready-to-go base operating systems with core cloud software.

ServerTemplates also allow you to designate any number of scripts that you want to run at boot time, upon demand, or when an event is triggered.

RightScripts allow you to specify packages that you want to install before a script is executed and even allow you to upload and attach files directly to scripts.

Amazon Machine Image (AMI)

• Use an existing AMI
• Create your own AMI from scratch

I was intrigued by an article on database virtualization that caught my eye early this morning and I wanted to find out what is it all about?

The business driver for database virtualization is the globalized economy where business transaction happen 24 x7 x 365 and business critical data must be available within the network boundary of a corporation, or through the Internet, spanning application downtime and IT maintenance windows.

Data virtualization is defined here as

to view data from disparate sources without knowing or caring where the data actually resides.

Data virtualization obviously leads to database virtualization, which is defined here as

the use of multiple instances of a DBMS, or different DBMS platforms, simultaneously and in a transparent fashion regardless of their physical location

James Kobielus, a Senior Abalyst with Forrester Research is predicting that real time information needs will drive database virtualization

the database as we know it is disappearing into a virtualization fabric of its own. In this emerging paradigm, data will not physically reside anywhere in particular. Instead, it will be transparently persisted, in a growing range of physical and logical formats, to an abstract, seamless grid of interconnected memory and disk resources; and delivered with subsecond delay to consuming applications.

He is making an interesting case that

Proof Points

Oracle acquired Tangosol in May 2007 and possesses a well-developed in-memory, distributed-caching technology called Coherence

Microsoft annonuced Project Velocity a year later in June 2008:

a distributed cache that allows any type of data (CLR object, XML document, or binary data) to be cached. “Velocity” fuses large numbers of cache nodes in a cluster into a single unified cache and provides transparent access to cache items from any client connected to the cluster.

xkoto was selling GRIDSCALE as a database load balancer in 2006. However, it is very smartly capitalizing on virtualization being a hot segment and has repositioned GRIDSCALE as a database virtualization product. This is a vaiid repositioning not only in the context of the definitions cited above but more importantly due to its validation by noted industry analysts:

• Robin Bloor, a database industry pundit and blogger, who describes its database virtualization capabilities
• Dan Kusnetzky, another influential industry analyst, author and blogger, whose review appears here

Gigaspaces XAP can be used for implementing a distributed cache (in-memory data grid)

GemStone Systems offers the GemFire Enterprise as a data fabric (distributed cache)

Scaleout Software also has a distributed cache offering

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

Updated on September 25, 2009: Please review the updated list of virtual appliances.

You may also be interested in reviewing virtual appliances for IT Administrators.

The original post has been retained as a historical reference. The download links for most of the appliances in the following table no longer work since VMware has revamped the Appliance Marketplace.

Thanks

********** ORIGINAL POST FOLLOWS **********

I have ranked the virtual appliances for IT Administrators available at the VMWare Virtual Appliance Marketplace (VVAM) as described below. Apart from the serial number in the first column and the computed Rank in the last column, all other data is from the VVAM; the titles are self-explanatory.

Methodology

• I accessed the listings for Security, sorted them by Rating and cut and pasted the first 100 listing into an Excel spreadsheet.
• The Star rating that is visible on the web page shows up as a decimal fraction in Excel, this is what you see in the Customer Rating column below.
• I computed a Rank for each listing by multiplying the Customer Rating x 100 x the number of ratings for that appliance.
• Finally I sorted the result by the rank to prepare the table that appears below. I have selected the top 20 of 174 appliances.
• The virtual appliances are ordered by their rank from the highest to the lowest
• I found it interesting that several appliances that rated a 5 had either one or no user each and the bulk of the ratings are for appliances having 4 and 3 stars.

The Microsoft Virtual Hard Disk (VHD) format is the common virtualization file format for

• Virtual PC (host: Windows XP or 2000, guest: multiple PC-based operating systems)
• Virtual Server (host: Windows 2003, Small Business Server or XP, guest: Windows 2003, XP, 2000 and NT4 SP6a)
• Windows Server 2008 with built-in hypervisor Hyper-V (host: Windows 2008 64-bit, 64-bit and 32-bit guests: Windows 2008, 2003, SuSe Enterprise Server 10, WIndows Vista, Windows XP)

Microsoft’s has introduced the Run IT on a Virtual Hard Disk program so that customers can try Microsoft’s and its partner’s products downloaded as a pre-configured virtual hard disk, i.e., virtual appliance, for free in their own environment , for 30, 60 or 90 days, without the need for dedicated servers or complex installations.

Windows Server 2008 VHD’s:

Pre-requisites: Microsoft Server 2008 Hyper-V must be installed on a physical machine. A 64-bit system with hardware-assisted virtualization enabled and data execution prevention (DEP) is required. You should have a clean install of x64 edition of Windows Server 2008 to be able to use the Hyper-V technology. Detailed pre-requisites are availabe here.

Full Install:

• Windows Server 2008 Standard Edition x86 (Full Install) VHD
• Windows Server 2008 Standard Edition x64 (Full install) VHD
• Windows Server 2008 Enterprise Edition x86 (Full Install) VHD
• Windows Server 2008 Enterprise Edition x64 (Full Install) VHD

 
Core Install:

• Windows Server 2008 Standard Edition x86 (Core Install) VHD
• Windows Server 2008 Standard Edition x64 (Core install) VHD
• Windows Server 2008 Enterprise Edition x86 (Core Install) VHD
• Windows Server 2008 Enterprise Edition x64 (Core Install) VHD

Application VHD’s:

Pre-requisites:: Microsoft Virtual PC or Virtual Server

None of Microsoft’s partners are offering pre-configured VHD’s on Microsoft’s site at the time of writing. I am wondering why this is so?

I have ranked the virtual appliances for App/Web Servers available at the VMWare Virtual Appliance Marketplace (VVAM) as described below. I am going through this exercise to gain an understanding of what types of appliances are popular, who is providing them and what they are composed of. Apart from the serial number in the first column and the computed Rank in the last column, all other data is from the VVAM; the titles are self-explanatory.

Methodology

• I accessed the listings for App/Web Server, sorted them by Rating and cut and pasted the first 100 listing into an Excel spreadsheet.
• The Star rating that is visible on the web page shows up as a decimal fraction in Excel, this is what you see in the Customer Rating column below.
• I computed a Rank for each listing by multiplying the Customer Rating x 100 x the number of ratings for that appliance.
• Finally I sorted the result by the rank to prepare the table that appears below. I have selected the top 25 of 269 appliances
• The virtual appliances are ordered by their rank from the highest to the lowest
• I found it interesting that several appliances that rated a 5 had either one or no user each and the bulk of the ratings are for appliances having 4 and 3 stars.

Be sure to lookup the other articles on the most popular VMWare Virtual Appliances on this blog.

Mike Culver, technology evangelist for Amazon Web Services announced the availability of a web-based management console for ec2, the Elastic Compute Cloud.

The key features are summarized by James Urquhart and by Saad Ali Abassi.

Alan Williamson offers a visual tour of the new UI.

As an AWS customer, I feel heartened by this investment on Amazon’s part because it emphasizes their commitment to commoditize AWS and make its services widely usable. The AWS services will become cheaper as usage ramps up on the commoditization curve and makes “cloud computing accessible for the masses“

Defragmentation is not necessary on Ubuntu, Fedora, or other Linux implementations when you use the ext3 file system. This is because it is designed to reduce fragmentation, it tries to allocate a free block nearest to others blocks for the same file.

Wikipedia says there are user space tools for defragmenting ext3, however, “A true defragmentation tool does not exist for ext3“.”

There are userspace defragmentation tools like Shake and defrag. Shake works by allocating space for the whole file bolt upright and hoping that it will make the newly allocated file less fragmented. It also tries to write files used at the same time next to each other. Defrag works by copying each file over itself. However they only work if the filesystem is reasonably empty. A true defragmentation tool does not exist for ext3.

How to find out if ext3 file system is fragmented?

• Overview: Fragmentation and Unix file systems
• Measuring fragmentation on ext3 in Linux
• Determining file fragmentation on ext3 file systems
• Measuring ext3 fragmentation – Tools and Techniques

ext4 : The next generation of the ext3 filesystem

• ext4 supports online defragmentation
• How to get ext4 for Ubuntu and Fedora
• Tools: ext4 defragemntation with e4defrag

Reiserfs

Wikipedia says

There are no programs to specifically defragment a ReiserFS file system, although tools have been written to automatically copy the contents of fragmented files hoping that more contiguous blocks of free space can be found. However, Reiser4 will have a repacker that optimizes file fragmentation

• Ubuntu forums: resierfs and defrag and fsck