shareVM- Share insights about using VM's

I wrote a Python 2.6 script to find and list VM’s older than 90 days on my Windows workstation, so that I could compress them, move them to a 1TB drive attached to my machine, or to a file server, or delete them.

find_old_vms is a tool to find and list old VM’s (vmdk’s, vhd’s) on your hard drives that are older than a given number of days.

Usage: find_old_vms in_this_directory_tree older_than_days

Example: find_old_vms “c:\\” 90

Download for Windows XP, 2003, and Linux. The script uses atime (latest file access time), which is not supported on Windows Vista and Windows 7.

Code:

#
# NOTE: This script uses atime - the last access time for deciding whether a VM
# ****  file is a candidate. On Windows XP, atime is updated every hour, whereas,
#          Windows Vista and Windows 7 do not provide an atime.

import os, sys, glob, time

# dtroot is the pathname for a node in a directory tree
# age is the number of days for which a file has not been accessed
# size in bytes is the maximum size of a file
def scan(dtroot, age, size):
	""" scan <dir> scans the <dir> on host for virtual images
	"""
	filecount = 0
	wctime = time.time(); # get current time

	for root, subdirs, files in os.walk(dtroot):
		# Build a list of filenames that have a suffix vmd* or vhd*
		vfiles = glob.glob(os.path.join(root,"*.v[mh]d*"))
		for f in vfiles:
			atime = os.path.getatime(f)
			elapsed_time = (wctime - atime)/(60*60*24)
			if elapsed_time > age:
				filecount = filecount + 1
				print f, " last accessed ", int(elapsed_time), " days ago\n",

if __name__ == "__main__":

	import sys

	# User asked for help
	if sys.argv[1] == '?':
		print "\nfind_old_vms in_this_directory_tree older_then_days\n",
		print "Find all files with the suffix .v[mh]d* that have not been accessed since older_than_days",
		print "through a recusrive descent starting from the root in_this_directory_tree",
		sys.exit()	

	# Validate that the first argument in_this_directory_tree is a valid path
	if not os.path.exists(sys.argv[1]):
		print sys.argv[1], " is not a valid directory. Please provide another",
		sys.exit()

	# Arbitrarily limit search to 10 years
	invalid_age = 0
	if int(sys.argv[2]) < 1:
		invalid_age = 1
	elif int(sys.argv[2]) > 3650:
		invalid_age = 1

	if invalid_age == 1:
		print sys.argv[2], " is invalid. Please provide between 1 and 3650 days",
		sys.exit()

	scan(sys.argv[1], int(sys.argv[2]), 0)

If you remove the restriction of searching for vImh]d* files, it will help you find other older files as well. I will appreciate your feedback.

Our readers are primarily asking questions like:

• How can I free up disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
• Where can I find the best virtual appliances/ Top 10 virtual appliances?
• How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
• Who are the competitors for ec2?

An analysis of the search terms shows interesting clusters:

If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.

Thank you. I appreciate your interest in this blog.

I have reviewed several appliances in the Secure Content and Threat Management and Identity, Access and Vulnerability Managament categories of the VMware Appliance Marketplace to identify the Top 20 appliances. Here are the steps I followed for selecting the appliances listed below:

• I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
• I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
• I also discarded several appliances rated 4 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.

Which virtual appliance do you use the most and why do you like it?

I have reviewed several appliances in the IT Administration and Systems Infrastructure categories of the VMware Appliance Marketplace to identify the Top Ten and more appliances. Here are the steps I followed for selecting the appliances listed below:

• I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
• I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
• I also discarded several appliances rated 4.5 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.

Which virtual appliance do you use the most and why do you like it?

During the past week, I was analyzing the recently revamped VMware Appliance Marketplace. My analysis is summarized below:

Salient Statistics

• 9% are top-rated, i.e., have a Star Rating of 5
• 41% have a Star Rating of between 3 and 4, i.e., this cluster may be providing the most value to its users, even though it is not top-rated.
• There is a steep drop-off in the number of appliances that have a Star Rating below 3
• 31% are unrated, which seems to indicate that a significant majority has been submitted recently
• Only 3% (40/1227) have Reviews – this is a very small number. The number of reviews have to grow substantially to indicate that a vibrant community, like Amazon.com reviews, has formed here

Transparency

I feel there is a greater need for transparency. What do the Star Ratings mean?

• Are they correlated with the number of downloads for that appliance?
• What is the impact of reviews on these ratings?

Beware Star Ratings alone! The following appliance rated 4.5 has 4 reviews, read them and you will know what I mean.

As a member of VMware’s Technology Network community, I will urge VMware to provide greater transparency around how these ratings are computed/awarded. It will greatly help the users understand the ratings system.

What do you think?

Data

Here are the Top 10 posts for Q1 2009, the numbers of views are in parentheses.

1. Defragment Ubuntu, Fedora, ext3, ext4 (2247)
2. Most popular VMWare Virtual Appliances for IT Administrators (2186)
3. VirtualBox – setup, share, shrink, convert (842)
4. How to convert a VMWare VMDK to a Microsoft, Xen VHD? (810)
5. How does shrink with vmware disk manager work? (614)
6. Most popular VMWare Virtual Appliances for Security (607)
7. Pre-configured VHD (Virtual Appliance) available from Microsoft (593)
8. Most popular VMWare Virtual Appliances for Web Apps (558)
9. Virtual Machine Disk Image Compression (320)
10. rsync vm, vhd for backup, disaster recovery, ec2 (317)

Defragmentation of virtual disk files remains the dominant theme. There is an equal amount of interest in virtual appliances, particularly those for system administrators.

Search terms:

• ext4 defrag ubuntu
• ext4 defrag
• convert vdi to vhd
• e4defrag ubuntu
• virtualbox shrink
• rsync vmdk
• wubi
• defrag ubuntu
• defrag ext3
• windows 7 virtual appliance
• defragment ext3
• vmware appliances
• defrag ext4
• xen vhd
• ubuntu ext4 defrag
• defrag ext4 ubuntu
• vmware firewall appliance
• vmware appliance
• “vdi to vhd”
• convert vhd to xen
• ext3 defrag
• windows 7 beta vmware virtual appliances
• defrag fedora
• ext3 defragmentation
• virtual appliance windows 7
• ubuntu defrag
• hercules load balancer virtual appliance
• fedora defrag
• convert vmdk to xen
• shrink vmware disk

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

One of my earliest posts on this blog was about installing and configuring Ubuntu 8.10 within a VM, and my feeling that I did springboard into the deep end of a frigid pool on a frosty winter day. I am delighted with the experience of installing and using wubi- the ubuntu installer for windows, on my Windows XP laptop.

User Experience

wubi is an innovative approach towards introducing Windows users to Ubuntu Linux. It preserves the user experience of installing a Windows application using a standard installer and uninstalling it later from Control Panel >  Add/Remove Programs seamlessly.

• It provides an installation wizard implemented as a  standard Windows executable (.exe), double click on it and Ubuntu gets installed; users don’t have to deal with ISO images
• wubi installs ubuntu on the desktop as a NTFS file and uses the ext3 filesystem for its contents within (escalating file defragmentation needs, many of our readers visit this blog prescisly for this topic)
• NTFS-3G (Linux NTFS) driver with write support
• Grub4Dos as a boot loader – every time the laptop reboots, I am presented with a choice of whether I want to start Windows XP or Ubuntu
• Ubuntu appears as a program in the Control Panel > Add/Remove Programs.
• Uninstall removes all the artifacts cleanly

Pros and Cons:

• The Ubuntu GUI looks stunning , however, it is sluggish compared with the response time I get on XP (Harvey Su, remember him?, has a ghoulish imagination and he has by now trained all of us to view the background pattern as a giant lion that  has a void in its skull and a  skeleton dangling from its open jaws. Definitely PG-17 material)
• Ubuntu found the printers on our LAN (this blew me away), it even found the PostScript profile (.ppd file?) for our HP LaserJet automatically, however, I when I tried to install it, the installation failed with no diagnostics.
• It could not find the Postscript profile  for another Canon printer/copier/scanner, however, unlike Windows, it did not lead me to a website where I could download it from.
• It found a Broadcom wireless driver for my Dell laptop, I was able to install and activate it, however, some interaction with our Active Directory authentication prevented me from getting access to our secure wireless network. Once again, there was no diagnostic to indicate what went wrong.
• I would be glad to send the wubi developers log files except that I don’t know what to send to whom, however, it would be nice to have a utility that gathers all the relevant diagnostics and beams it up to “mother ship” over http.
• Update: It would be cool if I could share a folder between XP and Ubuntu so that I could install the ISO’s I had downloaded earlier on  XP.

In conclusion

There is an excellent how-to guide prepared by parthodeep for your reference.

Kudos to Agostino Russo and team for an outstanding job.

 Ubuntu JeOS

• How to develop virtual appliances using Ubuntu JeOS

VMWare Studio

• Studio: A Free Virtual Appliance Authoring Tool With Robust Management Features
• Videos: Learn how to build a VMWare virtual appliance
• Create your own Linux appliances

Parallels

• How to build a Parallel’s virtual appliances

rPath rBuilder

• rBuilder Appliance creator
• Best practices for building virtual appliances

Rightscale

RightImages provide ready-to-go base operating systems with core cloud software.

ServerTemplates also allow you to designate any number of scripts that you want to run at boot time, upon demand, or when an event is triggered.

RightScripts allow you to specify packages that you want to install before a script is executed and even allow you to upload and attach files directly to scripts.

Amazon Machine Image (AMI)

• Use an existing AMI
• Create your own AMI from scratch

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.