Find VM’s older than N days to free disk space
I wrote a Python 2.6 script to find and list VM’s older than 90 days on my Windows workstation, so that I could compress them, move them to a 1TB drive attached to my machine, or to a file server, or delete them.
find_old_vms is a tool to find and list old VM’s (vmdk’s, vhd’s) on your hard drives that are older than a given number of days.
Usage: find_old_vms in_this_directory_tree older_than_days
Example: find_old_vms “c:\\” 90
Download for Windows XP, 2003, and Linux. The script uses atime (latest file access time), which is not supported on Windows Vista and Windows 7.
Code:
# # NOTE: This script uses atime - the last access time for deciding whether a VM # **** file is a candidate. On Windows XP, atime is updated every hour, whereas, # Windows Vista and Windows 7 do not provide an atime. import os, sys, glob, time # dtroot is the pathname for a node in a directory tree # age is the number of days for which a file has not been accessed # size in bytes is the maximum size of a file def scan(dtroot, age, size): """ scan <dir> scans the <dir> on host for virtual images """ filecount = 0 wctime = time.time(); # get current time for root, subdirs, files in os.walk(dtroot): # Build a list of filenames that have a suffix vmd* or vhd* vfiles = glob.glob(os.path.join(root,"*.v[mh]d*")) for f in vfiles: atime = os.path.getatime(f) elapsed_time = (wctime - atime)/(60*60*24) if elapsed_time > age: filecount = filecount + 1 print f, " last accessed ", int(elapsed_time), " days ago\n", if __name__ == "__main__": import sys # User asked for help if sys.argv[1] == '?': print "\nfind_old_vms in_this_directory_tree older_then_days\n", print "Find all files with the suffix .v[mh]d* that have not been accessed since older_than_days", print "through a recusrive descent starting from the root in_this_directory_tree", sys.exit() # Validate that the first argument in_this_directory_tree is a valid path if not os.path.exists(sys.argv[1]): print sys.argv[1], " is not a valid directory. Please provide another", sys.exit() # Arbitrarily limit search to 10 years invalid_age = 0 if int(sys.argv[2]) < 1: invalid_age = 1 elif int(sys.argv[2]) > 3650: invalid_age = 1 if invalid_age == 1: print sys.argv[2], " is invalid. Please provide between 1 and 3650 days", sys.exit() scan(sys.argv[1], int(sys.argv[2]), 0)
If you remove the restriction of searching for vImh]d* files, it will help you find other older files as well. I will appreciate your feedback.
Vizioncore vOptimizer Pro to shrink or enlarge VM’s
You can download vOptimizer WasteFinder – 2.2 free from Vizioncore to analyze your VMware vCenter Server / ESX hosts and report how much free disk space can be recovered from over-allocated virtual storage. It also locates all VM’s that are not properly aligned on Windows 64K partition boundaries and as a result experience decreased I/O throughput and higher latency.
In case you wish to reclaim the free disk space and align VM’s on Windows 64K partition boundaries, you will have to purchase vOptimizer Pro 2.2.248.0 (free 14 day trial).
vOptimizer Pro can not only shrink VM’s to reclaim the over-allocated virtual storage, perhaps more importantly, it can also enlarge VM’s that are running out of storage, effectively preventing painful and costly VM outages.
I just wonder, wouldn’t it be better to simply offer a 14 day free trail that lets users test drive vOptimizer that induces them to buy it?
A year in review: What are our readers looking for?
Our readers are primarily asking questions like:
- How can I free disk space, on Windows, and on ext4, ext3 on Ubuntu and Linux, within virtual disks like vmdk, vhd and vdi?
- Where can I find the best virtual appliances/ Top 10 virtual appliances?
- How can I convert from one virtual disk (vmdk to vhd, or vdi to vhd) to another?
- Who are the competitors for ec2?
An analysis of the search terms shows interesting clusters:
|
Serial |
Topic |
% of queries |
Search terms |
|
1 |
ext4 defragmentation |
23% |
ext4 defrag, defrag ext4, ext4 defragment, defragment ext4 |
|
2 |
ubuntu ext4 defragmentation |
14% |
ext4 defrag ubuntu, ext4 ubuntu defrag, ubuntu ext4 defrag, ubuntu defrag ext4, defrag ext4 ubuntu, defrag ubuntu ext4 |
|
3 |
vmware virtual appliance |
14% |
vmware virtual appliance, vmware virtual appliances, top vmware appliances, top 10 vmware appliances, best vmware appliances |
|
4 |
virtual appliance |
5% |
virtual appliance, virtual appliances, top appliances, top 10 appliances, best appliances |
|
5 |
vmware firewall appliance |
5% |
vmware firewall appliance, vmware appliance firewall |
|
6 |
ubuntu defragmentation |
4% |
defrag ubuntu, ubuntu defrag, defragment ubuntu, ubuntu defragment |
|
7 |
ec2 competitors |
4% |
amazon ec2 competitors, ec2 competitors |
|
8 |
windows 7 virtual appliance |
4% |
windows 7 virtual appliance, virtual applaince windows 7 |
|
9 |
ext3 defragmentation |
4% |
ext3 defrag, defrag ext3, ext3 defragment, defragment ext3 |
|
10 |
convert vdi to vhd |
3% |
convert vdi to vhd, vdi to vhd |
If I abstract it out, our readers are primarily interested in learning how to free disk storage and where to find the best / Top 10 vmware, Xen and Windows virtual applainces.
Thank you. I appreciate your interest in this blog.
Top 10 VMware Virtual Appliances for Security
I have reviewed several appliances in the Secure Content and Threat Management and Identity, Access and Vulnerability Managament categories of the VMware Appliance Marketplace to identify the Top 20 appliances. Here are the steps I followed for selecting the appliances listed below:
- I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
- I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
- I also discarded several appliances rated 4 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.
|
S. No. |
Virtual Appliance |
What is it used for? |
Download Link |
Average Customer Rating |
Number of Reviews |
Pricing |
|
1 |
Protects Internal Networks from Malicious Traffic in Demanding Virtual Environments |
5 |
0 |
Free trial with registration |
||
|
2 |
Web application firewall with automated adaptive learning and HTTP load balancing |
5 |
0 |
Free Trial, USD 2950 per subscription |
||
|
Web application firewall with automated adaptive learning, load balancing and XML support. PCI DSS and OWASP Top Ten compliant |
Free Trial, USD 5950 per subscription |
|||||
|
3 |
The only VTL solution that improves the quality and efficiency of tape backup in virtual enviroments. |
5 |
0 |
Registration 30 day trial |
||
|
4 |
The WiKID Strong Authentication Enterprise Edition VMware 3.3.8. Support for Google SSO/SAML has been added |
5 |
0 |
USD 24 per user |
||
|
5 |
1st Purpose-Built Virtual Firewall |
5 |
0 |
Free trial with registration |
||
|
6 |
HyTrust Appliance provides a single point of control for hypervisor configuration, compliance, and access management. |
5 |
0 |
Free |
||
|
7 |
Total Web Security software for any organisation |
5 |
0 |
Free Trail, USD 5 per |
||
|
8 |
Provides data integrity protection by centralizing and preserving sensitive data,making it tamper-evident at the highest detail. |
5 |
0 |
Free Trial, EUR 10,000 |
||
|
9 |
AEP Netilla SSL VPN is a secure application access gateway that enables secure, web browser access to a range of business apps. |
5 |
0 |
Free trial, USD 1 |
||
|
10 |
Comprehensive email security gateway reduces TCO with immediate protection from spam, phishing, malware and data leaks |
5 |
0 |
Free Trial, USD 19.67 per user |
||
|
11 |
InterScan Web Security Virtual Appliance applies real-time web reputation, flexible content scanning and powerful URL filtering. |
5 |
0 |
Free Trial, USD 13.45 per user |
||
|
12 |
BackTrack is a penetration testing oriented live CD and is the result of the merger of WHAX and Auditor. |
4.5 |
0 |
Free |
||
|
13 |
Symantec Brightmail™ Gateway Virtual Edition (formerly Mail Security 8300) |
Inbound and outbound messaging security, antispam and antivirus protection, advanced content filtering, and data loss prevention |
4.5 |
0 |
Free Trail, USD 15 per user |
|
|
14 |
Internet Privacy Appliance : Encrypts your Internet traffic, hides your IP address, and is easy to setup. |
4.5 |
0 |
Free |
||
|
15 |
*SpamTitan allows you create a Email Security Appliance for your Gateway |
4.5 |
0 |
Free hosted trial, USD 395 per subscription |
||
|
16 |
gateProtect solutions combine state of the art security and network features such as firewalls, bridging, VLAN, single sign-on, traffic shaping, QoS, IPSec/SSL (X.509), IDS/IPS, web filters, virus filters, real-time spam detection and HTTPS proxy in one system |
4.5 |
0 |
Free |
||
|
17 |
Secure File Transfer Virtual Appliance – secure, economical and easy to use secure file transfer for today’s global enterprises |
4.5 |
13 |
Free hosted trial |
||
|
18 |
Best-of-breed open source network security applications with supporting scripts and a web-based front-end management interface. |
4.0 |
0 |
Free |
||
|
19 |
LogLogic Security Change Manager- Great for Firewall Coversions! |
Streamline the design and deployment of network security rules for firewalls, routers, switches, VPN, and IPS’s. |
4.0 |
0 |
Free |
|
|
20 |
Proven Security for Virtual Environments |
4.0 |
0 |
Free Trial |
Which virtual appliance do you use the most and why do you like it?
Top 10 VMware Virtual Appliances for IT Administrators
I have reviewed several appliances in the IT Administration and Systems Infrastructure categories of the VMware Appliance Marketplace to identify the Top Ten and more appliances. Here are the steps I followed for selecting the appliances listed below:
- I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
- I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
- I also discarded several appliances rated 4.5 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.
|
S. No. |
Virtual Appliance |
What is it used for? |
Download Link |
Average Customer Rating |
Number of Reviews |
Pricing |
|
1 |
WebGUI is an open source content management system built to give business users the ability to build and maintain complex web sites. |
5 |
0 |
Free |
||
|
2 |
Opsview 2.12 Virtual Machine – Network and Application Monitoring |
Opsview is enterprise network and application monitoring software designed for scalability, flexibility and ease of use. |
5 |
0 |
Free |
|
|
3 |
Web Interface for managing Nessus Vulnerability Scans and results |
5 |
0 |
Free |
||
|
4 |
Open Source Business Intelligence Suite |
5 |
0 |
Free |
||
|
5 |
up.time allows you to monitor, measure and manage your physical and virtual IT infrastructure from a single centralized console. |
5 |
5 |
$695 per license |
||
|
6 |
Web application firewall with automated adaptive learning, load balancing and XML support. PCI DSS and OWASP Top Ten compliant |
5 |
0 |
Free Trial, $5950 per license |
||
|
7 |
The only VTL solution that improves the quality and efficiency of tape backup in virtual enviroments. |
5 |
0 |
Registration 30 day trial |
||
|
8 |
Virtual WAN Optimization Controller |
5 |
0 |
Free Trial |
||
|
9 |
The AllardSoft Secure Filetransfer Appliance allows you to send very large files securely using a standard web browser. |
5 |
5 |
Free trial, $79 per license |
||
|
10 |
FOG is a computer imaging/cloning solution with many advanced features includes web gui and client service. |
5 |
0 |
Free |
||
|
11 |
High-Class scalable anti spam solution from small business to enterprise. Developed in Europe/Austria |
5 |
0 |
Free |
||
|
12 |
*Most popular Firewall appliance* All-in-one firewall package upgraded with VMXnet, heartbeat and MUI-control. |
4.5 |
0 |
Free |
||
|
13 |
Mail Server appliance |
4 |
0 |
Free |
Which virtual appliance do you use the most and why do you like it?
How To expand your vmdk file
What should you do when you run out of virtual disk space and have to expand it? Here is an iinsightful article that describes the gymanstics.
VMware Appliance Marketplace Ratings Analysis
During the past week, I was analyzing the recently revamped VMware Appliance Marketplace. My analysis is summarized below:
Salient Statistics
- 9% are top-rated, i.e., have a Star Rating of 5
- 41% have a Star Rating of between 3 and 4, i.e., this cluster may be providing the most value to its users, even though it is not top-rated.
- There is a steep drop-off in the number of appliances that have a Star Rating below 3
- 31% are unrated, which seems to indicate that a significant majority has been submitted recently
- Only 3% (40/1227) have Reviews – this is a very small number. The number of reviews have to grow substantially to indicate that a vibrant community, like Amazon.com reviews, has formed here
Transparency
I feel there is a greater need for transparency. What do the Star Ratings mean?
- Are they correlated with the number of downloads for that appliance?
- What is the impact of reviews on these ratings?
Beware Star Ratings alone! The following appliance rated 4.5 has 4 reviews, read them and you will know what I mean.
As a member of VMware’s Technology Network community, I will urge VMware to provide greater transparency around how these ratings are computed/awarded. It will greatly help the users understand the ratings system.
What do you think?
Data
|
VMware Marketplace Ratings |
Number of User Reviews |
|||||||||
|
Star Rating |
Total for this Rating |
% of Overall |
14 |
6 |
5 |
4 |
3 |
2 |
1 |
Total Reviews |
|
5 |
115 |
9% |
|
|
2 |
|
|
3 |
9 |
14 |
|
4.5 |
40 |
3% |
|
|
|
1 |
|
|
2 |
3 |
|
4 |
151 |
12% |
|
|
|
|
1 |
|
3 |
4 |
|
3.5 |
158 |
13% |
|
1 |
|
|
|
|
1 |
2 |
|
3 |
196 |
16% |
|
|
|
|
|
1 |
2 |
3 |
|
2.5 |
64 |
5% |
|
|
|
|
1 |
1 |
|
2 |
|
2 |
24 |
2% |
1 |
|
|
|
|
1 |
3 |
5 |
|
1.5 |
13 |
1% |
|
|
|
|
|
|
|
0 |
|
1 |
87 |
7% |
|
|
|
|
|
|
7 |
7 |
|
0 |
379 |
31% |
|
|
|
|
|
|
0 |
0 |
|
Total |
1227 |
100% |
1 |
1 |
2 |
1 |
2 |
6 |
27 |
40 |
DropBox dedup only in the cloud
I had observed in my earlier article that DropBox performs de-duplication in the cloud. This would mean that de-duplication is not performed at the client. In order to test my hypothesis, I performed the following experiment:
I first looked at the size of the DropBox folder on Windows and found it to be 1,723,871,232 bytes.
Next, in the DropBox client, I opened the DropBox folder and simply duplicated the contents of the Public folder by copying the 1.68MB file and pasting it as its copy. I looked at the size of the folder once again and it had doubled to 3,446,513,664 bytes.
If DropBox had been performing dedup at the client, then it should have detected the duplicate blocks between the parent and its copy at source and the folder should not have grown in size at all. As a result, my conclusion is that DropBox dedup’s only in the cloud but not at the client.
Wait, there’s more:
I repeated the same experiment on the Mac after deleting the duplicate file. Here’s what I started out with:
Last login: Thu Sep 17 15:30:58 on ttys000
mace1s:~ paule1s$ du -k DropBox
1152 DropBox/Photos/Sample Album
1516 DropBox/Photos
1682636 DropBox/Public
368 DropBox/sharevm
1684880 DropBox
Notice that the total size of the folder (the last line of the listing above) is 1.68GB.
Next, in the DropBox client, I opened the DropBox folder and simply duplicated the contents of the Public folder by copying the 1.68MB file and pasting it as its copy. I looked at the size of the folder once again and saw:
mace1s:~ paule1s$ du -k DropBox
1152 DropBox/Photos/Sample Album
1516 DropBox/Photos
2600140 DropBox/Public
368 DropBox/sharevm
2602384 DropBox
This is very interesting. I had expected the storage requirements to double to 3,369,760 however, they grew by approx. 1GB. What happened to the remaining 682MB? Did the DropBox client truncate the file? If so, why?
Readers, can you shed some light?
Compressed VM file transfer using DropBox
I am using DropBox for transferring compressed files including VM’s between my environment at home, a Mac running Windows XP SP3 in VMware Fusion 2.0.5 and the test machine, a Windows XP SP3 system located in the office lab. Each machines has a DropBox folder linked to the same account.
Neat product!
I love the simplicity and ease of use. A lot of thought has gone into making the product easy to install, the integration with the host OS (Windows and Mac) is seamless and sets a benchmark for how UI’s for downloadable products should be designed.
Usage model
I compress each file using the Mac’s native file compression and drop into into my DropBox folder. DropBox seems to follow a two-step file transfer process:
- It first uploads the file completely from the source DropBox folder to the DropBox folder in the cloud
- After the upload is complete, the file is then downloaded from the DropBox folder in the cloud to the destination DropBox folders.
Setup
Speed ratings are from here. I have been able to correlate these speeds with the end-to-end transfer times.
|
Transfer Type |
Speed Rating for my ISP |
Observed DropBox Transfer Rate |
|
Upload |
120 KB/sec |
70 KB/sec |
|
Download |
360 KB/sec |
210 KB/sec |
Near real-time transfer for uncompressed files
DropBox transfers uncompressed files almost instantaneously between the two machines. The files are transferred sequentially and seem to arrive in order. For example, I transferred a 1.72 GB folder containing 400 photographs and the photos started appearing sequentially 10 – 15 seconds apart.
Compressed files
Compressed files are transferred as a unit, although dedup applies to blocks contained within it. The transfer times are as recorded below:
|
Original Size |
Compressed Size |
Upload Time |
Download Time |
Total Time |
|
4.30 GB |
1.6800 GB |
6h 40m |
2h 12m |
8h 52m |
|
2.15 GB |
0.6714 GB |
2h 27m |
0h 48m |
3h 15m |
|
1.10 GB |
0.2371 GB |
0h 56m |
0h 18m |
1h 14m |
Dedup works well with compressed files
DropBox examines the file to be transferred and builds an index of blocks to be transferred. Its de-duplication technology is smart enough to figure out when not to transfer blocks that are duplicates, i.e., have already been transferred before. For example, when I tried to transfer two clones, the first one took a long time to transfer ( a few hours), but the second transfer was very rapid (under five minutes).
Since I am using the free account, I deleted a 2GB VM from my DropBox folder in order to begin my next transfer. I was pleasantly surprised to see that the next VM transfer was very rapid. I suspect this was because the VM that was transferred earlier was still residing in DropBox’s cache even though I had deleted it, so that DropBox discovered common/duplicate blocks and did not upload them from my Mac.
Summary
Nifty tool. Love it. Will use it a lot.
A few feature requests
- Subfolders: I would like to organize the files by date and category.
- Timers: I would like to time the uploads and downloads easily.
- Profile my usage and suggest how long an end-to-end transfer will take
- Speed up compressed file transfers – improve my effective transfer rate from ~60% to ~80%- I would like to saturate the available bandwidth for uploads and downloads
Thanks 
gzip vs dedup: I shrink, therefore I am
[reposted from rosensharma.wordpress.com]
I stole “I shrink, therefore I am” from my wife’s good friend Arun Verma, who is incredibly creative, and makes some of the best lamps ever. He also does websites and ads if you are interested.
I have a macbook and use vmware fusion to run a windows XP VM. I keep all my data on a hosted folder on the mac’s operating system. So the VM is basically programs and user settings. In addition I have several images which I work with: Red Hat Enterprise, Ubuntu, Win 2K3 etc. Not atypical of someone who either develops or tinkers with technology.
My problem is that out of a 120GB hard disk, I am upto 100GB, and a whopping 60GB of that is virtual images. I have about 8. So I wanted to see if I could compress the virtual images in some fashion. I decided to run a small test of how much dedup would buy me over gzip
w2k3.vhd: Original size: 1.6GB
w2k3.vhd.gz: 712 MB
Further Analysis of the image showed that there were
14K Zero Filled Blocks, and
About 40K blocks occurred more than once
gzip wxp.vhd –> 921 MB
43K Additional Blocks Repeated between this and previous image
Dedup Optimization: 66K*4K ~ 250MBClearly gzip would win over a simple dedup. Even with two images xp and w2k3 I guess there are just not enough blocks to make dedup shine. Less than 10% of the blocks are being found. Cloning in some sense avoids large matches in a small set of images like on the desktop.
