Share Virtual Machines

I have reviewed several appliances in the Secure Content and Threat Management and Identity, Access and Vulnerability Managament categories of the VMware Appliance Marketplace to identify the Top 20 appliances. Here are the steps I followed for selecting the appliances listed below:

• I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
• I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
• I also discarded several appliances rated 4 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.

Which virtual appliance do you use the most and why do you like it?

I have reviewed several appliances in the IT Administration and Systems Infrastructure categories of the VMware Appliance Marketplace to identify the Top Ten and more appliances. Here are the steps I followed for selecting the appliances listed below:

• I have relied on the Average customer Rating, expressed as a 5 star, or a 4 star, etc., rating (you may wish to review my analysis of VMware’s ratings)
• I discarded all virtual appliances that solely package OS distributions, primarily, ubuntu, fedora, etc. My rationale is that an OS by itself provides low business value to an IT Administrator. While an IT administrator can use these just as if they were using a ghost image, these virtual appliances neither package applications in a usable form, nor simplify the task of installing and configuring the applications that provide business value. Besides, the base OS virtual appliances are available in a category by themselves
• I also discarded several appliances rated 4.5 star or less, which are present in the directory but have either broken or stubbed out download links. They seem to have been retained in the directory to beef up the appliance count, however, they are not useful to the community.

Which virtual appliance do you use the most and why do you like it?

During the past week, I was analyzing the recently revamped VMware Appliance Marketplace. My analysis is summarized below:

Salient Statistics

• 9% are top-rated, i.e., have a Star Rating of 5
• 41% have a Star Rating of between 3 and 4, i.e., this cluster may be providing the most value to its users, even though it is not top-rated.
• There is a steep drop-off in the number of appliances that have a Star Rating below 3
• 31% are unrated, which seems to indicate that a significant majority has been submitted recently
• Only 3% (40/1227) have Reviews – this is a very small number. The number of reviews have to grow substantially to indicate that a vibrant community, like Amazon.com reviews, has formed here

Transparency

I feel there is a greater need for transparency. What do the Star Ratings mean?

• Are they correlated with the number of downloads for that appliance?
• What is the impact of reviews on these ratings?

Beware Star Ratings alone! The following appliance rated 4.5 has 4 reviews, read them and you will know what I mean.

As a member of VMware’s Technology Network community, I will urge VMware to provide greater transparency around how these ratings are computed/awarded. It will greatly help the users understand the ratings system.

What do you think?

Data

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (765)
2. http://networksecuritytoolkit.org/nst/index.html (566)
3. http://dabcc.com/article.aspx?id=9653 (149)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (111)
5. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (101)
6. http://stumbleupon.com/refer.php?url=http://sharevm.wordpress.com/2009/01/19/most-popular-vmware-virtual-appliances-for-it-administrators/ (84)
7. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (67)
8. http://thedarkmaster.wordpress.com/2007/03/12/vmware-virtual-machine-to-virtual-box-conversion-how-to/ (66)
9. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (66)
10. http://prefetch.net/blog/index.php/2007/01/21/determining-file-fragmentation-on-ext3-file-systems/ (61)
11. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (52)
12. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (53)

Thank you for the referrals. Hope the content is meaningful for our readers

Total VM’s on VMWare Virtual Appliance Marketplace : 2560
Number of distinct download sites providing these appliances : 674

The following table lists the compression formats used for 408 appliances we have examined:

Amongst the compression formats, zip is the most popular one (42%), with 7z the next in sequence (23%) for Windows; gz is the most popular one for Linux (18%)

We intend to publish more statistics over the next few days. Watch this space.

When I think about “database virtualization” as

the use of multiple instances of a DBMS, or different DBMS platforms, simultaneously and in a transparent fashion regardless of their physical location

with the goal of providing “data virtualization“,

to view data from disparate sources without knowing or caring where the data actually resides.

this reminds me of location transaprency (ref: Jim Gray and Andreas Reuter)

The databse must insulate the application from the location of the data and the exact representation of the data

The real issue is that there is a lot of hype surrounding virtualization today and database virtualization, at least in the prevalent use of the term, is attempting to shoe horn itself into this space in order to leverage the hype cycle and stock valuations associated with this market segment.

xkoto has built a SQL mediation layer above commodity DBMS’ for the mid-market and IBM has been licensing their technology as a load balancer since 2006. Oracle’s acquisition of TangoSol, Microsoft’s Project Velocity are following HP NeoView’s usage of distributed caches for solving large BI queries. Strictly speaking these are not virtualization technology related innovations – they are simply replacing shared memory clustering with distributed, shared-nothing caches that provide location transparency for data. Tandem was doing this all along from the early 80’s through the mid 90’s. Seems to be old wine in a new bottle – the vintage may still be good, but let us acknowledge it for what it is.

The positioning and messaging innovation in recasting location transparency as “databse virtualization” is an incremental one. What groundbreaking technoloigy innovations can we expect in database virtualization that truly differentiates it and helps startups unveil new disruptive business models?

You may well ask, “Ok Paul, what’s your beef? We are solving real world business problems for our customers”. Your point is well taken, ultimately it is indeed about the customer. If they can map your solution as meeting the technical requirements to be fulfilled for solving their business problem, the marketing labels are a moot point. My concern is about whether the positioning obfuscates instead of providing that sort of clarity to customers.

virtualization.info is reporting that a glimpse of the Cisco, VMWare, EMC strategy has emerged in a post on the personal blog of Chad Sakac, Sr. Director VMWare Strategic Alliance at EMC.

Upon reading Chad’s blog, my impression is that a partnership ecosystem seems to be emerging between Cisco, VMWare and EMC for supporting private clouds within an Enterprise and public clouds, a la EC2, through

• A deep integration with the VMWare Hypervisor or “VMWare’s cloud operating system” using standard API’s
• A broad integration to provide a cross-vendor management fabric that spans across management tools of the respective vendors and enables management of VM’s/Virtual Appliances, the underlying host servers, storage and the network.
• VM/Virtual appliance portability across the private and public clouds

This management layer will permit control over individual VM’s and groups of VM’s within this cloud and permit applications (virtual appliances) to be deployed using Just Enough OS’s (JeOS)

EMC’s internal goals (and perhaps VMWare and Cisco’s, too) seem to be

1) To Drive 100% virtualization

Requires A Virtualization Layer that can literally meet the scaling, performance and availability goals of any x86 workload.

EVERY EMC product is being turned into a Virtual Appliance.

Physical adaptability (i.e. increase/decrease CPU/Memory model) needs to extend into the Networking and Storage stacks. People will REALLY start to see “purpose built servers/network/storage” for VMware in 2009

2)  To drive API Integration

Streamline the integration of existing management tools and capabiities with VMWare’s management tools and capabilties.  

These are about making sure that virtual world is able to do everything the phyiscal world can do.   They make sure that the datacenter CAN be 100% virtualized

3)  To create infrastructure that understands and responds to “VM/Application objects”

the next phase is where things really get blended – where thin provisioning is integrated, where management tasks are integrated, where “VM object awareness” is added, and where networking policy portability really takes off.

vCenter is surely a critical new management point – so expect to see core management capability for EMC storage integrated into vCenter in the very near term. … We’ll leverage existing open APIs to create plug-in extension models. BUT at the same time – we will continue to integrate into the vCenter APIs for integrated views in management frameworks that are “home” to people other than VMware Administrators.

Epilogue

Done right – the Private Cloud and Public Cloud can share the applications transparently, and the “Public Cloud” infrastructure layers can “read the same bar-codes”  Clearly the infrastructure needs to be a bit different (management model, federation, multi-tenancy, scale and price points are all different), but they need to be linked.

This ain’t about consolidating servers (though includes that too!).   It **IS** about the next big transformation we all see coming in the IT space we deal in.   We’re gearing up, and as leaders in our respective spaces, focusing our resources, and driving towards a vision. 

If Cisco, VMWare and EMC indeed work together on this, they will be able to dominate this market for years to come. Very cool, Chad! Thanks for sharing your views.

I was intrigued by an article on database virtualization that caught my eye early this morning and I wanted to find out what is it all about?

The business driver for database virtualization is the globalized economy where business transaction happen 24 x7 x 365 and business critical data must be available within the network boundary of a corporation, or through the Internet, spanning application downtime and IT maintenance windows.

Data virtualization is defined here as

to view data from disparate sources without knowing or caring where the data actually resides.

Data virtualization obviously leads to database virtualization, which is defined here as

the use of multiple instances of a DBMS, or different DBMS platforms, simultaneously and in a transparent fashion regardless of their physical location

James Kobielus, a Senior Abalyst with Forrester Research is predicting that real time information needs will drive database virtualization

the database as we know it is disappearing into a virtualization fabric of its own. In this emerging paradigm, data will not physically reside anywhere in particular. Instead, it will be transparently persisted, in a growing range of physical and logical formats, to an abstract, seamless grid of interconnected memory and disk resources; and delivered with subsecond delay to consuming applications.

He is making an interesting case that

Proof Points

Oracle acquired Tangosol in May 2007 and possesses a well-developed in-memory, distributed-caching technology called Coherence

Microsoft annonuced Project Velocity a year later in June 2008:

a distributed cache that allows any type of data (CLR object, XML document, or binary data) to be cached. “Velocity” fuses large numbers of cache nodes in a cluster into a single unified cache and provides transparent access to cache items from any client connected to the cluster.

xkoto was selling GRIDSCALE as a database load balancer in 2006. However, it is very smartly capitalizing on virtualization being a hot segment and has repositioned GRIDSCALE as a database virtualization product. This is a vaiid repositioning not only in the context of the definitions cited above but more importantly due to its validation by noted industry analysts:

• Robin Bloor, a database industry pundit and blogger, who describes its database virtualization capabilities
• Dan Kusnetzky, another influential industry analyst, author and blogger, whose review appears here

Gigaspaces XAP can be used for implementing a distributed cache (in-memory data grid)

GemStone Systems offers the GemFire Enterprise as a data fabric (distributed cache)

Scaleout Software also has a distributed cache offering

Here are the Top 12 referrers to our blog over the past 3 months, the numbers of referrals are in parentheses.

1. http://pro-linux.de/berichte/ext4/ext4.html (546)
2. http://dabcc.com/article.aspx?id=9653 (342)
3. http://networksecuritytoolkit.org/nst/index.html (110)
4. http://polishlinux.org/apps/cli/ext4-defragmentation-with-e4defrag/ (59)
5. http://communities.vmware.com/thread/189804?tstart=0 (49)
6. http://techblog.41concepts.com/2008/03/31/shrink-your-windows-disk-image-on-wmware-fusion-mac/ (42)
7. http://blog.rightscale.com/2009/01/09/amazon-launches-ec2-console/ (37)
8. http://wordpress.com/tag/vhd/ (33)
9. http://wordpress.com/tag/vmdk/ (32)
10. http://virtualgeek.typepad.com/virtual_geek/2009/01/updated-homebrew-esx-hardware-list.html (32)
11. http://blogs.msdn.com/heaths/archive/2005/07/30/445621.aspx (32)
12. http://kakku.wordpress.com/2008/06/23/virtualbox-shrink-your-vdi-images-space-occupied-disk-size/ (31)

Thank you for the referrals. Hope the content is meaningful for our readers.

I have been reading about SQL performance issues within the VM and have come to the realization that the performance of SQL queries on any database,  Oracle, SQL Server, mySQL, etc. running in a VM can never match the performance of the same DB running on the native host OS. I have seen questions about the performance issue in several Google searches, particularly for SQL Server within VMWare over the past couple of years. I would like to share my insight  about the root causes.

SQL Query Optimizer Support

The database’s cost-based optimizer creates and evaluates several  query execution plans and chooses the one that meets the optimization goal, e.g., fastest response time for returning the first row, or least overall time to compute the entire result, etc. at the the lowest cost. The cost is based on several parameters that govern the query plan execution environment. For example, the query optimizer can examine parameters such as the number of processors on the system, the processor type, the amount of physical memory, the amount of swap space available, the disk throughput in I/O per second, CPU load factors, etc.  The query optimizers of the current generation DB’s – Oracle 10g, Oracle 11i, SQL Server 2005, etc are not designed for computing query plans for the VM environment,  they have a single system image for the VM and do not understand that the underlying hardware resources will actually be shared by several VM’s.

VMWare seems to have begin affording a view of the underlying host system for the query optimizer’s benefit, although, at the time of writing I have not researched what Microsoft and Xen offer.  It is important for you to understand the tuning knobs VMWare and HyperV have to offer for deploying Oracle and SQL Server,  Xen is interesting for me because it is the underlying virtualization platform for Amazon’s ec2, which I am using currently. However, such a view is necessary but not sufficient, because it cannot support  the possibility of running large queries in parallel  (parallel query execution) over  multiple, distributed, databases.

Lower Transaction Throughput

But of course, the pathlengths are longer.   Let me state at the outset that I am willing to learn from experts who write such code for a living and are interested in providing clarity in depth. My goal here is to paraphrase and simplify to get the point across:

Imagine that the database has a relational storage manager that provides methods to open/close tables and set-oriented API’s that iterate over a set of rows for reading or updating them individually, or inserting a new row. The storage manager encapsulates a file system driver for managing reads from and writes to its proprietary data store. Imagine further that the file system driver is a kernel-mode component that  performs the low-level I/O on NTFS or Linux, as the case may be.  The nominal read/write call sequences and pathlengths  are optimized so that the databases perform well for routine use as well as for competitive TPC benchmarks. I am illustrating a notional  call stack below:

SQL application
|
DB query run time (user mode)
|
DB storage manager (user mode)
|
DB file system driver (kernel mode)
|
Host OS (kernel mode)
|
Disk driver

Now let’s see what happens when we we run the DB  within the VM.

SQL application
|
DB query run time (user mode)
|
DB storage manager (user mode)
|
DB file system driver (kernel mode)
|
Guest OS (kernel mode)  <= additional I/O path length
|
Hypervisor (kernel mode) <= additional path length
|
Host OS (kernel mode)
|
Disk driver

In the VM, the OS kernel-mode I/O path is traversed more than once (I am sure the optimizations that VMWare and Microsoft are furiously working on to reduce it from 2x to somewhere in the range of 1.6x – 1.8 x including the overhead of the hypervisor code).

Manageability overhead

The database manages its own proprietary storage structures that differ substantially from the files supported by the native OS. For example, most databases allocate large extents (think, collection of disk blocks) which contain several pages (think, disk blocks), and rows within pages. The management complexity lies in the fact that extents can contain pages of different sizes, e.g., 64K, 32K, 4K, 512 bytes, a set of pages may be linked in a list, some pages may be data pages, some only index pages, whereas some others may contain both data and index rows.  The rows in each page may either be of fixed length or variable lengths. Index pages split or combine based on the fill factors, rows get forwarded when they are updated but cannot fit within the current page, there are shadow rows that get deleted asynchronously after a transaction commits. The point is that the database buries a lot of complexity in dealing with storage efficiently in its proprietary internal representation.

All databases grow. The current best practice that I see on the Web is to pre-allocate the largest storage size you expect your DB to row to. Think about what will happen if your DB is sharing that storage with another app and outgrows it fater than anticipated.

OS/third party tools for de-fragmentation and compression are pretty much useless, or are marginally useful at best. You must use the administration utilities, e.g.,  SQL Server stored procedures, for de-fragmenting the database and effecting space recovery – the OS utilities for de-fragmentation and compression are not effective. Remember that any I/O operations performed within the VM suffer the additional kernel I/O pathlengths of the guest and host OS together with that for the hypervisor, as discussed in the earlier section.

From the manageability perspective, none of the databases except Tandem’s, can perform an online reorganization of data. SQL Server, and to the best of my knowledge Oracle too, requires you to bring the database offline, unload and reload the data in order to remove de-fragmentation. This is a real bummer.

Epilogue

Ultimately it is all about planning. Do your capacity planning and then validate it in the lab to some scale with the database residing in a VM. You should read the manuals and ensure that you truly understand and can justify the performance and manageability trade off for moving the DB to a VM.