Recently, The Planet published a white paper comparing Cloud Storage performance as offered by The Planet (which uses Nirvanix), Amazon S3 and Rackspace CloudFiles. It did a nice job of creating a performance-oriented benchmark, comparing Cloud Storage file upload and download time for the three services. While it is necessary to understand this factor associated with Cloud Storage, it is far from sufficient and much more is needed, if one wants to begin assembling metrics and from these make business and technology decisions.
While this does paint The Planet’s offering in a very positive light, one has to question the pertinence of the actual test. (Knowing how our offering would have performed in the specific test, as the CEO of Mezeo, I am a bit disappointed that we were not included in the test) Of course, at the end of the day we have to remember that this is after all a test conducted by one of the vendors who also somehow turned out to be the winner. A third party validation of the results would certainly be more credible. To that end we are assembling a slate of comparisons and seeking third party verifications, and hope to publish these results in late summer.
A notable omission in the test was the comparison of the price per GB of the storage. Based upon published prices, The Planet offering is significantly more expensive than the others.
This introduces the idea of price performance, and that is very applicable, since many Cloud Storage use cases do not contemplate that the accessing server is housed in the same data center as the Cloud Storage solution it is accessing. When access via the Internet is contemplated, the specific speeds of upload and download as revealed in the aforementioned test may be less pertinent, whereas price or other features may be the major differentiator of note.
How about the following metrics:
- availability: a fact based measurement
- reliability: based on the SLA offering
- ease of Web Services API access: length and complexity of required API usage against a defined set of standard actions
- access methods: WebDAV, NFS, CIFS
- feature/function richness and differentiation: sharing and collaboration, available clients, tagging, geo capabilities
- utility billing: true pay for use and on what time frames and at what minimums, and ultimate scalability (how much storage is immediately available).
All or any of these may be highly pertinent to a Cloud Storage decision.
And finally, how about running the test enough times so that the results are reliable and meaningful? Running the test for a total of four times and concluding that Amazon S3 has a 90+% variance is a bit of a stretch.
Please do not take this blog post as being too critical; I am grateful that The Planet is signaling a degree of market maturity as real, meaningful discussion of Cloud Storage attributes are brought to the marketplace. The Planet is to be congratulated for elevating the discussion. Now all Cloud Storage service providers, and infrastructure providers who enable storage cloud offerings, need to begin the hard work of defining the metrics, and publishing the results.
It is very interesting that we are still working out the definition of Cloud Storage and now we are faced with benchmarking competing service offerings. The Cloud Storage marketplace is growing rapidly, and all of us are engaged in bringing this new capability to market.
Thanks for reading this post, and stand by, more on this topic coming soon.