Managing Petabyte Scale Workloads In The Cloud

data_tsunami

The cloud tsunami is here. There is a massive move to the cloud with new applications being birthed in the cloud. The amount of data and applications in the cloud has risen exponentially. For most customers, it is not a matter of if, but when they move to the cloud.

We all know about the cloud promise. The cloud has promised us a safe, secure, agile and cost-effective computing environment. The question is ‘Are we there yet? Is the promise here?’ This question is even more pertinent when we talk about petabyte-scale workloads.

From a cost perspective, public clouds have done a phenomenal job helping customers move from CAPEX to OPEX model. New evolutions, such as pay-as-you-go and micro compute—like Lambda & Azure Functions—have helped dramatically reduce compute costs. You can now pay for a milli-second of compute and scale as you grow.

From the point of agility, it’s a mixed bag. The agility of the cloud from being able to provide burst computing, is unmatched. You can fire up a 1000 virtual machines in the cloud to run analytics and then shut it down—this is something you just can’t imagine being able to do on-prem. However, there are many locks – vendor locks, architecture locks and cloud locks—which prevent you from being entirely agile. Some new technologies, like micro-compute, not only have steep learning curves, but also require you to completely rewrite the application.  The cloud is agile, as long as you stay within your framework.

Let’s take the example of a Fortune 500 organization that has eight petabytes of data. The task of ingesting and making the data online in the cloud is a mammoth task in and of itself. An average virtual machine on Azure can mount about 40 TB of data. Yes, there are bigger machines which cost a lot more. This would require you to fire up 250 VMs just to make the data online before you even do something meaningful. And before you even make it available, it’s your task to manually slice and dice this data. It’s a Home Depot-style, DIY effort. Take this a step further, you now want to ingest this data to cloud services like Hadoop or Azure Media Services or many of the PaaS components available on Azure or AWS. This further requires you to manually manage more infrastructure in the cloud.

No doubt, the wise men (CIOs) with large amounts of data are still on the fence till a better framework comes by to help ingest, manage and ignite petabyte-scale dark data.

Vijay Ramaswamy @ The CloudLanes Team

Advertisements

Chain of Custody

chain-of-custody-2

Chain of custody (CoC), refers to the act, manner, handling, supervision and/or control of media or information (usually, but not always, tape). The ultimate goal of successful chain of custody is to preserve the integrity of the resource (tape, video etc.) while providing a detailed audit log of who has touched it, when and where.  Chain of Custody is usually offered by remote vaulting providers like Iron Mountain and requires implementation of several best practices described below:

  • Tracking: Media should be tracked by unique bar codes and reports should be generated detailing the current location of the media. If the media is stored across cloud providers, a best practice is to report on a regular cadence on what media reside on which clouds.
  • Security/Encryption: The off-site location and the process used to access the media should be analyzed for security practices. Media should be placed in locked containers before leaving the data center and subsequent tracking done at the tape container level. For cloud-based vaulting, similar measures need to be in place. End to end encryption is a must and data must not be left unencrypted at any time.
  • En-Route Security / In-Flight: Whether it’s you or the physical media recipient who handles the arrangement, make sure the transportation is sufficiently secure. This includes bonding and background checks on the drivers and other personnel, using reasonably secure vehicles and making sure that the media will be carefully tracked through the entire transportation process. Ideally, the tapes will be locked in secure containers and the keys will be held securely at the other end of the trip. Just like security policies need to be in place while media is being vaulted from the data center to remote location, encryption of data before it leaves the premises and revalidation of the data needs to happen once data is vaulted in the cloud.
  • Verification: On a regular cadence, rotating media need to be rotated and verified to ensure they can be read. With physical tapes, it is impossible to verify all the tapes that are vaulted on a regular cadence. The agility and elasticity of the cloud provides an advantage allowing customers the option to verify all their tapes.
  • Movement of Media: Tracking should be performed by scanning bar codes every time the media container is moved, including at data centers and at off-site locations. Media containers should be signed for and never left exposed for someone to take. For cloud based vaulting, similar measures need to be in place. Media needs to be tracked as they are moved between tiers of cloud or between cloud providers.
  • WORM: Option should be available make data immutable. One should be able to read the data but not accidentally erase or overwrite it. Physical tapes have a tab which is broken to make the tape read-only.
  • Guard against Malicious Intent: Safety rails need to be in place for protecting against malicious intent. Detailed logs should be available of where and when things happened and under whose control.
  • Audit Logs: Detailed audit logs need to be provided tracking every step of chain of custody irrespective of where the media is vaulted at a remote site or on the cloud.
  • End of Life & Shredding: Once the media has reached obsolescence or can no longer be relied upon for its integrity, the media must be appropriately destroyed. The destruction of magnetic media is usually accomplished by applying some destruction process to the cartridge, either scrambling the data on the tape or destroying the tape altogether, rendering it useless. For cloud-based solutions, all copies of data need to be shredded.

CloudLanes is the leading provider of digitally archived media in the cloud with ‘Chain of Custody’ and enables implementation of several best practices described above.

-Vijay Ramaswamy @ The CloudLanes Team

Tape: Don’t Burry Me. I Am Alive & Kicking Butt!

 

rip-tapeFor years, disk backup vendors have thrown FUD at tape-based backup solutions and tape media itself.  Even many from the analyst community concurred and prematurely declared “Tape is dead”.  However, the truth is far from it. Approximately 5 million LTO drives and more than 280 million LTO tape cartridges have been shipped since the format’s inception and the amount of data being stored on tape continues to grow.

lto-consumption-chart

The Cloud Changes Everything

The latest innovations in cloud storage have enabled a new era for backup, recovery and archiving. There are now new ‘cool’ and ‘cold’ tiers of storage being offered by leading cloud providers including AWS and Azure.  Though some of the back end technologies being used are not externally disclosed, it is generally agreed upon that they are a perfect use case for long-term data archival, traditionally being served by tape media.  It is anybody’s  guess what that back end media needs to be for the cloud provider in the long term to achieve those cost economics. After all, it’s tape media which wins the storage economics hands down.

Tape Media vs Tape Format

We at CloudLanes believe that the jury is out there about longevity of the tape media itself. However, the tape format has a compelling use case. LTFS based systems offer a unit of archival which no other storage unit offers. An archival atomic unit of storage in tape format also naturally lends itself to chain of custody tracking which is a huge additional benefit. A digitized archival unit now in tape format can now be managed with meta data around it which offers compelling value proposition as data can then be archived in the cloud, across clouds, managed and analyzed at scale.

Tape as a technology ain’t dead yet. It’s just getting better with age.

Vijay Ramaswamy @ The CloudLanes Team 

Would appreciate your thoughts on cloud, archiving, and tape technologies.

 

Azure Cool: The Cool Kid On the Block

Azure Cool

Recently, Azure (MSFT) announced another storage tier in the cloud called ‘Azure Cool’. As the name suggests it is targeted at ‘cool data’.  It seems the industry focus has moved from high performing hot data to glacier age cold data to now cool data.  So what exactly is ‘cool data’?

Cool Data

MSFT with Azure Cool is now targeting non-mission critical data. Example use cases for cool storage include backups, media content, scientific data, compliance and archival data. In general, any data which lives for a longer period of time and is accessed less than once a month is a perfect candidate for cool storage. Cool data, in our opinion, forms a significant portion of the larger secondary data storage market.

Ecosystem

It’s good to see Microsoft working with the eco system partners to build out solutions around Azure Cool. At launch, Microsoft did announce integration with leading data protection providers including Commvault, Cohesity, and Veritas besides others. The API for cool storage is the same as hot blob storage which reduces the complexity of integrating with such a solution.

Performance, Availability & Pricing

Interestingly the performance of Cool is the same as Hot tier with a lower level of availability (99% vs 99.9%).  This is an interesting twist to the positioning of Cool tier, as it provides a higher performance compared to similar offerings from AWS and Google. Of course, the pricing for Cool is cheaper at  $.01/ GB/mo. I am sure most enterprise customers have a better deal than what is advertised via their ELAs (enterprise agreements).

 So where do you start?

Cool tier has lower cost of storing data but higher cost of accessing data and it should be used for data that is not frequently accessed and has lower availability SLA requirements. If you do not know the access pattern for your data, MSFT recommends that you start with the Hot tier for at least a month and then determine whether you can save costs by moving to Cool storage based on your actual usage.  Surely, something easier said than done.

What would be interesting is to have lifecycle policies provided by Azure around the storage management of the objects in the hot tier such that they are automatically migrated to the cold tier or between tiers when access patterns change or when certain cost thresholds are crossed etc.

Cool Kid, welcome to the neighborhood. Some interesting use cases for you around the corner.

Let’s stay cool till then!

-The CloudLanes Team