Virtualization Technology News and Information
HPC Applications in the Cloud

A Contributed Article by Kris Bliesner, co-founder and CTO, 2nd Watch

It's a game-changing time in the history of computing to be running large-scale, compute intensive applications like HPC. Companies that need to run risk analysis applications, financial reporting, compliance and auditing applications or conduct clinical research requiring massive parallel processing haven't had cost-effective options in the past. Yet today, companies and organizations are running these jobs in the public cloud.

Consider the economics. Instead of acquiring and managing millions of dollars in infrastructure to run an HPC application, a company can spin up cloud services for a few thousand dollars per hour. When the job's done a few hours later, the project manager can turn off the environment. That's bringing considerable flexibility for companies that want to do large-scale demographic analysis of customers or test and retest a theory or prototype. Why HPC in the cloud is a wise decision:

1. Cost: Managing HPC applications on internal infrastructure is exorbitantly expensive to acquire and maintain given that the environment needs to handle an unpredictable scale of processing. It's even harder to justify procuring this technology when it's not used regularly. HPC applications, after all, are typically used periodically for a specific purpose, not for everyday business needs. So a company is paying for infrastructure that's possibly only being used 5-10% of the time. With the on-demand "pay by the drink" structure of the public cloud, companies purchase hourly chunks of thousands of CPU cores and petabytes of RAM. There's no need to overprovision. Smarter use of resources in the cloud is saving companies easily 70-80%, compared with setting up and managing an on-premise HPC environment.

2. Time-to-market: After moving a financial reporting application to the public cloud, one of our enterprise customers saw a reduction in processing time from 24 days to 3 days every month. That's an 8x gain in opportunity cost, if you consider the additional days the company now has to analyze the data or run a new analysis and gain more information for business decision-making.  In a world where there are fixed deadlines for everything from financial reporting to monthly regulatory hurdles - Businesses that have more time flexibility with infrastructure have more business flexibility.

3. Business opportunity: HPC in the cloud means that companies can crunch data for potential competitive gain - not just run workloads that are required for an audit or to support a critical business operation. This could have enormous potential in healthcare (consider genomics sequencing for hard to treat diseases or faster clinical trials), marketing (analyzing television ad performance during a new prime time series), or storm tracking (running real-time analysis of climate data to more accurately predict the path of a major weather event). The speed and affordability of the cloud for HPC enables an organization to run more analyses, more frequently, and re-run workloads without delay if needed.

What to do before you jump

Running a high performance computing application in the cloud is not a simple activity, even though some cloud providers have developed environments fine-tuned for running HPC.  Consider the help of an outside expert. Consultants and other service providers that have migrated HPC workloads to the public cloud can bring ample experience and on-the-street advice for working with the major cloud providers. They can help design and set up your environment properly and recommend best practices for monitoring and management.  HPC computing tasks are by nature highly automated and this type of environment requires care to be taken to ensure adequate security and cost controls are in place to ensure predictable results.

Software platforms are also evolving to assist companies with HPC in the cloud. These are job scheduling and job management platforms from companies such as Cycle Computing, which provide value in orchestrating workloads and resources according to requirements. Job scheduling tools are critical to avoid over provisioning. No company can afford to waste $6000 by accidentally running a job for an hour more than necessary.

Tracking costs is also crucial. Commercial tools for monitoring resource usage and costs comparing against budget, such as 2W Insight, take the pain out of this task and prevent career-killing mistakes.  When you get to the level of resources needed to run HPC, it's not hard to rack up $1 million in a single month when your budget is $200,000. There are no refunds in the cloud - and no easy way to explain to the boss how you snoozed on the job to the cloud provider's enormous benefit.

Research and compare cloud providers on their HPC-ready features. AWS, for instance, offers Cluster Compute-optimized and GPU instance types which have been designed for high performance networking and can scale to tens of thousands of instances on-demand. AWS C4 instances are available in five sizes, offering up to 36 vCPUs. C4 instances are based on Intel Xeon E5-2666 v3 processors that can deliver speeds as high as 3.5 GHz with Intel ® Turbo Boost.  

HPC in the cloud is a breakthrough for companies and organizations that have been hampered by funding, time, staff or all three. With proper planning, monitoring and expert help, companies will no longer view HPC as a cost center-but instead, a business enabler.


About the Author 

Kris Bliesner is co-founder and CTO at 2nd Watch, the enterprise workload management company for the public cloud and an AWS Premier Consulting Partner.


Published Wednesday, March 18, 2015 6:40 AM by David Marshall
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<March 2015>