Author Pawel Plaszczak
Pawel also blogs regularly at bigdatamatters.com

In certain sectors that were the early adopters of Grids, migration to the Cloud is bound to happen soon.
Pharmaceuticals is a good example. As Bob Cohen pointed out in a recent presentation:

• Eli Lilly has already tried using the Amazon EC2 external Cloud,
• GlaxoSmithKline is looking at using internal Clouds

As I remember, years ago Glaxo was among the early Grid users. Like many other pharmas, they used
software from UnivaUD to distribute protein docking simulations over a large number of machines. Now UnivaUD also sells Cloud services.

This supposedly common movement from Grid to Cloud is thought provoking. What does this “evolutionary step” really mean? Something conceptually quite simple: The Grid makes it possible to manage processes in many physical machines. The Cloud offers even an greater potential: to manage processes in many virtual machines, or even to manage those virtual machines like they were processes. This is what VMware vSphere offers or what internally powers Amazon EC2. So:

Cloud = set of virtual machines managed by a scheduler (Grid).

All those above named are great products. If you want an internal Cloud. The thing is: in solving large data challenges, the Cloud is no less limited than its predecessor, the Grid. Chris Dagidigian of gridengine.info said at BioIT that he “solved real problems” on the Cloud. This is not surprising (BioTeam once teamed with Univa to demonstrate Grid Engine on AWS), but such a statement needs an explicit remark: problems solvable on the Cloud are still a small subset of the World’s important data processing challenges.

Virtualized Cloud environments are perfectly isolated from each other. If you have one, pray that you only happen to compute tasks that can be domain-decomposed into millions of perfectly independent pieces.

Protein docking, mentioned earlier, is like that: thousands of simulations, one per each set of chemical
compounds, that do not need to communicate. But most apps, in most industry sectors (not just bioinformatics) do not share these characteristics: they require intensive database querying and/or data sharing. Genomics is like that. The Cloud will not help here. Clouds may even make it more difficult. I also agree here with another of Chris’s statements: Cloud data ingest is a pain.
The answer to large data challenges is a puzzle of three pieces:

1. efficient distributed processing
2. efficient data provisioning
3. efficient storage

Workload management engines (Grids, Clouds, you name it) provide the first point. Today’s data intensive apps need the full stack, an efficient integration of (1), (2) and (3). A fully scalable data integration. Where does the challenge lie?
Efficient processing is easy. With many great scheduling vendors, this is not rocket science any more.
Efficient storage is becoming commonplace too, with interesting examples of federated storage distributions. The trick is in the middle layer: an efficient connection between these two. And that is really difficult. There is certainly no universal solution, but we have recently had some successes here.

By Adrian Mouat

Grid computing was born in academia and was originally designed to support scientific and research computing. In contrast, Cloud computing has a business background and is designed to enable the delivery of scalable Web applications.

The BEinGRID project has looked into how Grid is appropriate for business use (and has run several successful business experiments proving this proposition), but what about looking at whether the Cloud is useful for academia? Can it be effectively used to run scientific codes, such as those found in climate modelling, fluid dynamics or molecular physics simulations, which have traditionally required the use of supercomputers?

On the face of it, Cloud services offer a compelling, simple and relatively cost-effective HPC proposition – just pay for as many CPUs as you want, when you want them. Of course, the truth isn’t quite as simple as that.

Take a look at the Google Cloud offering, App Engine. Users access App Engine through an API, which places quite a lot of restrictions on the code that can be run, including:

• It must be written either in Python or Java (or use a JVM based interpreter or compiler) – meaning any C or Fortran codes have to be ported
• It can’t start any threads (instead the API is used to start a new task)
• Any single request/task must complete in 30 seconds
• It has to stay within quotas on CPU, bandwidth and storage usage

For full details see the App Engine documentation. Note that there are different quotas for the free and billed service, and that it is possible to negotiate increases to the quotas.

This doesn’t make scientific computing impossible, but it does put in place a lot of barriers. It would be interesting to see what could be achieved computationally, given the above restrictions, by an academic research group that chose to use App Engine. However, the bottom line is that Google App Engine is more suited to creating dynamic web applications, such as photo and document editing tools, than to processing long-lived scientific computations.

Amazon’s offering, EC2, is a lot more promising. EC2 gives users much more access and control over the system through the use of virtualisation. Users are free to install whatever software and applications they need on EC2.

Users provide “virtual images”, instances of which can be launched at any time and will normally be running in under 10 minutes. By default, only 20 instances (per region) can normally be launched, but users can apply to increase this limit, potentially allowing thousands of instances to be launched. Amazon also supports Hadoop, Condor and OpenMPI for batch/parallel processing. The Data Wrangling blog has some in-depth information on using Amazon to set up an MPI cluster.

For data storage, the Amazon S3 service can be used to store the enormous amounts of data produced by some scientific applications. Access to the data is controlled via Access Control Lists (ACLs) and data is encrypted during transmission using SSL. Users are encouraged to encrypt any sensitive data being stored in S3. It is important to note that Amazon does not guarantee that data will not be lost or compromised (see point 7.2 of the AWS Customer Agreement).

So, it should be fairly easy to get most scientific computing codes running in parallel on EC2. But what’s the performance like? There has been some research and the results are mixed. Comparing a roughly equivalent amount of CPU resources, super-computer clusters are typically much faster at processing scientific codes, largely due to having a better interconnect (see this article by Edward Walker). However, if we include the amount of time it takes to get the code running (i.e. to request and boot the images on EC2 and to wait in the queue on a super-computer cluster), EC2 is likely to be faster in many cases, dependent on the size of the job and scheduling policies (as shown by Ian Foster on his blog). In the future, EC2 may offer an even more competitive service if they upgrade their systems.

I haven’t taken into account Microsoft Azure, which is still in “Community
Technology Preview” at the time of writing, but may be interesting for any .NET based scientific codes. The offering is similar to EC2, the main difference being that users must use a supplied Windows 2008 Server VM.

With all the money and time being invested in Cloud computing, it will be interesting to the see the effect it has on HPC resource providers over the next decade. Will the emergence of cloud lead to a greater trust in outsourcing compute resources and a direct boost to HPC resource providers? Will there be a level of symbiosis where Cloud resources can be built on top of or alongside HPC computing resources? Or will they just be direct competition? One things for sure; the rules of the HPC game are being questioned.

For more information on using Cloud platforms for scientific computing, see the HPCCloud Google Group.