|
September 22, 2008 Article #20541 Volume 127, Issue 4 HPC
|
It’s been said that the most profound software systems are those that disappear 
Virtual Clustering at TACC with MyCluster
Remotely Managed Clusters Enable Resource Sharing at HPC Facilities
|
|
Aaron Dubrow's article on Virtual Clustering at the Texas Advanced Computing Center discusses the locally developed MyCluster software, which allows researchers to aggregate resources across the TeraGrid and into the clouds.
“In the academic grid computing space, sharing is key,” Edward Walker, researcher in the Distributed and Grid Computing group at the Texas Advanced Computing Center (TACC), told Dubrow. “You’re trying to allow a user to aggregate and leverage federated resources from across institutional boundaries to do large-scale science.”
Jeffrey P. Gardner, senior research scientist in high-performance computing at the University of Washington, offered some historical perspective concerning his experience in an effort to run his research team’s large-scale cosmic simulations on the TeraGrid.
“We needed to run a humongous number of independent jobs,” Gardner recalled. “One of our users had several million computations. The advantage of having a large number of independent tasks is that you can farm them out anywhere. But the TeraGrid had no real way to do that.”
At just that moment Walker introduced the first version of the MyCluster software tool (then called GridShell), Dubrow writes.
“MyCluster was instantly useful,” Gardner said, “and allowed researchers to spend a lot more time on the science and the computing rather than the management of the project.”
According to Dubrow, MyCluster creates a virtual cluster, a group of linked computers that work together closely and can be managed remotely. Interacting with the user’s preferred job management system (with which most scientists are familiar), the tool sets up job proxies on whatever system can accomplish the computing tasks most quickly. When these jobs run, the provisioned computing resources appear on the user’s personal computer as a virtual cluster that dynamically shrinks or expands over time.
“All the user has to worry about is submitting his own jobs into his own personal cluster,” Walker said. “The user doesn’t have to worry about the infrastructure or deal with the heterogeneous usage policies at the different supercomputing sites. The importance of this development is illustrated by Gardner's experience. As he noted, the ability to coordinate resources at multiple supercomputing centers and at departmental systems, from a single virtual cluster, enables scientists to accomplish projects that otherwise could not be attempted, Dubrow writes.
“What I really want to be able to do is just submit my job to ‘The TeraGrid,’ and have the scheduler send some jobs to TACC, and some to Pittsburgh Supercomputing Center, to get my computation done as quickly as possible,” Gardner said. “And MyCluster accomplishes this meta-scheduling solution. From an ease of use standpoint, MyCluster simplifies your job tremendously because you let the tool make all the scheduling decisions for you,” Gardner said.
MyCluster has also helped Michael Deem (Rice University) run a million jobs relating to a zeolite crystal structure simulation on the TeraGrid in just six months — no small feat. In addition, the tool has helped the National Virtual Observatory create a way for astronomical researchers to find, retrieve and analyze data from ground- and space-based telescopes across the globe.
MyCluster also allows users with large numbers of serial, single-processor jobs to use TeraGrid machines, which, because of policies at some centers, is not otherwise possible.
MyCluster 2.0, set for release in Spring 2009, will expand on the original feature set, adding a dynamic, virtual global file system and compatibility with cloud computing solutions like Amazon EC2. It will also provide an even more transparent and unobtrusive infrastructure.
“It’s been said that the most profound software systems are those that disappear,” Walker stated. “The interface is the job management system that the user picks, and as much as possible, we want the user to not interact with the software. MyCluster is this transparent tool that goes out and does something for the user.”
Send to a Colleague
Printer-Friendly Version
Virtual Clustering at TACC with MyCluster
|
|
 
 
 
 
 
 
|
|
|
Customized news reports about Sun Microsystems. Just the news you need, none of what you don't.
50,000+ Members. 20,000+ Articles Published since 1998. |
Copyright © 1998-2008 System News, Inc.