Virtual Supercomputer as basis of Scientific Computing

Nowadays supercomputer centers strive to provide their computational resources as services, however, present infrastructure is not particularly suited for such a use. First of all, there are standard application programming interfaces to launch computational jobs via command line or a web service, which work well for a program but turn out to be too complex for scientists: they want applications to be delivered to them from a remote server and prefer to interact with them via graphical interface. Second, there are certain applications which are dependent on older versions of operating systems and libraries and it is either non-practical to install those old systems on a cluster or there exists some conflict between these dependencies. Virtualization technologies can solve this problem, but they are not too popular in scientific computing due to overheads introduced by them. Finally, it is difficult to automatically estimate optimal resource pool size for a particular task, thus it often gets done manually by a user. If the large resource pool is requested for a minor task, the efficiency degrades. Moreover, cluster schedulers depend on estimated wall time to execute the jobs and since it cannot be reliably predicted by a human or a machine their efficiency suffers as well.

Applications delivery, efficient operating system virtualization and dynamic application resource pool size defining constitute the two problems of scientific computing: complex application interfaces and inefficient use of resources available — and virtual supercomputer is the way to solve them. The research shows that there are ways to make virtualization technologies efficient for scientific computing: the use of lightweight application containers and dynamic creation of these containers for a particular job are both fast and transparent for a user. There are universal ways to deliver application output to a front-end using execution of a job on a cluster and presenting its results in a graphical form. Finally, an application framework can be developed to decompose parallel application into small independent parts with easily predictable execution time, to simplify scheduling via existing algorithms.

The aim of this chapter is to promote the key idea of a virtual supercomputer: to harness all available HPC resources and provide users with convenient access to them. Such a challenge can be effectively faced using contemporary virtualization technologies. They can materialize the long-term dream of having a supercomputer at your own desk.

• Preprint

@inbook{bogdanov2015vsc,
  title={Virtual Supercomputer as basis of Scientific Computing},
  author={Alexander Bogdanov and Alexander Degtyarev and Vladimir Korkhov and Vladimir Gaiduchok and Ivan Gankevich},
  publisher={Nova Science Publishers},
  howpublished={Horizons in Computer Science Research},
  year={2015},
  month={01},
  language={english},
  isbn={978-1-63482-499-6},
  pages={159--198},
  volume={11},
  chapter={5},
  editor={Thomas S. Clary},
  type={inbook}
}