Distributed Fault-Tolerant Computing with SBN-Python on a Real Company Case (Распределенные отказоустойчивые вычисления с SBN-Python на реальном кейсе компании)

Distributed computing today is in demand in batch data processing tasks, but current solutions that allow to use them in Python either are too specific, or do not guaranteefull fault tolerance. As a part of the final qualification work, a high-level Python interface (hereinafter SBN-Python) was developed for the new C++ framework called Subordination, in which the last problem was solved. The interface was implemented as anextension of the Python interpreter to achieve low-level compatibility and adaptation of all functionality. The purpose of this work was to test the possibility of using the new interface on a real case of LLC «Gazpromneft–Digital Solution» and to demonstrate the its core principles. To achieve this goal the current solution was analysed, a new architecture using SBN-Python was thought out and implemented, and eventually, the resulting solution was deployed at the company's facilities. As a result of thework, it turned out that using SBN-Python on a real case also scales performance with the number of nodes in the cluster, gives the ability to process various scenarios of node failure in a limited time, as well as some architectural advantages in organizing calculations. In the future, it is planned to expand the boundaries of the new interface, implementing the possibility of building distributed web services on its basis.

• Full text
• Published version

@inproceedings{tereschenko2021sbnpython,
  title={Distributed Fault-Tolerant Computing with SBN-Python on a Real Company Case (Распределенные отказоустойчивые вычисления с SBN-Python на реальном кейсе компании)},
  author={Dmitrii Tereshchenko and Ivan Gankevich},
  publisher={RWTH Aahen University},
  booktitle={Proceedings of GRID'21},
  url={http://ceur-ws.org/Vol-3041/519-524-paper-96.pdf},
  year={2021},
  month={01},
  doi={10.54546/MLIT.2021.84.27.001},
  language={russian},
  volume={3041},
  series={CEUR Workshop Proceedings},
  issn={1613-0073},
  editor={Vladimir Korenkov and Andrey Nechaevskiy and Tatiana Zaikina},
  type={inproceedings}
}