Efficient Data Redistribution to Speedup Big Data Analytics in Large Systems

Long ChengLong Cheng, Tao LiTao Li

ISBN: 978-1-5090-5411-4
Download
DOI Link

Long Cheng, Tao Li
Efficient Data Redistribution to Speedup Big Data Analytics in Large Systems
Proc. 23rd IEEE International Conference on High Performance Computing (HiPC'16), 91-100, December 2016. IEEE

Details
Bibtex

KurzfassungAbstract
The performance of parallel data analytics systems becomes increasingly important with the rise of Big Data. An essential operation in such environment is parallel join, which always incurs significant cost on network communication. State-of-the-art approaches have achieved performance improvements over conventional implementations through minimizing network traffic or communication time. However, these approaches still face performance issues in the presence of big data and/or large-scale systems, due to their heavy overhead of data redistribution scheduling. In this paper, we propose near-join, a network-aware redistribution approach targeting to efficiently reduce both network traffic and communication time of join executions. Particularly, near-join is lightweight and adaptable to processing large datasets over large systems. We present the details of our algorithm and its implementation. The experiments performed on a cluster of up to 400 nodes and datasets of about 100GB have demonstrated that our scheduling algorithm is much faster than the state-of-the-art methods. Moreover, our join implementation can also achieve speedups over the conventional approaches.
Projekt:Project: Cfaed, DIAMOND, HAEC, HAEC B08
Forschungsgruppe:Research Group: Wissensbasierte SystemeKnowledge-Based Systems

@inproceedings{CL2016,
  author    = {Long Cheng and Tao Li},
  title     = {Efficient Data Redistribution to Speedup Big Data Analytics in
               Large Systems},
  booktitle = {Proc. 23rd {IEEE} International Conference on High Performance
               Computing (HiPC'16)},
  publisher = {IEEE},
  year      = {2016},
  month     = {December},
  pages     = {91-100},
  doi       = {10.1109/HiPC.2016.020}
}