Practical DevOps for Big Data / Monitoring Course

What is Practical DevOps for Big Data / Monitoring?

Big Data technologies have become an ever more present topic in both academia and industrial world. These technologies enable businesses to extract valuable insight from their available data, hence more and more SMEs are showing increasing interest in using these types of technologies. Distributed frameworks for processing large amounts of data, such as Apache Hadoop[1], Spark[2], or Storm[3] gained in popularity and applications developed on top of them are more and more prevalent. However, developing soft- ware that meets these high-quality standards expected for business-critical Cloud applications remains a challenge for SMEs. In this case model-driven development (MDD) paradigm and popular standards such as UML, MARTE, TOSCA[4] hold strong promises to tackle this challenge. During development of Big Data applications it is important to monitor performance for each version of the application. Information obtained can be used by software architects and developers to track the evolution in time of the developed application. Monitoring is also useful in determining main factors that impact the quality of different application versions. Throughout the development stage, running applications tend to be more verbose in terms of logged information so that developers can get insights about the developed application. Due to verbosity of logs, data- intensive applications produce large amounts of monitoring data, which in turn need to be collected, pre-processed, stored and made available for high-level queries and visualization. It is clear that there is a need for a scalable, highly available and easy deployable platform for monitoring multiple Big Data frameworks. Which can collect resource-level metrics, such as CPU, memory, disk or network, together with framework level metrics collected from Apache HDFS, YARN, Spark and Storm.

Content

• Introduction
• Motivations
• Existing solutions
• How the tool works
• Open Challenges
• Application domains