Skip to main content

Apache StreamPark™

Make stream processing easier!

🚀 What is Apache StreamPark™​

Apache StreamPark™ is a streaming application development framework that provides a development framework for developing stream processing application with Apache Flink® and Apache Spark™, Also, StreamPark is a professional management platform for streaming application, Its core capabilities include application development, debugging, deployment, operation, etc. Drastically simplifies streaming apps’ development and operations, enabling enterprises to derive immediate insight from their data in real-time.

Why Apache StreamPark™?​

Apache Flink® and Apache Spark™ are widely used as the next generation of big data streaming computing engines. Based on a foundation of excellent experiences combined with best practices, we extracted the task deployment and runtime parameters into the configuration files. This way, an easy-to-use RuntimeContext with out-of-the-box connectors can bring an easier and more efficient task development experience. It reduces the learning cost and development barriers so developers can focus on the business logic.

However, enterprises may struggle to use Apache Flink® and Apache Spark™ without a professional management platform for Flink and Spark tasks during the deployment phase. StreamPark offers a professional task management platform to address this need.

🎉 Features​

  • Easy-to-use Apache Flink® and Apache Spark™ application development framework
  • One-stop stream processing operation platform
  • Support multiple versions of Apache Flink® & Apache Spark™
  • Out-of-the-box connectors
  • Support catalog、olap、streaming-warehouse etc.

🏳‍🌈 Architecture of Apache StreamPark™​

The overall architecture of Apache StreamPark is shown in the following figure. Apache StreamPark has two parts, streampark-core and streampark-console.

StreamPark Archite

1. streampark-core​

streampark-core is a framework used during development. It supports coding development, regulates configuration files, and follows the 'convention over configuration' principle. streampark-core provides development-time Runtime Content and a series of out-of-the-box Connectors. Cumbersome operations are simplified by extending DataStream-related methods and integrating DataStream and the Flink SQL API. This improves development efficiency and developer experience because users can focus on the business logic.

2. streampark-console​

streampark-console is a comprehensive real-time Low Code data platform that can manage Flink tasks more convenient. It integrates the experience of many best practices and integrates many functions such as project compilation, release, parameter configuration, startup, savepoint, Flink SQL, monitoring, etc., which greatly simplifies the daily operation of Flink tasks and maintenance. The ultimate goal is to create a one-stop big data platform, which can provide a solution that integrates flow and batch, and integrates lake and warehouse.

This platform uses technologies including, but not limited to:

Thanks for the support and inspiration given by the above excellent open source projects and many other excellent open source projects not mentioned here!