Skip to main content

11 posts tagged with "StreamPark"

View All Tags

Although the Hadoop system is widely used today, its architecture is complicated, it has a high maintenance complexity, version upgrades are challenging, and due to departmental reasons, data center scheduling is prolonged. We urgently need to explore agile data platform models. With the current popularization of cloud-native architecture and the integration between lake and warehous, we have decided to use Doris as an offline data warehouse and TiDB (which is already in production) as a real-time data platform. Furthermore, because Doris has ODBC capabilities on MySQL, it can integrate external database resources and uniformly output reports.

Introduction:This article mainly introduces the architecture upgrade and evolution of the self-migrating MySQL data to Hive, the original architecture involves many components, complex links, and encounters many challenges, and effectively solves the dilemmas and challenges encountered in data integration after using the combination of StreamPark + Paimon, and shares the specific practical solutions of StreamPark + Paimon in practical applications, as well as the advantages and benefits brought by this rookie combination solution.

StreamPark: https://github.com/apache/streampark

Paimon: https://github.com/apache/paimon

Welcome to follow, star, fork, and participate in contributions

Contributor|Beijing Ziru Information Technology Co., Ltd.

Authors of the article|Liu Tao, Liang Yansheng, Wei Linzi

Article compilation|Yang Linwei

Content proofreading|Pan Yuepeng

Abstract: This article is compiled from the sharing of Mu Chunjin, the head of China Union Data Science's real-time computing team and Apache StreamPark Committer, at the Flink Forward Asia 2022 platform construction session. The content of this article is mainly divided into four parts:

  • Introduction to the Real-Time Computing Platform Background
  • Operational Challenges of Flink Real-Time Jobs
  • Integrated Management Based on StreamPark
  • Future Planning and Evolution

Foreword: This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon.

  • Introduction to Company Business
  • Pain Points and Selection in Big Data Technology
  • Production Practice
  • Troubleshooting Analysis
  • Future Planning

Preface: This article primarily discusses the challenges encountered by Shunwang Technology in using the Flink computation engine, and how StreamPark is leveraged as a real-time data platform to address these challenges, thus supporting the company's business on a large scale.

  • Introduction to the company's business
  • Challenges encountered
  • Why choose StreamPark
  • Implementation in practice
  • Benefits Brought
  • Future planning