Skip to main content

Quick Start

Note: This section is designed to provide a convenient process for submitting Flink jobs using the StreamPark platform through simple operational steps.

1_config_flink_home

2_add_flink_home_info

Click "OK" to save

3_display_flink_home_config

Depending on the Flink deployment mode and resource management method, StreamPark supports the following six job modes:

  • Standalone Session
  • Yarn Session
  • Yarn Per-job
  • Yarn Application
  • K8s Session
  • K8s Application

For this guide, choose the simpler Standalone Session mode (indicated by the green line in the image below) for a quick start.

4_flink_deployment_execute_model

start-cluster.sh

5_start_flink_standalone_session_service

Page access: http://vm:8081/

6_flink_standalone_session_web_ui

7_config_flink_cluster_menu.png

8_add_flink_cluster_info.png

9_add_flink_cluster_successfully

10_display_flink_cluster_list

Create Job

11_add_flink_job_menu

Main Parameters

  • Development Mode: Choose “Flink SQL”
  • Execution Mode: Choose “remote”
  • Flink Version: Select "flink-1.17", as configured in “1.1 Configure FLINK_HOME”
  • Flink Cluster: Select “myStandalonSession”, as configured in “1.2 Configure FLINK Cluster”
  • Flink SQL: See example below
  • Application Name: Job name

Create Job

12_add_flink_job_param_1

13_add_flink_job_param_2

14_add_flink_job_param_3

Save Job

Click the blue “Submit” button to submit the job

15_save_flink_job

16_display_flink_job_list

Build Job

17_build_flink_job

Build successful

18_build_flink_job_successfully

Start Job

19_start_flink_job

Start Checkpoint Settings

20_set_checkpoint_when_start_flink_job

Submit Job

21_submit_flink_job_1 22_submit_flink_job_2

Check Job Status

View via Apache StreamPark™ Dashboard

StreamPark dashboard

23_flink_job_dashboard

View job details

24_display_flink_job_details

25_display_flink_job_details_page

26_display_native_flink_job_web_ui

![27_display_native_flink_job_web_ui_2](/doc/image_en/platform-usage/27_display_native_flink_job

_web_ui_2.png)

With this, the process of submitting a Flink job using the StreamPark platform is essentially complete. Below is a brief summary of the general process for managing Flink jobs on the StreamPark platform.

28_streampark_process_workflow

Stopping, modifying, and deleting Flink jobs through the StreamPark platform is relatively simple and can be experienced by users themselves. It is worth noting that: If a job is in a running state, it cannot be deleted and must be stopped first.

Apache StreamPark™ System Module Introduction

System Settings

Menu location

29_streampark_system_menu

User Management

For managing users of the StreamPark platform

30_streampark_user_management_menu

Token Management

Allows users to operate Flink jobs in the form of Restful APIs

31_streampark_token_management_menu

32_streampark_token_management_menu_usage

curl -X POST '/flink/app/cancel' \
-H 'Authorization: 69qMW7reOXhrAh29LjPWwwP+quFqLf++MbPbsB9/NcTCKGzZE2EU7tBUBU5gqG236VF5pMyVrsE5K7hBWiyuLuJRqmxKdPct4lbGrjZZqkv5lBBYExxYVMIl+f5MZ9dbqqslZifFx3P4A//NYgGwkx5PpizomwkE+oZOqg0+c2apU0UZ9T7Dpnu/tPLk9g5w9q+6ZS2p+rTllPiEgyBnSw==' \
-H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
--data-urlencode 'savePoint=' \
--data-urlencode 'id=100001' \
--data-urlencode 'savePointed=false' \
--data-urlencode 'drain=false' \
-i

Role Management

User roles: Currently, there are two types, develop and admin.

33_streampark_role_menu

Team Management

Teams: Used to distinguish and manage jobs of different teams in an enterprise.

34_streampark_team_menu

Member Management

(Team) member management

35_streampark_member_menu

Managing system menus

36_streampark_menu_management

Apache StreamPark™ Menu Modules

Project

StreamPark integrates with code repositories to achieve CICD

37_streampark_project_menu

To use, click "+ Add new," configure repo information, and save.

38_streampark_project_menu_usage_1

39_streampark_project_menu_usage_2

Application

Core Module: Used for full lifecycle management (creation, build, start, stop, delete, etc.) of Flink jobs.

40_application_menu

Variable

Variable management: Manage variables that can be used when creating Application jobs.

41_variable_menu_usage_1

42_variable_menu_usage_2

43_variable_menu_usage_3

Setting

System Setting

For system configurations: Maven, Docker, alert email, Ingress

44_system_menu_1

45_system_menu_2

46_system_menu_3

47_system_menu_4

Alert Setting

Supports multiple alert notification modes

48_alert_setting_menu

To be improved】Can perform some operations on Flink jobs, such as validation of Flink SQL, etc.

49_flink_home_list

50_flink_home_usage

51_flink_home_usage_2

52_add_flink_cluster_info

53_display_flink_cluster_list

54_visit_flink_cluster_web_ui

Using Native Flink with Apache StreamPark™

To be improved】In fact, a key feature of StreamPark is the optimization of the management mode for native Flink jobs at the user level, enabling users to rapidly develop, deploy, run, and monitor Flink jobs using the platform. Meaning, if users are familiar with native Flink, they will find StreamPark even more intuitive to use.

How to Use in Apache StreamPark™

Session Mode

  1. Configure Flink Cluster

61_flink_session_cluster_usage_1

  1. When creating a job, select the corresponding resource manager's model and an established Flink Cluster in Execution Mode

62_flink_session_cluster_usage_2

Application Mode

63_flink_application_model_usage_1

64_flink_application_model_usage_2

65_flink_application_model_usage_3

66_flink_application_model_usage_4

Setting Job Parameters

Official website: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/

Native submission command (with parameters)

flink run-application -t yarn-application \
-Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
hdfs://myhdfs/jars/my-application.jar

How to Use in Apache StreamPark™

When creating or modifying a job, add in “Dynamic Properties” as per the specified format

67_dynamic_params_usage

Alert Strategy

To be improved

Reference: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/task_failure_recovery/

68_native_flink_restart_strategy

How to Use in Apache StreamPark™

To be improved】Generally, alerts are triggered when a job fails or an anomaly occurs

  1. Configure alert notifications

69_streampark_alert_settings_1

70_streampark_alert_settings_2

  1. When creating or modifying a job, configure in "Fault Alert Template" and “CheckPoint Failure Options”

71_streampark_alert_usage

cp/sp

To be improved

cp: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/fault-tolerance/checkpointing/ sp: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/savepoints/

72_native_flink_save_checkpoint_gramma

How to Configure Savepoint in Apache StreamPark™

Users can set a savepoint when stopping a job

73_streampark_save_checkpoint

74_streampark_set_checkpoint_location

75_streampark_save_checkpoint_log

View savepoint

76_show_checkpoint_file_name

77_show_checkpoint_file_name_2

How to Restore a Job from a Specified Savepoint in Apache StreamPark™

Users have the option to choose during job startup

78_usage_checkpoint_in_streampark

Job Status

To be improved

Reference: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/internals/job_scheduling/#jobmanager-data-structures

Job Status in Apache StreamPark™

To be improved

Job Details

View through “Apache Flink® Dashboard

80_native_flink_job_details_page

Job Details in Apache StreamPark™

81_streampark_flink_job_details_page

In addition, for jobs in k8s mode, StreamPark also supports real-time display of startup logs, as shown below

82_streampark_flink_job_starting_log_info

Integration with Third-Party Systems

Native Flink provides a REST API Reference: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/rest_api/

How Apache StreamPark™ Integrates with Third-Party Systems

StreamPark also provides Restful APIs, supporting integration with other systems. For example, it offers REST API interfaces for starting and stopping jobs.

83_streampark_restful_api_1 84_streampark_restful_api_2