Skip to main content

Flink on K8s

StreamPark Flink Kubernetes is based on Flink Native Kubernetes and support deployment modes as below:

  • Native-Kubernetes Application
  • Native-Kubernetes Session At now, one StreamPark only supports one Kubernetes cluster.You can submit Fearure Request Issue , when multiple Kubernetes clusters are needed.

Environments requirement

Additional operating environment to run StreamPark Flink-K8s is as below:

  • Kubernetes
  • Maven(StreamPark runNode)
  • Docker(StreamPark runNode)

StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed on node out of Kubernetes cluster when there are smooth network between the node and cluster.


Preparation for integration

configuration for connecting Kubernetes

StreamPark connects to Kubernetes cluster by default connection credentials ~/.kube/config .User can copy .kube/config from Kubernetes node to StreamPark nodes,or download it from Kubernetes provided by cloud service providers.If considering Permission constraints, User also can generate custom account`s configuration by themselves.

kubectl cluster-info

configuration for coKubernetes RBAC

User can configure RBAC for K8s Namespace by referring to Flink-Docs:https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/resource-providers/native_kubernetes/#rbac

When Flink Namespace is flink-dev and there are no needed to explicitly specify K8s accounts, user can allocate resource to clusterrolebinding by the way as below

kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=flink-dev:default

Configuration for remote Docker service

On Setting page of StreamPark, user can configure the connection information for Docker service of Kubernetes cluster.

docker register setting

Building a Namespace named streampark(other name should be set at Setting page of StreamPark) at remote Docker.The namespace is push/pull space of StreamPark Flink image and Docker Register User should own pull/push permission of this namespace.

# verify access
docker login --username=<your_username> <your_register_addr>
# verify push permission
docker pull busybox
docker tag busybox <your_register_addr>/streampark/busybox
docker push <your_register_addr>/streampark/busybox
# verify pull permission
docker pull <your_register_addr>/streampark/busybox

## Job submit

Application Job release

k8s application submit

parameter descriptions are as below:

  • Flink Base Docker Image: Base Flink Docker Image Tag can be obtained from DockerHub - offical/flink .And user can also use private image when Docker Register Account owns pull permission of it.

  • Rest-Service Exposed Type:Description of candidate values for native Flink K8s configuration kubernetes.rest-service.exposed.type

    • ClusterIP:ip that StreamPark can access;
    • LoadBalancer:resource of LoadBalancer should be allocated in advance, Flink Namespace own permission of automatic binding,and StreamPark can access LoadBalancer`s gateway;
    • NodePort:StreamPark can access all K8s nodes;
  • Kubernetes Pod Template: It`s Flink custom configuration of pod-template.The container-name must be flink-main-container. If the k8s pod needs a secret key to pull the docker image, please fill in the information about the secret key in the pod template file.The example pod-template is as below:

    ```
    apiVersion: v1
    kind: Pod
    metadata:
    name: pod-template
    spec:
    serviceAccount: default
    containers:
    - name: flink-main-container
    image:
    imagePullSecrets:
    - name: regsecret
    ```
  • Dynamic Option:It`s dynamic parameter of Flink on k8s(part of parameters can also be defined in pod-template)which should start with -D.Details are inFlink on Kubernetes parameters

After the job is started, it is supported to directly access the corresponding Flink Web UI page on the Detail page of the task:

k8s app detail

Session job release

The additional configuration of Flink-Native-Kubernetes Session Job will be decided by Flink-Session cluster.More details can be find in Flink-Doc:https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/resource-providers/native_kubernetes


other configuration

StreamPark parameter related to Flink-K8s in applicaton.yml are as below.And in most condition, it is no need to change it.

Configuration itemDescriptionDefault value
streampark.docker.register.image-namespacenamespace of Remote docker service repository, flink-job image will be pushed herenull
streampark.flink-k8s.tracking.polling-task-timeout-sec.job-statustimeout in seconds of flink state tracking task120
streampark.flink-k8s.tracking.polling-task-timeout-sec.cluster-metrictimeout in seconds of flink metrics tracking task120
streampark.flink-k8s.tracking.polling-interval-sec.job-statusinterval in seconds of flink state tracking task.To maintain accuracy, please set below 5s, the best setting is 2-3s5
streampark.flink-k8s.tracking.polling-interval-sec.cluster-metricinterval in seconds of flink metrics tracking task10
streampark.flink-k8s.tracking.silent-state-keep-secfault tolerance time in seconds of silent metrics60