Skip to main content

Platform Deployment using Docker

This tutorial for docker deploy StreamPark

Prepare

Docker 1.13.1+ Docker Compose 1.28.0+

1. Install docker

To start the service with docker, you need to install docker first

2. Install docker-compose

To start the service with docker-compose, you need to install docker-compose first

Apache StreamPark™ Deployment

1. Apache StreamPark™ deployment based on h2 and docker-compose

This method is suitable for beginners to learn and become familiar with the features. The configuration will reset after the container is restarted. Below, you can configure Mysql or Pgsql for persistence.

2. Deployment

wget https://raw.githubusercontent.com/apache/incubator-streampark/dev/docker/docker-compose.yaml
wget https://raw.githubusercontent.com/apache/incubator-streampark/dev/docker/.env
docker-compose up -d

Once the service is started, StreamPark can be accessed through http://localhost:10000 and also through http://localhost:8081 to access Flink. Accessing the StreamPark link will redirect you to the login page, where the default user and password for StreamPark are admin and streampark respectively. To learn more about the operation, please refer to the user manual for a quick start.

Note:When configuring the flink-sessin cluster address, the ip address is not localhost, but the host network ip, which can be obtained through ifconfig

Use existing Mysql services

This approach is suitable for enterprise production, where you can quickly deploy StreamPark based on docker and associate it with an online database Note: The diversity of deployment support is maintained through the .env configuration file, make sure there is one and only one .env file in the directory

wget https://raw.githubusercontent.com/apache/incubator-streampark/dev/deploy/docker/docker-compose.yaml
wget https://raw.githubusercontent.com/apache/incubator-streampark/dev/deploy/docker/mysql/.env
vim .env

First, you need to create the "streampark" database in MySQL, and then manually execute the corresponding SQL found in the schema and data for the relevant data source.

After that, modify the corresponding connection information.

SPRING_PROFILES_ACTIVE=mysql
SPRING_DATASOURCE_URL=jdbc:mysql://localhost:3306/streampark?useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8
SPRING_DATASOURCE_USERNAME=root
SPRING_DATASOURCE_PASSWORD=streampark
docker-compose up -d

Use existing Pgsql services

wget https://raw.githubusercontent.com/apache/incubator-streampark/dev/deploy/docker/docker-compose.yaml
wget https://raw.githubusercontent.com/apache/incubator-streampark/dev/deploy/docker/pgsql/.env
vim .env

Modify the corresponding connection information

SPRING_PROFILES_ACTIVE=pgsql
SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/streampark?stringtype=unspecified
SPRING_DATASOURCE_USERNAME=postgres
SPRING_DATASOURCE_PASSWORD=streampark
docker-compose up -d

Build images based on source code for Apache StreamPark™ deployment

git clone https://github.com/apache/incubator-streampark.git
cd incubator-streampark/deploy/docker
vim docker-compose
    build:
context: ../..
dockerfile: deploy/docker/console/Dockerfile
# image: ${HUB}:${TAG}

docker-compose up -d

Docker-Compse Configuration

The docker-compose.yaml file will reference the configuration from the env file, and the modified configuration is as follows:

version: '3.8'
services:
## streampark-console container
streampark-console:
## streampark image
image: apache/streampark:latest
## streampark image startup command
command: ${
RUN_COMMAND}
ports:
- 10000:10000
## Environment configuration file
env_file: .env
environment:
## Declare environment variable
HADOOP_HOME: ${
HADOOP_HOME}
volumes:
- flink:/streampark/flink/${
FLINK}
- /var/run/docker.sock:/var/run/docker.sock
- /etc/hosts:/etc/hosts:ro
- ~/.kube:/root/.kube:ro
privileged: true
restart: unless-stopped
networks:
- streampark

## flink-jobmanager container
flink-jobmanager:
image: ${
FLINK_IMAGE}
ports:
- "8081:8081"
command: jobmanager
volumes:
- flink:/opt/flink
env_file: .env
restart: unless-stopped
privileged: true
networks:
- streampark

## streampark-taskmanager container
flink-taskmanager:
image: ${
FLINK_IMAGE}
depends_on:
- flink-jobmanager
command: taskmanager
deploy:
replicas: 1
env_file: .env
restart: unless-stopped
privileged: true
networks:
- streampark

networks:
streampark:
driver: bridge

volumes:
flink:

Finally, execute the start command:

cd deploy/docker
docker-compose up -d

You can use docker ps to check if the installation was successful. If the following information is displayed, it indicates a successful installation:

Uploading Configuration to the Container

In the previous env file, HADOOP_HOME was declared, with the corresponding directory being "/streampark/hadoop". Therefore, you need to upload the /etc/hadoop from the Hadoop installation package to the /streampark/hadoop directory. The commands are as follows:

## Upload Hadoop resources
docker cp entire etc directory streampark-docker_streampark-console_1:/streampark/hadoop
## Enter the container
docker exec -it streampark-docker_streampark-console_1 bash
## Check
ls

In addition, other configuration files, such as Maven's settings.xml file, are uploaded in the same manner.