Preface

This article belongs to the column 《 Big data technology system 》, This column was originally written by the author , Please indicate the source of the quotation , Please point out the shortcomings and mistakes in the comment area , thank you ！

For the table of contents and references of this column, please refer to Big data technology system

Catalog

Apache Flink Of YARN Session Submission process

Apache Flink Of Per-Job Submission process

Apache Flink Of K8s Session Submission process

Text

In general , stay Yarn Use... On the cluster Session Mode submission Flink The process of homework is divided into 3 Stages ：

• First, in the Yarn Start the Flink Session Cluster of patterns ;
• Secondly through Flink Client Submit the assignment ;
• Finally, carry out job scheduling and execution .

The complete process is as follows ：

1. Start cluster

1. Use bin/yarn-session.sh Submit a session mode job

• If you submit to an existing cluster , Then get Yarn Cluster information 、 application ID, And prepare to submit homework .
• If you start a new Yarn Session colony , Then go to step （2）.

2. Yarn Start new Flink colony

• If there is no cluster , Creates a new one Session Cluster of patterns .

First apply the configuration （flink-conf.yaml、logback.xml、log4j.properties） And related documents ( Flink Jar、 Configuration class file 、 user Jar file 、 JobGraph Object etc. ） Upload to distributed storage （ Such as HDFS） Application staging Directory .

• adopt Yarn Client towards Yarn Submit Flink Application for cluster creation ,Yarn Allocate resources , In the application of Yarn Container Initialize and start Flink JobManager process , stay JobManager Running in process YarnSessionClusterEntrypoint As the entry for cluster startup （ Different cluster deployment modes have different ClusterEntrypoint Realization ）, initialization Dispatcher、ResourceManager, Start related RPC service , wait for Client adopt Rest Interface submission job .

2. Homework submission

Yarn When the cluster is ready , Start job submission .

1. Flink Client adopt Rest towards Dispatcher Submit JobGraph .

2. Dispatcher yes Rest Interface , Not responsible for the actual scheduling 、 Implementation work , When I received JobGraph after , Create a... For the job JobMaster , Give the job to JobMaster ( Be responsible for job scheduling 、 Manage jobs and Task Life cycle of ), structure ExecutionGraph ( JobGraph Parallel version of , The core data structure of the scheduling layer ).

After these two steps , The job enters the scheduling execution stage .

3 . Job scheduling execution

1. JobMaster towards YarnResourceManager Application resources , Start scheduling ExeeutionGraph perform , towards YarnResourceManager Application resources ; There is no TaskManager , At this time, there are insufficient resources , Start applying for resources .
2. YarnResourceManager received JobMaster Resource request for , If you are currently free Slot Will Slot Assigned to JobMaster , otherwise YarnResourceManager To YarnMaster Request creation TaskManager .
3. YarnResourceManager Add the resource request to the waiting request queue , And through the heartbeat to YARN ResourceManager Apply for a new Container Resources to start TaskManager process ; Yarn Assign a new Container to TaskManager
4. YarnResourceManager start-up , And then from HDFS load Jar Relevant resources required by documents , Start... In the container TaskManager .
5. TaskManager After starting , towards ResourceManager register , And put your own Slot Report resources to ResourceManager .
6. ResourceManager Remove... From the waiting queue Slot request , towards TaskManager Confirm resource availability , And let us know TaskManager take Slot To which JobMaster .
7. TaskManager towards JobMaster Provide Slot , JobMaster Dispatch Task To TaskManager This Slot On the implementation .

At this point, the job enters the execution stage .