How to write a resume and project

star How to write a project ：

Shenzhen Langjian Liaoda Co., Ltd  2018.3- so far

Log center construction   Project cycle 2020.2.10-2020.4.20

Brief introduction of achievement experience

2020.2.10-2020.5.20, In Shenzhen Langjian Liaoda Data Co., Ltd. completed the construction of log center together with the team , Improve the company Log query , The efficiency of fault detection and treatment .

Situation:

1. Shenzhen Langjian Liaoda Data Co., Ltd. includes official website , game app, Online retailers app, Official account background ,redis Such as storage system and service background , Logs need to be collected and processed in a unified way .

2. Need to provide a log collection that is easy to deploy agent, Real time log collection . Support 8w+ strip /s Data volume .

3. To design and provide a unified log retrieval center , It is convenient for users to query logs , Avoid login machine queries .

4. Need to design a rules engine , Users can easily write rules , Then the rules are executed on the log data in real time , Timely discovery of log exceptions , In order to deal with it in time .

5. We need to build an alarm system , Year on year , Chain ratio , Frequency and other functions , Statistics of the same alarms and combined alarms , Reduce the number of invalid alarms .

6. Acquisition system deployment , Rule configuration , Alarm configuration , Log retrieval , Need to design a unified UI, Add user rights management function .

Task：

The task and goal of the team ：

1. Design a log collector , Real time collection of application logs , Report to kafka.

2. Research design rule engine , Rule writing should be simple , Support grammar to be diverse , To take advantage of existing real-time computing platforms , Such as flink.

3. Research and design alarm system , Support year-on-year , Chain ratio , loop , frequency , Fixed interval and alarm aggregation statistical function , Reduce invalid alarms , And the alarm should be real-time .

4. Design web UI And the corresponding backstage , It is convenient for users to deploy the acquisition system with one key , Issue Log rules and alarm rules and convenient query log location problem . 

My mission and goal

1. Research the requirements of user rule writing .

2. Research existing rule engines , Choose the most appropriate engine based on the user's needs .

3. Design the implementation kernel architecture of rule engine , Integrating real-time computing engines .

4. Deploy , test , Pressure test rule engine , For performance optimization , Make it meet the performance requirements .

5. Write script to realize the rule engine failure automatically pull up . 

Action：

My job and mission ：

1. It took two days , research , Collect user's log rule requirements .

2. It took a week , research , Compared with existing rule engines ,easyrules,groovy,drools etc. . Finally, write the complexity according to the grammar , Performance etc. , Be sure to use groovy To implement your own rules engine .

3. In three days , Realize to groovy As a rule engine, research carefully , Design groovy As a rule engine, the overall architecture and determination of the rules and flink Integrated solutions .

4. Take a week , Complete the core code of the rule engine , Successfully integrated flink.

5. Take three days , Do a preliminary test , Pressure measurement and tuning , At the same time, complete the monitoring script .

6. Take a day , Complete the joint debugging with other colleagues .

Result：

The overall results of the project ：

After two months of hard work , Completed all functions of the company's log Center , With the rule engine and alarm system , Application exception can be found and alerted in time , Improve the efficiency of exception handling , The use of log center greatly improves the efficiency of user log retrieval . 

My results ：

1. Independent research , Design , The real-time rule engine system is implemented .

2. Rules are issued and take effect in real time .

3. At present and support 2k+ The rules , The amount of data processed per second 1w strip .

4. Assist the company to find some application anomalies , It avoids the loss caused by the huge fault .

5. In the process of further mastering the technology of subscription publishing system Apollo,flink, Multithreading ,groovy kernel , Concurrent collections and so on .

 

Project brief introduction

The company has a lot of services , For example, the storage system ,app Backstage, etc , Need to provide a log query system , Easy to retrieve logs ; Log rules engine   It is convenient for different applications to configure different rules , To find the exception in the log ; Log abnormal alarm system , So that even if the task is found to be abnormal , Convenient troubleshooting service .

Project responsibilities ：

It is mainly responsible for the rule engine , The main contents and achievements are as follows ：

1. Research common rule engines ,easyrules,drools,qlexpress,groovy Engine, etc. , Through performance , Ease of use , Easy to maintain, etc , Finally decided to use groovy.

2. The design is based on groovy The kernel architecture of rule engine , The main design content ：

1. Rule subscription publishing system ,Apollo.

2. Rule concurrent processing model , Multithreading and concurrent collections .

3. GroovyClassloader Kernel and rule loading mechanism , And memory optimization , Precompilation, etc .

4. Integrate flink.

3. Do it on your own , A functional test , Pressure measurement , Deploy , And automatic fault handling script writing .

4. result ： So far online rules 1k+, Handle 1w strip /s, It can be very simple to scale out .

The wrong way to write ：

Be honest and modest , diligence , endure hardships and be capable of hard work , A sense of teamwork , Strong sense of responsibility , Good at communication , Good sense of teamwork ; Solid technical background , Have strong research spirit and learning ability ; I am optimistic and outgoing , Like playing basketball , Billiards .

Correct writing ：

Love sports , Climbing the mountain , Basketball , Table tennis .

geek 、 Love technology 、 Love open source

programing language ： master java,scala,shell, And for multithreading , High concurrency ,jvm I have some experience in tuning .

Basic framework ： master netty,dubbo,mysql,redis,mongodb etc. .

Spark： Master . I have read it carefully spark core ,spark sql spark streaming Source code , Have done secondary development .

Flink： Master . Read over flink Source code , Use flink Did real-time OLAP platform ,DDL Syntax parsing , Real graph , Virtual columns support .

Hadoop： master .

Hbase： master . be familiar with hbase Storage structure ,rowkey Experience in design , Have used hbase Store 100 TB Level table ..

Hive： master .hive The experience of doing digital warehouse , Yes hive We have rich experience in data skew processing , Deal with a hundred TB Data of the above levels .

Kafka： Master . I have read it carefully kafka Source code , For the underlying storage design , The principle and model of production and consumption are deeply understood , Handled 100w/s Data volume .

Clickhouse： master . Use clickhouse As the underlying layer of indicator storage , Speed up the presentation .

ElasticSearch： master .