Realization of data warehouse technology

1 Traditional warehouse ：

Developed from stand-alone database , It is composed of relational database MPP（ Large scale parallel processing ） colony

Split a large table into nodes for storage , Sub database and table storage （ Hash ）

When the data volume is small , Superior performance 、 Once the amount of data reaches a certain level , There are limitations .

shortcoming ： 1） Limited scalability ,2） Hotspot issues （ Data skew ）-- It can be solved by adding salt to the data

2 Big data warehouse

Relying on big data technology , Take advantage of the natural scalability of big data , Complete the storage of massive data .

take SQL Convert to big data computing engine task , Complete data analysis . 

Concurrent computing  

advantage ： 

Scalable , Distributed file system split storage , When calculating, restore the file to the original table structure ;

Security ： More secure

shortcoming ： 

1 SQL rate of support

2 Things support

3 When the amount of data does not reach a certain level , Low computational efficiency . 

MPP The difference between architecture and distributed structure

MPP ：

1. Common technical architecture of traditional data warehouse , Stand alone database nodes form a cluster , Improve overall processing performance
2. The nodes are non shared （share nothing）, Each node has an independent disk storage system and memory system
3. No data nodes are connected to each other through private networks or commercial general Networks , Work together to calculate , Provide services as a whole
4. Design priority C, Second, consider A, Try to be good at P

advantage ： 

The operation method is fine , Low latency , Low throughput

Suitable for medium-sized structured data processing

shortcoming ： 

The storage location is opaque , Determine the physical node of the data through hash , The query task is executed on all nodes * Architecture decision ）

In parallel computing , The single node bottleneck will become a short board of the whole system , Low fault tolerance

The time limit of distributed transactions will reduce scalability

Distributed architecture （hadoop framework / Batch Architecture ）

1. Each node realizes site autonomy （ Local applications can be run separately ）, Data is shared globally and transparently in the cluster （ And MPP The biggest difference ）
2. Each node is connected through Wan or LAN , The communication overhead between nodes is large , Strive to reduce data movement in Computing （ Mobile computing, not mobile data ）
3. Give priority to P（ Partition tolerance ）, then A（ Usability ）, Last C（ Uniformity ）.

A piece of data will be split into multiple pieces , A shard will store multiple copies , Solve the single point of failure problem .

The distributed processing method is rough （ Store data as a file ）,mpp It's fine . 

advantage ：

It's very scalable , Handle structured and semi-structured data , Suitable for massive data storage and processing .

MPP+ Distributed architecture

Storage layer ： Distributed storage , Adopted by upper layer MPP Fine processing , Reduce the delay .