Spark advanced features, 220720,

RangerPartition

Scala in ： Construct random values as Key, utilize hash Partition reallocation

Pyspark: Optimized , Small amount of data , There is no need to re divide , In the case of data volume , Will be re divided

map、foreach： A cycle

xxxxPartition： Two layers of circulation

The sentence -> words

concat_ws（" Separator ", list ）

  data verification

Compare the analysis results with the analysis results in the original data

first TRUE disk

Second memory

The third out of heap memory

The fourth serialization  

alt+enter, Guide pack ,

Call in pairs ,

persist Unless the last step ,

If the cache is lost , The cache can still be restored through the blood mechanism

persist cache , Memory or disk ,

RDD All the dependencies of ,Driver There are

object = data + Blood dependence ,

dict[Key], If there is no correspondence, an error will be reported , therefore get,

Avoid every 1 individual task download ,

MR、Hive:Map Join

Spark : Broadcast Join

 Driver Medium sum It should be equal to all Task The sum of the copies in

review,

Tomorrow,

preview, ？？？？？？？？？？？？day07,

 day06 Watch the course review video ？？？？？