Creation of sparksql dataset

Spark SQL framework

• Spark SQL yes Spark One of the core components of （2014.4 Spark1.0）
• Direct access to existing Hive data
• Provide JDBC/ODBC Interface for third-party tools Spark Data processing
• It provides a higher-level interface to process data conveniently
• Support multiple operation modes ：SQL、API Programming
• Support multiple external data sources ：Parquet、JSON、RDBMS etc.
• Catalyst The optimizer is Spark SQL At the heart of

Dataset The creation of

stay Spark shell When created in , Packages that need to be imported ：

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import spark.implicits._
import org.apache.spark.sql.types._

Creation method 1 ：

val dt=spark.createDataset(1 to 5)
dt.show

Creation mode 2

spark.createDataset(List(("a",1),("b",2),("c",3))).show

Custom column names .

Creation method 3 ：
Can be in Dataset put RDD.

spark.createDataset(sc.parallelize(List(("a",1,2),("b",2,3),("c",3,4)))).show

Conclusion ：

1. createDataset() The parameter of can be :Seq、Array、RDD;

2. The above three lines of code generate Dataset Namely ：
   Dataset[Int]、Dataset[(String,Int)]、Dataset[(String,Int,Int)];

3. Dataset=RDD+Schema, therefore Dataset And RDD There are most common functions , Such as map、filter etc.

Use Cass Class establish Dataset

case class Point(name:String,age:Int,score:Int)
val stu=Seq(Point("zs",18,90),Point("ls",19,85))
// convert to Dataset
val stuInfo=stu.toDS
stuInfo.show

This method creates Dataset It can correspond to the corresponding column name .
In addition to calling the sample class at creation , We can also go through map To name the column .

val stu2=spark.createDataset(List(("sam",15,79),("john",17,80)))

stu2 Data and type ：

Use map function ：

 val stu3=stu2.map(x=>Point(x._1,x._2,x._3))
 stu3.show

Use select see ：

stu3.select("name","score").show

Use SQL Statement to query information
In addition to the above methods, you can query information , We can also use SQL Syntax to query data information ：
Spark DataFrame Provides registerTempTable Such an interface , You can save data objects as temporary tables , It is convenient for subsequent query operations . Such as select, join etc. .

stu3.registerTempTable("info")
spark.sql("select * from info").show

Add ：
Scala In the class Add... Before keywords case keyword This class becomes the sample class , The difference between sample class and ordinary class ：

1. Unwanted new You can generate objects directly
2. The default implementation is serialization interface
3. Default auto overwrite toString()、equals()、hashCode()