Start cluster

1. Starting the cluster for the first time , You need to format the cluster

If the cluster is starting for the first time , Need to be in hadoop102 Node format NameNode（ Be careful ： Format turn NameNode, There will be new clusters id, Lead to NameNode and DataNode The cluster of id atypism , Cluster search No past data . If the cluster reports an error during operation , Need to reformat NameNode Words , Make sure to stop first stop namenode and datanode process , And delete all machines data and logs Catalog , Then format turn .） Format command ：[[email protected] hadoop-3.1.3]$ hdfs namenode -format

start-up HDFS：[[email protected] hadoop-3.1.3]$ sbin/start-dfs.sh

because ResourceManager Configure in hadoop103 On ; All because in hadoop103 The above to start YARN.                start-up YARN command ：[[email protected] hadoop-3.1.3]$ sbin/start-yarn.sh

Correct troubleshooting （ Premise bin Contents and etc The directory is not damaged ）

1 Kill the process first

[[email protected] hadoop-3.1.3]$ sbin/stop-dfs.sh

[[email protected] hadoop-3.1.3]$ sbin/stop-yarn.sh

2. Delete data and logs

[[email protected] hadoop-3.1.3]$ rm -rf data/ logs/
 
[[email protected] hadoop-3.1.3]$ rm -rf data/ logs/
 
[[email protected] hadoop-3.1.3]$ rm -rf data/ logs/
------------------------------------------------------------

3. Format cluster

[[email protected] hadoop-3.1.3]$ hdfs namenode -format

4. Start cluster

[[email protected] hadoop-3.1.3]$ sbin/start-dfs.sh

Configure history server

1） To configure mapred-site.xml

Just in mapred-site.xml Add two parameters to the configuration file ：
 

[[email protected] hadoop-3.1.3]$ vim mapred-site.xml
 
 
Add the following configuration to this file .
 
Input i Enter editor ;ESC Exit the editor ;shift+： Save and exit
<!-- Historical server address -->
<property>
 <name>mapreduce.jobhistory.address</name>
 <value>hadoop102:10020</value>
</property>
<!-- History server web End address -->
<property>
 <name>mapreduce.jobhistory.webapp.address</name>
 <value>hadoop102:19888</value>
</property>
2） Distribute the configuration to each node

[[email protected] hadoop]$ xsync mapred-site.xml

stay hadoop102 Start the history server

[[email protected] hadoop]$ mapred --daemon start historyserver

Check whether the history server is started successfully ：[[email protected] hadoop]$ jps

Configure the aggregation of logs

The concept of log aggregation ： After the application runs , Upload the program running log information to HDFS On the system

The benefits of log aggregation ： You can easily view the details of program operation , Convenient for development and debugging

Be careful ： Turn on the log aggregation function , Reboot required NodeManager、ResourceManager、HistoryServer.

1. To configure yarn-site.xml

[[email protected] hadoop]$ vim yarn-site.xml

Configure the following contents under the file

Distribution configuration ：[[email protected] hadoop]$ xsync yarn.site.xml

Because it changed yarn Configuration of , So we need to restart yarn;HDFS The configuration has not been modified , So there is no need to restart HDFS

So we need to restart yarn and historysever

[[email protected] hadoop-3.1.3]$ sbin/stop.yarn.sh
close YARN
[[email protected] hadoop-3.1.3]$ sbin/start-yarn.sh
start-up YARN
because yarn Collocation in hadoop103 On , So we need to hadoop103 Start the

[[email protected] hadoop-3.1.3]$mapred --daemon stop historyserver
Turn off the history server
[[email protected] hadoop-3.1.3]$mapred --daemon start historyserver
Start history server

Cluster start / The way to stop is summarized （ Commonly used ）

Each module starts separately / stop it （ To configure ssh It's the premise ）

（1） Overall start up / stop it HDFS ：start-dfs.sh/stop-dfs.sh

（2） Overall start up / stop it YARN ：start-yarn.sh/stop-yarn.sh

Each service component starts one by one / stop it

To write Hadoop Common cluster scripts

Hadoop Cluster start stop script （ contain HDFS,Yarn,Historyserver）：myhadoop.sh

stay bin Create a document under the directory myhadoop.sh

[[email protected] ~]$ cd /home/atguigu/bin
[[email protected] bin]$ vim myhadoop.sh

Configure the following contents under the new file

i  Enter editor ;Esc Exit the editor ; shift+ ：+ wq Save and exit 
 Add the following ：
 
#!/bin/bash
if [ $# -lt 1 ]
then
 echo "No Args Input..."
 exit ;
fi
case $1 in
"start")
 echo " ===================  start-up  hadoop  colony  ==================="
 echo " ---------------  start-up  hdfs ---------------"
 ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"
 echo " ---------------  start-up  yarn ---------------"
ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"
 echo " ---------------  start-up  historyserver ---------------"
 ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start 
historyserver"
;;
"stop")
 echo " ===================  close  hadoop  colony  ==================="
 echo " ---------------  close  historyserver ---------------"
 ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop 
historyserver"
 echo " ---------------  close  yarn ---------------"
 ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"
 echo " ---------------  close  hdfs ---------------"
 ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;;
*)
 echo "Input Args Error..."
;;
esac

Modify script permissions

[[email protected] bin]$ chmod 777 myhadoop.sh

Distribute scripts

[[email protected] bin]$ xsync myhadoop.sh

The test script

[[email protected] bin]$ myhadoop.sh start    Open cluster
[[email protected] bin]$ myhadoop.sh stop    Shut down the cluster

Configure to view three servers Java Script process ：jpsall

stay bin Create a file in the directory jpsall

[[email protected] ~]$ cd /home/atguigu/bin
[[email protected] bin]$ vim jpsall

The configuration is as follows

#!/bin/bash
for host in hadoop102 hadoop103 hadoop104
do
 echo =============== $host ===============
 ssh $host jps 
done

Modify script permissions

[[email protected] bin]$ chmod 777 jpsall

Distribute scripts

[[email protected] bin]$ xsync jpsall

test jpsall Script

[[email protected] bin]$ jpsall

Check whether the configuration on each node is consistent

Common mistakes and solutions

1、 The firewall is not closed 、 Or it doesn't start YARN

2、 Host name configuration error

3、IP Address configuration error

4、ssh Not configured properly

5、root Users and atguigu The cluster started by two users is not unified

6、 The configuration file is not modified carefully

7、 Unrecognized host name

java.net.UnknownHostException: hadoop102: hadoop102
 at 
java.net.InetAddress.getLocalHost(InetAddress.java:1475)
 at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(Job
Submitter.java:146)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
 at java.security.AccessController.doPrivileged(Native 
Method)
at javax.security.auth.Subject.doAs(Subject.java:415)

resolvent ：

1） stay /etc/hosts Add... To the file 192.168.10.102 hadoop102

2） The host name should not start with hadoop hadoop000 Other special names

8、DataNode and NameNode Processes can only work one at a time

9、 The execution order does not take effect , Paste word When the command is in , encounter - And long _ No distinction . Cause the command to fail

resolvent ： Try not to paste word The code in , Or reconfigure the file