Hadoop-单机伪分布式部署

1. 配置网络环境
   • vi /etc/hosts: 192.168.140.130 hadoop
   • vi /etc/hostname: hadoop
   • 关闭防火墙
     • systemctl stop firewalld.service
     • systemctl disable firewalld.service
     • firewall-cmd --state
   • 配置免密钥登陆
     • ssh-keygen -t rsa
     • ssh-copy-id hadoop
2. 安装Java, 并配置环境变量:（Linux-yum安装jdk11）
3. 解压 Hadoop 软件包: tar -zxvf hadoop-3.3.5.tar.gz -C /opt/hadoop
4. 修改配置文件: /opt/hadoop/hadoop-3.3.5/etc/hadoop/
   • hadoop-env.sh
     • export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.18.0.10-2.el9_1.x86_64
   • core-site.xml

     <configuration>
         <property>
             <name>fs.defaultFS</name>
             <value>hdfs://hadoop:9000</value>
         </property>
         <property>
             <name>hadoop.tmp.dir</name>
             <value>/tmp/hadoop</value>
         </property>
     </configuration>

   • hdfs-site.xml

     <configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
        <property>
            <name>dfs.secondary.http.address</name>
            <value>hadoop:50090</value>
        </property>
     </configuration>
     

   • mapred-site.xml (指定使用 yarn 方式运行 mr)

     <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
     </configuration>
     

   • yarn-site.xml

     <configuration>
         <property>
             <name>yarn.resourcemanager.hostname</name>
             <value>hadoop</value>
         </property>
         <property>
             <name>yarn.nodemanager.aux-services</name>
             <value>mapreduce_shuffle</value>
         </property>
     </configuration>
     

   • workers (slaves in Hadoop2): hadoop
5. 格式化: hdfs namenode -format(或 hadoop namenode -format)
6. 修改启动脚本 (使用 root 用户启动): /opt/hadoop/hadoop-3.3.5/sbin/
   • start-dfs.sh and stop-dfs.sh

     HDFS_DATANODE_USER=root
     HADOOP_SECURE_DN_USER=hdfs
     HDFS_NAMENODE_USER=root
     HDFS_SECONDARYNAMENODE_USER=root
     

   • start-yarn.sh and stop-yarn.sh

     YARN_RESOURCEMANAGER_USER=root
     HADOOP_SECURE_DN_USER=yarn
     YARN_NODEMANAGER_USER=root
     

7. 配置环境变量
   • vi /etc/profile
   • export HADOOP_HOME=/opt/hadoop/hadoop-3.3.5
   • export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   • source /etc/profile
8. 启动 Hadoop
   • start-dfs.sh
   • start-yarn.sh
9. web 访问端口
   • 50070(Hadoop2) | 9870(Hadoop3)
   • 8088