JZTXT

Hadoop - 分布式部署

发布时间 2023-06-06 01:23:21作者: HOUHUILIN

Zookeeper的分布式部署 >> Hadoop的分布式部署

 

 

集群规划

安装部署

 

 

 

 

集群规划

序号 主机名 JDK Zookeeper NameNode JournalNode DataNode ResourceManager NodeManager
1 node01 JDK ZK NN JN DN RM NM
2 node02 JDK ZK 2NN JN DN RM NM
3 node03 JDK ZK   JN DN   NM

 

 

 

安装部署

1、将 hadoop-2.5.2.tar.gz 上传到node01、node02、node03的 /opt/software目录下

2、将 hadoop-2.5.2.tar.gz 解压到 /opt/module 目录下

[root@node01 software]# tar -zxvf hadoop-2.5.2.tar.gz -C /opt/module/
hadoop-2.5.2/
hadoop-2.5.2/bin/
hadoop-2.5.2/bin/hadoop
hadoop-2.5.2/bin/hdfs
hadoop-2.5.2/bin/mapred
hadoop-2.5.2/bin/yarn.cmd
hadoop-2.5.2/bin/hadoop.cmd
hadoop-2.5.2/bin/hdfs.cmd
hadoop-2.5.2/bin/mapred.cmd
......
[root@node01 software]#

3、修改配置文件/opt/module/hadoop-2.5.2/etc/hadoop/hadoop-env.sh

......
# 配置JDK
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.372.b07-1.el7_9.x86_64/

......
# 定义一些变量
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_ZKFC_USER=root

通过whereis java找到JDK的bin目录

也可以为JDK设置环境变量，即修改/etc/profile文件，添加export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.372.b07-1.el7_9.x86_64，然后执行source /etc/profile。这样的话，hadoop-env.sh里的export JAVA_HOME=${JAVA_HOME}就可以取到值。也就不用在配置文件里把JAVA_HOME的路径写死了

4、修改配置文件/opt/module/hadoop-2.5.2/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<!-- 核心的hdfs协议访问方式 -->
	<property>
        <name>fs.defaultFS</name>
        <value>hdfs://nameservice1</value>
    </property>
	
	<!-- 所有hadoop的存储目录 -->
	<property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop/data/hadoop-${user.name}</value>
    </property>
    
    <!-- 指定Hadoop依赖的Zookeeper列表 -->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>node01:2181,node02:2181,node03:2181</value>
    </property>
</configuration>

5、修改配置文件/opt/module/hadoop-2.5.2/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 副本数;默认3个 -->
	<property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>

    <!-- 权限检查 -->
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>

    <!-- dfs.namenode.name.dir:namenode的目录放的路径在hadoop.tmp.dir之上做了修改
		file://${hadoop.tmp.dir}/dfs/name

		dfs.datanode.data.dir:namenode的目录放的路径在hadoop.tmp.dir之上做了修改
		file://${hadoop.tmp.dir}/dfs/data
     -->
    <!-- 为nameservice起一个别名 -->
    <property>
        <name>dfs.nameservices</name>
        <value>nameservice1</value>
    </property>
    
    <property>
        <name>dfs.ha.namenodes.nameservice1</name>
        <value>nn1,nn2</value>
    </property>

    <property>
        <name>dfs.namenode.rpc-address.nameservice1.nn1</name>
        <value>node01:8020</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.nameservice1.nn1</name>
        <value>node01:9870</value>
    </property>
    
    <!-- 第二个成员 -->
    <property>
        <name>dfs.namenode.rpc-address.nameservice1.nn2</name>
        <value>node02:8020</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.nameservice1.nn2</name>
        <value>node02:9870</value>
    </property>
    
    <!-- journalnode:负责hadoop与zk进行沟通 -->
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://node01:8485;node02:8485;node03:8485/nameservice1</value>
    </property>
    
    <!-- 自动切换namenode -->
    <property>
        <name>dfs.client.failover.proxy.provider.nameservice1</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    
    <!-- journal的存储位置 -->
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/data/hadoop/data/journal/</value>
    </property>

    <!-- 大哥挂了,自动切换到二哥上
        启动故障转移
     -->
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>

    <!-- （ssh免密码登录） -->
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_rsa</value>
    </property>
</configuration>

dfs.replication：副本数、

dfs.permissions.enabled：权限检查,true:在HDFS中启用权限检查.false:关闭权限检查

dfs.nameservices：以逗号分隔的主节点列表（namenode组）

dfs.namenode.rpc-address.nameservice1.nn1：处理所有客户端请求的RPC地址

dfs.namenode.http-address.nameservice1.nn1：DFS NameNodeWebUI将侦听的地址和基本端口。

dfs.namenode.rpc-address.nameservice1.nn2：处理所有客户端请求的RPC地址

dfs.namenode.http-address.nameservice1.nn2：DFS NameNodeWebUI将侦听的地址和基本端口。

dfs.namenode.shared.edits.dir：HA集群中多个namenodes之间共享存储的目录。此目录将由活动目录写入，由备用目录读取，以保持名称空间同步。在非HA集群中保持为空。

dfs.client.failover.proxy.provider.nameservice1： 主机配置的故障转移代理提供程序的类名的前缀(加上所需的名称服务ID)。有关更详细的信息，请参阅HDFS高可用性文档的“配置详细信息”部分。

dfs.journalnode.edits.dir：存储日志编辑文件的目录。

dfs.ha.automatic-failover.enabled：是否启用自动故障转移。

dfs.ha.fencing.methods：免密登录

dfs.ha.fencing.ssh.private-key-files：私钥目录（/root/.ssh/id_rsa）

 

6、 修改配置文件：/opt/module/hadoop-2.5.2/etc/hadoop/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- yarn -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <!-- ctrl+shift+/ -->
    <!--  <property>
       <name>mapreduce.application.classpath</name>
       <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property> -->
  
    <!-- 一旦启动了yarn,建议换成必须设置最大内存 -->
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>200</value>
    </property>
    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx200M</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>200</value>
    </property>
    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>-Xmx200M</value>
    </property>
</configuration>

7、修改配置文件： /opt/module/hadoop-2.5.2/etc/hadoop/yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
        <!-- 配置yarn -->
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.nodemanager.env-whitelist</name>
                <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
        </property>
        <!-- yarn开启ha -->
        <property>
                <name>yarn.resourcemanager.ha.enabled</name>
                <value>true</value>
        </property>
        <!-- yarn董事会的名字 -->
        <property>
                <name>yarn.resourcemanager.cluster-id</name>
                <value>yarn1</value>
        </property>
        <!-- 董事会列表 -->
        <property>
                <name>yarn.resourcemanager.ha.rm-ids</name>
                <value>rm1,rm2</value>
        </property>
        <!-- hostname,webapp-->
        <property>
                <name>yarn.resourcemanager.hostname.rm1</name>
                <value>node01</value>
        </property>
        <property>
                <name>yarn.resourcemanager.webapp.address.rm1</name>
                <value>node01:8088</value>
        </property>
        <!-- 第二台 -->
        <property>
                <name>yarn.resourcemanager.hostname.rm2</name>
                <value>node02</value>
        </property>
        <property>
                <name>yarn.resourcemanager.webapp.address.rm2</name>
                <value>node02:8088</value>
        </property>
        <!-- zookeeper -->
        <property>
                <name>hadoop.zk.address</name>
                <value>node01:2181,node02:2181,node03:2181</value>
        </property>
</configuration>

8、将解压后的hadoop-2.5.2分发到node02、node03节点的/opt/module目录下

[root@node01 module]# scp -r -p hadoop-2.5.2/ root@node02:$PWD
[root@node01 module]# scp -r -p hadoop-2.5.2/ root@node03:$PWD

 

 

首次启动初始化

1、启动zookeeper（在node01、node02、node03节点执行）

cd /opt/module/zookeeper-3.4.5/bin
./zkServer.sh restart

 2、启动journalnode（在node01、node02、node03节点执行）

sbin/hadoop-daemon.sh start journalnode

脚本路径：/opt/module/hadoop-2.5.2/sbin/hadoop-daemon.sh

 3、格式化namenode

[root@node01 hadoop-2.5.2]# bin/hdfs namenode -format
23/06/06 01:02:22 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node01/192.168.56.121
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.5.2
......

脚本路径： /opt/module/hadoop-2.5.2/bin/hdfs

 4、将格式化后的元数据分发到node02（2nn）的 /data/hadoop/data目录下

[root@node01 hadoop-2.5.2]# cd /data/hadoop/data
[root@node01 data]# scp -r -p hadoop-root/ root@node02:$PWD
VERSION                                                                                                                                              100%  206   335.4KB/s   00:00
seen_txid                                                                                                                                            100%    2     2.0KB/s   00:00
fsimage_0000000000000000000.md5                                                                                                                      100%   62    37.9KB/s   00:00
fsimage_0000000000000000000                                                                                                                          100%  351     1.1MB/s   00:00
[root@node01 data]#

5、启动node01节点的namenode（nn）

[root@node01 hadoop-2.5.2]# sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /opt/module/hadoop-2.5.2/logs/hadoop-root-namenode-node01.out
[root@node01 hadoop-2.5.2]# jps
2032 JournalNode
2294 Jps
1943 QuorumPeerMain
2223 NameNode
[root@node01 hadoop-2.5.2]#

6、在node02节点执行 bin/hdfs namenode -bootstrapStandby

[root@node02 hadoop-2.5.2]# bin/hdfs namenode -bootstrapStandby
23/06/06 01:10:01 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node02/192.168.56.122
STARTUP_MSG:   args = [-bootstrapStandby]
STARTUP_MSG:   version = 2.5.2
...
23/06/06 01:10:01 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
23/06/06 01:10:01 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
=====================================================
About to bootstrap Standby ID nn2 from:
           Nameservice ID: nameservice1
        Other Namenode ID: nn1
  Other NN's HTTP address: http://node01:9870
  Other NN's IPC  address: node01/192.168.56.121:8020
             Namespace ID: 391110921
            Block pool ID: BP-1236811567-192.168.56.121-1685984543491
               Cluster ID: CID-5f5f7221-5415-49f5-b42b-3931a21825f4
           Layout version: -57
=====================================================
Re-format filesystem in Storage Directory /data/hadoop/data/hadoop-root/dfs/name ? (Y or N) y
23/06/06 01:10:09 INFO common.Storage: Storage directory /data/hadoop/data/hadoop-root/dfs/name has been successfully formatted.
23/06/06 01:10:10 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
23/06/06 01:10:10 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
23/06/06 01:10:10 INFO namenode.TransferFsImage: Opening connection to http://node01:9870/imagetransfer?getimage=1&txid=0&storageInfo=-57:391110921:0:CID-5f5f7221-5415-49f5-b42b-3931a21825f4
23/06/06 01:10:10 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds
23/06/06 01:10:10 INFO namenode.TransferFsImage: Transfer took 0.00s at 0.00 KB/s
23/06/06 01:10:10 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 351 bytes.
23/06/06 01:10:10 INFO util.ExitUtil: Exiting with status 0
23/06/06 01:10:10 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node02/192.168.56.122
************************************************************/

7、 启动node02的namenode（2nn）

# 启动namenode
[root@node02 hadoop-2.5.2]# sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /opt/module/hadoop-2.5.2/logs/hadoop-root-namenode-node02.out
[root@node02 hadoop-2.5.2]# jps
2080 Jps
1795 JournalNode
2009 NameNode
1647 QuorumPeerMain
[root@node02 hadoop-2.5.2]#

8、在其中一个zookeeper节点初始化 ZKFC（注意在zookeeper服务运行的情况下执行该操作）

[root@node01 hadoop-2.5.2]# bin/hdfs zkfc -formatZK
23/06/06 01:14:15 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at node01/192.168.56.121:8020
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:host.name=node01
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_372
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Red Hat, Inc.
...
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/module/hadoop-2.5.2/lib/native
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:user.name=root
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/module/hadoop-2.5.2
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=node01:2181,node02:2181,node03:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@797badd3
23/06/06 01:14:15 INFO zookeeper.ClientCnxn: Opening socket connection to server node01/192.168.56.121:2181. Will not attempt to authenticate using SASL (unknown error)
23/06/06 01:14:15 INFO zookeeper.ClientCnxn: Socket connection established to node01/192.168.56.121:2181, initiating session
23/06/06 01:14:15 INFO zookeeper.ClientCnxn: Session establishment complete on server node01/192.168.56.121:2181, sessionid = 0xff888c7b849e0000, negotiated timeout = 5000
23/06/06 01:14:15 INFO ha.ActiveStandbyElector: Session connected.
23/06/06 01:14:15 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/nameservice1 in ZK.
23/06/06 01:14:15 INFO zookeeper.ZooKeeper: Session: 0xff888c7b849e0000 closed
23/06/06 01:14:15 INFO zookeeper.ClientCnxn: EventThread shut down

9、群起/停 namenode、journalnode、datanode

sbin/start-dfs.sh
sbin/stop-dfs.sh