Heartbeat + DRBD + NFS 高可用文件共享

时间 2021-02-28

Heartbeat为我们提供了廉价的、可伸缩的高可用集群方案。我们通过heartbeat+drbd+nfs在Linux下创建一个高可用(HA)的集群服务器。

一．DRBD是一种块设备，可以被用于高可用(HA)之中。它类似于一个网络RAID-1功能。当你将数据写入本地文件系统时，数据还将会被发送到网络中另一台主机上。以相同的形式记录在一个文件系统中。本地(主节点)与远程主机(备节点)的数据可以保证实时同步。当本地系统出现故障时，远程主机上还会保留有一份相同的数据，可以继续使用。

二．案例拓扑：

一．Heratbeat 配置

1.修改hosts文件

[[email protected] ~]# vim /etc/hosts

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

192.168.10.1 node1.a.com

192.168.10.2 node2.a.com

2.修改network文件

[[email protected] ~]# hostname node1.a.com

[[email protected] ~]# vim /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=no

HOSTNAME=localhost.localdomain

NISDOMAIN=node1.a.com

3.配置yum源

[[email protected] ~]# hwclock -s //同步时间

[[email protected] ~]# mkdir /mnt/cdrom

[[email protected] ~]# mount /dev/cdrom /mnt/cdrom

[[email protected] ~]# vim /etc/yum.repos.d/rhel-debuginfo.repo

[rhel-server]

name=Red Hat Enterprise Linux server

baseurl=file:///mnt/cdrom/Server

enabled=1

gpgcheck=0

gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release

[rhel-cluster]

name=Red Hat Enterprise Linux cluster

baseurl=file:///mnt/cdrom/Cluster

enabled=1

gpgcheck=0

gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release

node1与node2 的配置同样。

[[email protected] ~]# scp /etc/yum.repos.d/rhel-debuginfo.repo node2.a.com:/etc/yum.repos.d/

Heartbeat

4.安装heartbeat所需的包。

[[email protected] ~]# yum localinstall heartbeat-2.1.4-9.el5.i386.rpm heartbeat-lnoded-2.1.4-9.el5.i386.rpm heartbeat-pils-2.1.4-10.el5.i386.rpm heartbeat-stonith-2.1.4-10.el5.i386.rpm libnet-1.1.4-3.el5.i386.rpm perl-MailTools-1.77-1.el5.noarch.rpm --nogpgcheck -y

[[email protected] ~]# cd /usr/share/doc/heartbeat-2.1.4/

[[email protected] heartbeat-2.1.4]# cp authkeys haresources ha.cf /etc/ha.d/

[[email protected] heartbeat-2.1.4]# vim ha.cf

94 bcast eth1 //产生广播监听eth1

213 node node1.a.com //定义节点

214 node node2.a.com

5.验证使用md5 认证密文

[[email protected] ~]# echo "123" |openssl md5

ba1f2511fc30423bdbb183fe33f3dd0f

[[email protected] heartbeat-2.1.4]# vim authkeys

auth 3

3 md5 ba1f2511fc30423bdbb183fe33f3dd0f

[[email protected] ha.d]# chmod 600 authkeys //修改权限

6.修改资源文件/etc/ha.d/haresources，定义HA群集的资源

[[email protected] ~]# cp /usr/share/doc/heartbeat-lnoded-2.1.4/lnoded.cf /etc/ha.d/

[[email protected] ha.d]# vim haresources

node1.a.com IPaddr::192.168.10.254/24/eth0 drbddisk::web Filesystem::/dev/drbd0::/public::ext3 killnfsd

[[email protected] ~]# vim /etc/exports

/public 192.168.10.0/24(rw,sync)

二．DRBD的配置

1 简介：

Distributed Replicated Block Device(DRBD)是一个用软件实现的、无共享的、服务器之间镜像块设备内容的存储复制解决方案。Node1，node2上DRBD 配置

1.添加一块磁盘分区

[[email protected] ~]# fdisk /dev/sdb

Command (m for help): n

Command action

e extended

p primary partition (1-4)

Partition number (1-4): 1

First cylinder (1-1044, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1044, default 1044): +1G

Command (m for help): P

Disk /dev/sdb: 8589 MB, 8589934592 bytes

255 heads, 63 sectors/track, 1044 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sdb1 1 123 987966 83 Linux

Command (m for help): W

The partition table has been altered!

2.修改hosts

[[email protected] ~]# vim /etc/hosts

192.168.10.1 node1.a.com

192.168.10.2 node2.a.com
[[email protected] ~]# scp /etc/hosts 192.168.20.2:/etc/

[[email protected] ~]# rpm -ivh drbd83-8.3.8-1.el5.centos.i386.rpm //需要网上下载。

[[email protected] ~]# rpm -ivh kmod-drbd83-8.3.8-1.el5.centos.i686.rpm

3.编辑配置文件

[[email protected] ~]# vim /etc/drbd.conf //在里面输入如下

:r /usr/share/doc/drbd83-8.3.8/drbd.conf

[[email protected] ~]# cd /etc/drbd.d/

[[email protected] drbd.d]# vim global_common.conf // 1,$d 干掉所有添如下内容

global {

usage-count yes;

# minor-count dialog-refresh disable-ip-verification

}

common {

protocol C;

startup {

wfc-timeout 120;

degr-wfc-timeout 120;

}

disk {

on-io-error detach;

fencing resource-only;

}

net {

cram-hmac-alg "sha1";

shared-secret "mydrbdlab";

}

syncer {

rate 100M;

}

[[email protected] drbd.d]# vim web.res

resource web {

on node1.a.com {

device /dev/drbd0;

disk /dev/sdb1;

address 192.168.10.1:7789;

meta-disk internal;

}

on node2.a.com {

device /dev/drbd0;

disk /dev/sdb1;

address 192.168.10.2:7789;

meta-disk internal;

}

[[email protected] drbd.d]# scp * node2.a.com:/etc/drbd.d //拷贝到配置到节点2上。

5.开始初始化双方借点上都要执行

[[email protected] drbd.d]# drbdadm create-md web

6.在双方的节点上启动服务 [[email protected] drbd.d]#service drbd start

7. 查看状态 [[email protected]1~]# cat /proc/drbd

version: 8.3.8 (api:88/proto:86-94)

GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by [email protected], 2010-06-04 08:04:16

0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:987896

都为second 状态，没有同步

也可以

[[email protected] ~]# drbd-overview

0:web Connected Secondary/Secondary Inconsistent/Inconsistent C r----

8.在主机点上执行

[[email protected] ~]# drbdadm -- --overwrite-data-of-peer primary web

然后查看同步过程 watch -n 1 'cat /proc/drbd' //一般速度比较快不容易发现

[[email protected] ~]# drbd-overview //节点1是主节点2是备

0:web Connected Primary/Secondary UpToDate/UpToDate C r----

[[email protected] ~]# drbd-overview

0:web Connected Secondary/Primary UpToDate/UpToDate C r----

9.创建文件系统（在主节点上实现）

[[email protected] ~]# mkfs -t ext3 -L drbdweb /dev/drbd0 //-L 表示卷标

[[email protected] ~]# mkdir /mnt/public

[[email protected] ~]# mount /dev/drbd0 /mnt/public

[[email protected] ~]# cd /mnt/public

10.执行I/O操作

[[email protected] public]# cp /etc/grub.conf ./

[[email protected] public]# ll

-rw------- 1 root root 746 01-19 21:11 grub.conf

drwx------ 2 root root 16384 01-19 21:03 lost+found

[[email protected] public]# cd

[[email protected] ~]# umount /mnt/public

[[email protected] ~]# drbdadm secondary web

[[email protected] ~]# drbd-overview

0:web Connected Secondary/Secondary UpToDate/UpToDate C r----

[[email protected] ~]# drbdadm primary web //手动设置节点2为主

[[email protected] ~]# drbd-overview

0:web Connected Primary/Secondary UpToDate/UpToDate C r---- //本身设备主/对方设备辅

[[email protected] ~]# mount /dev/drbd0 /mnt/public

[[email protected] ~]# cd /mnt/public

[[email protected] 1]# ll

总计 20

-rw------- 1 root root 746 2013-01-19 grub.conf

drwx------ 2 root root 16384 2013-01-19 lost+found

[[email protected] public]# cp /etc/passwd ./

[[email protected] public# cd

[[email protected] ~]# umount /mnt/1

[[email protected] ~]# drbdadm secondary web

11.切换到主节点1上查看

[[email protected] ~]# drbdadm primary web

[[email protected] ~]# mount /dev/drbd0 /public

[[email protected] ~]# cd /public

[[email protected] public]# ll

总计 24

-rw------- 1 root root 746 01-19 21:11 grub.conf //节点1上的I/O操作

drwx------ 2 root root 16384 01-19 21:03 lost+found

-rw-r--r-- 1 root root 1560 01-09 13:43 passwd //节点2上的I/O操作
只有手动将一个节点设置为主节点时，才可访问该节点下的内容，不符合智能化的要求，可以使用heartbeat工具，将drbd作为其资源，实现主节点和备份节点的自动切换。在两个节点上都安装nfs服务，将分区共享出来，这样客户端就可以看到这些内容

三．NFS搭建

1.配置node1.a.com的NFS server： node2上做同样的配置

NFS服务是默认安装的，使用时直接启动即可；

# service nfs start # service portmap start

2.编写共享清单：

[[email protected] ~]# vim /etc/exports

/public 192.168.10.0/24(rw,sync)

# exportfs -rv 导出共享清单：

exporting 192.168.10.0/24:/public -r 导出； -v 详细信息；

3.修改nfs启动脚本：

# vim /etc/init.d/nfs

122 killproc nfsd -9

4.改变nfs服务的资源控制文件的权限：

# chmod 755 /etc/ha.d/resource.d/killnfsd

5.创建nfs服务的资源控制文件：

# vim /etc/ha.d/resource.d/killnfsd

killall -9 nfsd ; /etc/init.d/nfs restart ; exit 0

6.改变nfs服务的资源控制文件的权限：

# chmod 755 /etc/ha.d/resource.d/killnfsd

启动node1.a.com节点的heartbeat的服务：

# service heartbeat start

启动node2.a.com节点的heartbeat的服务：

# service heartbeat start

7.查看heartbeat两节点的状态：

[[email protected] ~]# drbd-overview

0:web Connected Primary/Secondary UpToDate/UpToDate C r---- /public ext3 950M 18M 885M 2%

四．测试：

1. 开启一个客户端：

配置一个临时地址： # ifconfig eth0 192.168.10.3

创建一个挂载点：

# mkdir /mnt/nfs # mount 192.168.10.254:/public /mnt/nfs

查看挂载的状态：

2 当heartbeat正常运行时：查看NFS网络文件系统：

[[email protected] ~]# cd /mnt/nfs

[[email protected] nfs]# ll

-rw------- 1 root root 746 01-19 16:11 grub.conf

drwx------ 2 root root 16384 01-19 16:03 lost+found

-rw-r--r-- 1 root root 1560 01-09 08:43 passw

3.在node1.a.com中将heartbeat服务关闭：

[[email protected] ~]# service heartbeat stop

[[email protected] ~]# drbd-overview

[[email protected] ~]# ifconfig

[[email protected] ~]# mount

nfsd on /proc/fs/nfsd type nfsd (rw)

/dev/drbd0 on /public type ext3 (rw)

4.NFS网络文件系统仍然可以使用

[[email protected] ~]# cd /mnt/nfs

[[email protected] nfs]# ll

总计 24

-rw------- 1 root root 746 01-19 16:11 grub.conf

drwx------ 2 root root 16384 01-19 16:03 lost+found

-rw-r--r-- 1 root root 1560 01-09 08:43 passwd

通过上述案例可以实现高可用性，以及通过heartbeart控制DRDB的自动切换，而且保证文件系统的高可用性。

4.定义资源

转载于:https://blog.51cto.com/dg123/1123488