从部署mysql聊一聊有状态服务和PV及PVC

前言

部署mysql之前我们需要先了解一个概念有状态服务。这是一种特殊的服务,简单的归纳下就是会产生需要持久化的数据,并且有很强的I/O需求,且重启需要依赖上次存储到磁盘的数据。如典型的mysql,kafka,zookeeper等等。

在我们有比较优秀的商业存储的前提下,灰常推荐使用有状态服务进行部署,计算和存储分离那是相当的爽的。在实际生产中如果没有这种存储,才有localPV也是不错的选择,当然local pv其实呢和hostPath是一样的。当然我们在开发测试环境也是可以自己搭建一套简单的如NFS服务,来享受存储和计算分离的爽快感。

kubernetes中定义一种了资源类型Stateful Service即有状态服务,有状态服务需要的持久化数据动态绑定我们可以利用存储的API PersistentVolume(PV)和PersistentVolumeClaim(PVC)来进行需要的相关数据的绑定和存储。

PV & PVC

PV就好比是一个仓库,我们需要先购买一个仓库,即定义一个PV存储服务,例如CEPH,NFS,Local Hostpath等等。
PVC就好比租户,pv和pvc是一对一绑定的,挂载到POD中,一个pvc可以被多个pod挂载。
大致一个流程如下,
可以从这里,以及官网看到更多的额关于PV的细节

1
创建PV --> 创建PVC --> 绑定 --> 可写入数据

有了这个理解之后,我们接下来实战一下(mysql(pvc)+NFS(PV))

NFS Server

首先我们需要创建一个nfs server。我从hub.docker找到一个nfs-server。下面我们将他部署到kubernetes中。

部署nfs到kubernetes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# nfs-server.yaml
apiVersion: v1
kind: Service
metadata:
name: nfs-service
labels:
app: nfs-service
spec:
ports:
- port: 2049
name: nfs-service
clusterIP: 10.96.0.14
selector:
app: nfs-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-service
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-service
template:
metadata:
labels:
app: nfs-service
spec:
# 采用node选择器
nodeName: node1
containers:
- name: nfs-service
image: itsthenetwork/nfs-server-alpine:latest
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
capabilities:
add:
- SYS_ADMIN
- SETPCAP
livenessProbe:
tcpSocket:
port: 2049
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: 2049
timeoutSeconds: 5
ports:
- containerPort: 2049
env:
- name: SHARED_DIRECTORY
value: /nfsshare
volumeMounts:
- mountPath: /nfsshare
name: nfsshare
volumes:
- name: nfsshare
hostPath:
path: /nfsshare
type: DirectoryOrCreate

部署到集群

script
1
kubectl apply -f nfs-server.yaml

查看日志

script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@server1:~# kubectl logs -f deploy/nfs-service
Starting rpcbind...
Displaying rpcbind status...
program version netid address service owner
100000 4 tcp 0.0.0.0.0.111 - superuser
100000 3 tcp 0.0.0.0.0.111 - superuser
100000 2 tcp 0.0.0.0.0.111 - superuser
100000 4 udp 0.0.0.0.0.111 - superuser
100000 3 udp 0.0.0.0.0.111 - superuser
100000 2 udp 0.0.0.0.0.111 - superuser
100000 4 local /var/run/rpcbind.sock - superuser
100000 3 local /var/run/rpcbind.sock - superuser
Starting NFS in the background...
rpc.nfsd: knfsd is currently down
rpc.nfsd: Writing version string to kernel: -2 -3 +4 +4.1 +4.2
rpc.nfsd: Created AF_INET TCP socket.
Exporting File System...
exporting *:/nfsshare
/nfsshare <world>
Starting Mountd in the background...These
Startup successful.

验证

安装nfs-client。
CentOS

script
1
sudo yum install nfs-utils

Ubuntu

script
1
sudo apt-get install nfs-common -y

挂载到node的目录上

script
1
sudo mount -t nfs -o vers=4,minorversion=0,noresvport 10.96.0.14:/ /nfs

向挂载的NFS木写入数据,

script
1
echo 1 > /nfs/1.txt

前往node1查看,看到如下结果表示NFS服务器搭建成功。

script
1
2
3
4
root@node1:/nfsshare# ls
1.txt
root@node1:/nfsshare# cat 1.txt
1

静态PV

搭建完了nfs之后,我们就可以创建PV了

部署pv

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-static-pv
spec:
capacity:
storage: 10Gi
storageClassName: mysql-scn
#ReadWriteOnce - 卷可以由单个节点以读写方式挂载
#ReadOnlyMany - 卷可以由许多节点以只读方式挂载
#ReadWriteMany - 卷可以由许多节点以读写方式挂载
accessModes:
- ReadWriteOnce
#Retain,不清理, 保留 Volume(需要手动清理)
#Recycle,删除数据,即 rm -rf /thevolume/*(只有 NFS 和 HostPath 支持)
#Delete,删除存储资源,比如删除 AWS EBS 卷(只有 AWS EBS, GCE PD, Azure Disk 和 Cinder 支持)
persistentVolumeReclaimPolicy: Retain
nfs:
path: /
server: 10.96.0.14
mountOptions:
- vers=4
- minorversion=0
- noresvport

部署PV到集群中

script
1
2
$ kubectl apply -f mysql-pv.yaml 
persistentvolume/mysql-static-pv created

查看PV状态,看到status为Available表示PV创建成功。

script
1
2
3
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
mysql-static-pv 10Gi RWO Retain Available mysql-scn 3s

部署PVC

script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mysql-static-pvc
spec:
#RWO - ReadWriteOnce
#ROX - ReadOnlyMany
#RWX - ReadWriteMany
accessModes:
- ReadWriteOnce
storageClassName: mysql-scn
resources:
requests:
storage: 10Gi

部署到集群中

script
1
2
$ kubectl apply -f mysql-static-pvc.yaml 
persistentvolumeclaim/mysql-static-pvc created

查看状态,看到STATUS=Bound表示PVC和PV绑定成功了。可以看到我们有一个名称为mysql-static-pvc的pvc可以用了。

script
1
2
3
freemandeMacBook-Pro:pv-pvc freeman$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-static-pvc Bound mysql-static-pv 10Gi RWO mysql-scn 4s

部署mysql使用pvc存储

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# mysql-server.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
ports:
- port: 3306
name: mysql
clusterIP: 10.96.0.15
selector:
app: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: mysql/mysql-server:8.0.17
imagePullPolicy: IfNotPresent
livenessProbe:
tcpSocket:
port: 3306
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: 3306
timeoutSeconds: 5
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql-data
readOnly: false
env:
- name: MYSQL_ROOT_PASSWORD
value: "root"
volumes:
- name: mysql-data
persistentVolumeClaim:
# 这里名称需要和pvc一一对应
claimName: mysql-static-pvc

部署到集群

script
1
kubectl apply -f mysql-server.yaml

查看日志

1
2
3
4
5
6
7
$ kubectl logs -f deploy/mysql
...
[Entrypoint] Starting MySQL 8.0.17-1.1.12
2019-08-22T09:44:49.137554Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 1
2019-08-22T09:44:49.754020Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-08-22T09:44:49.795415Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.17' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL.
2019-08-22T09:44:49.943852Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '0.0.0.0' port: 33060

查看运行状态

script
1
2
$ kubectl get pods -owide | grep mysql
mysql-f475dc6c8-xnb4x 1/1 Running 0 5m20s 172.224.3.158 node1 <none> <none>

查看node1 nfs目录下的数据,可以看到mysql的文件都存储到了我们的nfs属猪鸡的文件系统中了。而我们在部署mysql时,是不需要指定node也没有挂载hostpath,这意味着我们的mysql计算是可以运行在任意集群中的noe上。

script
1
2
3
4
root@node1:/nfsshare# ls /nfsshare/
auto.cnf binlog.index client-cert.pem ibdata1 ibtmp1 mysql.ibd performance_schema server-cert.pem undo_001
binlog.000001 ca-key.pem client-key.pem ib_logfile0 '#innodb_temp' mysql.sock private_key.pem server-key.pem undo_002
binlog.000002 ca.pem ib_buffer_pool ib_logfile1 mysql mysql.sock.lock public_key.pem sys

测试mysql 迁移到其他node运行

我们先看一下目前mysql是运行在哪个节点上,可以看到目前是运行在node1上。

script
1
2
$ kubectl get pods -owide | grep mysql
mysql-f475dc6c8-xnb4x 1/1 Running 0 5m20s 172.224.3.158 node1 <none> <none>

为了方便测试我修改一下mysql的yaml文件加入node选择器,将mysql强行调度到node2运行。

1
spec.template.spec.nodeName: node2

重新apply

1
2
3
$ kubectl apply -f mysql-server.yaml 
service/mysql unchanged
deployment.apps/mysql configured

查看mysql是否还运行正常,可以看到运行正常,这就是存储和计算分离的爽点。

script
1
2
$ kubectl get pods -owide | grep mysql
mysql-6f5944cbcd-nd7c4 1/1 Running 0 38s 172.224.4.18 node2 <none> <none>

授权远程访问

script
1
2
3
4
5
6
7
8
$ kubectl get pods | grep mysql
mysql-6f5944cbcd-nd7c4 1/1 Running 0 7m58s
$ kubectl exec -it mysql-6f5944cbcd-nd7c4 bash
$ mysql -uroot -proot
$ CREATE USER 'root'@'%' IDENTIFIED BY 'root';
$ GRANT ALL ON *.* TO 'root'@'%';
$ ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'root';
$ FLUSH PRIVILEGES;

在开发机进行访问

script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ mysql -uroot -proot -h 10.96.0.15
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 136
Server version: 8.0.17 MySQL Community Server - GPL

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

到此我们完成了基于Mysql+nfs的部署工作。

推荐文章