openstack之安装cinder服务

title: openstack之安装cinder服务 #标题tags: openstack #标签
date: 2021-02-15
categories: openstack # 分类

cinder块存储服务就是给openstack虚机中提供硬盘的，其中包含如下角色：

• cinder-api：接收和响应外部有关块存储请求。
• cinder-volume：提供实际存储空间。
• cinder-scheduler daemon：调度器，决定将要分配的空间由哪一个cinder-volume提供。
• cinder-backup daemon：备份存储，专用于备份cinder-volume，一般很少用。
• Messaging queue：消息队列，在块存储进程之间路由信息。

cinder支持多种存储驱动，如：lvm、nfs、gfs、ceph、san等。

参考官方文档。

注：此博文是基于openstack queens部署进行配置的。

安装配置控制节点

接下来的配置在控制节点执行。

数据库创库授权

$ mysql -uroot -p123.com
CREATE DATABASE cinder;
GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' \
  IDENTIFIED BY '123.com';

keystone创建系统用户

$ source admin-openrc 
# 创建用户
$ openstack user create --domain default --password 123.com cinder
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | df07de5bcab54c1381ba15552c6fca27 |
| name                | cinder                           |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+
# 关联项目
$ openstack role add --project service --user cinder admin

keystone创建服务和注册api

# 创建cinderv2和cinderv3服务实体（块存储服务需要两个服务实体）：
$ openstack service create --name cinderv2 \
 --description "OpenStack Block Storage" volumev2
openstack service create --name cinderv3 \
 --description "OpenStack Block Storage" volumev3
# 注册API
$ openstack endpoint create --region RegionOne   volumev2 public http://controller:8776/v2/%\(project_id\)s
openstack endpoint create --region RegionOne   volumev2 internal http://controller:8776/v2/%\(project_id\)s
openstack endpoint create --region RegionOne   volumev2 admin http://controller:8776/v2/%\(project_id\)s
openstack endpoint create --region RegionOne   volumev3 public http://controller:8776/v3/%\(project_id\)s
openstack endpoint create --region RegionOne   volumev3 internal http://controller:8776/v3/%\(project_id\)s
openstack endpoint create --region RegionOne   volumev3 admin http://controller:8776/v3/%\(project_id\)s

安装cinder

$ yum -y install openstack-cinder

修改cinder配置文件

# 定义控制节点的IP(将下面IP替换为你的cinder服务所在节点的IP)
My_ip=192.168.20.2
# 备份配置文件
mv /etc/cinder/cinder.conf{,.bak}
egrep -v '^$|^#' /etc/cinder/cinder.conf.bak > /etc/cinder/cinder.conf
# 修改配置文件
openstack-config --set /etc/cinder/cinder.conf database connection mysql+pymysql://cinder:123.com@controller/cinder
openstack-config --set /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:123.com@controller
openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/cinder/cinder.conf DEFAULT my_ip ${My_ip}
openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_api_servers http://controller:9292
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller:5000
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller:5000
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken memcached_servers controller:11211
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_type password
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_domain_id default
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken user_domain_id default
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name service
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password 123.com
openstack-config --set /etc/cinder/cinder.conf oslo_concurrency lock_path /var/lib/cinder/tmp

同步数据库

# 同步数据库
$ su -s /bin/sh -c "cinder-manage db sync" cinder
# 可能会输出如下警告信息，可以忽略
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".


# 确认库已新增如下表
$ mysql cinder -uroot -p123.com -e 'show tables;'
+----------------------------+
| Tables_in_cinder           |
+----------------------------+
| attachment_specs           |
| backup_metadata            |
| backups                    |
| cgsnapshots                |
| clusters                   |
| consistencygroups          |
| driver_initiator_data      |
| encryption                 |
| group_snapshots            |
| group_type_projects        |
| group_type_specs           |
| group_types                |
| group_volume_type_mapping  |
| groups                     |
| image_volume_cache_entries |
| messages                   |
| migrate_version            |
| quality_of_service_specs   |
| quota_classes              |
| quota_usages               |
| quotas                     |
| reservations               |
| services                   |
| snapshot_metadata          |
| snapshots                  |
| transfers                  |
| volume_admin_metadata      |
| volume_attachment          |
| volume_glance_metadata     |
| volume_metadata            |
| volume_type_extra_specs    |
| volume_type_projects       |
| volume_types               |
| volumes                    |
| workers                    |
+----------------------------+

修改计算服务配置文件

$ openstack-config --set  /etc/nova/nova.conf cinder os_region_name RegionOne

# 重启计算服务
$ systemctl restart openstack-nova-api

启动cinder服务并加入开机自启

$ systemctl enable openstack-cinder-api openstack-cinder-scheduler
systemctl start openstack-cinder-api openstack-cinder-scheduler

确认服务已正常

# 需要等待一会，确保state的值为up
$ cinder service-list       
+------------------+------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host       | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | controller | nova | enabled | up    | 2021-02-15T09:37:34.000000 | -               |
+------------------+------------+------+---------+-------+----------------------------+-----------------+

安装配置存储节点

接下来的配置在cinder存储节点执行。

安装lvm包

$ yum -y install lvm2 device-mapper-persistent-data

启动lvm元数据服务并加入开机自启

$ systemctl enable lvm2-lvmetad && systemctl start lvm2-lvmetad

制作vg

这里仅以做出效果为目的，如果想要了解更多lvm的知识，请自行查阅相关资料。

自行准备新的硬盘。如果没有识别到新添加的硬盘请执行命令 echo '- - -' > /sys/class/scsi_host/host0/scan， 然后确认已添加（host0目录不固定，请根据实际来定）。

$ lsblk      # 确认新增硬盘已被系统识别到
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0  100G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   99G  0 part 
  ├─centos-root 253:0    0   50G  0 lvm  /
  ├─centos-swap 253:1    0    2G  0 lvm  [SWAP]
  └─centos-home 253:2    0   47G  0 lvm  /home
sdb               8:16   0   50G  0 disk 
sdc               8:32   0   20G  0 disk 
sr0              11:0    1  4.3G  0 rom  
# 创建pv
$ pvcreate /dev/sdb /dev/sdc
  Physical volume "/dev/sdb" successfully created.
  Physical volume "/dev/sdc" successfully created.

# 创建vg
$ vgcreate cinder-ssd /dev/sdb
  Volume group "cinder-ssd" successfully created

$ vgcreate cinder-sata /dev/sdc 
  Volume group "cinder-sata" successfully created

修改lvm配置文件

至于为何要修改lvm的配置文件，官方解释如下（自行阅读吧）：

# 修改配置文件130行
$ vim /etc/lvm/lvm.conf +130      # 增加如下内容（将你系统上所有做了lvm的磁盘都加上）
filter = [ "a/sda/", "a/sdb/" "a/sdc/", "r/.*/"]

安装cinder客户端

$ yum -y install openstack-cinder targetcli python-keystone

修改cinder配置文件

# 定义本机IP
My_ip=192.168.20.3

# 备份配置文件
mv /etc/cinder/cinder.conf{,.bak}
egrep -v '^$|^#' /etc/cinder/cinder.conf.bak > /etc/cinder/cinder.conf

# 修改配置文件
openstack-config --set /etc/cinder/cinder.conf database connection mysql+pymysql://cinder:123.com@controller/cinder
openstack-config --set /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:123.com@controller
openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone
openstack-config --set /etc/cinder/cinder.conf DEFAULT my_ip ${My_ip}
openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends ssd,sata
openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_api_servers http://controller:9292
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller:5000
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller:5000
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken memcached_servers controller:11211
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_type password
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_domain_id default
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken user_domain_id default
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name service
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder
openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password 123.com
openstack-config --set /etc/cinder/cinder.conf oslo_concurrency lock_path /var/lib/cinder/tmp
openstack-config --set /etc/cinder/cinder.conf ssd volume_driver cinder.volume.drivers.lvm.LVMVolumeDriver
openstack-config --set /etc/cinder/cinder.conf ssd volume_group cinder-ssd
openstack-config --set /etc/cinder/cinder.conf ssd iscsi_helper lioadm
openstack-config --set /etc/cinder/cinder.conf ssd iscsi_protocol iscsi
openstack-config --set /etc/cinder/cinder.conf ssd volume_backend_name ssd
openstack-config --set /etc/cinder/cinder.conf sata volume_driver cinder.volume.drivers.lvm.LVMVolumeDriver
openstack-config --set /etc/cinder/cinder.conf sata volume_group cinder-sata
openstack-config --set /etc/cinder/cinder.conf sata iscsi_helper lioadm
openstack-config --set /etc/cinder/cinder.conf sata iscsi_protocol iscsi
openstack-config --set /etc/cinder/cinder.conf sata volume_backend_name sata

# 确认修改后的配置文件（注意下面的提示，可能需要根据你的实际情况去更改）
$ cat /etc/cinder/cinder.conf
[DEFAULT]
transport_url = rabbit://controller:123.com@controller
auth_strategy = keystone
my_ip = 192.168.20.3
enabled_backends = ssd,sata  # 这里的ssd、sata对应配置文件末尾的两段配置名称
glance_api_servers = http://controller:9292
[backend]
[backend_defaults]
[barbican]
[brcd_fabric_example]
[cisco_fabric_example]
[coordination]
[cors]
[database]
connection = mysql+pymysql://cinder:123.com@controller/cinder
[fc-zone-manager]
[healthcheck]
[key_manager]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = 123.com
[matchmaker_redis]
[nova]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[profiler]
[service_user]
[ssl]
[vault]
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_helper = lioadm
iscsi_protocol = iscsi


# 下面的是和上面enabled_backends = ssd,sata遥相呼应，请根据实际情况修改
[ssd]          # 这里要和上面对应上
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-ssd         # 指定用哪个卷组
iscsi_helper = lioadm
iscsi_protocol = iscsi
volume_backend_name = ssd          # 指定此后端存储卷的名称


[sata]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-sata
iscsi_helper = lioadm
iscsi_protocol = iscsi
volume_backend_name = sata

启动cinder服务

$ systemctl enable openstack-cinder-volume target
systemctl start openstack-cinder-volume target

# 查看日志（需要确保无error信息，有时候即使上面启动没报错，也会有问题的，以日志为准）
$ tailf /var/log/cinder/volume.log

验证cinder服务

接下来的配置在控制节点执行。

# 控制节点查询cinder服务列表，可以输出如下则表示没有问题
$ openstack volume service list
+------------------+----------------+------+---------+-------+----------------------------+
| Binary           | Host           | Zone | Status  | State | Updated At                 |
+------------------+----------------+------+---------+-------+----------------------------+
| cinder-scheduler | controller     | nova | enabled | up    | 2021-02-15T11:39:26.000000 |
| cinder-volume    | compute01@ssd  | nova | enabled | up    | 2021-02-15T11:39:29.000000 |
| cinder-volume    | compute01@sata | nova | enabled | up    | 2021-02-15T11:39:30.000000 |
+------------------+----------------+------+---------+-------+----------------------------+

注：如果由于配置错误，导致查出来的volume服务列表有down掉的volume记录，想干掉，可以通过如下命令：

$ cinder service-list      # 由于刚开始操作失误，导致错误的volume信息注册了进来
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host           | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+----------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | controller     | nova | enabled | up    | 2021-02-15T11:42:56.000000 | -               |
| cinder-volume    | compute01@sata | nova | enabled | up    | 2021-02-15T11:42:50.000000 | -               |
| cinder-volume    | compute01@sate | nova | enabled | down  | 2021-02-15T11:21:37.000000 | -               |
| cinder-volume    | compute01@ssd  | nova | enabled | up    | 2021-02-15T11:42:59.000000 | -               |
+------------------+----------------+------+---------+-------+----------------------------+-----------------+


# 执行如下命令即可删除
$ cinder-manage service remove cinder-volume compute01@sate
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
已移除主机 compute01@sate 上的服务 cinder-volume。


# 再次查看即可
$ openstack volume service list
+------------------+----------------+------+---------+-------+----------------------------+
| Binary           | Host           | Zone | Status  | State | Updated At                 |
+------------------+----------------+------+---------+-------+----------------------------+
| cinder-scheduler | controller     | nova | enabled | up    | 2021-02-15T11:44:16.000000 |
| cinder-volume    | compute01@ssd  | nova | enabled | up    | 2021-02-15T11:44:19.000000 |
| cinder-volume    | compute01@sata | nova | enabled | up    | 2021-02-15T11:44:20.000000 |
+------------------+----------------+------+---------+-------+----------------------------+

创建卷类型

接下来的配置在控制节点执行。

由于我们上面添加了两个vg组，如果不进行其他配置，那么我们是无法控制lv卷创建在哪个vg上的，所以还需设置卷类型，如下：

参考学习OpenStack之 (2)：Cinder LVM 配置。

命令行创建卷类型

# 创建卷类型
$ cinder type-create ssd
cinder type-create sata

# 设置volume type 的backend name
# 下面set后面的键值对，就是我们在cinder存储节点上指定的volume_backend_name那个键值对
$ cinder type-key ssd set volume_backend_name=ssd  
cinder type-key sata set volume_backend_name=sata   


# 附件命令：删除卷类型
$ cinder type-delete ssd

web界面创建卷类型

web界面创建比较简单，如下：

控制台创建卷

web页面操作如下：

到存储节点上，执行 lvs 命令，即可看到新增的lv卷：

其id和web页面查看到的详细信息一致，如下：

至此，可自行进行创建虚机等操作，所有实例将使用lvm存储池。