通过 Rally 进行 OpenStack Tempest 测试

Rally 基本介绍

Rally 是OpenStack社区推出开源测试工具,可用于对OpenStack各个组件进行性能测试。通过使用Rally组件,用户可完成OpenStack云计算平台的安装部署、功能验证、大规模负载测试(性能测试)、输出测试报告等一系列动作。
Rally 的概况和结构如下图所示:
image
Rally 主要包括三大部分:

  • Deploy engine:这不是一个真的部署工具,它只是一个插件形式的东西,它可以和其他部署工具(比如 DevStack,Fuel,Anvil 等)一起工作来简化和统一部署流程。
  • Verification:使用tempest验证已经部署的openstack云环境的功能。
  • Benchmark engine:性能测试

Tempest 基本介绍

Tempest 是一个旨在为云计算平台 OpenStack 提供集成功能测试的开源项目,包含了 Openstack 基本组件(nova, keystone, glance, neutron, cinder 等)的 API 测试用例与场景。它是基于 unittest2 和 nose 建立的,灵活且易于扩展及维护,使得 OpenStack 相关测试效率得到大幅度提升。

安装 Rally

  1. 安装依赖包
# yum install python-pip lsb_release gcc gmp-devel libffi-devel libxml2-devel libxslt-devel openssl-devel postgresql-devel python-devel redhat-rpm-config
  1. 安装 rally 最简单的方法就是使用下面的安装脚本
wget -q -O- https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash
# or using curl:
curl https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash

如果用普通用户执行脚本,Rally 会在 ==~/rally/== 下创建一个新的虚拟环境并安装在这里,使用 sqlite作为数据库后端。如果使用 root 用户执行脚本,Rally 会安装在系统路径,更多的安装选项,可以参考安装页面

Rally 配置

  1. 创建 openstack 环境变量文件,加入以下内容,注意修改用户名、密码、认证地址、region_name 等内容。
# vim admin-openrc

unset OS_SERVICE_TOKEN
    export OS_USERNAME=admin
    export OS_PASSWORD=admin
    export OS_AUTH_URL=http://192.168.3.222:5000/v3

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
  1. 加载环境变量
# . admin-openrc
  1. 注册一个 Openstack deployment,注册成功后,将会默认使用这个 deployment,同时在主目录下会有一个新的目录出现:.rally。
# rally deployment create --fromenv --name=openstack

2017-07-31 15:44:12.509 20293 INFO rally.deployment.engines.existing [-] Save deployment 'openstack' (uuid=3403b234-76ae-4afb-9d96-49ef2d872069) with 'openstack' platform.
+--------------------------------------+---------------------+-----------+------------------+--------+
| uuid                                 | created_at          | name      | status           | active |
+--------------------------------------+---------------------+-----------+------------------+--------+
| 3403b234-76ae-4afb-9d96-49ef2d872069 | 2017-07-31T07:44:12 | openstack | deploy->finished |        |
+--------------------------------------+---------------------+-----------+------------------+--------+
Using deployment: 3403b234-76ae-4afb-9d96-49ef2d872069
~/.rally/openrc was updated

HINTS:

* To use standard OpenStack clients, set up your env by running:
    source ~/.rally/openrc
  OpenStack clients are now configured, e.g run:
    openstack image list
  1. 检查一下刚注册的 deployment 是否存在。
# rally deployment list
+--------------------------------------+---------------------+-----------+------------------+--------+
| uuid                                 | created_at          | name      | status           | active |
+--------------------------------------+---------------------+-----------+------------------+--------+
| 3403b234-76ae-4afb-9d96-49ef2d872069 | 2017-07-31T07:44:12 | openstack | deploy->finished | *      |
+--------------------------------------+---------------------+-----------+------------------+--------+
  1. 检查 deployment 是否可行
# rally deployment check

--------------------------------------------------------------------------------
Platform openstack:
--------------------------------------------------------------------------------

Available services:
+-------------+----------------+-----------+
| Service     | Service Type   | Status    |
+-------------+----------------+-----------+
| __unknown__ | alarming       | Available |
| __unknown__ | compute_legacy | Available |
| __unknown__ | event          | Available |
| __unknown__ | placement      | Available |
| __unknown__ | volumev2       | Available |
| __unknown__ | volumev3       | Available |
| cinder      | volume         | Available |
| glance      | image          | Available |
| gnocchi     | metric         | Available |
| keystone    | identity       | Available |
| neutron     | network        | Available |
| nova        | compute        | Available |
+-------------+----------------+-----------+

关于 service 显示 unknown 的问题可以参看以下文章:
rally deployment check is giving unknown under services
OpenStack Rally 性能测试

通过 Tempest verifier 验证云环境

  1. 创建 Tempest verifier
#  rally verify create-verifier --type tempest --name tempest-verifier
  1. 验证是否安装完成
# rally verify list-verifiers
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
| UUID                                 | Name             | Type    | Namespace | Created at          | Updated at          | Status    | Version | System-wide | Active |
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
| 4f4db99c-3930-442e-b592-bed5f428814e | tempest-verifier | tempest | openstack | 2017-07-31T05:24:09 | 2017-07-31T05:25:28 | installed | master  | False       | ✔      |
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
  1. 配置 Tempest verifier
    执行以下命令为当前部署配置 Tempest verifier
# rally verify configure-verifier

2017-07-31 15:56:33.940 20338 INFO rally.api [-] Configuring verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069).
2017-07-31 15:56:35.945 20338 INFO rally.api [-] Verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e) has been successfully configured for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!

查看配置信息

# rally verify configure-verifier --show

[DEFAULT]
debug = True
use_stderr = False
log_file = 

[auth]
use_dynamic_credentials = True
admin_username = admin
admin_password = admin
admin_project_name = admin
admin_domain_name = default
…………
[service_available]
cinder = True
glance = True
heat = False
ironic = False
neutron = True
nova = True
sahara = False
swift = False

[validation]
run_validation = True
image_ssh_user = cirros
connect_method = floating

[volume-feature-enabled]
bootable = True

开始验证

  1. 执行以下命令开始验证
# rally verify start
2017-07-31 16:02:14.679 20417 INFO rally.api [-] Starting verification (UUID=ddca5b4b-03a9-49e4-8c91-1d53943ad10b) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:02:25.381 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent ... success [0.752s]
2017-07-31 16:02:25.972 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent ... success [0.588s]
2017-07-31 16:02:26.458 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents ... success [0.486s]
2017-07-31 16:02:27.335 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter ... success [0.877s]
2017-07-31 16:02:27.975 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent ... success [0.639s]
2017-07-31 16:02:37.491 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list ... success [0.498s]
2017-07-31 16:02:38.042 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list_detail ... success [0.551s]

默认情况下,以上命令会为当前部署执行完整的 tempest测试。

  1. 可以使用 --pattern 选项只执行部分tempest测试
# rally verify start --pattern set=compute
2017-07-31 16:07:12.163 20459 INFO rally.api [-] Starting verification (UUID=4e36a2fb-5780-4db0-86bf-fe2b0ab92bf2) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:07:17.189 20459 INFO tempest-verifier [-] {1} tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest ... skip: The microversion range[2.37 - latest] of this test is out of the configuration range[None - None].
2017-07-31 16:07:21.786 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent ... success [0.836s]
2017-07-31 16:07:23.170 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent ... success [1.382s]
2017-07-31 16:07:25.590 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents ... success [2.422s]
2017-07-31 16:07:27.447 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter ... success [1.856s]
2017-07-31 16:07:28.135 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent ... success [0.686s]

如 --pattern set=compute 选项,只会执行compute相关的测试。当前可供选择的测试内容有 full, smoke, compute, identity, image, network, object_storage, orchestration, volume, scenario

  1. 用户可以使用正则表达式运行某些的测试集
# rally verify start --pattern tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON
2017-07-31 16:25:55.659 20502 INFO rally.api [-] Starting verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:26:03.792 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram ... success [0.683s]
2017-07-31 16:26:04.703 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_verify_entry_in_list_details ... success [0.910s]
2017-07-31 16:26:05.478 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_int_id ... success [0.774s]
2017-07-31 16:26:06.230 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_none_id ... success [0.750s]
2017-07-31 16:26:06.906 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_uuid_id ... success [0.677s]
2017-07-31 16:26:08.224 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_list_flavor_without_extra_data ... success [1.317s]
2017-07-31 16:26:09.264 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_server_with_non_public_flavor ... success [1.038s]
2017-07-31 16:26:13.144 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_is_public_string_variations ... success [3.873s]
2017-07-31 16:26:14.477 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_non_public_flavor ... success [1.336s]
2017-07-31 16:26:15.548 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_public_flavor_with_other_user ... success [1.067s]
2017-07-31 16:26:22.767 20502 INFO rally.api [-] Verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) has been successfully finished for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!

======
Totals
======

Ran: 10 tests in 14.768 sec.
 - Success: 10
 - Skipped: 0
 - Expected failures: 0
 - Unexpected success: 0
 - Failures: 0

Using verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) as the default verification for the future operations.

只会运行compute中和flavor相关的测试

  1. 以这种方式,可以从某个目录或类运行测试,甚至可以运行单个测试
# rally verify start --pattern tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram
2017-07-31 16:30:28.586 20533 INFO rally.api [-] Starting verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:30:36.781 20533 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram ... success [0.772s]
2017-07-31 16:30:42.064 20533 INFO rally.api [-] Verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) has been successfully finished for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!

======
Totals
======

Ran: 1 tests in 2.734 sec.
 - Success: 1
 - Skipped: 0
 - Expected failures: 0
 - Unexpected success: 0
 - Failures: 0

Using verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) as the default verification for the future operations.

查看结果

我们可以报错结果为html、json等格式,一般保存为 html 格式,可以在浏览器中直观的查看

  1. 获得 verify id
# rally verify list
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+
| UUID                                 | Tags | Verifier name    | Deployment name | Started at          | Finished at         | Duration | Status   |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+
| db55c49c-9316-4353-94db-e0c777831157 | -    | tempest-verifier | openstack       | 2017-07-31T08:37:28 | 2017-07-31T14:40:46 | 6:03:18  | failed   |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+

如果进行了多次测试会有多条劫夺,可以根据时间来区分,每次测试结束的时候都会提示本次测试的UUID

  1. 导出为 html 文件
# rally verify report --uuid db55c49c-9316-4353-94db-e0c777831157 --type html --to export-name.html
  1. 在浏览器中查看
    如图,可以显示所有的测试用例,错误的用例也会给出详细的错误信息
    image

使用 docker 快速部署 ceph

系统环境

  • 至少需要三台虚拟机或者物理机,这里使用虚拟机
  • 每台虚拟机至少需要两块硬盘(一块系统盘,一块OSD),本例中有三块硬盘
  1. 部署流程(博客使用的markdown解析器不支持流程图使用图片代替)
    liuchengtu.png

  2. 主机规划
    biaoge.png

安装 docker

登录 https://cr.console.aliyun.com/#/accelerator 获取自己的阿里云 docker 加速地址

  1. 安装升级 docker 客户端
# curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh -
  1. 使用 docker 加速器
    可以通过修改 daemon 配置文件 /etc/docker/daemon.json 来使用加速器,注意修改使用自己的加速地址
# mkdir -p /etc/docker
# tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://******.mirror.aliyuncs.com"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker

启动 MON

  1. 下载 ceph daemon 镜像
# docker pull ceph/daemon
  1. 启动第一个 mon
    在 node1 上启动第一个 mon,注意修改 MON_IP
# docker run -d \
        --net=host \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        -e MON_IP=192.168.3.123 \
        -e CEPH_PUBLIC_NETWORK=192.168.3.0/24 \
        ceph/daemon mon

查看容器

# docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED              STATUS              PORTS               NAMES
b79a02c40296        ceph/daemon         "/entrypoint.sh mon"   About a minute ago   Up About a minute                       sad_shannon

查看集群状态

# docker exec b79a02 ceph -s
    cluster 96ae62d2-2249-4173-9dee-3a7215cba51c
     health HEALTH_ERR
            no osds
     monmap e2: 1 mons at {node01=192.168.3.123:6789/0}
            election epoch 4, quorum 0 node01
        mgr no daemons active 
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating
  1. 复制配置文件
    将 node1 上的配置文件复制到 node02 和 node03,复制的路径包含/etc/ceph和/var/lib/ceph/bootstrap-*下的所有内容。
# ssh root@node2 mkdir -p /var/lib/ceph
# scp -r /etc/ceph root@node2:/etc
# scp -r /var/lib/ceph/bootstrap* root@node2:/var/lib/ceph

# ssh root@node3 mkdir -p /var/lib/ceph
# scp -r /etc/ceph root@node3:/etc
# scp -r /var/lib/ceph/bootstrap* root@node3:/var/lib/ceph
  1. 启动第二个和第三个 mon
    在 node02 上执行以下命令启动 mon,注意修改 MON_IP
# docker run -d \
        --net=host \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        -e MON_IP=192.168.3.124 \
        -e CEPH_PUBLIC_NETWORK=192.168.3.0/24 \
        ceph/daemon mon

在 node03 上执行以下命令启动 mon,注意修改 MON_IP

# docker run -d \
        --net=host \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        -e MON_IP=192.168.3.125 \
        -e CEPH_PUBLIC_NETWORK=192.168.3.0/24 \
        ceph/daemon mon

查看在 node01 上集群状态

# docker exec b79a02 ceph -s
    cluster 96ae62d2-2249-4173-9dee-3a7215cba51c
     health HEALTH_ERR
            64 pgs are stuck inactive for more than 300 seconds
            64 pgs stuck inactive
            64 pgs stuck unclean
            no osds
     monmap e4: 3 mons at {node01=192.168.3.123:6789/0,node02=192.168.3.124:6789/0,node03=192.168.3.125:6789/0}
            election epoch 12, quorum 0,1,2 node01,node02,node03
        mgr no daemons active 
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating

可以看到三个 mon 已经正确启动

启动 OSD

每台虚拟机准备了两块磁盘作为 osd,分别加入到集群,注意修改磁盘

# docker run -d \
        --net=host \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        -v /dev/:/dev/ \
        --privileged=true \
        -e OSD_FORCE_ZAP=1 \
        -e OSD_DEVICE=/dev/sdb \
        ceph/daemon osd_ceph_disk
# docker run -d \
        --net=host \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        -v /dev/:/dev/ \
        --privileged=true \
        -e OSD_FORCE_ZAP=1 \
        -e OSD_DEVICE=/dev/sdc \
        ceph/daemon osd_ceph_disk

按照同样方法将 node02 和 node03 的 sdb、sdc 都加入集群

查看集群状态

# docker exec b79a ceph -s
    cluster 96ae62d2-2249-4173-9dee-3a7215cba51c
     health HEALTH_OK
     monmap e4: 3 mons at {node01=192.168.3.123:6789/0,node02=192.168.3.124:6789/0,node03=192.168.3.125:6789/0}
            election epoch 12, quorum 0,1,2 node01,node02,node03
        mgr no daemons active 
     osdmap e63: 6 osds: 6 up, 6 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v157: 64 pgs, 1 pools, 0 bytes data, 0 objects
            212 MB used, 598 GB / 599 GB avail
                  64 active+clean

可以看到 mon 和 osd 都已经正确配置,切集群状态为 HEALTH_OK

创建 MDS

使用以下命令在 node01 上启动 mds

# docker run -d \
        --net=host \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        -e CEPHFS_CREATE=1 \
        ceph/daemon mds

启动 RGW ,并且映射 80 端口

使用以下命令在 node01 上启动 rgw,并绑定 80 端口

# docker run -d \
        -p 80:80 \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/ceph/:/var/lib/ceph/ \
        ceph/daemon rgw

集群的最终状态

# docker exec b79a02 ceph -s
    cluster 96ae62d2-2249-4173-9dee-3a7215cba51c
     health HEALTH_OK
     monmap e4: 3 mons at {node01=192.168.3.123:6789/0,node02=192.168.3.124:6789/0,node03=192.168.3.125:6789/0}
            election epoch 12, quorum 0,1,2 node01,node02,node03
      fsmap e5: 1/1/1 up {0=mds-node01=up:active}
        mgr no daemons active 
     osdmap e136: 6 osds: 6 up, 6 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v1460: 136 pgs, 10 pools, 3829 bytes data, 223 objects
            254 MB used, 598 GB / 599 GB avail
                 136 active+clean

参考文章:
使用Docker部署Ceph
Demo: running Ceph in Docker containers

devstack dashboard 开启开发者选项 和 OpenStack Profiler

在ocata的版本中,引入了一个新的“openstack profiler”的面板,启用openstack profiler可以方便的看到访问horizon页面时的API调用情况。如下图所示:
image
下面介绍如何启用 openstack profiler,首先需要一个正常运行的devstack环境,启用方法如下

安装mongoDB

Horizon会将API调用过程的数据都保存到mongodb中,mongodb可以安装在本机,也可以在本机能够访问的任意一台机器上。

  1. 安装软件包

    # yum install mongodb-server mongodb -y
  2. 编辑文件 /etc/mongod.conf 并完成如下动作:
    • 配置 bind_ip 使用本机 ip 或者 0.0.0.0。
      bind_ip = 192.168.3.222
    • 默认情况下,MongoDB会在/var/lib/mongodb/journal 目录下创建几个 1 GB 大小的日志文件。如果你想将每个日志文件大小减小到128MB并且限制日志文件占用的总空间为512MB,配置 smallfiles 的值:
      smallfiles = true
  3. 启动MongoDB 并配置它随系统启动
    # systemctl enable mongod.service
    # systemctl start mongod.service

配置 horizon

  1. 复制文件
    $ cd /opt/stack/horizon
    $ cp openstack_dashboard/contrib/developer/enabled/_9001_developer.py openstack_dashboard/local/enabled/
    $ cp openstack_dashboard/contrib/developer/enabled/_9030_profiler.py openstack_dashboard/local/enabled/
    $ cp openstack_dashboard/contrib/developer/enabled/_9010_preview.py openstack_dashboard/local/enabled/
    $ cp openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py.example openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py
  2. 编辑 _9030_profiler_settings.py 文件,修改 mongoDB 相关配置
    修改 OPENSTACK_HOST 为mongoDB所在地址

    $ vim openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py
    
    OPENSTACK_PROFILER.update({
      'enabled': True,
      'keys': ['SECRET_KEY'],
      'notifier_connection_string': 'mongodb://192.168.3.222:27017',
      'receiver_connection_string': 'mongodb://192.168.3.222:27017'
    })
  3. 重启 horizon,重新登录 dashboard ,会发现右上角有一个 Profile 下拉菜单,如下图:
    image
    如果要获取当前页面的API调用数据,点击 Profile Current Page 会重新刷新页面,加载完成后,到 Developer 下面的 OpenStack Profiler 页面就会看到页面加载过程的详细数据。

参考文章:
孔令贤-OpenStack Horizon Profiling
OpenStack Installation Guide for Red Hat Enterprise Linux and CentOS

ssh 无密码访问的问题

ssh 无密码登录失败

虚拟机 resize 需要配置计算节点之间 nova 用户无密码访问,但是在配置过程中有一台始终不能用密钥登录,对比了正常可以无密码登录的日志如下。

# 正常登录
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/nova/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
# 异常报错
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/nova/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /var/lib/nova/.ssh/id_dsa
debug3: no such identity: /var/lib/nova/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /var/lib/nova/.ssh/id_ecdsa
debug3: no such identity: /var/lib/nova/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /var/lib/nova/.ssh/id_ed25519
debug3: no such identity: /var/lib/nova/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

分析问题

  1. 找个一个类似报错的 CentOS SSH公钥登录问题 ,文中是由于seliunx导致的,我查看了本地的selinux发现已经关闭,不适用我的情况

  2. 使用 journalctl _COMM=sshd 命令查看日志,发现如下权限问题
May 10 17:11:11 compute01 sshd[26498]: pam_systemd(sshd:session): Failed to release session: Interrupted system call
May 10 17:11:11 compute01 sshd[26498]: pam_unix(sshd:session): session closed for user root
May 10 17:12:28 compute01 sshd[2297]: Authentication refused: bad ownership or modes for directory /var/lib/nova
May 10 17:13:09 compute01 sshd[2297]: Connection closed by 192.168.101.105 [preauth]
May 10 17:13:33 compute01 sshd[4103]: Authentication refused: bad ownership or modes for directory /var/lib/nova
May 10 17:25:21 compute01 sshd[23157]: Authentication refused: bad ownership or modes for directory /var/lib/nova
May 10 17:25:25 compute01 sshd[23157]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=compute02  user=nova
  1. 对比无问题主机的 /var/lib/nova 权限
正常主机
drwxr-xr-x   8 nova    nova     118 May 10 16:59 nova
异常主机
drwxrwxrwx. 11 nova           nova            4096 May 10 17:07 nova
  1. 解决办法
    修改 /var/lib/nova 目录权限为 755 后,可以正常无密码登录
# chmod -R 755 /var/lib/nova/

openstack HA模式下控制台无法访问的问题

控制台无法访问,多次刷新才能访问,nova有如下报错

2017-02-09 17:09:51.311 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] "GET /websockify HTTP/1.1" 101 -
2017-02-09 17:09:51.312 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] 192.168.170.41: Plain non-SSL (ws://) WebSocket connection
2017-02-09 17:09:51.313 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] 192.168.170.41: Version hybi-13, base64: 'False'
2017-02-09 17:09:51.313 57467 INFO nova.console.websocketproxy [-] 192.168.170.41 - - [09/Feb/2017 17:09:51] 192.168.170.41: Path: '/websockify'
2017-02-09 17:09:51.382 57467 INFO nova.console.websocketproxy [req-f51929d9-8c9b-4df0-abeb-247ce6ef5d65 - - - - -] handler exception: The token '1dfc9af9-8a49-44b3-a955-5196197bc8f7' is invalid or has expired

原因分析

When running a multi node environment with HA between two or more controller nodes(or controller plane service nodes), nova consoleauthservice must be configured with memcached.  
If not, no more than one consoleauth service can berunning in active state, since it need to save the state of the sessions. Whenmemcached is not used, you can check that can connect to the vnc console only afew times when you refresh the page. If that occurs means that the connectionis handled by the consoleauth service that currently is issuing sessions.    
To solve your issue, configure memcached as backend tonova-consoleauth service.  
To solve your issue add this line to nova.conf:  
memcached_servers = 192.168.100.2:11211,192.168.100.3:11211  
This should work to solve your issue.  

解决

M版在增加memcached_servers选项

# vim /etc/nova/nova.conf

[DEFAULT]
# "memcached_servers" opt is deprecated in Mitaka. In Newton release oslo.cache
# config options should be used as this option will be removed. Please add a
# [cache] group in your nova.conf file and add "enable" and "memcache_servers"
# option in this section. (list value)
memcached_servers=controller01:11211,controller02:11211,controller03:11211

如果是N版的话,memcached_servers已经废弃,需要按照如下修改

[cache]
enabled=true
backend=oslo_cache.memcache_pool
memcache_servers=controller01:11211,controller02:11211,controller03:11211