通过 Rally 进行 OpenStack Tempest 测试
Rally 基本介绍
Rally 是OpenStack社区推出开源测试工具，可用于对OpenStack各个组件进行性能测试。通过使用Rally组件，用户可完成OpenStack云计算平台的安装部署、功能验证、大规模负载测试(性能测试)、输出测试报告等一系列动作。
Rally 的概况和结构如下图所示：
Rally 主要包括三大部分：
- Deploy engine：这不是一个真的部署工具，它只是一个插件形式的东西，它可以和其他部署工具（比如 DevStack，Fuel，Anvil 等）一起工作来简化和统一部署流程。
- Verification：使用tempest验证已经部署的openstack云环境的功能。
- Benchmark engine：性能测试
Tempest 基本介绍
Tempest 是一个旨在为云计算平台 OpenStack 提供集成功能测试的开源项目，包含了 Openstack 基本组件（nova, keystone, glance, neutron, cinder 等）的 API 测试用例与场景。它是基于 unittest2 和 nose 建立的，灵活且易于扩展及维护，使得 OpenStack 相关测试效率得到大幅度提升。
安装 Rally
- 安装依赖包
# yum install python-pip lsb_release gcc gmp-devel libffi-devel libxml2-devel libxslt-devel openssl-devel postgresql-devel python-devel redhat-rpm-config- 安装 rally 最简单的方法就是使用下面的安装脚本
wget -q -O- https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash
# or using curl:
curl https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash如果用普通用户执行脚本，Rally 会在 ==~/rally/== 下创建一个新的虚拟环境并安装在这里，使用 sqlite作为数据库后端。如果使用 root 用户执行脚本，Rally 会安装在系统路径，更多的安装选项，可以参考安装页面。
Rally 配置
- 创建 openstack 环境变量文件，加入以下内容，注意修改用户名、密码、认证地址、region_name 等内容。
# vim admin-openrc
unset OS_SERVICE_TOKEN
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://192.168.3.222:5000/v3
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne- 加载环境变量
# . admin-openrc- 注册一个 Openstack deployment，注册成功后，将会默认使用这个 deployment，同时在主目录下会有一个新的目录出现：.rally。
# rally deployment create --fromenv --name=openstack
2017-07-31 15:44:12.509 20293 INFO rally.deployment.engines.existing [-] Save deployment 'openstack' (uuid=3403b234-76ae-4afb-9d96-49ef2d872069) with 'openstack' platform.
+--------------------------------------+---------------------+-----------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+---------------------+-----------+------------------+--------+
| 3403b234-76ae-4afb-9d96-49ef2d872069 | 2017-07-31T07:44:12 | openstack | deploy->finished | |
+--------------------------------------+---------------------+-----------+------------------+--------+
Using deployment: 3403b234-76ae-4afb-9d96-49ef2d872069
~/.rally/openrc was updated
HINTS:
* To use standard OpenStack clients, set up your env by running:
source ~/.rally/openrc
OpenStack clients are now configured, e.g run:
openstack image list- 检查一下刚注册的 deployment 是否存在。
# rally deployment list
+--------------------------------------+---------------------+-----------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+---------------------+-----------+------------------+--------+
| 3403b234-76ae-4afb-9d96-49ef2d872069 | 2017-07-31T07:44:12 | openstack | deploy->finished | * |
+--------------------------------------+---------------------+-----------+------------------+--------+- 检查 deployment 是否可行
# rally deployment check
--------------------------------------------------------------------------------
Platform openstack:
--------------------------------------------------------------------------------
Available services:
+-------------+----------------+-----------+
| Service | Service Type | Status |
+-------------+----------------+-----------+
| __unknown__ | alarming | Available |
| __unknown__ | compute_legacy | Available |
| __unknown__ | event | Available |
| __unknown__ | placement | Available |
| __unknown__ | volumev2 | Available |
| __unknown__ | volumev3 | Available |
| cinder | volume | Available |
| glance | image | Available |
| gnocchi | metric | Available |
| keystone | identity | Available |
| neutron | network | Available |
| nova | compute | Available |
+-------------+----------------+-----------+关于 service 显示 unknown 的问题可以参看以下文章：
rally deployment check is giving unknown under services
OpenStack Rally 性能测试
通过 Tempest verifier 验证云环境
- 创建 Tempest verifier
# rally verify create-verifier --type tempest --name tempest-verifier- 验证是否安装完成
# rally verify list-verifiers
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
| UUID | Name | Type | Namespace | Created at | Updated at | Status | Version | System-wide | Active |
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+
| 4f4db99c-3930-442e-b592-bed5f428814e | tempest-verifier | tempest | openstack | 2017-07-31T05:24:09 | 2017-07-31T05:25:28 | installed | master | False | ✔ |
+--------------------------------------+------------------+---------+-----------+---------------------+---------------------+-----------+---------+-------------+--------+- 配置 Tempest verifier
执行以下命令为当前部署配置 Tempest verifier
# rally verify configure-verifier
2017-07-31 15:56:33.940 20338 INFO rally.api [-] Configuring verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069).
2017-07-31 15:56:35.945 20338 INFO rally.api [-] Verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e) has been successfully configured for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!查看配置信息
# rally verify configure-verifier --show
[DEFAULT]
debug = True
use_stderr = False
log_file =
[auth]
use_dynamic_credentials = True
admin_username = admin
admin_password = admin
admin_project_name = admin
admin_domain_name = default
…………
[service_available]
cinder = True
glance = True
heat = False
ironic = False
neutron = True
nova = True
sahara = False
swift = False
[validation]
run_validation = True
image_ssh_user = cirros
connect_method = floating
[volume-feature-enabled]
bootable = True开始验证
- 执行以下命令开始验证
# rally verify start
2017-07-31 16:02:14.679 20417 INFO rally.api [-] Starting verification (UUID=ddca5b4b-03a9-49e4-8c91-1d53943ad10b) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:02:25.381 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent ... success [0.752s]
2017-07-31 16:02:25.972 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent ... success [0.588s]
2017-07-31 16:02:26.458 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents ... success [0.486s]
2017-07-31 16:02:27.335 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter ... success [0.877s]
2017-07-31 16:02:27.975 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent ... success [0.639s]
2017-07-31 16:02:37.491 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list ... success [0.498s]
2017-07-31 16:02:38.042 20417 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list_detail ... success [0.551s]默认情况下，以上命令会为当前部署执行完整的 tempest测试。
- 可以使用 --pattern 选项只执行部分tempest测试
# rally verify start --pattern set=compute
2017-07-31 16:07:12.163 20459 INFO rally.api [-] Starting verification (UUID=4e36a2fb-5780-4db0-86bf-fe2b0ab92bf2) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:07:17.189 20459 INFO tempest-verifier [-] {1} tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest ... skip: The microversion range[2.37 - latest] of this test is out of the configuration range[None - None].
2017-07-31 16:07:21.786 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent ... success [0.836s]
2017-07-31 16:07:23.170 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent ... success [1.382s]
2017-07-31 16:07:25.590 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents ... success [2.422s]
2017-07-31 16:07:27.447 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter ... success [1.856s]
2017-07-31 16:07:28.135 20459 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent ... success [0.686s]如 --pattern set=compute 选项，只会执行compute相关的测试。当前可供选择的测试内容有 full, smoke, compute, identity, image, network, object_storage, orchestration, volume, scenario
- 用户可以使用正则表达式运行某些的测试集
# rally verify start --pattern tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON
2017-07-31 16:25:55.659 20502 INFO rally.api [-] Starting verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:26:03.792 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram ... success [0.683s]
2017-07-31 16:26:04.703 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_verify_entry_in_list_details ... success [0.910s]
2017-07-31 16:26:05.478 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_int_id ... success [0.774s]
2017-07-31 16:26:06.230 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_none_id ... success [0.750s]
2017-07-31 16:26:06.906 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_uuid_id ... success [0.677s]
2017-07-31 16:26:08.224 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_list_flavor_without_extra_data ... success [1.317s]
2017-07-31 16:26:09.264 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_server_with_non_public_flavor ... success [1.038s]
2017-07-31 16:26:13.144 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_is_public_string_variations ... success [3.873s]
2017-07-31 16:26:14.477 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_non_public_flavor ... success [1.336s]
2017-07-31 16:26:15.548 20502 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_public_flavor_with_other_user ... success [1.067s]
2017-07-31 16:26:22.767 20502 INFO rally.api [-] Verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) has been successfully finished for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!
======
Totals
======
Ran: 10 tests in 14.768 sec.
- Success: 10
- Skipped: 0
- Expected failures: 0
- Unexpected success: 0
- Failures: 0
Using verification (UUID=84fce1ca-304b-4663-bba5-185f24d013a1) as the default verification for the future operations.只会运行compute中和flavor相关的测试
- 以这种方式，可以从某个目录或类运行测试，甚至可以运行单个测试
# rally verify start --pattern tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram
2017-07-31 16:30:28.586 20533 INFO rally.api [-] Starting verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069) by verifier 'tempest-verifier' (UUID=4f4db99c-3930-442e-b592-bed5f428814e).
2017-07-31 16:30:36.781 20533 INFO tempest-verifier [-] {0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram ... success [0.772s]
2017-07-31 16:30:42.064 20533 INFO rally.api [-] Verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) has been successfully finished for deployment 'openstack' (UUID=3403b234-76ae-4afb-9d96-49ef2d872069)!
======
Totals
======
Ran: 1 tests in 2.734 sec.
- Success: 1
- Skipped: 0
- Expected failures: 0
- Unexpected success: 0
- Failures: 0
Using verification (UUID=181d37bd-d9a7-46fa-9311-ffe09d81e84c) as the default verification for the future operations.查看结果
我们可以报错结果为html、json等格式，一般保存为 html 格式，可以在浏览器中直观的查看
- 获得 verify id
# rally verify list
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+
| UUID | Tags | Verifier name | Deployment name | Started at | Finished at | Duration | Status |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+
| db55c49c-9316-4353-94db-e0c777831157 | - | tempest-verifier | openstack | 2017-07-31T08:37:28 | 2017-07-31T14:40:46 | 6:03:18 | failed |
+--------------------------------------+------+------------------+-----------------+---------------------+---------------------+----------+----------+如果进行了多次测试会有多条劫夺，可以根据时间来区分，每次测试结束的时候都会提示本次测试的UUID
- 导出为 html 文件
# rally verify report --uuid db55c49c-9316-4353-94db-e0c777831157 --type html --to export-name.html- 在浏览器中查看
如图，可以显示所有的测试用例，错误的用例也会给出详细的错误信息
