Tags: srinisakh/rally
Tags
rally 0.4.0 release meta:version: 0.4.0 meta:series: independent meta:release-type: release meta:announce: [email protected] meta:pypi: yes meta:first: no
============ Rally v0.3.3 ============ Information ----------- +------------------+-----------------------+ | Commits | **20** | +------------------+-----------------------+ | Dev cycle | **10 days** | +------------------+-----------------------+ | Release date | **3/24/2016** | +------------------+-----------------------+ Details ------- A half of patches relate to Cleanup. We have once again proved that ideal stuff can be improved. :) Specs & Feature Requests ~~~~~~~~~~~~~~~~~~~~~~~~ * `[Spec][Introduced] Improve atomic actions format`__ __ https://github.com/openstack/rally/blob/master/doc/specs/in-progress/improve_atomic_actions_format.rst Plugins ~~~~~~~ * **Cleanups**: * Use proper attribute to get heat stack name * Always assign a name to created images. This is necessary for name-based cleanup. If a name is not specified, one will be generated automatically. * Improve filtering glance images in case of V2 API * Delete only images created by images context Since the images context allows creating images with arbitrary names, name-based cleanup won't work for it, so we have to delete the exact list of images that it created instead. * New config option to set cleanup threads Allow the user to change the number of cleanup threads via the rally config. When scaling out to thousands of instances, the cleanup can take forever with the static 20 threads. * Add inexact matching to name_matches_object This will support places where we create resources with names that start with a given name pattern, but include some additional identifier afterwards. For instance, when bulk creating instances, Nova appends a UUID to each instance name. * **Scenarios**: * Add sample of template for testing for testing heat caching. * Introduced new scenario `Dummy.dummy_random_action`__. It is suitable for demonstration of upcoming trends report. __ http://rally.readthedocs.org/en/latest/plugin/plugin_reference.html#dummy-dummy-random-action-scenario * **Contexts**: `api_versions`__ context was extended to support switch between Keystone V2 and V3 API versions. Now it is possible to use one Rally deployment to check both Keystone APIs. __ http://rally.readthedocs.org/en/latest/plugin/plugin_reference.html#api-versions-context * **Newcomer in the family**: All ResourceType classes are pluggable now and it is much easier to use and extend them. .. warning:: Decorator ``rally.task.types.set`` is deprecated now in favor of ``rally.task.types.convert``. Bug fixes ~~~~~~~~~ * #1536172: rally deployment destroy failed with traceback for failed deployments. At current moment it is impossible to delete deployment if for some reason deployment engine plugin cannot be found, because exception will be thrown. Documentation ~~~~~~~~~~~~~ * Remove extra link in `All release notes` Previously, two links for latest release were presented. * Update release notes for 0.3.2 * Fixed indents for warning messages * Fixed all references Thanks ~~~~~~ To Everybody!
============
Rally v0.3.2
============
Information
-----------
+------------------+-----------------------+
| Commits | **55** |
+------------------+-----------------------+
| Dev cycle | **25 days** |
+------------------+-----------------------+
| Release date | **3/14/2016** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
CLI changes
~~~~~~~~~~~
* .. warning:: [Modified] Option '--tempest-config' for 'rally verify
reinstall' command was deprecated for removal.
* .. warning:: [Removed] Option `--system-wide-install` was removed from
`rally verify` commands in favor of `--system-wide` option.
* .. warning:: [Modified] Step of installation of Tempest during execution of
the `rally verify start` command was deprecated and will be removed in the
future. Please use `rally verify install` instead.
* Rework commands.task.TaskCommands.detailed. Now output of the command
contains the same results as in HTML report.
Rally Verify
~~~~~~~~~~~~
* Re-run failed Tempest tests
Add the ability to re-run the tempest tests that failed in the last test
execution. Sometimes Tempest tests fail due to a special temporary condition
in the environment, in such cases it is very useful to be able to re-execute
those tests.
Running the following command will re-run all the test that failed during
the last test execution regardless of what test suite was run.
.. code-block:: bash
rally verify start --failing
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* [Spec][Introduced] Refactoring scenario utils
`https://github.com/openstack/rally/blob/master/doc/specs/in-progress
/refactor_scenario_utils.rst`_
* [Spec] Deployment unification
Plugins
~~~~~~~
* **Scenarios**:
* [updated] Fix flavor for cloudera manager
Cloudera manager need master-node flavor
* [added] Expand Nova API benchmark in Rally
Add support for listing nova hosts, agents, availability-zones
and aggregates.
* [updated] Make sure VolumeGenerator uses the api version info while cleanup
* Designate V2 - Add recordset scenarios
Add create_and_(list|delete)_recordset scenarios
Remove the test also that checks the allowed methods, this is in order for
us to be able to have a private method _walk_pages that will do fetching of
pages for us vs attempting to fetch 1 giant list at once.
* unify *_kwargs name in scenarios
When running a scenario, `kwargs` is used as default key-word arguments.
But in some scenarios, there are more and one services being called, and
we use xxx_kwargs for this case.
However, some xxx_kwargs are not unifed for same usage[0]. Unifying
these could avoid midleading for end users. Another improvement is to
add xxx_kwargs with empty settings for scenario config files.
[0] http://paste.openstack.org/show/489505/
* .. warning:: Deprecated arguments 'script' and 'interpreter' were removed
in favor of 'command' argument.
VM task scenarios executes a script with a interpreter provided through a
formatted argument called 'command' which expects the remote_path or
local_path of the script and optionally an interpreter with which the
script has to be executed.
Miscellaneous
~~~~~~~~~~~~~
* Avoid using `len(x)` to check if x is empty
This cases are using `len()` to check if collection has items. As
collections have a boolean representation too, directly check for true /
false. And fix the wrong mock in its unit test.
* Fix install_rally.sh to get it to work on MacOSX
On MacOSX, `mktemp` requires being passed a template. This change modifies
the calls to `mktemp` to explicitly pass a template so that the code works
on both MacOSX and linux.
* Use new-style Python classes
There are some classes in the code that didn't inherited from
nothing and this is an old-style classes. A "New Class" is the
recommended way to create a class in modern Python.A "New Class"
should always inherit from `object` or another new-style class.
Hacking rule added as well.
* Make Rally cope with unversioned keystone URL
With the change, the client version that's returned is now determined by
the keystoneclient library itself based on whether you supply a URL with a
version in it or not.
* Fix rally-mos job to work with mos-8.0
Also remove hardcoded values for some other jobs.
* Add name() to ResourceManager
This will allow us to perform cleanup based on the name.
* Add task_id argument to name_matches_object
This will be used to ensure that we are only deleting resources for a
particular Rally task.
* Extend api.Task.get_detailed
Extend api.Task.get_detailed with ability to return task data as dict with
extended results.
Bug fixes
~~~~~~~~~
**The most critical fixed bugs are**:
* #1547624: Wrong configuration for baremetal(ironic) tempest tests
* #1536800: openrc values are not quoted
The openrc file created after rally deployment --fromenv did not quote the
values for environment variables that will be exported.
* #1509027: Heat delete_stack never exits if status is DELETE_FAILED
* #1540545: Refactored atomic action in authenticate scenario
* #1469897: Incompatible with Keystone v3 argument in service create scenario
* #1550262: Different results in ``rally task detailed``, ``rally task report``
and ``rally task status`` commands.
* #1553024: Backward incompatible change in neutronclient(release 4.1.0) broke
Tempest config generation to support latest neutronclient.
Documentation
~~~~~~~~~~~~~
* Add documentation for DB migration
Changes:
* Add descriptive docstrings for plugins based on OutputChart
* Register these plugins in doc/ext/plugin_reference.py
so plugin/plugin_reference.htmlwill have a documentation
chapter based on added docstrings
* Documentation tox fix
Added information about debugging unit test with tox. Replace 3 references
to py26 with py34 to reflect current rally tox configuration.
* Change structure of rally plugin and plugin references page
* Update the scenario development, runner and context sections
* The design of 'Rally Plugins Reference' page was improved (see
`http://docs.openstack.org/developer/rally/plugin/plugin_reference.html`_
with new design)
* New page was added - CLI references
`http://docs.openstack.org/developer/rally/cli/cli_reference.html`_
Thanks
~~~~~~
To Everybody!
Rally v0.3.1 Information ----------- +------------------+-----------------------+ | Commits | **9** | +------------------+-----------------------+ | Bug fixes | **6** | +------------------+-----------------------+ | Dev cycle | **2 days** | +------------------+-----------------------+ | Release date | **2/18/2016** | +------------------+-----------------------+ Details ------- This release is more about bug-fixes than features. .. warning:: Please, update 0.3.0 to latest one. Features ~~~~~~~~ * Pass api_versions info to glance images context * [Verify] Don't create new flavor when flavor already exists Bug fixes ~~~~~~~~~ **6 bugs were fixed, the most critical are**: * #1545889: Existing deployment with given endpoint doesn't work anymore * #1547092: Insecure doesn't work with Rally 0.3.0 * #1547083: Rally Cleanup failed with api_versions context in 0.3.0 release * #1544839: Job gate-rally-dsvm-zaqar-zaqar fails since the recent Rally patch * #1544522: Non-existing "called_once_with" method of Mock library is used
Rally v0.3.0
Information
-----------
+------------------+-----------------------+
| Commits | **69** |
+------------------+-----------------------+
| Bug fixes | **7** |
+------------------+-----------------------+
| Dev cycle | **29 days** |
+------------------+-----------------------+
| Release date | **2/16/2016** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
.. warning:: In this release Rally DB schema migration is introduced. While
upgrading Rally from previous versions it is required now to run
``rally-manade db upgrade``. Please see 'Documentation' section for details.
CLI changes
~~~~~~~~~~~
* [Removed] ``rally info`` in favor of ``rally plugin *``.
It was deprecated for a long time.
* [Modified] ``rally deployment check`` now prints services, which don't have
names, since such services can be used via api_versions context.
* [Modified] ``rally verify [re]install`` option --no-tempest-venv for was
deprecated in favor of --system-wide
* [Added] ``rally-manage db revision`` displays current revision of
Rally database schema
* [Added] ``rally-manage db upgrade`` upgrades pre-existing Rally
database schema to the latest revision
* [Added] ``rally-manage db downgrade`` to downgrades existing Rally
database schema to previous revision
* [Added] ``rally task export`` exports task results to external
services (only CLI command introduced, no real service support
implemented yet, however one could write own plugins)
* [Added] ``rally verify export`` exports verification results to
external services (only CLI command introduced, no real service support
implemented yet, however one could write own plugins)
Rally Deployment
~~~~~~~~~~~~~~~~
* ``fuel`` deployment engine is removed since it was outdated and lacked
both usage and support
Rally Task
~~~~~~~~~~
Add custom labels for "Scenario Output" charts
* X-axis label can be specified to add_output() by
"axis_label" key of chart options dict.
The key is named "axis_label" but not "x_label"
because chart can be displayed as table, so we explicitly
mention "axis" in option name to make this parameter
useless for tables
* Y-axis label can be specified to add_output() by
"label" key of chart options dict
In some cases this parameter can be used for rendering
tables - it becomes column name in case if chart with
single iteration is transformed into table
* As mentiones above, if we have output chart
with single iteration, then it is transformed to table,
because chart with single value is useless
* OutputLinesChart is added, it is displayed by
NVD3 lineChart()
* Chart "description" is optional now. Description is
not shown if it is not specified explicitly
* Scenario Dummy.add_output is improved to display labels
and OutputLinesChart
* Fix: If Y-values are too long and overlaps chart box,
then JavaScript updates chart width in runtime to fit
width of chart graphs + Y values to their DOM container
Rally Certification
~~~~~~~~~~~~~~~~~~~
None.
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* [Spec][Introduced] Export task and verification results to external services
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/task_and_verification_export.rst
* [Spec][Implemented] Consistent resource names
https://github.com/openstack/rally/blob/master/doc/specs/implemented/consistent_resource_names.rst
* [Feature request][Implemented] Tempest concurrency
https://github.com/openstack/rally/blob/master/doc/feature_request/implemented/add_possibility_to_specify_concurrency_for_tempest.rst
Plugins
~~~~~~~
* **Scenarios**:
- [added] VMTasks.workload_heat
- [added] NovaFlavors.list_flavors
- [updated] Flavors for Master and Worker node groups are now
configured separately for SaharaCluster.* scenarios
* **Context**:
- [improved] sahara_cluster
Flavors for Master and Worker node groups are now
configured separately in ``sahara_cluster`` context
Miscellaneous
~~~~~~~~~~~~~
* Cinder version 2 is used by default
* Keystone API v3 compatibility improved
- Auth URL in both formats http://foo.rally:5000/v3
and http://foo.rally:5000 is supported for Keystone API v3
- Tempest configuration file is created properly according
to Keystone API version used
* ``install_rally.sh --branch`` now accepts all git tree-ish,
not just branches or tags
* VM console logs are now printed when Rally fails to connect to VM
* Add support for Rally database schema migration (see 'Documentation' section)
Bug fixes
~~~~~~~~~
**7 bugs were fixed**:
* #1540563: Rally is incompatible with liberty Neutron client
The root cause is that in Neutron Liberty client,
the _fx function doesn't take any explicit keyword parameter
but Rally is passing one (tenant_id).
* #1543414: The `rally verify start` command fails when running
a verification against Kilo OpenStack
* #1538341: Error in logic to retrieve image details in image_valid_on_flavor
Documentation
~~~~~~~~~~~~~
* Add documentation for DB migration
https://github.com/openstack/rally/blob/master/rally/common/db/sqlalchemy/migrations/README.rst
Thanks
~~~~~~
2 Everybody!
Rally v0.0.2
Information
-----------
+------------------+-----------------+
| Commits | **100** |
+------------------+-----------------+
| Bug fixes | **18** |
+------------------+-----------------+
| Dev cycle | **45 days** |
+------------------+-----------------+
| Release date | **12/Mar/2015** |
+------------------+-----------------+
Details
-------
This release contains new features, new benchmark plugins, bug fixes,
various code and API improvements.
New Features
~~~~~~~~~~~~
* rally task start **--abort-on-sla-failure**
Stopping load before things go wrong.
Load generation will be interrupted if SLA criteria stop passing.
* Rally verify command supports multiple Tempest sources now.
* python34 support
* postgres DB backend support
API changes
~~~~~~~~~~~
* [new] **rally [deployment | verify | task] use** subcommand
It should be used instead of root command **rally use**
* [new] Rally as a Lib API
To avoid code duplication between Rally as CLI tool and Rally as a Service
we decide to make Rally as a Lib as a common part between these 2 modes.
Rally as a Service will be a daemon that just maps HTTP request to Rally
as a Lib API.
* [deprecated] **rally use** CLI command
* [deprecated] Old Rally as a Lib API
Old Rally API was quite mixed up so we decide to deprecate it
Plugins
~~~~~~~
* **Benchmark Scenario Runners**:
[improved] Improved algorithm of generation load in **constant runner**
Before we used processes to generate load, now it creates pool of
processes (amount of processes is equal to CPU count) after that in
each process use threads to generate load. So now you can easily
generate load of 1k concurrent scenarios.
[improved] Unify code of **constant** and **rps** runners
[interface] Added **abort()** to runner's plugin interface
New method **abort()** is used to immediately interrupt execution.
* **Benchmark Scenarios**:
[new] DesignateBasic.create_and_delete_server
[new] DesignateBasic.create_and_list_servers
[new] DesignateBasic.list_servers
[new] MistralWorkbooks.list_workbooks
[new] MistralWorkbooks.create_workbook
[new] Quotas.neutron_update
[new] HeatStacks.create_update_delete_stack
[new] HeatStacks.list_stacks_and_resources
[new] HeatStacks.create_suspend_resume_delete_stac
[new] HeatStacks.create_check_delete_stack
[new] NeutronNetworks.create_and_delete_routers
[new] NovaKeypair.create_and_delete_keypair
[new] NovaKeypair.create_and_list_keypairs
[new] NovaKeypair.boot_and_delete_server_with_keypair
[new] NovaServers.boot_server_from_volume_and_live_migrate
[new] NovaServers.boot_server_attach_created_volume_and_live_migrate
[new] CinderVolumes.create_and_upload_volume_to_image
[fix] CinderVolumes.create_and_attach_volume
Pass optional \*\*kwargs only to create server command
[fix] GlanceImages.create_image_and_boot_instances
Pass optional \*\*kwargs only to create server command
[fix] TempestScenario.\* removed stress cleanup.
Major issue is that tempest stress cleanup cleans whole OpenStack.
This is very dangerous, so it's better to remove it and leave some
extra resources.
[improved] NovaSecGroup.boot_and_delete_server_with_secgroups
Add optional \*\*kwargs that are passed to boot server comment
* **Benchmark Context**:
[new] **stacks**
Generates passed amount of heat stacks for all tenants.
[new] **custom_image**
Prepares images for benchmarks in VMs.
To Support generating workloads in VMs by existing tools like: IPerf,
Blogbench, HPCC and others we have to have prepared images, with
already installed and configured tools.
Rally team decide to generate such images on fly from passed to avoid
requirements of having big repository with a lot of images.
This context is abstract context that allows to automate next steps:
1) runs VM with passed image (with floating ip and other stuff)
2) execute abstract method that has access to VM
3) snapshot this image
In future we are going to use this as a base for making context that
prepares images.
[improved] **allow_ssh**
Automatically disable it if security group are disabled in neutron.
[improved] **keypair**
Key pairs are stored in "users" space it means that accessing keypair
from scenario is simpler now:
self.context["user"]["keypair"]["private"]
[fix] **users**
Pass proper EndpointType for newly created users
[fix] **sahara_edp**
The Job Binaries data should be treated as a binary content
* **Benchmark SLA**:
[interface] SLA calculations is done in additive way now
Resolves scale issues, because now we don't need to have whole
array of iterations in memory to process SLA.
This is required to implement **--abort-on-sla-failure** feature
[all] SLA plugins were rewritten to implement new interface
Bug fixes
~~~~~~~~~
**18 bugs were fixed, the most critical are**:
* Fix **rally task detailed --iterations-data**
It didn't work in case of missing atomic actions. Such situation can occur
if scenario method raises exceptions
* Add user-friendly message if the task cannot be deleted
In case of trying to delete task that is not in "finished" status users get
traces instead of user-friendly message try to run it with --force key.
* Network context cleanups networks properly now
Documentation
~~~~~~~~~~~~~
* Image sizes are fixed
* New tutorial in "Step by Step" relate to **--abort-on-sla-failure**
* Various fixes
Rally v0.1.2
Information
-----------
+------------------+-----------------------+
| Commits | **208** |
+------------------+-----------------------+
| Bug fixes | **37** |
+------------------+-----------------------+
| Dev cycle | **77 days** |
+------------------+-----------------------+
| Release date | **23/December/2015** |
+------------------+-----------------------+
Details
-------
This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.
.. warning:: Release 0.1.2 is the last release with Python 2.6 support.
Deprecations
~~~~~~~~~~~~
* Class `rally.common.objects.Endpoint` was renamed to `Credentials`. Old
class is kept for backward compatibility. Please, stop using the old class
in your plugins.
.. warning:: dict key was changed too in user context from "endpoint" to "credential"
* rally.task.utils: wait_is_ready(), wait_for(), wait_for_delete() deprecated
you should use wait_for_status() instead.
Rally Verify
~~~~~~~~~~~~
* Added possibility to run Tempest tests listed in a file(--tests-file argument in ``verify start``)
* Added possibility to upload Tempest subunit stream logs into data base
* Improvements in generating Tempest config file
* Reworked subunit stream parser
* Don't install Tempest when `rally verify [gen/show]config`
* Rally team tries to simplify usage of each our component.
Now Rally verification has some kind of a context like in Tasks.
Before launching each verification, Rally checks existence of required
resources(networks, images, flavours, etc) in Tempest configuration file and
pre-creates them. Do not worry, all these resources will not be forgotten
and left, Rally will clean them after verification.
Rally Task
~~~~~~~~~~
* Add --html-static argument to ``rally task report`` which allows to
generate HTML reports that doesn't require Internet.
* Rally supports different API versions now via api_versions context:
.. code:: none
CinderVolumes.create_and_delete_volume:
-
args:
size: 1
runner:
type: "constant"
times: 2
concurrency: 2
context:
users:
tenants: 2
users_per_tenant: 2
api_versions:
cinder:
version: 2
service_name: cinderv2
* Move rally.osclients.Clients to plugin base
Rally OSclients is plugable now and it is very easy to extend OSClients for
your cloud out of Rally tree.
* Add 'merge' functionality to SLA
All SLA plugins should implement merge() method now.
In future this will be used for distributed load generation.
Where SLA results from different runners will be merged together.
* New optional_action_timer decorator
Allows to make the methods that can be both atomic_action or regular
method. Method changes behavior based on value in extra key "atomic_action"
Rally Certification
~~~~~~~~~~~~~~~~~~~
* Fix Glance certification arguments
* Add Neutron Quotas only if Neutron service is available
Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~
* Spec consistent-resource-names:
Resource name is based on Task id now. It is a huge step to persistence
and disaster cleanups.
* Add a spec for distributed load generation:
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/distributed_runner.rst
* Improvements for scenario output format
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/improve_scenario_output_format.rst
* Task and Verify results export command
https://github.com/openstack/rally/blob/master/doc/specs/in-progress/task_and_verification_export.rst
Plugins
~~~~~~~
* **Scenarios**:
* [new] NovaServers.boot_and_get_console_output
* [new] NovaServers.boot_and_show_server
* [new] NovaServers.boot_server_attach_created_volume_and_resize
* [new] NovaServers.boot_server_from_volume_and_resize
* [new] NeutronSecurityGroup.create_and_delete_security_groups
* [new] NeutronSecurityGroup.create_and_list_security_groups
* [new] NeutronSecurityGroup.create_and_update_security_groups
* [new] NeutronLoadbalancerV1.create_and_delete_healthmonitors
* [new] NeutronLoadbalancerV1.create_and_list_healthmonitors
* [new] NeutronLoadbalancerV1.create_and_update_healthmonitors
* [new] SwiftObjects.list_and_download_objects_in_containers
* [new] SwiftObjects.list_objects_in_containers
* [new] FuelNodes.add_and_remove_node
* [new] CeilometerMeters.list_matched_meters
* [new] CeilometerResource.list_matched_resources
* [new] CeilometerSamples.list_matched_samples
* [new] CeilometerStats.get_stats
* [new] Authenticate.validate_monasca
* [new] DesignateBasic.create_and_delete_zone
* [new] DesignateBasic.create_and_list_zones
* [new] DesignateBasic.list_recordsets
* [new] DesignateBasic.list_zones
* [fix] CinderVolumes.create_nested_snapshots_and_attach_volume
Remove random nested level which produce different amount of atomic
actions and bad reports.
* Support for Designate V2 api
* A lot of improvements in Sahara scenarios
* **Context**:
* [new] api_versions
Context allows us to setup client to communicate to specific service.
* [new] swift_objects
Context pre creates swift objects for future usage in scenarios
* [update] sahara_cluster
It supports proxy server which allows to use single floating IP for
whole cluster.
* [fix] cleanup
Fix cleanup of networks remove vip before port.
Bug fixes
~~~~~~~~~
**37 bugs were fixed, the most critical are**:
* Follow symlinks in plugin discovery
* Use sed without -i option for portability (install_rally.sh)
* Fixed race in rally.common.broker
* Fixed incorrect iteration number on "Failures" Tab
* Fixing issue with create_isolated_networks = False
* Fix docker build command
Documentation
~~~~~~~~~~~~~
Fixed some minor typos and inaccuracies.
Thanks
~~~~~~
We would like to thank Andreas Jaeger for ability to provide Python 2.6 support in this release.
Rally v0.1.1
Information
-----------
+------------------+-----------------------+
| Commits | **32** |
+------------------+-----------------------+
| Bug fixes | **9** |
+------------------+-----------------------+
| Dev cycle | **11 days** |
+------------------+-----------------------+
| Release date | **6/October/2015** |
+------------------+-----------------------+
Details
-------
This release contains new features, new 6 plugins, 9 bug fixes,
various code and API improvements.
New Features
~~~~~~~~~~~~
* **Rally verify generates proper tempest.conf file now**
Improved script that generates tempest.conf, now it works out of box for
most of the clouds and most of Tempest tests will pass without hacking it.
* **Import Tempest results to Rally DB**
``rally verify import`` command allows you to import already existing Tempest
results and work with them as regular "rally verify start" results:
generate HTML/CSV reports & compare different runs.
API Changes
~~~~~~~~~~~~
**Rally CLI changes**
* [add] ``rally verify import`` imports raw Tempest results to Rally
Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~
There is no new specs and feature requests.
Plugins
~~~~~~~
* **Scenarios**:
[new] NeutronNetworks.create_and_list_floating_ips
[new] NeutronNetworks.create_and_delete_floating_ips
[new] MuranoPackages.import_and_list_packages
[new] MuranoPackages.import_and_delete_package
[new] MuranoPackages.import_and_filter_applications
[new] MuranoPackages.package_lifecycle
[improved] NovaKeypair.boot_and_delete_server_with_keypair
New argument ``server_kwargs``, these kwargs are used to boot server.
[fix] NeutronLoadbalancerV1.create_and_delete_vips
Now it works in case of concurrency > 1
* **Contexts**:
[improved] network
Network context accepts two new arguments:
``subnets_per_network`` and ``network_create_args``.
[fix] network
Fix cleanup if nova-network is used. Networks should be dissociate from
project before deletion
[fix] custom_image
Nova server that is used to create custom image was not deleted if
script that prepares server failed.
Bug fixes
~~~~~~~~~
**9 bugs were fixed, the most critical are**:
* Fix install_rally.sh script
Set 777 access to /var/lib/rally/database file if system-wide method of
installation is used.
* Rally HTML reports Overview table had few mistakes
* Success rate was always 100%
* Percentiles were wrongly calculated
* Missing Ironic, Murano and Workload(vm) options in default config file
* ``rally verify start`` failed while getting network_id
* ``rally verify genconfig`` hangs forever if Horizon is not available
Documentation
~~~~~~~~~~~~~
* **Fix project maintainers page**
Update the information about Rally maintainers
* **Document rally --plugin-paths CLI argument**
* **Code blocks in documentation looks prettier now**
Rally v0.1.0
============
Information
-----------
+------------------+-----------------------+
| Commits | **355** |
+------------------+-----------------------+
| Bug fixes | **90** |
+------------------+-----------------------+
| Dev cycle | **132 days** |
+------------------+-----------------------+
| Release date | **25/September/2015** |
+------------------+-----------------------+
Details
-------
This release contains new features, new 42 plugins, 90 bug fixes,
various code and API improvements.
New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* **Improved installation script**
* Add parameters:
* ``--develop`` parameter to install rally in editable (develop) mode
* ``--no-color`` to switch off output colorizing
useful for automated output parsing and terminals that don't
support colors.
* Puts rally.conf under virtualenv etc/rally/ so you can have several
rally installations in virtualenv
* Many fixes related to access of different file, like: rally.conf,
rally db file in case of sqlite
* Update pip before Rally installation
* Fix reinstallation
* **Separated Rally plugins & framework**
Now plugins are here:
https://github.com/openstack/rally/tree/master/rally/plugins
Plugins are as well separated common/* for common plugins
that can be use no matter what is tested and OpenStack related
plugins
* **New Rally Task framework**
* All plugins has the same Plugin base:
rally.common.plugin.pluing.Plugin They have the same mechanisms for:
discovering, providing information based on docstrings, and in future
they will use the same deprecation/rename mechanism.
* Some of files are moved:
* rally/benchmark -> rally/task
*This was done to unify naming of rally task command and
actually code that implements it.*
* rally/benchmark/sla/base.py -> rally/task/sla.py
* rally/benchmark/context/base.py -> rally/task/context.py
* rally/benchmark/scenarios/base.py -> rally/task/scenario.py
* rally/benchmark/runners/base.py -> rally/task/runner.py
* rally/benchmark/scenarios/utils.py -> rally/task/utils.py
This was done to:
* avoid doing rally.benchamrk.scenarios import base as scenario_base
* remove one level of nesting
* simplify framework structure
* Some of classes and methods were renamed
* Plugin configuration:
* context.context() -> context.configure()
* scenario.scenario() -> scenario.configure()
* Introduced runner.configure()
* Introduced sla.configure()
This resolves 3 problems:
* Unifies configuration of different types of plugins
* Simplifies plugin interface
* Looks nice with new modules path:
>>> from rally.task import scenario
>>> @scenario.configure()
* Atomic Actions were changed:
* New rally.task.atomic module
This allow us in future to reuse atomic actions in Context plugins
* Renames:
rally.benchmark.scenarios.base.AtomicAction
-> rally.task.atomic.ActionTimer
rally.benchmark.scenarios.base.atomic_action()
-> rally.task.atomic.action_timer()
* **Context plugins decide how to map their data for scenario**
Now Context.map_for_scenario method can be override to decide
how to pass context object to each iteration of scenario.
* Samples of NEW vs OLD context, sla, scenario and runner plugins:
* Context
.. code-block:: python
# Old
from rally.benchmark.context import base
@base.context(name="users", order=100)
class YourContext(base.Context):
def setup(self):
# ...
def cleanup(self):
# ...
# New
from rally.task import context
@context.configure(name="users", order=100)
class YourContext(context.Context):
def setup(self):
# ...
def cleanup(self):
# ...
def map_for_scenario(self):
# Maps context object to the scenario context object
# like context["users"] -> context["user"] and so on.
..
* Scenario
.. code-block:: python
# Old Scenario
from rally.benchmark.scenarios import base
from rally.benchmark import validation
class ScenarioPlugin(base.Scenario):
@base.scenario()
def some(self):
self._do_some_action()
@base.atomic_action_timer("some_timer")
def _do_some_action(self):
# ...
# New Scenario
from rally.task import atomic
from rally.task import scenario
from rally.task import validation
# OpenStack scenario has different base now:
# rally.plugins.openstack.scenario.OpenStackScenario
class ScenarioPlugin(scenario.Scenario):
@scenario.configure()
def some(self):
self._do_some_action()
@atomic.action_timer("some_action")
def _do_some_action(self):
# ...
..
* Runner
.. code-block:: python
## Old
from rally.benchmark.runners import base
class SomeRunner(base.ScenarioRunner):
__execution_type__ = "some_runner"
def _run_scenario(self, cls, method_name, context, args)
# Load generation
def abort(self):
# Method that aborts load generation
## New
from rally.task import runner
@runner.configure(name="some_runner")
class SomeRunner(runner.ScenarioRunner):
def _run_scenario(self, cls, method_name, context, args)
# Load generation
def abort(self):
# Method that aborts load generation
..
* SLA
.. code-block:: python
# Old
from rally.benchmark import sla
class FailureRate(sla.SLA):
# ...
# New
from rally.task import sla
@sla.configure(name="failure_rate")
class FailureRate(sla.SLA):
# ...
..
* **Rally Task aborted command**
Finally you can gracefully shutdown running task by calling:
.. code:: bash
rally task abort <task_uuid>
..
* **Rally CLI changes**
* [add] ``rally --plugin-paths`` specify the list of directories with plugins
* [add] ``rally task report --junit`` - generate a JUnit report
This allows users to feed reports to tools such as Jenkins.
* [add] ``rally task abort`` - aborts running Rally task
when run with the ``--soft`` key, the ``rally task abort`` command is
waiting until the currently running subtask is finished, otherwise the
command interrupts subtask immediately after current scenario iterations
are finished.
* [add] ``rally plugin show`` prints detailed information about plugin
* [add] ``rally plugin list`` prints table with rally plugin names and titles
* [add] ``rally verify genconfig`` generates tempest.conf without running it.
* [add] ``rally verify install`` install tempest for specified deployment
* [add] ``rally verify reinstall`` removes tempest for specified deployment
* [add] ``rally verify uninstall`` uninstall tempest of specified deployment
* [fix] ``rally verify start --no-use`` --no-use was always turned on
* [remove] ``rally use`` now each command has subcommand ``use``
* [remove] ``rally info``
* [remove] ``rally-manage tempest`` now it is covered by ``rally verify``
* **New Rally task reports**
* New code is based on OOP style which is base step to make plugable Reports
* Reports are now generated for only one iteration over the resulting data
which resolves scalability issues when we are working with large
amount of iterations.
* New Load profiler plot that shows amount of iterations that are working
in parallel
* Failed iterations are shown as a red areas on stacked are graphic.
Non backward compatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* [remove] ``rally use`` cli command
* [remove] ``rally info`` cli command
* [remove] ``--uuid`` parameter from ``rally deployment <any>``
* [remove ``--deploy-id`` parameter from:
``rally task <any>``, ``rally verify <any>``, ``rally show <any>``
Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~
[feature request] Explicitly specify existing users for scenarios
[feature request] Improve install script and add --unistall and --version
[feature request] Allows specific repos & packages in install-rally.sh
[feature request] Add abbility to caputre logs from tested services
[feature request] Check RPC queue perfdata
[spec] Refactoring Rally cleanup
[spec] Consistent resource names
Plugins
~~~~~~~
* **Scenarios**:
[new] CinderVolumes.create_volume_backup
[new] CinderVolumes.create_and_restore_volume_backup
[new] KeystoneBasic.add_and_remove_user_role
[new] KeystoneBasic.create_and_delete_role
[new] KeystoneBasic.create_add_and_list_user_roles
[new] FuelEnvironments.list_environments
[new] CinderVolumes.modify_volume_metadata
[new] NovaServers.boot_and_delete_multiple_servers
[new] NeutronLoadbalancerV1.create_and_list_pool
[new] ManilaShares.list_shares
[new] CeilometerEvents.create_user_and_get_event
[new] CeilometerEvents.create_user_and_list_event_types
[new] CeilometerEvents.create_user_and_list_events
[new] CeilometerTraits.create_user_and_list_trait_descriptions
[new] CeilometerTraits.create_user_and_list_traits
[new] NeutronLoadbalancerV1.create_and_delete_pools
[new] NeutronLoadbalancerV1.create_and_update_pools
[new] ManilaShares.create_and_delete_share
[new] ManilaShares.create_share_network_and_delete
[new] ManilaShares.create_share_network_and_list
[new] HeatStacks.create_and_delete_stack
[new] ManilaShares.list_share_servers
[new] HeatStacks.create_snapshot_restore_delete_stack
[new] KeystoneBasic.create_and_delete_ec2credential
[new] KeystoneBasic.create_and_list_ec2credentials
[new] HeatStacks.create_stack_and_scale
[new] ManilaShares.create_security_service_and_delete
[new] KeystoneBasic.create_user_set_enabled_and_delete
[new] ManilaShares.attach_security_service_to_share_network
[new] IronicNodes.create_and_delete_node
[new] IronicNodes.create_and_list_node
[new] CinderVolumes.create_and_list_volume_backups
[new] NovaNetworks.create_and_list_networks
[new] NovaNetworks.create_and_delete_network
[new] EC2Servers.list_servers
[new] VMTasks.boot_runcommand_delete_custom_imagea
[new] CinderVolumes.create_and_update_volume
* **Contexts**:
[new] ManilaQuotas
Add context for setting up Manila quotas:
shares, gigabytes, snapshots, snapshot_gigabytes, share_networks
[new] ManilaShareNetworks
Context for share networks that will be used in case of usage
deployment with existing users. Provided share networks via context
option "share_networks" will be balanced between all share creations
of scenarios.
[new] Lbaas
Context to create LBaaS-v1 resources
[new] ImageCommandCustomizerContext
Allows image customization using side effects of a command execution.
E.g. one can install an application to the image and use these image
for 'boot_runcommand_delete' scenario afterwards.
[new] EC2ServerGenerator
Context that creates servers using EC2 api
[new] ExistingNetwork
This context lets you use existing networks that have already been
created instead of creating new networks with Rally. This is useful
when, for instance, you are using Neutron with a dumb router that is
not capable of creating new networks on the fly.
* **SLA**:
[remove] max_failure_rate - use failure_rate instead
Bug fixes
~~~~~~~~~
**90 bugs were fixed, the most critical are**:
* Many fixes related that fixes access of rally.conf and DB files
* Incorrect apt-get "-yes" parameter in install_rally.sh script
* Rally bash completion doesn't exist in a virtualenv
* Rally show networks CLI command worked only with nova networks
* RPS runner was not properly generating load
* Check is dhcp_agent_scheduler support or not in network cleanup
* NetworkContext doesn't work with Nova V2.1
* Rally task input file was not able to use jinja2 include directive
* Rally in docker image was not able to
* Rally docker image didn't contain samples
* Do not update the average duration when iteration failed
Documentation
~~~~~~~~~~~~~
* **Add plugin reference page**
:ref:`Rally Plugins Reference page <plugin_reference>` page contains a
full list with
* **Add maintainers section on project info page**
:ref:`Rally Maintainers section <project_info>` contains information
about core contributors of OpenStack Rally their responsibilities and
contacts. This will help us to make our community more transparent and open
for newbies.
* **Added who is using section in docs**
* **Many small fixes**
============
Rally v0.0.4
============
Information
-----------
+------------------+-----------------+
| Commits | **87** |
+------------------+-----------------+
| Bug fixes | **21** |
+------------------+-----------------+
| Dev cycle | **30 days** |
+------------------+-----------------+
| Release date | **14/May/2015** |
+------------------+-----------------+
Details
-------
This release contains new features, new benchmark plugins, bug fixes, various code and API improvements.
New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* Rally now can generate load with users that already exist
Now one can use Rally for benchmarking OpenStack clouds that are using LDAP, AD or any other read-only keystone backend where it is not possible to create any users. To do this, one should set up the "users" section of the deployment configuration of the ExistingCloud type. This feature also makes it safer to run Rally against production clouds: when run from an isolated group of users, Rally won’t affect rest of the cloud users if something goes wrong.
* New decorator *@osclients.Clients.register* can add new OpenStack clients at runtime
It is now possible to add a new OpenStack client dynamically at runtime. The added client will be available from osclients.Clients at the module level and cached. Example:
.. code-block:: none
>>> from rally import osclients
>>> @osclients.Clients.register("supernova")
... def another_nova_client(self):
... from novaclient import client as nova
... return nova.Client("2", auth_token=self.keystone().auth_token,
... **self._get_auth_info(password_key="key"))
...
>>> clients = osclients.Clients.create_from_env()
>>> clients.supernova().services.list()[:2]
[<Service: nova-conductor>, <Service: nova-cert>]
* Assert methods now available for scenarios and contexts
There is now a new *FunctionalMixin* class that implements basic unittest assert methods. The *base.Context* and *base.Scenario* classes inherit from this mixin, so now it is possible to use *base.assertX()* methods in scenarios and contexts.
* Improved installation script
The installation script has been almost completely rewritten. After this change, it can be run from an unprivileged user, supports different database types, allows to specify a custom python binary, always asks confirmation before doing potentially dangerous actions, automatically install needed software if run as root, and also automatically cleans up the virtualenv and/or the downloaded repository if interrupted.
Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~
* [Spec] Reorder plugins
The spec describes how to split Rally framework and plugins codebase to make it simpler for newbies to understand how Rally code is organized and how it works.
* [Feature request] Specify what benchmarks to execute in task
This feature request proposes to add the ability to specify benchmark(s) to be executed when the user runs the *rally task start* command. A possible solution would be to add a special flag to the *rally task start* command.
Plugins
~~~~~~~
* **Benchmark Scenario Runners**:
* Add limits for maximum Core usage to constant and rps runners
The new 'max_cpu_usage' parameter can be used to avoid possible 100% usage of all available CPU cores by reducing the number of CPU cores available for processes started by the corresponding runner.
* **Benchmark Scenarios**:
* [new] KeystoneBasic.create_update_and_delete_tenant
* [new] KeystoneBasic.create_user_update_password
* [new] NovaServers.shelve_and_unshelve_server
* [new] NovaServers.boot_and_associate_floating_ip
* [new] NovaServers.boot_lock_unlock_and_delete
* [new] NovaHypervisors.list_hypervisors
* [new] CeilometerSamples.list_samples
* [new] CeilometerResource.get_resources_on_tenant
* [new] SwiftObjects.create_container_and_object_then_delete_all
* [new] SwiftObjects.create_container_and_object_then_download_object
* [new] SwiftObjects.create_container_and_object_then_list_objects
* [new] MuranoEnvironments.create_and_deploy_environment
* [new] HttpRequests.check_random_request
* [new] HttpRequests.check_request
* [improved] NovaServers live migrate benchmarks
add 'min_sleep' and 'max_sleep' parameters to simulate a pause between VM booting and running live migration
* [improved] NovaServers.boot_and_live_migrate_server
add a usage sample to samples/tasks
* [improved] CinderVolumes benchmarks
support size range to be passed to the 'size' argument as a dictionary
*{"min": <minimum_size>, "max": <maximum_size>}*
* **Benchmark Contexts**:
* [new] MuranoPackage
This new context can upload a package to Murano from some specified path.
* [new] CeilometerSampleGenerator
Context that can be used for creating samples and collecting resources for benchmarks in a list.
* **Benchmark SLA**:
* [new] outliers
This new SLA checks that the number of outliers (calculated from the mean and standard deviation of the iteration durations) does not exceed some maximum value. The SLA is highly configurable: the parameters used for outliers threshold calculation can be set by the user.
Bug fixes
~~~~~~~~~
**21 bugs were fixed, the most critical are**:
* Make it possible to use relative imports for plugins that are outside of rally package.
* Fix heat stacks cleanup by deleting them only 1 time per tenant (get rid of "stack not found" errors in logs).
* Fix the wrong behaviour of 'rally task detailed --iterations-data' (it lacked the iteration info before).
* Fix security groups cleanup: a security group called "default", created automatically by Neutron, did not get deleted for each tenant.
Other changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* Streaming algorithms that scale
This release introduces the common/streaming_algorithms.py module. This module is going to contain implementations of benchmark data processing algorithms that scale: these algorithms do not store exhaustive information about every single benchmark iteration duration processed. For now, the module contains implementations of algorithms for computation of mean & standard deviation.
* Coverage job to check that new patches come with unit tests
Rally now has a coverage job that checks that every patch submitted for review does not decrease the number of lines covered by unit tests (at least too much). This job allows to mark most patches with no unit tests with '-1'.
* Splitting the plugins code (Runners & SLA) into common/openstack plugins
According to the spec "Reorder plugins" (see above), the plugins code for runners and SLA has been moved to the *plugins/common/* directory. Only base classes now remain in the *benchmark/* directory.
Documentation
~~~~~~~~~~~~~
* Various fixes
* Remove obsolete *.rst* files (*deploy_engines.rst* / *server_providers.rst* / ...)
* Restructure the docs files to make them easier to natigate through
* Move the chapter on task templates to the 4th step in the tutorial
* Update the information about meetings (new release meeting & time changes)
PreviousNext