Aug 12, 2016

Document Binary Package Dependencies - not only for OpenStack Python Packages

Python developers record their dependencies on other Python packages in requirements.txt and test-requirements.txt. But some packages havedependencies outside of python and we should document thesedependencies as well so that operators, developers, and CI systems
know what needs to be available for their programs.

Bindep is a solution to this, it allows a repo to document binarydependencies in a single file. It even enablies specification of which distribution the package belongs to - Debian, Fedora, Gentoo, openSUSE, RHEL, SLES and Ubuntu have different package names - and allows profiles, like a test profile.

Bindep is one of the tools the OpenStack Infrastructure team has written and maintains. It is in use by already over 130 repositories.

For better bindep adoption, in the just released bindep 2.1.0 we have changed the name of the default file used by bindep from other-requirements.txt to bindep.txt and have pushed changes to master branches of repositories for this.

Projects are encouraged to create their own bindep files. Besides documenting what is required, it also gives a speedup in running tests since you install only what you need and not all packages that some other project might need and are installed  by default. Each test system comes with a basic installation and then we either add the repo defined package list or the large default list.

In the OpenStack CI infrastructure, we use the "test" profile for installation of packages. This allows projects to document their run time dependencies - the default packages - and the additional packages needed for testing.

Be aware that bindep is not used by devstack based tests, those have their own way to document dependencies.

A side effect is that your tests run faster, since they have less packages to install. A Ubuntu Xenial test node installs 140 packages and that can take between 2 and 5 minutes. With a smaller bindep file, this can change.

Let's look at the log file for a normal installation with using the default dependencies:
2 upgraded, 139 newly installed, 0 to remove and 41 not upgraded
Need to get 148 MB of archives.
After this operation, 665 MB of additional disk space will be used.

Compare this with the openstack-manuals repostiry that uses bindep - this example was 20 seconds and not minutes:
0 upgraded, 17 newly installed, 0 to remove and 43 not upgraded.
Need to get 35.8 MB of archives.
After this operation, 128 MB of additional disk space will be used.

If you want to learn more about bindep, read the Infra Manual on package requirements 
 
If you have questions about bindep, feel free to ask the Infra team on #openstack-infra.
 
Thanks to Anita for reviewing and improving this blog post and to the OpenStack Infra team that maintains bindep, especially to Jeremy Stanley and Robert Collins.

Aug 11, 2016

Testing OpenStack with always updating python package versions

With any software package, you will need additional packages to run it. Often, there's a tight coupling: The software package will only run with specific other package versions. This dependency information is sometimes found in README files, in code, or in package metadata. If you install the package, you need to figure out the dependency and
handle it properly.

The Python package installer pip uses a list of requirements to install dependent Python packages. This list not only contains the name of packages but also limits which versions to use, or not to use.
In OpenStack we handle these dependencies in a global requirements list and use it for most of the repositories. During initial testing a specific package version is tested but at a later point, another one might be used, or during deployment again another one.

To document what was tested, give guidenance for deployment, and help to figure out breakage by upstream projects, the OpenStack requirements projects maintains a set of constraints with packages pinned to specific package versions that are known to be working.
These are in the upper-constraints.txt file.

Devstack already handles upper-constraints.txt when installing packages and I'm happy to say that tox, the Python testing framework used in OpenStack, can now handle upper-constraints as well everywhere.


Constraints for tox based jobs

To use constraints, change in tox.ini the install command to:

install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}

Caveat

Note that constraints are used for the installation of each packages, so if you want to install a package from source and have constraints for a specific version in the constraints file, it will not work. This happens with some of  the OpenStack python client packages: When they install their dependencies, those might
have a dependency on the client package itself. And this then will cause an error since the client package should get installed from source.

So, projects need to remove the constraints file for themselves if they run into this. Packages like python-novaclient and python-glanceclient therefore use a wrapper (tools/tox_install.sh) as
install command to edit the constraints file first and remove their own project from it.

Also, be aware that this only for those jobs that have been enabled for it in the project-config repository. It's done for all the generic tox enabled targets and should be done for all custom tox targets as well. Some repositories are not using constraints like project-config
itself, so those jobs are not set up.

Constraints for DevStack jobs

Devstack-gate takes care using constraints, there is nothing for a repository to do to honor constraints.

Check the devstacklog.txt file, if constraints are in use it will use lines like:

Collecting oslo.context===2.7.0 (from -c /opt/stack/new/requirements/upper-constraints.txt (line 204))

References

To learn more about constraints read the requirements documents. There is also a spec that explains all the steps that where needed for this.


Thanks

As usual in OpenStack, such work is a team work of many people. I'd like to thank especially:

  • Robert Collins 'lifeless': For writing the initial spec, implementation work, and giving guideance on many of these details.
  • Sean Dague: He was bold enough to try using constraints everywhere and showing us where it failed.
  • Sachi King for making zuul-cloner usable in the post queue. This was a missing part in the last months.
  • The OpenStack infra team for many reviews and design discussions - especially to Jeremy Stanley and Jim Blair.

Feb 8, 2016

Templates in OpenStack's Zuul

This is a followup to my post  "Creating new test jobs in OpenStack CI". Last time I covered the basic setup of jobs by Jenkins and Zuul. Since many OpenStack projects run the same jobs, the Zuul developers have introduced templates to easily group and reuse these jobs.

Let's look at one common example, it's the python-jobs template. Like all examples in this article, it is defined in file zuul/layout.yaml in the openstack-infra/project-config repository and to use it, you need to edit the same file.

Defining your own templates

Since you now know how to use a template, let's explain how they really look like. A template consists of a name, definitions for which jobs should run in which queue and allows to substitute the name of the repository in the job, so {name} gets replaced by your repository, in the example by amazing-repo.

Let's look at the python-jobs template:



  - name: python-jobs
    check:
      - 'gate-{name}-pep8'
      - 'gate-{name}-docs'
      - 'gate-{name}-python27'
    gate:
      - 'gate-{name}-docs'
      - 'gate-{name}-pep8'
      - 'gate-{name}-python27'
    post:
      - '{name}-branch-tarball'


The template has the name python-jobs, adds three jobs to the check queue and the same jobs also to the gate queue. An additional job is added to the post queue. Jobs in the check get queue get triggered when a change gets submitted, jobs in the gate queue get triggered when a change gets approved by a core reviewer and jobs in the post queue get triggered after a change has merged.
If you are adding the same class of jobs to several repositories, create a template for it. A template can contain of a single job that is associated with one queue, or contain several jobs in several queues like the example above.

Using a template

So, if your project amazing-project wants to reuse the python-jobs template as is, just add it as template:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: python-jobs

You can also limit, on which branches those are jobs are triggered. For example, to run the docs job only on stable/liberty and newer branches, you can add a condition:

  - name: gate-amazing-project-docs
    branch: ^(?!stable/kilo).*$



So, instead of saying run on liberty and newer, we block it on older supported branches, in this case kilo is the only older supported branch.

If you're introducing jobs, best practice is to add them first to the experimental queue, and then add them as non-voting, and only finally as voting. In this case, the templates do not help you at all for the first two steps, you have to look at their definition and add them manually.

First step, using the jobs in the experimental queue:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: noop-jobs
    experimental:
      - gate-amazing-repo-pep8
      - gate-amazing-repo-docs
      - gate-amazing-repo-python27


Note that we use noop-jobs as a template, so that both check and gate queue have at least one job. The noop jobs do nothing but are important since Zuul requires at least one job to run with success, otherwise you will not be able to merge anything.

With this definition, you can now submit a change and add as review comment "check experimental" and the jobs are run and the results are reported.

Later, the manually triggered jobs run fine, so it's time to run them on each change but keep them non-voting to not block any merges:

  - name: gate-amazing-repo-docs
    voting: false

  - name: gate-amazing-repo-pep8
    voting: false

  - name: gate-amazing-repo-python27
    voting: false
....

  - name: openstack/amazing-repo
    template:
      - name: merge-check
    check:
      - gate-amazing-repo-pep8
      - gate-amazing-repo-docs
      - gate-amazing-repo-python27
    gate:
      - noop

Here we added the noop job to the gate since otherwise no job would run in the gate and Zuul requires at least one job to run.

Once the jobs all run fine, you can add them to the gate as well - and for that case, let's finally use the template:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: python-jobs



Defining your own templates

Since you now know how to use a template, let's explain how they really look like. A template consists of a name, definitions for which jobs should run in which queue and allows to substitute the name of the repository in the job, so {name} gets replaced by your repository, in the example by amazing-repo.
Let's review the python-jobs template again:

 
  - name: python-jobs
    check:
      - 'gate-{name}-pep8'
      - 'gate-{name}-docs'
      - 'gate-{name}-python27'
    gate:
      - 'gate-{name}-docs'
      - 'gate-{name}-pep8'
      - 'gate-{name}-python27'
    post:
      - '{name}-branch-tarball'


The template has the name python-jobs, adds three jobs to the check queue and the same jobs also to the gate queue. An additional job is added to the post queue. Jobs in the check get queue get triggered when a change gets submitted, jobs in the gate queue get triggered when a change gets approved by a core reviewer and jobs in the post queue get triggered after a change has merged.
If you are adding the same class of jobs to several repositories, create a template for it. A template can contain of a single job that is associated with one queue, or contain several jobs in several queues like the example above.

References

For more information about templates, you can look at the file zuul/layout.yaml on definitions and usage.  Zuul has been written for OpenStack CI and has its own documentation. For information about Zuul's OpenStack instance, read the Project Config Infrastructure page about Zuul. The best starting place learn about using the OpenStack CI infrastructure is the Infra Manual.

Followup?

If you liked this post and like to learn more about OpenStack CI, please leave a comment with details.

Feb 6, 2016

Creating new test jobs in OpenStack CI

Reviewing patches for the OpenStack CI infrastructure, there's one piece that often confuse contributors: The question how Zuul and Jenkins configuration are working together.

While we have the Infra Manual with a whole page on how to create a project - and I advise everyone to read it - , let me try to tackle the specific topic of adding new jobs from a different angle.

What we're discussing here are job, or tests, that are run. Jenkins actually runs these jobs. Zuul watches for changes in gerrit (URL for OpenStack is review.openstack.org) to trigger the appropriate jobs so that Jenkins runs them.


To understand the relationship between these two systems, let's try as an analogy programming languages: As a developer, you create a library of functions that do a variety of actions. You also write a script that uses this library to execute them. Jenkins can be considered the library of test functions. But just defining these is not enough, you have to call them. Zuul takes care of calling them, so in the analogy is your script.

So, to actually get a job running for a repository, you first need to define it in the Jenkins "library", and then you trigger its invocation in Zuul. You can also add certain conditions to limit when the job runs or whether it is voting.

If you dig deeper into Jenkins and Zuul, keep in mind that these are two different programming languages, even if both use YAML as format. Jenkins runs jobs and these are defined as text files using the Jenkins job builder. To define them, you can write a job, or use a job-template and instantiate it, or group several job-template in a job-group and instantiate that job-group to create with a few lines many jobs. Zuul uses these jobs and has as syntactic sugar templates to reuse jobs and the queues they run in.

Let's look at a simple examples, adding a new docs job to your repository called amazing-repo:

  1. Check out the project-config repository and make it ready for patch submission like creating a branch where you work on.
  2. Since for the docs job already a template exists, you can reuse it. It is called'gate-{name}-docs', so add it to your repository in file jenkins/jobs/projects.yaml:
    - project:
      name: amazing-repo
      node: bare-trusty
      jobs:
          - gate-{name}-docs
  3. Now define how to trigger the job. Edit file zuul/layout.yaml and update your repository entry to add the job:

    - name: openstack/amazing-repo
      template:
        - name: merge-check
      check:
        - gate-amazing-repo-docs
      gate:
        - gate-amazing-repo-docs

    This adds the job to both the check and gate queue. So, it will notonly be run when a patch is initially submitted for review in the check queue but also after a patch gets approved in the gate queue. Since your tree might be different when you submitted a change and when it merges, we run jobs in both situations so that the tree istested exactly as it merges.
  4. Let's go one step back: Your repository is not ready yet to havethe docs job voting, so you only want to run it as non-voting.
    In that case add a condition in the jobs section of zuul/layout.yaml:

    - name: gate-amazing-repo-docs
        voting: false


    And in your repository, only add it to the check queue. Non-voting jobs should not be in the gate, they get ignored completely and just waste resources:

      - name: openstack/amazing-repo
        template:
          - name: merge-check
        check:
          - gate-amazing-repo-docs
        gate:
          - ...

So, these are simple jobs. Stay tuned for a followup article that will cover how to use templates in Zuul - and how to modify your repository in the context of templates.

Thanks to Doug Fish for reviewing the text and giving suggestions on how to improve it - and urging me to write a follow-up.

P.S. Follow-up article is called "Templates in OpenStack's Zuul".

Apr 18, 2014

OpenStack Icehouse released - and available now for openSUSE and SUSE Linux Enterprise

OpenStack Icehouse has been released and packages are available for openSUSE 13.1 and SUSE Linux Enterprise 11 SP3 in the Open Build Service.
The packages are developed in the Cloud:OpenStack:Icehouse project and can be download from the openSUSE download server:
  • For openSUSE 13.1, you can add the Icehouse repository with:
    zypper addrepo -f obs://Cloud:OpenStack:Icehouse/openSUSE_13.1 Icehouse
  • For SLES 11 SP3, add the icehouse repository with:
    zypper addrepo -f obs://Cloud:OpenStack:Icehouse/SLE_11_SP3 Icehouse
If you like to install from packages, follow the OpenStack Installation Guide for openSUSE and SUSE Linux Enterprise which has been updated for Icehouse.
With Icehouse, you can use the dashboard now in German and also use a new wizard for network creation.
Additional information about the release is available on the OpenStack web page.
My personal highlights of the new release are:
  • The translation of the dashboard to German and the new accordion navigation in the dashboard
  • The updated Installation Guides that have been greatly improved.
  • The new Database module (trove) that allows easy creation and manipulation of Databases for usage by virtual machines and thus gives you Database-as-a-Service.
  • The Compute component now allows migrations and upgrades - you can first update the controller nodes and then the compute nodes and run thus a mixture of old and new compute nodes.
  • The great progress that Orchestration makes with better integration of projects and giving the users now full control of manipulation of stacks.
  • Learning about the "most insane CI infrastructure". The more I learn about the CI infrastructure and interact with the infra team, my appreciation about their great work growth. Thanks a lot, Clark, Elizabeth, Fungi, Monty, James, Sergey, et al.!
Also, thanks to the Documentation team, it was again a lot of fun to work together with all of you and release Icehouse documentation and improve OpenStack! Team, I look forward to drink with you the 104 beers that Gauvain promised for Atlanta ;)
Now on to get Icehouse integrated into SUSE Cloud for our next release...


Apr 17, 2014

Changes for importing translations to OpenStack

Most OpenStack projects send after each commit updated files for translation to transifex. Also, every morning any translated files get merged back to the projects as a "proposal" - a normal review with all the changes in it.
Quite a lot projects had broken translation files - files with duplicate entries that transifex rejected -, and thus no update happened for some time. The problem is that these jobs run automatically and nobody gets notified when they fail.

Clark Boylan, Devananda van der Veen and myself have recently looked into this and produced several fixes to get this all working properly:
  • The broken translation files (see bug 1299349) have been fixed in time for the Icehouse release.
  • The gating has been enhanced so that no broken files can go in again.
  • The scripts that send the translations to and retrieve them from transifex have been greatly improved.
The scripts have been analyzed and a couple of problems fixed so that no more obsolete entries should show up anymore. Additionally, the proposal scripts - those that download from transifex - have been changed to not propose any files where only dates or line numbers have changed. This last change is a great optimization for the projects. For example, the sahara project got every morning a proposal that contained two line changes for each .po file - just updates of the dates. Now they do not get any update at all unless there is a real change of the translation files. A real change is either a new translation or a change of strings. For example, today's update to the operations-guide contained only a single file (see this review) since the only change was a translation update by the Czech translation team - and sahara did not get an update at all.

New User: OpenStack Proposal Bot

Previously the translation jobs where run as user "Jenkins" and now have been changed to "OpenStack Proposal Bot".

Now, another magic script showed up and welcomed the Proposal Bot to OpenStack:
Unfortunately, the workflow did not work for most projects - the bot forgot to sign the contributor license agreement:
I'm sure the infra team will find a way to have the bot sign the CLA and starting tomorrow, the import of translations will work again.

Mar 27, 2014

Horizon is translated to German for OpenStack Icehouse!

After my call for help two days ago, a few new people started helping with the German translations and we've achieved the goal of translating 100 per cent - that's all 2141 strings!
This was done by Christian Berendt, Marita Werner, Robert Simai, Sascha Peilicke, User "Schwefel", User "transistor" and myself! Thanks for getting this done!

I'm impressed that we got this final step done so quickly!

A special applause to Marita for doing the final 60 strings - I consider the last the hardest ones since those are the ones that we all skipped ;)