OpenStack RequirementsWhat we are doing, what to expect

and

what’s next

Swapnil Kulkarni@coolsvap

Davanum Srinivas@dims

Agenda

● What is requirements project○ Why global requirements○ Lifecycle of requirements

● Global Requirements and Upper Constraints● Accomplishments● Contribution Summary● Periodic Updates● Current Challenges● Whats Next?● How you can contribute● Resources

What is requirements project?

Requirements project help maintain a central list of all the requirements (global-requirements.txt) that are allowed in

OpenStack projects.

Dependencies in a typical project

requirements.txt

test-requirements.txt

setup.cfg

Global Requirements and Upper Constraints

Why global requirements?

Different projects may have different version ranges for same python library

Order of project installs may break randomly

Multiple installs of same library say during a single devstack run

What exactly are we testing in our CI?

Long term maintenance issues

Is the library maintained?

Is it under a compatible license?

Are there existing libraries in use that already have similar functionality?

Does this already exist in popular distros?

Is it python3 compatible?

So we need a single place to maintain version ranges and blessed versions

Lifecycle of Requirements

New requirements are added to g-r by teams

Component version ranges are updated in g-r as required

New component is released on pypi triggers updates to u-c

Updates to u-c affects all CI jobs for projects that are subscribed to g-r

Bot proposes updates to all subscribed projects

Requirements team considers pruning components from g-r/u-c

Works with individual projects to remove dependencies

Cleans up g-r and u-c, when all projects have removed those components

openstack-infra/project-config/zuul/layout.yaml

requirements change for oslo-utils merged in requirements

Subsequent change proposed by bot in nova

bot to update requirements versions

Individual project team reviewers then take decision on merging this requirement.

G-R and U-C maintenance

Bot picks up new releases from pypi and updates u-c

Bot picks up new indirect dependencies (of things in g-r) and updates u-c

Humans may propose the following (all of which will need updates to u-c):

New components

New version ranges of components

Exclusion of specific versions

Gate/Check jobs to validate and merge Bot and Human proposed updates

Another Bot proposes checks each project and proposes updates to requirements.txt, test-requirements.txt and setup.cfg

Gate/Check jobs in each project block team members from straying away from g-r

Humans approve bot proposed updates to their projects

openstack-infra pypi mirror

Periodic job to check upper constraints

requirements project team core reviewers then take decision on merging this requirement.

Cross-project jobs to verify upper-constraint changes work

How do i subscribe to global requirements process?

Compare with global-requirements.txt

Submit reviews for missing requirements that you need for your project

Sync your requirements.txt with the version ranges in global-requirements.txt

Add check-requirements job for the repository in project-config

Add an entry in projects.txt in requirements repository

Watch for reviews from OpenStack CI Bot

Adding/upgrading requirements

- Why adhere to the requirements questionnaire for adding new requirements?

- Upgrading requirements versions

- upper-constraint changes

Review Guidelines

- Early feedback of broken/unstable upstream requirements

- Known minimum versions

- Better commit messages

- What are blacklist requirements? Why we need them? When to use them?

Contribution Matrix (Newton)

Commits #570 Reviews #2600

Accomplishments (in Newton)

- Creation of formal team and Tony Breeds (tonyb) elected as PTL

- Formation of active core-team for requirements

- Improved cross-project integration tests with projects using upper constraints

- Improved reviews and stakeholders to actively track down issues arising with new requirements

- Better co-ordination with release engineering team for releases of openstack projects and related libraries.

- Removed most unused requirements from the list.

Pitfalls

Some projects start using a “feature” just added to u-c without updating the lower bound of g-r.

We don’t deal well with forks (pycrypto vs pycryptodome), Difficult to switch projects from one to another

We do not test lower bounds of version ranges (u-c by definition is latest and greatest versions usable)

Reverts trigger a lot of projects to be updated.

How do we get a larger set of jobs (to prevent bad versions getting in). Since the CI jobs test a small subset of tests.

Current Challenges

- Requirements sanitation

- Requirements with duplicate functionality

- Requirements without active maintainers and finding replacements (if possible)

- Advocate project teams to use upper-constraints

- Optimize propose changes

Looking ahead (Ocata priorities)

- Introduce lower bounds and testing in projects

- Python 3 compatibility and testing

- https://wiki.openstack.org/wiki/Python3

- Better communication and impact analysis when merging changes related to complex binaries.

Ways to contribute

- Manage global requirements for your projects in requirements repo

- Reviews

- Project integrations for u-c

Resources

GitHub: https://github.com/openstack/requirements

Developer Docs: http://docs.openstack.org/developer/requirements/

Gerrit Reviews: https://review.openstack.org/#/q/project:openstack/requirements

To-dos: https://etherpad.openstack.org/p/requirements-tasks

Weekly IRC meeting : https://wiki.openstack.org/wiki/Meetings/Requirements