Jump to: navigation, search

Simultaneous Release:Nitrogen Release Plan

Introduction

This is a Simultaneous Release Plan for the Nitrogen release of OpenDaylight. This is the seventh release of OpenDaylight and this release plan has some differences from it predecessors. Please read it carefully to understand the requirements to participate and for each milestone.

Projects may choose to participate or not based upon their readiness and desire to join the Simultaneous Release. This plan is structured as laid out in the OpenDaylight Project Lifecycle.

Definitions

APIs

  • APIs: Application Programing Interface. For the purposes of OpenDaylight an API is any form of interface that is exposed to other ODL apps/services and/or exposed to third party services or external applications. This includes all Java classes/interfaces declared public and exported from an OSGi bundle, all YANG models, all Karaf feature names/locations, all config file YANG schemas, and all REST/RESTCONF calls including the full URL with options, etc.
  • API Classifications:For the purposes of being declared Stable, Provisional, or Tentative, an API is a collection of code that provides some high-level functionality, e.g., flow programming or MD-SAL data store access. When listed on a release plan, APIs should be given a short name, classified into one of the three categories, and have the supporting bundles (if/when they exist) listed as well.
    • Stable API: An API that can be accessed external to your project, existed in a previous version of ODL, and will continue to exist in the current version of ODL with no changes. By definition, all Stable APIs are frozen throughout this entire release cycle. Note that all APIs are assumed to be Stable APIs unless called out as Provisional and/or Tentative in a release plan.
    • Provisional API: An API that can be accessed external to your project and is introduced in the current release, or an externally accessible API that existed in a previous version of ODL but is being modified for the current release.
    • Tentative API: A Provisional API that will be provided in a best effort manner, but which may or may not be delivered in the final release. The Go/NoGo status of Tentative APIs must be made by M3.

Milestones

  • Functionality Freeze (M3): (used to be called feature freeze) No new externally visible functionality is to be added to the current release of ODL. All provisional APIs are at least functional (at a beta-quality level) if not yet fully tested.
  • API Freeze (M4): All externally accessible APIs (Stable and Provisional) may not be modified. An API exception process (similar to the one used in helium) will allow for critical changes to APIs after API freeze with the consent of the consuming projects. APIs include, but are not limited to, all Java classes/interfaces declared public and exported from an OSGi bundle, all YANG models, all Karaf feature names/locations, all config file YANG schemas, and all REST/RESTCONF calls including the full URL with options.
  • Code Freeze (M5): No new features/functionality are to be allowed into the current release. Only errors/bugs identified in the bugzilla system are allowed. The exceptions to this include new tests, and documentation. Distribution (Karaf) packaging must be complete. Errors/bugs found after Code Freeze are still bugs and they may be created and worked on. This includes packaging bugs found as well.
  • String Freeze (M5): All text strings used within ODL may not be changed. Final documentation and localization teams may rely on these strings not changing for the current release.
  • Release Candidate (RC): A fully-built, complete version of the current ODL release. This includes all code, documentation, and packaging that make up the final user-deliverable set of artifacts. After RC0, new RCs will be continually built, e.g., once per day, to enable rapid testing and fixing of bugs.
    • Note that this definition makes the dates for RCs and the final release as targets, but they may need to be adjusted based on project readiness and any remaining blocking bugs.
    • While we will build daily release candidates, the notion of RC#s (increasing in number at a longer cadence, e.g., weekly) will remain to aid in planning when bug fixes are expected.
    • During the RC process blocking bugs will be tracked both on a spreadsheet and in bugzilla.
    • During the RC process regular, e.g., daily, meetings will be held on IRC to identify and address critical issues as they arise.
    • Projects not in the autorelease and the distribution feature index by RC0 cutoff will be dropped from the Nitrogen release.

Features

  • Feature: A logical grouping of code and functionality in a project. While a Feature is usually a Karaf feature, it could also be any other component or grouping.
  • Top-Level Feature: A Feature that provides one of the major pieces of functionality delivered by a project. In general, this should not require an understanding of the project's internals to know when/how to install it. Most projects will have a small number of, maybe even only one, Top-Level Features. In many cases the Top-Level Feature will actually only be a meta-feature grouping together lower-level features, which would be less obvious for an outsider to consume.
  • User-Facing Feature: A Top-Level Feature that somebody looking to install and run OpenDaylight should know about. They should be able to install the Feature and should be able to tell that it's been installed in the form of new user interface elements, support for new southbound devices, or other mechanisms. For example, odl-mdsal-broker is not a user-facing feature as it does nothing on it's own. However, odl-mdsal-apidocs likely is as it generates a user interface. Features that don't provide explicit user interfaces can still be user-facing. For example, odl-bgpcep-bgp-all provides no explicit user interface, but does enable BGP support that a user might care about.

Stable and Extended Features

  • Stable Feature: A Top-Level Feature that meets the following criteria. The TSC is expected to review each potentially stable feature during the project's release review and ensure that it meets these requirements. Note: Any of these requirements can be waived by the TSC for a given feature. However, the TSC is encouraged to find a way to make the test fit the situation before granting a waiver.
    • The Feature must lie within the declared scope of the project.
    • The feature is part of a Mature project (as defined in the project lifecycle).
    • The feature was present in the previous release.
    • The feature is only dependent on other stable features (or sub-components of other stable features).
    • The feature and its dependencies must meet the criteria for being accepted into Maven Central.
    • The feature provides adequate documentation.
    • The feature has 75% or higher test coverage as reported by Sonar.
      • Note: You can find individual module/feature test coverage numbers by: (i) going to a sonar report, (ii) click on the percentage under "Overall Coverage", (iii) use the scrollable list on the left to find the relevant module/bundle. It helps if your feature's code is all under a single subdirectory and that there is nothing else in that directory.
    • All externally-facing, i.e., accessible from outside the JVM, APIs in the feature must be protected by community-accepted authentication/authorization mechanisms, e.g., using aaa-authn to protect RESTCONF APIs
    • The feature must have at least one automated test for each of:
      • functionality (previously called a system test) to show the basic functionality works
      • cluster compatibility to show the feature works in a 3-node cluster using the clustered data store
      • scalability to show how large a system, e.g., number of hosts, switches, or links, the feature can handle
      • performance to show how many operations, e.g., transactions, flows, linkstate events, per second the feature can handle
      • longevity/stability to show the feature can run for a period of time under load without ill effect
      • In each case, the tests must show no unexplained regressions from previous releases.
    • The feature is backward compatible with the previous release of the feature, e.g., any APIs that were not deprecated in the previous release still exist with the same signatures.
      • Note: This does not prohibit adding new functions, REST URLs, or data items, but typically would prohibit removing or changing existing such things.
    • The feature has no known vulnerabilities that are older than a week and classified as important by the security response team or high by their CVSS score in a CVE. If a fix for such a vulnerability lies outside of OpenDaylight, the TSC may choose to relax the requirement on a case-by-case basis.
    • The feature commits to providing a migration strategy from the previous release. This will ideally take the form of scripts or automatic upgrade support, but could also come in the forum of documentation.
  • Extended Feature: A top-level feature that is a part of the Nitrogen release and does not meet the Stable Feature criteria.

Release Distributions

  • Nitrogen Stable Distribution: A Karaf distribution containing the collection of Stable Features as they are compiled into the Nitrogen Stable Release Feature Repository hosted in the Integration project.
  • Nitrogen Extended Distribution: A Karaf distribution containing the collection of both Stable and Extended Features as they are compiled in the Nitrogen Extended Release Feature Repository hosted in the Integration project.

Project Offsets

Projects are classified into one of 3 offsets:

  • offset zero: deadlines are at the dates listed
  • offset one: deadlines are generally1 at the listed dates + 2 weeks
  • offset two: deadlines are generally1 at the listed dates + 4 weeks
    • Note that the intent is that offset two projects have no other projects depending on them in this release

This is intentionally flattening the actual dependency graph

  • The full project-level graph is at least 10 levels
    • e.g., odlparent => yangtools => controller => openflowjava => openflowplugin => ovsdb => sfc => groupbasedpolicy => nic => integration
  • The idea is to hit an 80/20 point where projects can have some lag to get new APIs from those they depend on
    • If projects are in the same offset but need APIs from each other this should be noted and planned (possibly by asking for them sooner than would be required) as part of the API request negotiation at M2

The intent is for projects that form key infrastructure of all other projects (e.g., odlparent, yangtools, and controller) to operate at offset zero, projects which provide key network services (e.g., OpenFlow and OVSDB) to operate at offset one, and projects that others don't depend on to operate at offset two.

1Deadlines for Release Candidates (RC0, RC1 and RC2) and the release are the same regardless of offset. Deadlines for M1 and M2 are offset by +1 week and +2 weeks. Full details can be found in the dates listed in the Schedule table.

Requirements for Participation

In order to participate in the simultaneous release, a project must do the following things.

  1. Planning
    • Projects must declare their intent to participate in the Simultaneous Release by M1. This is accomplished by adding the project to the table in participating projects and sending the first milestone readout mail.
    • Participating projects must publish a candidate Release Plan by M1 and declare their final Release Plan by M2
      • Participating project Release Plans must contain Milestones that minimally line up with the Simultaneous Release Plan Milestones
      • Release plans should contain a complete list of the exposed APIs including the properties defined above, e.g., the name of the Java interface or YANG-file, a short name, and the list of supporting bundles.
      • Per-project release plans now include sections for cross-project negotiation of provided APIs and for noting cross-project incompatibilities.
        • Projects are required to negotiate cross-project dependencies for any new or modified APIs.
        • Projects are encouraged to think about and cross-project incompatibilities and how to resolve them, if possible, as part of their release plans.
  2. Leadership & Communication
    • Each project must elect a Project Lead as described in the TSC charter, section 7.
      • Phil Robb will help projects with this process and it must be completed by M1.
      • The results of the election, and other changes to the project lead during this release, should be reported by
        1. Updating the project facts template for the project on its main wiki page
        2. Updating the participating projects table of this release
        3. Sending an e-mail to the <project>-dev, release, and tsc lists
    • The Project Lead is expected to be responsible for the the project meeting deadlines, interacting with other projects, and interacting with the TSC
    • The Project Lead will be subscribed to the release mailing list and must respond to requests send to the a timely fashion—defined as two business days.
      • If a Project Lead is not be able to do so, they should (i) have somebody else stand in and do this on their behalf, (ii) send a mail to the release mailing list indicating this and the time period, and (iii) note the same information in the participating projects section of the release plan.
    • The project lead is expected to, at a minimum, read the release mailing list, read the TSC meeting minutes, and read the minutes from the IRC release meetings. The project lead is strongly encouraged to attend these meetings if at all possible and some representative from the project is expected to attend each IRC meeting if at all possible.
    • In addition to the Project Lead, each project must designate a Test Contact and Documentation Contact to handle test related communication.
    • All release-critical correspondence that requires a response will have a subject line containing "PLEASE RESPOND BY <DATE> <UTC-TIME>"
      • Please limit traffic to correspondence directly relating to the release
      • The TSC collects response time metrics for projects both to inform our planning and to measure project maturity going forward.
  3. Service Release Participation
    • All projects participating in the release are also required to participate in the stability releases described in the schedule after the formal release.
  4. Modularity
    • Modules that are not intended to interface with the controller via REST/other non-Java RPC mechanism must be OSGi bundles.
    • OSGi bundles should be reasonably granular.
    • OSGi bundles should be grouped into Karaf features by M3 including possibly defining some features as user-facing.
      • Each feature should be tested in every appropriate jenkins job (at least -verify, -merge, and -integration) using the "SingleFeatureTest" as defined in the Karaf step-by-step guide
  5. Quality
    • No later than M2, each project must have a "-verify" Jenkins Job which verifies that the project builds and passes test for each new patch pushed to gerrit.
    • No later than M2 as part of the Gerrit/Jenkins merge process, i.e., the Jenkins "-merge" job, participating projects must push their binary artifacts to the Nexus repository
    • No later than M2, each project must have a Jenkins Job which rebuilds and retests to an appropriate level when a project it depends on publishes new artifacts, i.e., a Jenkins "-integration" job.
    • No later than M2, each project primarily written in Java must be reporting unit and/or integration test coverage via sonar. (See instructions on reporting test code coverage)
      • Projects, especially ones that form key infrastructure for other projects, are strongly encouraged to set goals for code coverage and reported bugs. Doing so will be seen favorably when evaluating projects for advancement in the Project Lifecycle.
      • Stable Features must have appropriate unit and/or integration test coverage of at least 75% prior to M5.
  6. Testing
    • In addition to setting up appropriate Jenkins -verify, -merge, and -integration jobs by M2, projects are expected to provide adequate unit, integration and system tests.
      • Stable Features must have established integration and system tests as required for Mature project Stable Features.
    • The coverage provided by unit tests and integration tests should be reported to sonar by M2. (See instructions on reporting test code coverage)
    • Participating projects must describe a basic system test per top-level feature and a comprehensive system test including functionality, cluster, scalability, performance, longevity/stability per stable feature prior on M3.
    • Participating projects must run at least one basic automated system test for each top-level feature and several automated system tests including functionality, cluster, scalability, performance, longevity/stability for each stable feature by M5.
      • Note: The system test requirements can be waived by the TSC for a given feature if for example the top-level feature is tested through another top-level feature.
      • Note: Projects running system test outside OpenDaylight (external Lab) are required to report system test results in a timely fashion after release creations, e.g., weekly, RC, and formal releases.
    • System tests are expected to reliably pass. If a system test turns out to be unstable and intermittently fails, it must be fixed or disabled. If intermittent system tests are seen as having value to the project, they can be written and run on-demand by the project, but won't be run as part of the automated CSIT suite.
    • Further details and requirements can be found in the schedule and Nitrogen project integration and test requirements below.
  7. Documentation
  8. Code Hygiene
    • No uses of System.out.println in non-test code.
    • No dependencies on 3rd party (non-ODL) snapshot versions
    • Willing to use agreed-upon versions for dependencies (both 3rd-party and ODL), i.e., to eliminate version skew
    • Willing to find source code for 3rd-party dependencies and/or move to versions or alternatives for which source code can be found.
  9. Distribution
    • All projects must support a Karaf-based distribution model including defining Karaf features and adding them to integration repository no later than M3.
    • No later than M3, each project must have a “-distribution" Jenkins Job to verify changes in the code do not break integration distribution.
  10. Meeting Deadlines
    • All projects are expected to meet the deadlines laid out in the schedule below.
      • To indicate this, the project lead/contact is expected to provide send a milestone readout to the release mailing list by 23:59:59 UTC on the date listed for the the appropriate offset at each milestone.
      • Most information will be communicated by filling out appropriate information in the Nitrogen Tracking Spreadsheet, but a mail should still be sent indicating that the information has been filled in. Any other information or questions can be included in that mail.
    • If a project cannot make a deadline, the project lead/contact must write a summary of what was missed, why, the course correction that will be taken, and it's impact on other projects.
      • For offset two project this is mainly intended to be reflective and to help inform the release process.
      • For offset zero and offset one projects, this should be completed with 24 hours of missing the deadline and must be presented to the TSC at the first TSC meeting after the deadline.
    • All Milestone deliverables will be verified by the Nitrogen release management staff and/or the TSC.
      • NOTE: For deliverables defined only in the project's release plan—and not as a requirement in this document—the release management staff and/or TSC will verify that the status of the deliverables has been reported. Nitrogen release management staff and/or the TSC may also, but are not required to, verify the delivered functionality.

Nitrogen Project Integration and Test Requirements

This section is excerpted from Carbon Traditional Release Project Integration Requirements, to see it in it's full context, go to that page.

Kinds of Tests

These are "the 3 classics" in OpenDaylight:

  • Unit Test (UT)
    • This is the test developers write in the code to verify bundle functionality (e.g. JUnit).
    • This test exercises the contoller internal Java methods and classes.
    • Runs automatically on project build.
    • Test results and code coverage reported to Sonar.
  • Integration Test (IT)
    • This is the test developers write in the code to verify some inter-bundle (feature) functionality (e.g. PAX-EXAM).
    • This test exercises the controller internal Java interfaces.
    • Runs automatically on project build.
    • Test results and code coverage reported to Sonar.
    • For more information about IT check Integration Test wiki .

  • System Test (ST)
    • This is the test to verify multiple features at system level (e.g. Robot FW).
      • We strongly recommend the use of Robot Framework for system test (ST), we already have extensive knowledge, examples and tutorials.
    • These tests exercise the controller external REST interface and plugin protocols.
    • Runs on a test bed made of robot, controller and device instances.
    • Requires a karaf distribution as System Under Test (SUT).
    • Test results reported in Jenkins at a minimum.
    • System Test Report based off of this template completed and linked to from here
    • Test cases meet the following expectations.
    • Top level features are deemed "experimental" until the above System Test requirements are met.


Nitrogen Project Documentation Requirements

This section is excerpted from Nitrogen Project Documentation Requirements, to see it in it's full context, go to that page.

Kinds of Documentation

These are the expected kinds of documentation and target audiences for each kind.

  • User/Operator: for people looking to use the feature w/o writing code
    • Should include an overview of the project/feature
    • Should include description of availbe configuration options and what they do
  • Developer: for people looking to use the feature in code w/o modifying it
    • Should include API documentation, e.g., enunciate for REST, Javadoc for Java, ??? for RESTCONF/models
  • Contributor: for people looking to extend or modify the feature's source code
    • Note: should be documented on the wiki not in reStructuredText
  • Installation: for people looking for instructions to install the feature after they have downloaded the ODL release
    • Note: audience is the same as User/Operator docs
    • For most projects, this will be just a list of top-level features and options
      • As an example, l2switch-switch as the top-level feature with the -rest and -ui options
      • We'd also like them to note if the options should be checkboxes (i.e., they can each be turned on/off independently) or a drop down (i.e., at most one can be selected)
      • What other top-level features in the release are incompatible with each feature
      • This will likely be presented as a table in the documentation and the data will likely also be consumed by automated installers/configurators/downloaders
    • For some projects, there is extra installation instructions (for external components) and/or configuration
      • In that case, there will be a (sub)section in the documentation describing this process.
  • HowTo/Tutorial: walk throughs and examples that are not general-purpose documentation
    • Generally, these should be done as a (sub)section of either user/operator or developer documentation.
    • If they are especially long or complex, they may belong on their own
  • Release Notes:
    • Release notes are required as part of each project’s release review. They must also be translated into reStructuredText for inclusion in the formal documentation.

Requirements for projects

Projects MUST do the following

  • Provide reStructuredText documentation including
    • Developer documentation for every feature
      • Most projects will want to logically nest the documentation for individual features under a single project-wide chapter or section
      • This can be provided as a single .rst file or multiple .rst files if the features fall into different groups
      • This should start with ~300 word overview of the project and include references to any automatically-generated API documentation as well as more general developer information (as described above).
    • User/Operator documentation for every every user-facing feature (if any)
      • Note: This should be per-feature, not per-project. User's shouldn't have to know which project a feature came from.
      • Intimately related features, e.g., l2switch-switch, l2switch-switch-rest, and l2switch-switch-ui, can be documented as one noting the differences
      • This can be provided as a single .rst file or multiple .rst files if the features fall into different groups
    • Installation documentation
      • Most projects will simply provide a list of user-facing features and options. See Kinds of Documentation above.
    • Release Notes (both on the wiki and reStructuredText ) as part of the release review.
  • This documentation will be contributed to the docs repo (or possibly imported from the project's own repo with tooling that is under development)
    • Projects MAY be ENCOURAGED to instead provide this from their own repository if the tooling is developed
    • Projects choosing to meet the requirement this way MUST provide a patch to docs repo to import the project's documentation
  • Projects MUST cooperate with the documentation group on edits and enhancements to documentation
    • Note that the documentation team will also be creating (or asking projects to create) small groups of 2-4 projects that will peer review each other's documentation. Patches which have seen a few cycles of peer review will be prioritized for review and merge by the documentation team.

Timeline for Deliverables from Projects

  • M3: Documentation Started
    • Identified the kinds of documentation that will be provided and for what features
      • Release Notes are not required until release reviews at RC2
    • Created the appropriate .rst files in the docs repository (or their own repository if the tooling is available)
    • Have an outline for the expected documentation in those .rst files including the relevant (sub)sections and a sentence or two explaining what will go there
      • Obviusly, providing actual documentation in the (sub)sections is encouraged and meets this requirement
    • Milestone readout should include
      1. the list of kinds of documentation
      2. the list of corresponding .rst files and their location, e.g., repo and path
      3. the list of commits creating those .rst files
      4. the current word counts of those .rst files
  • M4: Documentation Continues
    • The readout at M4 should include the word counts of all .rst files with links to commits
    • The goal is to have draft documentation complete so that the documentation group can comment on it.
  • M5: Documentation Complete
    • All (sub)sections in all .rst files have complete, readable, usable content.
    • Ideally, there should have been some interaction with the documentation group about any suggested edits and enhancements
  • RC2: Release notes
    • Projects must provide release notes as .rst pushed to integration (or locally in the project's repository if the tooling is developed)


Milestones, Release Candidates, and Service Releases

  • Milestones are spaced roughly 4 weeks apart taking into account significant holidays.
  • Release Candidates (RC) are spaced 2 weeks apart
  • Service Releases are roughly 6, 12, 24, and 36 weeks after the Formal Nitrogen Release and are intended to continue at least until the after the next formal release of the OpenDaylight, presumably Nitrogen.

Schedule Framework

This Simultaneous Release plan has been drafted based on the Schedule Framework

Schedule

2The deadline to meet and report the results of each milestone is at 23:59:59 UTC on the listed day. That corresponds to 4p or 5p pacific time.

Milestone Offset 0 Date2 Offset 1 Date2 Offset 2 Date2 Events
M0/M1 6/7/2017 6/14/2017 6/21/2017

Nitrogen Simultaneous Release Open

  1. Contact Freeze
    • Projects must have declared intent to participate in Simultaneous Release
    • Projects must have elected their Project Leads and specify a Test Contact
    • Participating Projects must have published a candidate Release Plan for public comment ( Release Plan Template )
  2. Note: the date for M0 will be at least one day after the TSC approves the Nitrogen release plan.
last call for project proposals 6/14/2017 6/21/2017 6/28/2017
  1. This is the latest date a project proposal can be sent to the project-proposals list and still have the required two week public comment period before its project creation review at the last TSC meeting before the M2/M3/M4 milestone. Project proposals submitted after this date will not be able to become formal projects by M2/M3/M4 and thus will not be able to participate in the Nitrogen release.3
M2/M3/M4 6/28/2017 7/7/2017 7/14/2017

API Freeze: See more information in the definition above.

Note that the release plan includes details about negotiating inter-project dependencies, expectations, and incompatibilities.

  1. Plan Freeze
    • Participating Projects must have declared their final Release Plan with all sections fully completed.
    • Projects that need extra configuration or resources other than those availble in the OpenDaylight CI infrastructure must have opened helpdesk tickets to add them.
    • Project Checklist completed (for all projects, not just new ones).
    • Projects may apply for a system test waiver if they think they have top-level features not requiring system test or covered by other top-level features test.
    • Projects must specify whether they plan to use OpenDaylight CI infrastructure for system test. It is recommended to use the OpenDaylight CI infrastructure unless there is some HW or SW resource that cannot be installed there. Projects running system test in external Labs are required to report system test results in a timely fashion after release creations, e.g., weekly, RC, and formal releases.
    • Project must get acknowledged from all projects that it depends on.
  2. Feature/Functionality Freeze
    • Final list of externally consumable APIs defined and documented
      • Projects must state for each TENTATIVE API they have (if any) whether they are formally planning to deliver it.
        • If so, it should be noted that it will be delivered.
        • If not projects requesting the API must be informed so that they can take corrective actions.
      • Externally consumable APIs are available at beta-quality
    • All inter-project dependencies are resolved (all project functionality is declared as either "In" or "Out" of this release)
    • Karaf Features defined
      • Instructions can be found in the Karaf:Step by Step Guide
        • Each feature should be tested in every appropriate jenkins job (at least -verify, -merge, and -integration) using the "SingleFeatureTest" as defined in the Karaf step-by-step guide
      • Any feature repositories containing features intended for release must be added to the main features.xml file in the integration git repository as explained in the Karaf step-by-step guide
        • Projects must have a distribution job to verify changes in code do not impact the integration distribution (this will be automatically setup by the releng/builder group).
      • Features that are intended to be "top-level", "user-facing" and/or "stable" must be called out in the milestone readout. These features will have additional requirements:
      • Changing the name of a Karaf feature or removing a Karaf feature should be handled via an API freeze waiver after this point
    • Documentation Started
      • Identified the kinds of documentation to be provided, created AsciiDoc files for them with outlines, and committed those files in an appropriate location. (See the documentation requirements section above for more details.)
    • Feature Test Started
  3. API Freeze: See more information in the definition above.
    • Documentation:
      • Project readouts MUST include a word count of each relevant .adoc file with a goal of draft documentation done.
      • Projects must have a maven site with automatically generated Javadoc per instructions from the RelEng/builder project
    • Projects are encouraged to meet the requirements to be included in maven central
      • Project readout MUST include whether or not this was accomplished
    • Feature Test Continues
      • Participating projects Projects must have all extra SW configuration and resources required for system test installed in the OpenDaylight CI4. More information in How To Install SW in CI.
M5 7/28/2017 8/7/2017 8/14/2017

Code Freeze (bug fixes only from here as defined above)

  1. Stability branch, i.e., stable/nitrogen, must be cut and local project versions bumped on master to avoid overwriting Nitrogen SNAPSHOTS
  2. String Freeze (all externally visible strings frozen to allow for translation & documentation)
  3. Documentation Complete: Only editing and and enhancing should take place after this point.
  4. Feature Test Complete
    • Stable features should fulfill quality requirements listed in definitions section
    • Projects must run at least one basic automated system test job for each top-level feature and several automated system test jobs including functionality, cluster, scalability, performance, longevity/stability for each stable feature4.
RC0 8/14/2017 N/A N/A
  1. The build for RC1 will start at 23:59:59 UTC
    • At the start of the build for RC0, all projects must be int eh distribution and autorelease.
  2. During the RC process, regular, e.g., daily, IRC meetings will take place to identify and address issues
  3. During the RC process, blocking bugs will be tracked in bugzilla and a common spreadsheet
RC1 8/21/2017 N/A N/A
  1. The build for RC1 will start at 23:59:59 UTC
    • At the start of the build for RC1, all stable/nitrogen branches will be locked and only release engineering staff will be able to merge patches.
  2. During the RC process, regular, e.g., daily, IRC meetings will take place to identify and address issues
  3. During the RC process, blocking bugs will be tracked in bugzilla and a common spreadsheet
RC2 8/28/2017 N/A N/A
  1. The build for RC2 will start at 23:59:59 UTC
    • At the start of the build for RC2, the release engineering staff will only merge patches that fix blocking bugs. All stable/nitrogen branches will remain locked and only release engineering staff will be able to merge patches and will only do so for patches that fix blocking bugs.
  2. During the RC process, regular, e.g., daily, IRC meetings will take place to identify and address issues
  3. During the RC process, blocking bugs will be tracked in bugzilla and a common spreadsheet
RC3 9/3/2017 N/A N/A
  1. Participating Projects must hold their Release Reviews, including User Facing Documentation.
  2. The version suffix for RC3 will be -Nitrogen instead of -Nitrogen-RCX or -Nitrogen-RCX-<timestamp> so that the build can be released if it is found to be free of blocking bugs.
  3. The build for RC3 will start at 23:59:59 UTC
  4. All stable/nitrogen branches will remain locked and only release engineering staff will be able to merge patches and will only do so for patches that fix blocking bugs.
  5. During the RC process, regular, e.g., daily, IRC meetings will take place to identify and address issues
  6. During the RC process, blocking bugs will be tracked in bugzilla and a common spreadsheet
Formal Nitrogen Release 9/7/2017 (actual: 9/26/2017) N/A N/A
  1. Formal Nitrogen Release
    • NOTE: The build to produce the formal release artifacts is likely to occur before 5/25/2016.
  2. After the release, except for projects that have opted-out, the release engineering staff will apply the release patch to the stable/nitrogen branch and bump versions.
    • Note: Any patches merged to stable/nitrogen after the auto-release build that produces the formal release artifacts, but before the release patch and version bumps are applied will have to be reverted and re-applied after the release and version bump patches. This shouldn't happen in Nitrogen as the stable/nitrogen branches will have been locked since RC0.
SR1 (Service Release 1 aka Nitrogen.1) 10/7/2017 (revised: 10/26/2017, actual: TBD) N/A N/A
  1. First Service Release for Nitrogen. NOTE: This date is provisional, but will not move earlier. Please note, event based Updates (security/critical bugs) are distinct and may occur at any point.
    • To allow time for testing, a release candidate will be built before the service release and projects are expected to not merge patches except for blocking bugs between that time and the actual service release.
    • Blocking bugs will be tracked via bugzilla and a spreadsheet.
  2. After the release, projects MUST apply the release patch to the stable/nitrogen branch and bump versions. Unless a project opts out, this will be done automatically by the release team after the release.
    • Note: Any patches merged to stable/nitrogen after the auto-release build that produces the formal release artifacts, but before the release patch and version bumps are applied will have to be reverted and re-applied after the release and version bump patches.
SR2 (Service Release 2 aka Nitrogen.2) 12/7/2017 N/A N/A
  1. Second Service Release for Nitrogen. NOTE: This date is provisional, but will not move earlier. Please note, event based Updates (security/critical bugs) are distinct and may occur at any point.
    • To allow time for testing, a release candidate will be built before the service release and projects are expected to not merge patches except for blocking bugs between that time and the actual service release.
    • Blocking bugs will be tracked via bugzilla and a spreadsheet.
  2. After the release, projects MUST apply the release patch to the stable/nitrogen branch and bump versions. Unless a project opts out, this will be done automatically by the release team after the release.
    • Note: Any patches merged to stable/nitrogen after the auto-release build that produces the formal release artifacts, but before the release patch and version bumps are applied will have to be reverted and re-applied after the release and version bump patches.
SR3 (Service Release 3 aka Nitrogen.3) 2/7/2018 N/A N/A
  1. Third Service Release for Nitrogen. NOTE: This date is provisional, but will not move earlier. Please note, event based Updates (security/critical bugs) are distinct and may occur at any point.
    • To allow time for testing, a release candidate will be built before the service release and projects are expected to not merge patches except for blocking bugs between that time and the actual service release.
    • Blocking bugs will be tracked via bugzilla and a spreadsheet.
  2. After the release, projects MUST apply the release patch to the stable/nitrogen branch and bump versions. Unless a project opts out, this will be done automatically by the release team after the release.
    • Note: Any patches merged to stable/nitrogen after the auto-release build that produces the formal release artifacts, but before the release patch and version bumps are applied will have to be reverted and re-applied after the release and version bump patches.
SR4 (Service Release 4 aka Nitrogen.4) 3/21-5/7 N/A N/A
  1. Fourth Service Release for Nitrogen. NOTE: This date is provisional, but will not move earlier. Please note, event based Updates (security/critical bugs) are distinct and may occur at any point.
    • To allow time for testing, a release candidate will be built before the service release and projects are expected to not merge patches except for blocking bugs between that time and the actual service release.
    • Blocking bugs will be tracked via bugzilla and a spreadsheet.
  2. After the release, projects MUST apply the release patch to the stable/nitrogen branch and bump versions. Unless a project opts out, this will be done automatically by the release team after the release.
    • Note: Any patches merged to stable/nitrogen after the auto-release build that produces the formal release artifacts, but before the release patch and version bumps are applied will have to be reverted and re-applied after the release and version bump patches.

3Please note that the TSC reserves the right to allow projects to enter the Simultaneous Release for a reasonable period of time after the M1 date. For example, the TSC may allow additional time if a project is delayed by the IPR Review process.

4Projects running system tests outside the OpenDaylight CI infrastructure are not required to run system tests and report the results on "-merge" and "-integration" Jenkins jobs, although if they can this is ideal. They are required to report system test results in a timely fashion after release creations, e.g., weekly, RC, and formal releases.

Please also note that projects that would like to spin out parts of themselves into additional projects may have those new projects join the Simultaneous Release at any point prior to M3 provided:

  1. The TSC has been informed of this intent prior to M3
  2. The original project's release Release Plan is apportioned between the original and new projects with no parts missing
  3. The new projects have been proposed and approved by the TSC into one of the non-proposed life-cycle states in the normal manner by M3
  4. The new projects have completed the requirements for all milestones before they joined the release, e.g., M1 and/or M2

Lastly, note that as the new projects are joining the release prior to M3, they must meet all the requirements for M3 at the normal time.

Participating Projects

The list of participating projects can be found on the Nitrogen Tracking Spreadsheet

Offset 0 Projects

Project table with release plan, main page, and PTL/Project/Test/Doc contact information can be found on the Nitrogen Tracking Spreadsheet

Offset 1 Projects

Project table with release plan, main page, and PTL/Project/Test/Doc contact information can be found on the Nitrogen Tracking Spreadsheet

Offset 2 Projects

Project table with release plan, main page, and PTL/Project/Test/Doc contact information can be found on the Nitrogen Tracking Spreadsheet

Project Status

The status of projects is being tracked on the Nitrogen Tracking Spreadsheet

RC Download


SR1 Download

SR2 Download

  • TBD

SR3 Download

  • TBD

SR4 Download

  • TBD

Release Reviews

  • TBD

Project Dependency Diagram

  • TBD

Communication Channels

Mailing List

The release mailing list (release@lists.opendaylight.org) is the formal channel for communication about the Simultaneous Release.

Please limit mail to this list to things that directly concern the release as our goal is to keep it's volume at a level that allows the project lead/contact to read all of it.

Per-project Simultaneous Release Contact

Each project participating in the Simultaneous Release should designate a committer to be the contact for that project for that Simultaneous Release. It is expected that this be the project lead for most projects. Even though a primary contact other than the project lead can be designated, the project lead is still expected to be ultimately responsible for the project's participation in the release.

Cross Project Milestone and Release Candidate Reporting

At each milestone, each project is expected to send a readout to the release mailing list by 23:59:59 UTC on the date listed for the given milestone and offset. Most information will be reported via the release tracking spreadsheet, which can be found in the supporting documents section. While most information will be reported via the spreadsheet, projects should still send a mail indicating the information has been filled in, reporting any extra information, and possibly asking additional questions. Reported information will include things like links to gerrit patches, pointers to continuous integration Jenkins Jobs, and the like.

Negative statuses should be reported promptly. If a project is under threat of, or does miss an element on its release plan, the project contact/lead should report this as soon as it is known. They should not wait until the next milestone's readout.

It is the responsibility of each project's lead to report both positive and negative statuses. While they can delegate the task, the project lead is still ultimately responsible for the project's participation in the release.

Simultaneous Release Developer Meetings

One week prior to each Milestone or Release Candidate starting at M1, an IRC meeting for developer interested in the Simultaneous Release will be organized for real time coordination and check in. The Project for each project (or their delegate) should minimally be in attendance. This meeting should happen for each offset at each milestone.

The meeting will be held in #opendaylight-meeting on freenode. You can use an IRC client of your choice or the freenode web client if it is easier.

Bugs

Bugzilla is used to track all bugs in OpenDaylight. Bugs must be filed for the appropriate project. General guidelines and sample searches can be found on the OpenDaylight Bugs page.

During the release candidate process, all blocking bugs must be both logged on a bug-tracking spreadsheet (to be provided) and filed appropriately, e.g., with severity set to BLOCKING, in Bugzilla.

Weather Page

To track current ongoing issues and upcoming possible issues, it's worth checking and updating the Weather page since it provides an easier to find location for ongoing things than the mailing list.

Cross Project Meetings

Supporting Documents

Milestone Readout Templates

Note that any deliverable missed in a previous milestone should be reported on in all subsequent milestone readouts until the deliverable is completed. Also note that additional questions may be added if we need to gather it from all projects, e.g., in Lithium we asked about issues with the migration to Karaf 3.0.3.

M0: Declare Intent

(Project Name)
1. A statement to the effect: "The <Project Name> project formally joins the OpenDaylight Nitrogen Simultaneous Release and agrees to the activities and timeline documented on the Nitrogen Release Plan Page:https://wiki.opendaylight.org/view/Simultaneous_Release:Nitrogen_Release_Plan"
2. Project Offset: (Offset 0/Offset 1/Offset 2)
3. Project Category: (Kernel/Protocol/Services/Application/Support)
4. Project Labels: (List keywords and tags and fit the description of your project comma separated) 
5. Project PTL: (name/email/IRC)
6. Do you confirm that the list of Project Committers is updated and accurate? (Yes/No)

[1] https://wiki.opendaylight.org/view/Simultaneous_Release:Nitrogen_Release_Plan#M0:_Declare_Intent

M1: Draft Plan

(Project Name)
1. Project Lead Contact: (name/email/IRC)
   Review PTL Requirements [1].
2. Project Contact: (name/email/IRC)
3. Test Contact: (name/email/IRC)
4. Documentation Contact (name/email/IRC)
5. Draft Release Plan: (wiki link)

** FOR NEW PROJECTS ONLY **
6. Project Main Page: (wiki link)
   Use Project Facts Template [2]. 

[1] Be sure to read the responsibilities of being a project lead under Leadership & Communication in the Requirements for Participation section of the release plan: https://wiki.opendaylight.org/view/Simultaneous_Release:Nitrogen_Release_Plan#Requirements_for_Participation
[2] https://wiki.opendaylight.org/view/Template:Project_Facts

M2: Final Release Plan

(Project Name)
1. Does your project have any updates on any previously-incomplete items from prior milestone readouts?  (Yes/No)
   (If yes, list updates)
2. Were project-specific deliverables planned for this milestone delivered successfully? (No Deliverables/Yes/No)
   (If no, list incomplete deliverables)
3. Does your project have any special needs in CI Infrastructure [2]?  (Yes/No)
   (If yes, link to helpdesk ticket number)
4. Is your project release plan finalized?  (Yes/No)
   (If yes, link to final release plan wiki page)
   (If no, ETA to finalize release plan)
5. Do you have all APIs intended to be externally consumable listed? (Yes/No)
   Does each API have a useful short name? (Yes/No)
   Are the Java interface and/or YANG files listed for each API? (Yes/No)
   Are they labeled as tentative, provisional, or stable as appropriate for each API? (Yes/No)
   Do you call out the OSGi bundles and/or Karaf features providing the API for each API? (Yes/No)
6. Have all project dependencies requests on other project's release plans been acknowledged and documented by upstream projects?  (Yes/No)
   (List of all project dependencies and if they have been acknowledged, unacknowledged)
7. Will your project have top-level features not requiring system test? (Yes/No)
   (If yes, link to system test waiver request email) 
8. Will your project use the OpenDaylight CI infrastructure for testing top-level features requiring system test? (Yes/No)
   (If no, link to system test plan explaining why [3])
   (If no, link to system test plan identifying external lab testing [4])

** FOR NEW PROJECTS ONLY **
9. Have you completed the project checklist [1]? (Yes/No)
   (link to a merged patch in gerrit)
   (link to a mail from your mailing list)
   (link to a bug for your project; you can create a dummy one and close it if need be)
   (link to an artifact published from your project in nexus)
   (link to a sonar report)
   (link to your root pom file)

[0] https://wiki.opendaylight.org/view/Simultaneous_Release:Nitrogen_Release_Plan
[1] https://wiki.opendaylight.org/view/GettingStarted:Project_Main#New_Project_Checklist
[2] Special needs include tools or configuration.  Note that generally, the only available tools in CI are basic RHEL/CentOS linux images with Java. You should note and ask for anything beyond that here.  Email helpdesk@opendaylight.org
[3] It is recommended to use the OpenDaylight CI infrastructure unless there is some HW or SW resource that cannot be installed there.  Update the test plan with explanation on why your top-level features will not be using the OpenDaylight CI Infrastructure: https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Feature_Integration_System_Test_Template#Test_Infrastructure
[4] Projects running system test in external Labs are required to report system test results in a timely fashion after release creations, e.g., weekly, RC, and formal releases.  Update the test plan with plans on testing in external lab: https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Feature_Integration_System_Test_Template#Test_Infrastructure

M3: Functionality Freeze

<Project Name>

Please provide updates on any previously-incomplete items from prior milestone readouts.

Functionality Freeze:

  1. Final list of externally consumable APIs defined: Yes/No
    • If you had an Tentative APIs, have they been moved to Provisional or dropped? Yes/No (link to release plan)
    • If any of your Tentative APIs were dropped, have you notified all projects that were expecting them? Yes/No (link to e-mail)
      • Also please list all dropped APIs.
  2. Are all your inter-project dependencies are resolved (i.e., have the other projects you were counting on given you what you needed)? Yes/No
    • If no, please list the features you were expecting that haven't been delivered and the project you were expecting to receive them from.
    • Note that you can only reasonably hold a a project to something if you formally asked for it during the release planning process and they acknowledged that ask saying they would do it.
  3. Were there any project-specific deliverables planned for this milestone? Yes/No
    • If so, were they delivered? Yes/No

Karaf Features Defined:

  1. Are all your project's features that are intended for release added to the features.xml and checked into integration git repository. Yes/No (please provide link to the gerrit patch)
  2. List all top-level, user-facing, and stable Karaf features for your project.
    • For top-level and user-facing features, please provide a one-sentence description which a developer and/or user would find helpful.

Documentation:

  1. List the kinds of documentation you will provide including at least:
    • One user/operator guide section per user-facing feature.
    • One developer guide per top-level feature.
    • An installation guide for any top-level features that require more than feature:install <feature-name> to install.
    • Optional tutorials and how tos.
  2. Have you checked in an AsciiDoc outline to the docs repository? Yes/No (link to gerrit patch)

Integration and Test:

  1. Have you started automated system testing for your top-level features. Yes/No
    • If yes, link to test report
    • If no, why?
  2. Have you filled out basic system test plan template for each top-level feature (karaf and not karaf) and a comprehensive system test plan template including functionality, cluster, scalability, performance, longevity/stability for each stable feature? Yes/No
    • If yes, link to test plans
    • If no, why?

Project Specific:

  1. Were there any project-specific deliverables planned for this milestone? Yes/No
    • If so, were they delivered? Yes/No
  2. Have you updated your project facts with the project type category? Yes/No
  3. Do you acknowledge the changes to the RC Blocking Bug Policy for Nitrogen Release [1]? Yes/No

[1] https://lists.opendaylight.org/pipermail/tsc/2016-December/006468.html

M4: API Freeze

<Project Name>

  1. Please provide updates on any previously-incomplete items from prior milestone readouts.
  2. Has your project achieved API freeze such that all externally accessible Stable or Provisional APIs will not be modified after now? (Yes/No)
    • (Link to gerrit search for patches modifying the API [1])
  3. Do you have content in your project documentation? (Yes/No)
    • (For each document, provide current word count)
    • (For each document, link to the file in gerrit)
    • (Link to pending gerrit patches waiting approval)
  4. Has your project met the requirements to be included in Maven Central [2]? (Yes/No)
  5. Were project-specific deliverables planned for this milestone delivered successfully? (No Deliverables/Yes/No)
  6. Have you started automated system testing for your top-level features. (Yes/No)
    • (If yes, link to test report)
    • (If no, explain why)
  7. Does your project use any ports, including for testing? (Yes/No)
    • (If yes, list of ports used)
    • (If yes, have you updated the wiki [3] with all ports used? Yes/No)
  8. Does your project build successful in Autorelease?
    • (If yes, link to successful autorelease job [4])
    • (If not, explain why)


[1] Provide a link to a gerrit search for patches modifying the files defined as specifying the API. For example: https://git.opendaylight.org/gerrit/#/q/file:%255Eopendaylight/md-sal/sal-binding-api/.%252B+status:merged+project:controller

[2] http://central.sonatype.org/pages/requirements.html

[3] https://wiki.opendaylight.org/view/Ports

[4] https://wiki.opendaylight.org/view/RelEng/Autorelease/Project_Autorelease_Requirements

M5: Code Freeze

<Project Name>

  1. Please provide updates on any previously-incomplete items from prior milestone readouts.
  2. Has your project met code freeze, i.e., only bug fixes are allowed from now on? (Yes/No)
  3. Are all externally visible strings frozen to allow for translation & documentation? (Yes/No)
  4. Is your documentation complete such that only editing and enhancing should take place after this point? (Yes/No)
    • (For each document, link to the file in gerrit)
    • (Link to pending gerrit patches waiting approval)
  5. Were project-specific deliverables planned for this milestone delivered successfully? (No Deliverables/Yes/No)
  6. Are you running at least one basic automated system test job for each top-level feature? (Yes/No)
    • (If yes, link to test report)
    • (If not, explain why)
  7. Do you have any CLM violations?
    • (If yes, list your CLM violations and explain why each violation can be exempt)


Stables Features (Only for Projects with Stable Features)

  1. Do your stable features fulfill quality requirements (i.e. unit and/or integration test coverage of at least 75%)? (Yes/No)
    • (If yes, link to sonar report)
    • (If not, explain why)
  2. Are you running several automated system test jobs including functionality, cluster, scalability, performance, longevity/stability for each stable feature? (Yes/No)
    • (If yes, link to test reports)
    • (If not, explain why)

RCX: Release Candidate Testing

<Project Name>

  1. Have you tested your code in the release candidate? Yes/No (provide a link to the release candidate you tested)
    • If yes, did you find any issues?
    • If you found issues, do you believe any of them should block this release of OpenDaylight until they are resolved?
    • Please list all the issues and note if they are blocking.

Lessons learned during the Nitrogen release

  • TBD

Lessons learned during the Carbon release

  • Clustering breakage, see 6.b-d in the 3/24/17 TSC meeting minutes
    • Need more details and to figure out if there are lesson's learned beyond "it took 2 days to get a response and it hurt our jenkins queue" a lot.
  • An accelerated release plan helps new projects, as they often need more time to ramp up on technology, process, and setup. Recommend that the simultaneous release plan be approved earlier than M0 timeframe.

Lessons learned during the Beryllium release

  • Do we need enhanced requirements to be in a release and when to meet them? I don't think that we really need new requirements as much as we should probably start to more rigorously enforce the ones we have—predominantly around projects being responsive.
  • If you're planning to develop major new functionality or replace the implementation of major current functionality, be cognizant of your downstream consumers
    • have a migration plan for how people will start using the new functionality (or implementation)
    • understand how to help them deal with bugs as they come up
    • if you will have two different simultaneous implementations, make they can both be run at the same time
      • or ideally, don't do this, make sure everyone can make the jump to the new implementation
  • upgradability is critical, there's no sane way to do this without
    1. a per-project requirement to be upgradableThis is already baked into the concept of stable features, which are probably the right lever for this now.
    2. tests to confirm that this actually works
  • there have been requests to have upgrades work with hops between versions, e.g., Helium => Boron
  • we need stronger testing for upgradeablity between SRs
  • projects should have only stable tests, if we need to ask them about test failures consistently, they need to:
    • fix it
    • have a well known and clearly visible (in the test automation) bug for it
    • remove itAdded an extra bullet to testing requirements noting that system tests are expected to reliably pass.
  • projects should be required to generate a maven site with javadoc, and ideally REST API doc generated from YANG models
    • ideally this should be done in a way that autorelease can generate one big maven site added as a documentation requirement at M4 with instructions.
  • projects don't do system test even when they say they do
  • System Test is not very visible
  • We need to update release reviews and release notes to make us have to ask fewer questions
    • features (with links to the features.xml file and the right line for each feature to avoid listing non-existent features)
      • if there are any experimental features
      • user-facing features (with a clear definition of user-facing)
    • migration
    • compatibility (which is different from migration)
    • documentation (both wiki and AsciiDoc)
    • Ideally also add the wiki template to produce lists of release notes, release plans, and release reviews
  • It would be good to have documentation peer-review in a structured way
    • Maybe pairing up projects (or even forming small groups, e.g., 3 projects that have to do cross-review) Added to the documentation requirements that the documentation team will ask for projects to do peer review and preferentially review and merge patches that have been peer reviewed.
  • We need to start testing earlier, the observation seems to be that we don't start testing in earnest until RC0 (or sometimes RC1), which leaves 2-3 weeks for testing and fixing bugs.
    • How to do this isn't exactly clear. Some people call for having a longer RC time-frame. Others just say that we should be stricter and more disciplined with hitting functionality freeze (M3) and code freeze (M5)
  • We need more formalism around how to ship things that aren't Karaf features, e.g., NeXt, OpFlex and some of VTN
    • It seems like learning from VTN which is distributed inside the Karaf zip file might be a good thing
    • If not, we need to understand where the extra downloads will be listed on the downlaod page, how many there will be, etc.
  • We need minimum expectations for projects to be in and stay in autorelease (and thus be part of the release), e.g.,
    • respond to failed builds in a timely fashion This is already in the requirements for participation, we can just start enforcing it more.
  • Should we require projects to meet the minimum requirements for inclusion in maven central?
    • for everyone?
    • for stable features?

Lessons from Hydrogen, Helium and Lithium for future releases

Items that are struck out have been addressed. A comment will follow as to how it was addressed in blue if it seems like the right approach and in red if there may still be debate or questions. Similarly, comments in goldenrod note things that still need to be addressed, but are not blocking certifying the release plan.

  • Do we want to require system tests for all top-level features (and thus one per project) instead of just user-facing features? done, it's for top-level features
  • The release notes and release review templates need to be revised
    • Likely combined and to use a wiki template to make them more uniform and easier to suck out For now, the release notes have been updated to be more clearly per-project. The release review template is specified by the lifecycle document and so has been left the same and separate.
  • We need to track items that weren't green from previous milestone readouts so that they get reported in subsequent ones until they are green
    • A key example is getting features into integration
  • We need to require project feedback on RCs as part of the release plan.
  • Service releases should likely continue until some future release (either one or two releases in the future) rather than after a fixed number of releases. we've moved to four SRs with the note that the should go for at least one release in the future
    • How long after the new release do we wait?
    • Do we want to have a specified amount of overlap? 2 weeks? 6 weeks?
  • We desperately need pre-made templates for each milestone that make verifying requirements easy. We just missed things in Lithium M1 to M3 without that.
  • We need to mandate source jars generated in a canonical way, i.e., to be consumed by releng/autorelease.
  • We should also mandate javadoc generated in a canonical way, i.e., to be consumed by releng/autorelease. Thanh says releng is already doing this both in JJB for projects and auto-release.
  • We need to mandate that no projects can pull in 3rd-party dependencies without source jars.
  • Migration migration and backward compatibility requirements have been put out as part of mature projects and/or stable features, but could maybe use to be made more concrete
    • Do we want to require data schema translations?
    • Other issues?
  • The paperwork for M3 was substantial (and not easy see in advance) and should be streamlined or spread out to dos have been added above to copy documentation and integration requirements into this document to keep them from sneaking up on people
    • Consolidating all the requirements into one place would likely help.
  • We should make sure that people know where to produce and document known issues. In general, it's three places:
    • The release mailing list.
    • The Weather page.
    • The weekly IRC sync during the last part of the release. This is called out in #Communication Channels and any more forceful call out will likely have to happen not in this document
  • Make it clear what is expected of projects in terms of tracking what's going on in ODL.
    • Reading the release list.
    • Reading tsc list or at least the TSC meeting minutes.
    • Attending release IRC meetings or at least reading the minutes. This is called out in the Communication & Leadership requirement above and that is pointed out in the M1 milestone readout template
  • Adding a way to deal with docs-like projects that don't provide code-level negative interactions
    • This includes at least docs, toolkit (now mostly defunct), and coretutorials
    • Ideally, they might have laxer requirements for now, I think leaving these projects as leaf projects with minimal requirements seems reasonable
  • Deal with cutting branches and version bumps with offsets TODO: for now we've left the exact timing of branch cutting open to sometime between offset 0 M5 and offset 2 M5 as well as leaving open whether we'll do a synchronized cut or a cut staggered by offset.
    • If we cut branches and version bump at the same time, the only issue we have is slow projects that can hold things up, which has been fixed by automated version bumping that should happen for Helium.
    • However, if we cut branches and version bump at M5, there is an offset between different projects where upstream projects can merge patches that break downstream projects (regardless of whether the breakage is a bug in the upstream or downstream project) causing the version bump for downstream projects to fail.
    • The options seem to be:
      1. cut branches all at the same time at or near M5-offset2 or RC0, which has the disadvantage that offset 0 and offset 1 projects have somewhere between a 2-week and 6-week period after code freeze where they can't add new features.
      2. cut branches all at the same time at M5-offset0, which has the disadvantage that offset 1 and offset 2 projects will have to "double merge" patches to both branches during times that would normally have heavy coding.
      3. figure out how to deal with the issue that offset 1 and offset 2 projects may get hit with incompatibilities on version bumps
    • Also, cascading tests during staggered branch cutting breaks because the way JJB is set up right now it's not possible to trigger jobs across branches even when logically master of an offset 1 project is dependent on stable/lithium of an offset 0 project.
  • Docs improvements
    • We'd really like to be able to put project-specific docs in their repo
      • This will allow us to easier iterate on docs as patches go in, and allow committers to +2/-2 doc changes. (alagalah)
      • This should include the ability to directly pull code fragments from real code
    • We'd really like HTML versions of the docs that aren't so fragmented This will be folded in to the docs project's Nitrogen release plan

Lessons from Hydrogen/Helium that Should be Applied

Items that are struck out have been addressed. A comment will follow as to how it was addressed in blue if it seems like the right approach and in red if there may still be debate or questions. Similarly, comments in goldenrod note things that still need to be addressed, but are not blocking certifying the release plan.

  • The Release plan doesn't take into account project dependencies. e.g. M4 API Freeze. If a project is waiting on API freeze for a project it is dependent on, then that reduces the amount of time the "dependee" has to execute. - alagalah (Keith) Done, mainly by moving deadlines up by one step, e.g., M2 component free, M3 API freeze, M4 code freeze
    • We had offsets in Hydrogen, spaced at 2 days. We need 2-3 weeks between offsets for them to make sense,
    • With 6 offsets 2 weeks each we need additional 10 weeks to reach RC0 on all projects,
      • Can we can do it in 3 offsets: +0, +2 weeks, +4 weeks
        1. odlparent, yangtools, and controller
        2. openflowjava, openflowplugin, ovsdb, aaa
        3. everyone else
    • Which means lower-offset projects can (and need) to start their next-release while the SR process is finishing
  • We need a Feature Freeze milestone before the API freeze
    • It should occur at M3 with beta-quality APIs, so downstream projects can start consuming Currently at M2 instead, it will be ~M2.5 and ~M3 for most projects
  • We're using release@lists.opendaylight.org instead of discuss
  • We should make it easy for projects to convey and understand what APIs they are intending to make available vs. which ones are intended to be internal attempted as part of component/API freeze
  • We should make it clear that participation in Service Releases is not optional done, see #Requirements for Participation
  • We should make it clear what we expect in terms of timely responses from project primary contacts for a release done, see #Requirements for Participation
    • This involves identifying what mails that people should pay attention to, e.g., ones sent to release@lists.opendaylight.org with "PLEASE RESPOND" in the subject
    • It also involves identifying a time frame in which they should respond, e.g., two business days
      • One concrete stab at making this formal would be: "Technically, two business days will be defined as 48 hours not counting 2a UTC on Friday until 2a UTC on Sunday. This corresponds to 48 hours starting at 4p on Friday in the furthest ahead time zone (UTC+14). Note that this means if you want a response *this* week, you must send it before 2a UTC on Wednesday. That’s 6/7p pacific time on Tuesday in the Pacific time zone." The formal definition is currently left out
  • We need a longer time between code freeze and release candidates because developers don't focus on tests (especially system and integration tests) until after code freeze
  • Status reports for each milestone should include more than a Boolean for tests
  • We need to make it clear what tasks need to be done for docs, where and when handled by the deliverable from M2 from docs
    • Understanding the kinds of documentation we want to generate and who the audience is for each kind is going to be critical
      • e.g., one person's user is likely another's developer
    • The same is true about tests. handled by the deliverable from M2 from integration
  • We really need somebody who groks the things that need to be accomplished at each milestone and can take a glance at the code and jenkins jobs for each project to get an idea of whether they're on track or not. We need to make sure we do this for deadlines M3 and later, e.g., functionality freeze, karaf features defined, API freeze, and code freezeM milestone readout templates make this more doable
  • Requirements to meet at different stages (and especially RCs) should be set and enforced with clearly explained consequences for missing them ways to fix missed deadlines are now discussed by the TSC for offset zero and offset one projects as described in requirements for participation
    • Release throttle branch needs to be cut at RC0 at the latest done at M5 now
  • We need a standard way to track blocking issues: TODO: we still need this, but it's loosely defined in the Helium blocking bugs section here.
    • One suggestion is to treat them as bugs in bugzilla for easy tracking and querying
      • Projects would file bugs with severity as "critical", "blocker" with the target milestone being appropriate
      • Appropriate milestones are sometimes annoying, but generally, it should be "anything but the next release"
  • We need to pre-declare when RCs and final release artifacts will be cut (both dates and times for clarity) done at M5 and RCs
  • Need to add an EOL-plans section to release plan to understand user impact of EOLed features/components/APIs at the start of a development cycle done in release plan
    • What requirements do we want to place on projects? e.g., deprecated in one release and can remove in another?
    • plans for dealing with EOLed features should be incorporated into the release plan
  • We should reconsider when we set a release date done, there is a month of slack between the release and the ODL summit and the dates for RCs after RC0 and the formal release are stated to be tentative based on testing in the definition of RCs
    • Especially to the press, but also in other environments
    • For example, do we want to have a booked event giving us effectively zero wiggle room on the back end?
      • Maybe, because hard deadlines help get things done, but they also make for sub-optimal
  • We could use more automated release processes
    • For example, the auto-release is really, really nice as compared to spending 14+ hours on IRC cutting everything.
    • A similar process for post-release branch cutting and version bumping would be very helpful, e.g., take a 10+ day process and turn it into one that takes a few hours.
    • One problem is figuring out how to do this w/o requiring involvement from every project (at least on the critical path).
      • Solutions are (i) allowing for some scripts to commit changes to projects, which is likely bad, or (ii) automatically pushing patches for projects to review
      • Another solution is to switch to continuous delivery resolved thanks to Thanh and the autorelease project
  • We should avoid scheduling any major events, e.g., a design forum or summit, immediately after the release so that we can have some room for slippage without having to pull many developers out of the event into a "war" room. done. there is a month between the release and the ODL summit
  • More automated features testing TODO: yes, but this is technical debt, not directly related to the release plan
    • to really test things, we need to blow away the m2 repo before testing every features.xml file
  • Cyclic dependencies TODO: yes, but this is technical debt, not directly related to the release plan
    • We need to decide if we want to allow them, and if so what kind to allow
    • We need to provide documentation (or ideally scripts) that show how to build the code despite the circular dependencies (if we allow them)
    • We need tests to check for circular dependencies (either at all or new ones) so that we know about them
      • The simplest way to do this would be to have an offline auto-release which first clone all the repos and then tried to build them linearly without access to the nexus repos.