2013/10/23 - yocto - continuous integration

Yocto Project Experience: Continuous Integration Mark Hatle Senior Member of Technical Staff Wind River

Edinburgh, Scotland 23 Oct 2013

2 Yocto Project | The Linux Foundation

Agenda

•  Our experiences as an OSV, productizing the Yocto Project

•  Software Lifecycle •  Big-Bang Example •  Continuous Integration Example

•  Our recommendation

Yocto Project | The Linux Foundation

Productization

What does it take to turn the Yocto Project into a commercial product?


Yocto Project Productization

•  What does an OSVs customer’s require? •  Up-to-date kernel •  Up-to-date toolchain •  Up-to-date userspace •  One or more specific BSP (hardware support) •  Quality improvements •  Timely support



•  Up-to-date •  When the Yocto Project release is complete, it is generally

considered to be very Up-to-date (nothing older than 6 months)

•  Up-to-date in the customer world is roughly nothing older than one generation, or 12-18 months

•  Toolchain is at the current community supported version •  Kernel is at the generally accepted stable version (LTSI or

otherwise) •  …



•  Hardware Support •  Customers require the hardware of their choice to be

supported. •  Generally new hardware requires newer versions of the

Linux kernel •  Semiconductor specific optimizations for toolchains, drivers

and other components are often required.



•  Quality •  Anything that is released from an OSV needs to be at or

better than Open Source quality •  Requires significant test resources (people and machines)



•  Timely support •  When something doesn’t work, the OSV is expected to be

the expert on the problem! •  The OSV must understand the system as a whole •  The OSV must work with the community to find existing fixes •  The OSV must work with the community suggest new fixes


Software Lifecycle

How to manage the software lifecycle?


Software Lifecycle

•  Yocto Project Lifecycle

•  Commercial Product Lifecycle

•  Real World Examples


Yocto Project Lifecycle

•  6 month development cycle •  4 – 4 week development milestones •  5th milestone is stabilization •  Maintenance releases managed for roughly 1 year


Yocto Project Lifecycle 4 - 4 week development milestones

https://wiki.yoctoproject.org/wiki/Yocto_1.5_Schedule


Commercial Product Lifecycle Examples

Big-Bang Continuous Integration Start with community release Work in parallel with community Add ‘missing’ requirements Influence community work Add new value-add features Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features

QA/Verify OSS

QA/Verify value-add features Work to resolve bugs internally Work with community to fix bugs Release to customers Release to customers Maintain Release (5-10 years) Maintain Release (5-10 years)

Examples assume approx 12-18 month release cycles


Big-Bang Lifecycle Example

•  Big-Bang refers to the work •  Starts with a large amount of community software •  Need to learn how it works •  Learn what required product features need to be

implemented •  More of the traditional approach •  Follow Open Source…



Oct-Dec Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun

Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5 U7

•  No opportunity to influence design, must follow •  Enhancements should be contributed to the next

version, and backported •  Bugs found may or may not have been found by

community, may require additional resources to resolve

Customer Adoption




Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5 U6

•  At start of commercialization, product components are up to 6 months old

•  By the time of release, it’s nearly 12+ months old •  Has a “shelf life” of only 6-12 months from release •  Decision extend shelf-life or uprev?

Customer Adoption



Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun

Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5

Customer Adoption

Commercial

Jul-Sep Oct-Dec Jan-Mar

U1 U2 U3 U4 …

•  Extend shelf life? •  Update kernel, toolchain, BSPs and other critical elements •  Backport additional select features •  6 months work, only gains 6-12 months Customer Adoption •  No community help

Customer Adoption



Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun

Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5

Customer Adoption

Commercial

Jul-Sep Oct-Dec Jan-Mar

U1 U2 U3 U4 … •  Uprev? •  Rebase changes (15-45 days) •  Add commercialization time •  Finish date has now slipped back a bit more •  Still following, not leading…

Customer Adoption

Yocto Project Update Update EOL


Commercial Product Lifecycle Examples

Big-Bang Continuous Integration Start with community release Work in parallel with community Add ‘missing’ requirements Influence community work Add new value-add features Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features

QA/Verify OSS

QA/Verify value-add features Work to resolve bugs internally Work with community to fix bugs Release to customers Release to customers Maintain Release (5-10 years) Maintain Release (5-10 years)

Examples assume approx 12-18 month release cycles


Continuous Integration Lifecycle Example

•  Continuous Integration refers to tracking and contributing to the community development •  Work with the community on development •  Learn capabilities and feature deficits as development

continues •  Ability to influence the community by discussing

requirements and/or providing patches •  Ability to monitor OSS quality over a longer period of time •  Requires resources to follow the community •  Lead with the OSS community




Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5 U6

•  Ability to identify issues and work with the community to resolve them

•  Enhancements can be contributed during development

•  Bugs can be filed with the community and worked on as a group

Customer Adoption

…

Customer Adoption




Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5 U6

•  During commercialization components are current •  By the time of release, it’s only 6-7 months old •  Has a “shelf life” of 12-24 months from release •  No reason to extend the shelf-life!

Customer Adoption

…



Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar

Yocto Project

Commercial

Update Update EOL

U1 U2 U3 U4 U5 U6

Customer Adoption

…


Commercial U1 U2 U3 U4 U5 U6

Customer Adoption


•  Uprev! •  No rebase required… •  Ramp up developers


Real World Examples

•  Big-Bang •  Took a large product team 6 months to commercialize •  Required 6 one month cycles to add enhancements and

backport upstream features •  First cycle was devoted to investigation •  Required significant developer resources •  Release was approx time of next YP release


Real World Examples

•  Big-Bang Extended life •  Required 6 months development to update kernel, BSPs,

toolchain and other customer essential systems •  Required roughly the same development effort as a new

product •  Release occurred at appox time of EOL of the base Yocto

Project release


Real World Examples

•  Big-Bang Uprev •  Requires 45 days (of one engineer) to update the tree 2

Yocto Project releases. This 45 days simply enabled the main development team.

•  Projected to required 6 months of development to commercialize and add new features, QA, etc.

•  Release was now after the next YP release


Real World Examples •  Continuous Integration

•  Work done in parallel with the community. •  Able to ramp up a small team to full team over the course of

development. •  As bugs were found, many filed with the community and

fixed in a timely manner. (Many critical bug fixes were submitted to the community.)

•  As missing features were identified, worked with the community to implement functionality.



•  Required 1 full time resource to manage integration and tracking of the community.

•  Resource was the go-to person for questions about community quality, bug triage, etc.

•  Continuous uprev averaged every 1-2 weeks for the first 4 milestones. Bug fixes and QA took most of the two week time. (Longer than we expected.)

•  Unexpected change in the 5th milestone caused a single 4 week integration cycle.

•  Expect weekly uprevs after product release for point fixes.



•  Estimated to be the same amount of “improvement” and commercialization required

•  Smaller teams required, as less unexpected ‘reactionary’ work was required


Recommendation

What should you do?


Recommendations

•  Semiconductor Mfg – Kernel/BSP support •  ‘big-bang’ – address customers wanting a stable approach •  ‘continuous integration’ – keep changes timely, and ready to

release for the next stable release •  May depend on chip market, release schedules and

customer demand


Recommendations

•  OSV •  Use continuous integration. Quicker time to market, more

up-to-date software, and longer shelf-life. •  Should allow time to sync to semiconductor and customer

requirements.


Recommendations

•  ISV / Application Developers •  Follow the YP versions your customers need. Most likely to

‘follow’ the stable release, but for large applications the continuous-integration model may make sense.


Recommendations

•  Device Developers •  Look at what your needs are. •  If you don’t need ‘work-in-progress’ features, it’s better to

start with a stable release! •  If you expect to be updating the OS over the life of the

product, continuous integration may be useful.

Thank you for your participation!

2013/10/23 - yocto - continuous integration

Engineering