efl qa: where are we and where should we go?
TRANSCRIPT
1Samsung Open Source Group
Stefan SchmidtSamsung Open Source Group
QA: Where We Are And Where We Should Go
EFL Developer Day US 2015
2Samsung Open Source Group
Agenda
● Current QA Summary
● Missing QA Parts
● Steps Forward
● Q&A Topics
3Samsung Open Source Group
Current QA Summary
4Samsung Open Source Group
Current: Compile Time Testing
● Developers machine (One arch, make,
hopefully make check and distcheck)
● Machines of other developers
● Jenkins covers different architectures,
different compilers, different
dependencies and versions
5Samsung Open Source Group
Current: Unit Testing
● Test cases collected in test suite and
run with make check
● Should be run before committing
● Also run by Jenkins
● While unit testing is the main idea it also
had catched segfaults due to basic run
time testing of new code
6Samsung Open Source Group
Current: Miscellaneous
● Static Analysis
● Different EFL profiles
● Different configuration options
(Wayland, XCB)
● Different make targets (all, check,
distcheck, doc)
7Samsung Open Source Group
Missing QA Parts
8Samsung Open Source Group
Missing: Runtime Testing
● We rely on the developer testing his code before committing
● We rely on the community of developers and users to run the latest code and
report bugs
● No automation
9Samsung Open Source Group
Missing: Graphical Unit Tests
● No usage of graphical test cases
● Tooling is there: Excatness
● Almost no developer uses it, regularly
● Automation is highly needed
10Samsung Open Source Group
Missing: Code Coverage
● Only 39.2% of our code base covered by unit tests
● Newly added API's should be
mandantory to come with unit tests
● My goal would be that we increase coverage 5-10% per release
● How to motivate unit test writing?
11Samsung Open Source Group
Missing: Pre-push CI
● CI infrastructure only available post-push to master
● The various tooling only runs on code committed
to the master branch of our repos
● Testing the code before this stage could be
beneficial
● Testing developer branches on demand and
hooking it up with Phab to run it on incoming
submissions
12Samsung Open Source Group
Missing: Code Complexity
● More valid combinations for test matrix
– New jobs for xcb and wayland have
just been added
● What other configuration options should be officially supported?
13Samsung Open Source Group
Steps Forward
14Samsung Open Source Group
Steps Forward: Unit Testing
● Proposal: Make test cases for public API
additions (EAPI) mandatory from today
● Aim for 5% coverage increase in every
upcoming release
● Aim for 60 - 70% coverage at the end of
2015
● Not only NULL checks, but valid test cases
15Samsung Open Source Group
Steps Forward: Graphical Testing
● Identify the problems with exactness on
Jenkins
● Find a way to fix or work around them
● Run exactness during the nightlies
● So far screenshots for elementary_tests
and some edje examples.
● What else should be tested?
16Samsung Open Source Group
Steps Forward: API's and Breaks
● Run ABI/API checker every night
● Catch breaks
● Catch new additions
– Summary mail to list to highlight additions and maybe raise discussion if needed. (Shortly after the addition and not weeks later)
– Summary of new additions for release NEWS
– Notifications for manual bindings to catchup
– Check if new APIs have documentation
17Samsung Open Source Group
Steps Forward: Pre-push CI
● Two entry points we should cover:
– Developers: Allow dev branches to be run
through Jenkins
– Contributors: Run incoming phab submissions
through Jenkins and mark them as
passed/failed
● Both could have the same Jenkins jobs at the
backend
18Samsung Open Source Group
Q&A Topics
19Samsung Open Source Group
Q&A Topics
● Mandatory unit tests for new public APIs?
● 5% coverage increase per release a sane goal?
Who would help with this work?
● How to motivate the writing of unit tests?
● Anyone willing to drive the exactness integration
work until the end?
● Is there interest for build hooks for phab
submissions?
20Samsung Open Source Group
Thank you.
22Samsung Open Source Group
Attributions
● Brunoschalch: https://www.flickr.com/photos/bsgphotos/7594903774/in/photostream/
● Katja Schulz: https://www.flickr.com/photos/treegrow/14404553548
● Fernando Flores: https://www.flickr.com/photos/ferjflores/8697860914
● David Brossard: https://www.flickr.com/photos/string_bass_dave/8152835443
● Anne Reeves: https://www.flickr.com/photos/charlock/10677411025
● Andrew: https://www.flickr.com/photos/polandeze/741311758