magento marketplace's program marketplace... · test automation improvements foundation for...
TRANSCRIPT
Magento Marketplace's
New Extension Quality
Program
Erika TalbottProduct Manager - Marketplace
J Ravi MenonArchitect - Marketplace
Tom ErskineLead Engineer - MFTF
Extension Quality Program
Quality Extensions.
Rigorously Tested.
"Quality is an asset—not a cost or afterthought; it is insurance that the product will
meet the customer's true needs with uninterrupted delivery and ample performance."
Manual QA Then and Now
2017
700 Extensions
100 Submissions
/Week
2 QA Reviewers
1+ Month Lead Time
70 Submissions Reviewed/
Week
50% Failure Rate
2018
3400 Extensions
130 Submissions
/Week
6 QA Reviewers
~2 Week Lead Time
115 Submissions
Reviewed /Week
48% Failure Rate
Other Notable Achievements
• Implemented two expedited test paths
– Minor updates to Magento core
– Patch version submissions
• Introduced Varnish Cache testing
• Infrastructure improvements on Repo
• Increased communications to the community
• Commerce dev licenses and Cloud sandboxes available for all
We heard you
New EQP (Early Q3)
Parallel submission paths
Consolidated test reports
Improved documentation, contextual help
and tooltips
Extensions remain on store during updates
Improved dashboard and
reporting
Top 5 Complaints
Too SlowCan’t find Test
ReportsUnclear
requirements
Must remove extensions from store to update
Sales data is lacking
Technical and Marketing in Parallel
Consolidated Test Reports
Contextual Help and Tooltips
Improved Dashboard
But wait, there’s more!
Version pulled from Code package
Multiple Development
Lines
Withdraw a Submission
Cancel a Submission
Price changes on the fly
Preview in Product Page
Advanced Security Checks
Test Automation
Improvements
Foundation for MFTF
Responsive Design
Roadmap
New EQP
Support for M2.3
Quality based search
rankings
Authenticity certificates
NOW NEXT FUTURE
EQP automated functional
tests
Accept Developer MFTF
tests with submissions
Recurring billing
Expanded Developer payout
options
Advanced Developer
reports
Public EQP APIs
Merchant notification of
new versions
Technical EnhancementsJ Ravi Menon
Marketplace Architect
@jrmenon
EQP Technical Enhancements
• Support for multiple development lines.
• A new extensible and scalable infrastructure for automated
tests.
• Support for parallelization of technical (automated tests +
manual QA) and marketing tracks of the EQP Pipeline.
• REST API support for the EQP Process.
EQP Technical Enhancements
• Support for multiple development lines for a given extension:
– A gap in the current Developer Portal.
– This will now allow for bug-fixes/patches and feature releases on
any given line identified by the major version number (must follow
semver best practices).
– This allows the developers to support merchants still running their
older versions of extensions, while focusing on enhancing their
current line.
EQP Technical Enhancements
• A new extensible and scalable infrastructure for automated
tests:
– Ability to run automated tests in parallel.
– Ability to scale on-demand unlike in the current infrastructure,
where the test jobs are queued and handled sequentially.
– The current set of test tools enhanced to work in this new
infrastructure.
– A unified framework to add more new automated test tools.
EQP Technical Enhancements
• Setting up the foundation for the following features in some future release:
– Running extensions against upcoming Magento Releases and share reports to affected developers early on in the process.
– Ability to plug-in new automated test tools in a seamless manner; some examples upcoming ones in the roadmap:
• Support for MFTF tests
• Improved Security tools for static and dynamic tests.
• Tools around performance checks – i.e. did the installation of the extension caused a significant performance degradation from baseline Magento 2 setup.
EQP Technical Enhancements
• Parallelization of the technical (automated tests + manual QA) and marketing tracks of the EQP pipeline:
– A consolidated report highlighting all technical and marketing errors in one-shot, and being programmatically accessible.
– No more back-n-forth in a sequential manner as errors are encountered in either or both of the tracks.
– Foundation for better tools for the marketing track to automate certain processes, and speed up the time spent here.
• Support for future fast-track options based on historical metrics and MFTF.
APIs
EQP APIs - Introduction
• The new Marketplace Developer Portal will support REST API
access.
• The new Developer Portal website itself being built on top of
these same REST APIs – API First Design!
• Enables Programmatic access to the entire life-cycle of the
extension quality process.
• Sets the foundation for more feature development and
enhancements to the EQP process.
EQP APIs - Overview
Send feedback on which APIs you’d like to have available!
• REST API docs, Alpha version: https://devdocs.magento.com/guides/v2.2/marketplace/eqp/api.html
• Follows REST principles for all available resources:– GET: To retrieve a resource
– POST: To submit a new resource
– PUT: Update an existing resource
– DELETE: Delete an existing resource
• Not all HTTP commands may be available for certain resources – consult the documentation.
• Supports batch request and response models
• Error handling based on standard HTTP Response Codes with additional enhancements for batch responses.
EQP APIs – Use Cases
• Integrating the submission process to CICD pipelines at the developers end.
• Accessing reports/metrics periodically and incorporating them to developer preferred analytics services.
• Foundation for a more streamlined and enhanced EQP process:– With the upcoming support of MFTF, regressions can be run, and based on various
metrics, the submissions could by-pass manual QA.
• Integrating with in-house developer tools for extension management:– Example: Code in github, meta-info in Google spreadsheets, and assets in Dropbox/G-
Drive managed by different stakeholders. It can be programmatically combined and integrated to the EPQ pipeline.
• And much more! It sets the foundation for the community to build more tooling around the EQP process:
– Example: Community contribution in building developer sdks’ in various languages to use the APIs.
EQP API - /rest/v1/apps
• Requires Client application registration for API ID and Secret,
and obtaining session tokens.
• Enables Authentication and Authorization
• POST /rest/v1/apps/session/token
– https://devdocs.magento.com/guides/v2.2/marketplace/eqp/auth.ht
ml
EQP API - /rest/v1/users
• Profile:
– GET /rest/v1/users/:mage_id
– PUT /rest/v1/users/:mage_id
• Keys Management (M1 and M2):
– GET /rest/v1/users/:mage_id/keys
– POST /rest/v1/users/:mage_id/keys
– PUT /rest/v1/users/:mage_id/keys/:url_encoded_label_of_m2_key
– DELETE /rest/v1/users/:mage_id/keys/:url_encoded_label_of_m2_key
EQP API - /rest/v1/users
• User-owned extension related reports:
– GET /rest/v1/users/:mage_id/reports/pageviews
– GET /rest/v1/users/:mage_id/reports/totals
– GET /rest/v1/users/:mage_id/reports/sales
– GET /rest/v1/users/:mage_id/reports/refunds
• More reports to follow in future releases.
• https://devdocs.magento.com/guides/v2.2/marketplace/eqp/user
s.html
EQP API - /rest/v1/files
• Code (M1 Tarball, M2 ZIP files), PDF documentation and media
assets artifact management:
– GET /rest/v1/files/uploads/:file_upload_id
– POST /rest/v1/files/uploads
– DELETE /rest/v1/files/uploads/:file_upload_id
• Supports batch uploads
• https://devdocs.magento.com/guides/v2.2/marketplace/eqp/files.
html
EQP API - /rest/v1/products/packages• Package management and integration with the EQP process:
– GET /rest/v1/products/packages
– GET /rest/v1/products/packages/:submission_id
– GET /rest/v1/products/packages/skus
– GET /rest/v1/products/packages/skus/:url_encoded_sku
– GET /rest/v1/products/packages/items
– GET /rest/v1/products/packages/items/:item_id
– GET /rest/v1/products/packages/:submission_id/status
– GET /rest/v1/products/packages/sku/:url_encoded_sku/status
– GET /rest/v1/products/packages/item/:item_id/status
– POST /rest/v1/products/packages
– PUT /rest/v1/products/packages
– PUT /rest/v1/products/packages/:submission_id
– PUT /rest/v1/products/packages/:item_id
– DELETE /rest/v1/products/packages/:submission_id
– DELETE /rest/v1/products/packages/items/:item_id
EQP API - /rest/v1/products/packages
• Supports batch uploads and retrievals.
• Package submission is a 2-step process:
1. Upload Package code artifacts, PDF documents, and media assets via the /rest/v1/files resource endpoint and obtain associated file upload ids.
2. Submit package meta-info via /rest/v1/products/packages resource endpoints, linking file upload ids from step 1.
• Prior to submission in step 2, all malware checks on files must pass.
• The files can be periodically polled via the /rest/v1/files endpoint to check for malware status.
• Support for callback hooks instead of polling for all EQP state changes is being planned for some future release.
• https://devdocs.magento.com/guides/v2.2/marketplace/eqp/packages.html
EQP API - /rest/v1/products/packages
EQP API - /rest/v1/reports
• General purpose aggregated Marketplace reports:
– GET /rest/v1/reports/metrics
– GET /rest/v1/reports/metrics/:metric_name
• Example metrics: sessions, overall pageviews, category-specific pageviews, EQP process-related metrics etc..
• Marketplace Zeitgeist!
• Provides visibility to developers on where their extensions rank in various metrics compared to the overall trends in the Marketplace.
• https://devdocs.magento.com/guides/v2.2/marketplace/eqp/reports.html
MFTF Automation for
Extension DevelopmentTom Erskine
MFTF Lead Engineer
@TomMagento
What is MFTF
• MFTF is:
– For writing functional tests
– Using Selenium to drive a browser simulating user interaction
– A framework to structure testing
– A set of tools to aid rapid, reusable test creation
– Reporting tools and test metadata support
MFTF Technology
• PHP
• Composer
• XML
• Codeception (BDD-styled PHP testing framework)
• Allure (Reporting tool)
• Robo (PHP task runner)
Why use MFTF in extension development?
• Functional Testing is crucial for delivering quality
• MFTF is the future of functional testing in Magento
• As an extension developer:
– It allows simple reuse of existing Magento tests
– Will integrate with EQP
– Allows merging to existing tests
– Provides powerful reporting tools to measure and track quality
metrics
Making new tests
• Tests and all test objects are written in XML
• Test Objects:
– Tests
– Pages
– Sections
– Data
• Guides https://devdocs.magento.com/guides/v2.2/magento-
functional-testing-framework/release-2/introduction.html
Modifying existing tests
• As well as running Magento supplied tests, you can:
– Edit existing tests to run locally
– Write additional tests for your extension
– Create and edit suites of tests
– Reuse existing ActionGroups
– Merge test changes reflecting your extension functionality
Shipping tests for extensions
• Tests written for your extension
– New tests, merge files, associated data and objects
• These can all be packaged or delivered with your extension
• This allows your customers to verify your functionality in their
implementation
Custom ModulesPathing and Directory Structure
Where do I develop custom modules?
• Your tests should be in app/code/$vendor/$module/Test/MFTF
• Your custom module should follow the directory and naming
structure of existing MFTF modules:
– Test
– Page
– Section
– Data
– Metadata
– ActionGroup
Merging
Explanation of merging mechanism
Types of Merging
• Action addition
– Actions can be added. They should specify ordering.
• Action removal
• Action modification
– This is possible however:
– To avoid potential problems with sequence ordering, best practice
is to remove the line and replace via addition
• All references to Actions are done by that actions stepKey
MFTF – Start now!
• Magento MSI and PageBuilder projects are actively using MFTF
• Ongoing development – we’re just about to drop v2.2.0 with
expanded test suite support!
• Get ahead of the curve, start using MFTF now:
– You’ll increase your own quality, fast
– You’ll have a template for future EQP testing
– You’ll be set up to accelerate your EQP submission from day one
Next Steps
• Get Involved:
– Join the #mftf slack channel for active discussion and questions
– Log issues in MFTF Github project (https://github.com/magento/magento2-functional-testing-framework/issues)
– Join the #marketplace slack channel for active discussions and questions around EQP APIs’ and other topics.
– Email for EQP API feedbacks: [email protected]
• Further activities at Imagine 2018
– Join us at our Marketplace booth
– MFTF Workshop
• 1015-1100 Wednesday @ Montrachet 01
• Limited space! Email [email protected]
ResourcesEQP Guide
http://docs.magento.com/marketplace/user_guide/Resources/pdf/Extension_Quality_Program_Overview.pdf
User Guide
http://docs.magento.com/marketplace/user_guide/getting-started-seller.html
APIs
Https://devdocs.magento.com/guides/v2.2/marketplace/eqp/api.html
MFTF Guide
https://devdocs.magento.com/guides/v2.2/magento-functional-testing-framework/release-2/introduction.html
Contact us at [email protected]
Q&A