1

Anthony Wiles, ETSI Protocol and Testing Competence Centre (PTCC)

@metis Kick-off meeting

30-31 March 2005

Interoperability in the DNA of standards: Ensuring interoperability between building blocks

2

In Pursuit of Interoperability

Ultimate aim of ICT standardisation is interoperability

Likelihood of interoperability is increased with Well-defined, accurate and unambiguous standards Systematic testing of products based on those standards

ETSI produces Base Standards and Test Specifications

Development Testing

Test Specs

Specification

3

Unique Resources Available to ETSI Technical Bodies

TC MTS (Methods for Testing and Specification) Develops specification methodologies and techniques http://portal.etsi.org

ETSI PTCC (Protocol and Testing Competence Centre) Supports ETSI Technical Bodies with the development of:

• technology standards: protocols, services, APIs etc.• test specifications

http://www.etsi.org/ptcc

ETSI Plugtests Service Organises interoperability events http://www.etsi.plugtests

4

How Does the PTCC Help?

Assist ETSI Technical Bodies on the use of state of the art techniques for Specification, validation and testing Good working practices (Standards Engineering)

Pragmatic and flexible approach Based on experience

Help to develop usable methodologies For ETSI’s current and emerging needs

Knowledge transfer Quality through Continuity

5

Who PTCC Supports

Technical Bodies (TB) Technical Committees ETSI Projects Partnership Projects etc.

Chairmen, Rapporteurs, Individuals Working Groups (WG) STFs (Specialist Task Forces)

PTCC budget for test specifications (15-20 STFs per yr)

ETSI Secretariat

6

PTCC Support in Standards Development

Conformance Test Specifications

Interoperability Test Specifications

(Unit) Conformance Testing

Interoperability Testing

Specification, validation and simulation

Products mature from prototypes to commercial products

ETSI TB: Development of base standards

ETSI PTCC: Support on Protocol Specification and Testing

Good practice, pragmatic application of methodology, text complemented by the use of specification languages: UML, SDL MSC, ASN.1, IDL, XML etc.

Planning, methodology, test specifications, TTCN, ISO-9646, TS 102 237

Certification

Industry

7

Typical PTCC Areas of Activity

GSM and 3G (UMTS) terminals Wi-Fi: HiperLAN/2, HiperMAN, HiperACCESS Cordless phones: DECT Access terminals: FSK, SMS DSL Splitters ISDN, Broadband ISDN OSA (API, IDL) – web services NGN VoIP: H.225, H.248, H.245 (ITU), SIP (IETF) SIGTRAN Smartcards DECT IPv6: Core, IPSEC, Mobility, v4 -> v6 DSRC (ITS) DMR ... future: More telematics/ITS, more security?

8

Design for Interoperability

Interoperability takes place on external interfaces Normative (mandatory) features Options

• Consequences must be clear

Abnormal behaviour should be well-defined Robustness

Never specify how product is to be implemented internally Different levels of abstraction e.g., 3-stage approach

Methods like SDL, ASN.1, XML, MSC and UML used to specify these interfaces Data transferred on protocol interfaces

• syntax and encodings (ASN.1, XML)

Behaviour of modules as seen from these interfaces

TTCN and test methods used for black box testing No need to know how product is implemented External interface is tested according to specification

9

Software Interfaces: APIs

APIs are external interfaces to software specification of external interface:

• Protocol used at interface• Data exchanged on interface• Behaviour displayed by interface

API can be specified in high-level language like Java, C++ etc.

No specification of software system as a whole Interoperability is difficult without well-defined

external interfaces or connectors Standardised test procedures required to

demonstrate compliance with interface definition Use of TTCN (TTCN-3)

10

Profiling and Interworking

Many standards are ‘open’ Include many options

Interoperability requires profiling e.g. ETSI TISPAN NGN Options are screwed down (dropped) Works with ‘similar’ technologies Implement and test the profile

Significantly different technologies Interworking functions e.g., SIP – H.323

General reluctance to specify ‘profiles’ Indications that situation is changing

11

Different Kinds of ETSI Test Specifications

Conformance Robustness

PerformanceInteroperability

Network Integration

RF/EMC

12

Conformance Testing vs Interoperability Testing

A

TestSystem

latigid

Conformance testing(of a network element)

1 2 3

4 5 6

7 8 9

* 8 #

Conformance testing(of terminal equipment)

latigid

TestSystem

B

Interoperability testing(of terninal equipment)

1 2 3

4 5 6

7 8 9

* 8 #

1 2 3

4 5 6

7 8 9

* 8 #

13

Characteristics of Conformance Testing

Is unit testing Tests a single ‘part’ of a product (e.g., a protocol layer)

for conformity to the requirements specified in the base standard (specification)

Tests over standardised interfaces May not be available to ‘normal’ user

Works at a 'low' level Tests at the protocol (message/behaviour) level (bits)

High control and observability Means we can explicitly test error behaviour and other

'difficult' scenarios Repeatable

Requires a test system (and executable test cases) Can be expensive (e.g., radio-based test system)

Conformance Testing is DEEP and NARROW

14

Limitations of Conformance Testing

Does not prove end-to-end functionality (interoperability) between communicating systems Conformance tested implementations may still not

interoperate• this is often a specification problem rather than a testing

problem!• need minimum requirements or profiles

Does not test a complete product Tests the parts, not the whole

• often a product is more than the sum of its parts

Does not test proprietary features, functions etc. • but this may well be done by a manufacturer with own

conformance tests for proprietary item

15

Characteristics of Interoperability Testing

Do not confuse interoperability testing with interoperability demos

Shows that two products interoperate Within what may be a limited scenario

Tests Functionality (of an ‘entire’ product) Tests the ‘whole’, not the parts

• e.g, protocol stacks ...• or protocol stacks + applications

High-level (as perceived by users) Shows function is accomplished (but not how)

Does not necessarily require a test system Uses existing interfaces (standard/proprietary) Usually lower-cost than conformance But can (should be) automated => test drivers

Interoperability Testing is BROAD and SHALLOW

16

Limitations of Interoperability Testing

Does not prove interoperability with other implementations with which no testing has been done

Does not prove that a product conforms Interoperable products may still be non-conformant

Cannot explicitly test error behaviour or unusual scenarios Or other conditions that may need to be forced (lack of

controllability) Has limited coverage

May not be repeatable

17

Conformance Testing and Interoperability Testing complement each other

Not always a case of either ... or ... Though sometimes it has to be for practical or

economic reasons

But for the best probability of interoperability between products – do both!

Interop Testing

Conformance Testing

Interoperability

18

Summary

Standards can be designed for interoperability

Standards should be engineered

Plan for testing (early)

Do the right kind of testing and test in parallel

19

The End

Thank you for your attention

Interoperability in the DNA of standards: Ensuring interoperability between building blocks