© 2009 TASC, Inc.

Complex Networks Enabling Technologies Capability Development

2012 Annual IV&V Workshop

Jim Savarino, Jeff Northey, Shirley Savarino, Andy Chen, Bojan Cukic, Jesse Musgrove, John Ryan

© 2009 TASC, Inc.

IV&V Questions Addressed by CD

2

CD: Question addressed

Associated Tech Framework WBS

Develop modeling / simulation techniques capable of verifying that user needs are addressed.

2.2 System architecture meets user needs

For performance requirements, establish boundary conditions of how the system performs and assess robustness i.e.: margin against those requirements.

5.2 Ensure design provides required capability, reliably meets user needs, sufficiently stable for implementation

Develop a simulation that operates at varying levels of abstraction to address decomposition, interfaces and integration.

3.2 In-scope parent reqts. represented in appropriate child reqts., no unneeded capabilties added

3.4 Software interface reqts. adequate to meet needs of system

Define a data dictionary approach to address user scenarios that can feed into test analysis.

4.1.2 Ensure planned tests sufficient to confirm integrated system complies with system software reqts.

© 2009 TASC, Inc.

Scripting Products and Process Overview

3

Architecture (Simulink Model) - Varying levels of fidelity - Allows concatenation of Use Cases - Allows verification of interfaces

GUI Translates a use case to input data for each model.

The Simulink model reflects the system architecture at various levels of abstraction. Erlang distribution used for internal call processing parameters.

The Data Dictionary captures a Use Case and feeds the Simulink and Layered Queuing Network (LQN) models. We pulled all variables out of the model and made this aspect table driven

C# based Graphical User Interface (GUI) takes the parametric Use Case and turns it into individual calls and attributes for Simulink, call rate and call type probabilities for LQN.

Data Dictionary: Parameters and Variables for a particular instance (Use Case)

The LQN model allows rapid prototyping and is a “process model”. Inputs are parametric, characterizing the use case

Performance (LQN Model) - Performance across a time range - Evaluate Boundary Conditions

Models are validated against each other for correctness

© 2009 TASC, Inc.

A Representative System

4

911 Call

On- street

or In-

building call box

Computerized Triage Project

NYPD CAD Project

FDNY Fire/ EMS CAD Project

Brooklyn Hub Project (Redundancy to PSAC-1 Ties

to NYPD Radio Network)

Voice Alarms Project

ERS/ BARS

Project

Facilities Project

Logging and Recording

Project

Performance Monitoring

Project

Call Taker

Workforce Scheduling

Project

NYPD Radio Console/

ECS Project

FDNY Radio Console/

ECS Project

CPE/N

AC

D

Pro

ject

Network Projects (brown lines)

PSAC 2

Blue: Full Responsibility Projects Yellow: Oversight Coordination Projects

• We will perform IV&V on the integration and performance of the entire system (Pri 1)

• As required, this effort will evaluate other ECTP projects

• Additionally, there are seven projects which will receive “full” IV&V efforts

1. CPE/NACD (Pri 2)

2. Network (Pri 2)

3. NYPD Radio Console/ECS (Pri 3)

4. FDNY Radio Console/ECS (Pri 3)

5. FDNY Fire/EMS CAD (Pri 4)

6. Performance Monitoring (Pri 5)

7. Computerized Triage (Pri 6)

© 2009 TASC, Inc.

LQN of Police Department (stubbed out FD, EMS): Process Model

5

Spanish_Call_ Taker_Queue

Wait_for_Spa_

Call_Taker

[0s]

Verify_ Accidental_Call

[21s]

Hang_Up [3s]

CPE_NACD {inf}

Find_Call_ Taker [0s]

Get_ANI_ALI

[0s]

MSAG {inf}

Phone_to_ALI [2s]

English_Call_ Taker {50}

Opening [4s]

Get_Info_ Location

[28s]

Transfer_ Call [3s]

Spanish_Call_ Taker {8}

S_Get_ Location

[42s]

…

Get_Incident_Type

[13s]

Dial_ Appropriate

[12s]

Explain_ Dup [5s]

Continue_ Getting_Info_1

[20s]

Dropped_ Call_Redial

[3s]

Busy [3s]

No_Answer [15s]

Answer [5s]

P_MD P_U

Evaluate_ Language

[10s]

P_SPA

P_FOR

Ṗ_D

Ṗ_NE

F_Get _Info _Location

[84s]

F_Get_Incident_Type

[26s]

F_Dial_ Appropriate

[24s]

F_Explain_ Dup [10s]

Ṗ_D

Ṗ_NE

Contact_FLS [20s]

P_B P_A

English_Call_ Taker_Queue

Wait_for_Eng_ Call_Taker

[0s]

1-Ṗ_D

1-(P_FOR+P_SPA)

1-(P_B+P_A)

1-Ṗ_D

1-Ṗ_NE

1-Ṗ_POL

1-Ṗ_NE

1-(P_MD+P_U)

Classify_in_CAD [0s]

Continue_ Getting_Info_2

[77s]

F_Continue_ Getting_Info_1

[20s]

F_Continue_ Getting_Info_2

[174s]

English Call Takers

Spanish Call Takers

FLS and Accidental Calls

Emergency and Duplicate Calls

From Call Gen

To CAD

Dotted lines show Simulink model elements

© 2009 TASC, Inc.

LQN Performance Demonstration – Waiting Time

• For the first test, all values with the exception of Call_Rate were held constant for steady state values as listed in DataDictionary5, and the system was modeled with only the first section of the ECTP model as described in Design of the ECTP Layered Queueing Network10

• The Call_Rate value was tested at different values from 1250 to 12845 calls/hour in increasing increments of five

0

2

4

6

8

10

12

14

16

18

20

12115

12150

12185

12220

12255

12290

12325

12360

12395

12430

12465

12500

12535

12570

12605

12640

12675

12710

12745

12780

12815

Wait

Tim

e (

seco

nd

s)

Call Rate (Calls/Hour)

Wait Time in Queue for

English Call Taker

0

10

20

30

40

50

60

70

80

90

100

1255

1835

2415

2995

3575

4155

4735

5315

5895

6475

7055

7635

8215

8795

9375

9955

10535

11115

11695

12275

Call Rate (Calls/Hour)

Percent Utilization of

English Call Takers

© 2009 TASC, Inc.

Simulink Top Level Blocks (architecture model)

7

Conn1

Conn3

Conn5

Conn2

Conn4

Conn6

VESTA

PD Radio In

PD Report Back

FDNY Radio In

FDNY Report Back

EMD Radio In

EMD Report Back

PD Responders

FDNY Responders

EMD Responders

Radios

In Out

PD Responders

IN

NYPD Field Units Scope

In Out

FDNY Responders

IN

FDNY Field Units Scope

In Out

EMD Responders

IN

EMD Field Units Scope

Conn3

Conn1

Conn2

Conn4

E9-1-1

Calls Out

Call Generator

Conn1

Conn3

Conn5

Conn2

Conn4

Conn6

CAD

Call Generator

E911 VESTA CAD

Radios Reponders/ Field Units

© 2009 TASC, Inc.

Model Comparison – Simulink L2 Complex Network

8

2

Emergency Non-Duplicated

1

Call In

IN

Unintentional Calls

util

IN

OUT

Spanish Call Takers

in

Spa CT Util ization

#n

wIN

OUT

Spa CT FIFO Queue

in

Spa Avg Wait Time

IN1

IN2

IN3

OUT

Path Combiner

IN

Non-Emergency Calls

IN OUT

NYPD-SPN Calls Scope

IN OUT

NYPD-FLS Calls

IN OUT

NYPD-English Calls Scope

IN

Mis-Dialed Calls

IN

OUT1

OUT2

OUT3

OUT4

OUT5

Language and Mistake Calls

IN OUT

Foreign Language Service

util

IN

OUT

Eng Call Takers

in

Eng CT Util ization

#n

wIN

OUT

Eng CT FIFO Queue

in

Eng Avg Wait Time

IN

OUT1

OUT2

Emergency Switch

IN

Emergency Duplicated CallsIN

OUT1

OUT2

Duplicate Switch

IN OUT

Delayed Calls

in

# of Calls in Spa CT Queue

in

# of Calls in Egn CT Queue

English Call Takers

Spanish Call Takers

FLS and Accidental Calls

Emergency and Duplicate Calls

From Call Gen

To CAD

Dotted lines show LQN model elements

© 2009 TASC, Inc.

Simulink and LQN Results at 13k Calls/hour for 24 hrs (we wanted to make sure models were in sync)

9

CallRate AvgECTQueueTime ECT_Utilization EngProbOfDelay EngAcceptablePercent AvgSCTQueueTime SCT_Utilization SpaProbOfDelay SpaAcceptablePercent

13300 7.1544 0.97782 0.612330436 0.885415365 0.0001 0.465530556 6.55E-05 0.99999948

Simulink Results

LQN Results

This data avail in Simulink, not LQN

This data avail in Simulink, not LQN

Note: some variation due to translation (e.g. LQN had 13300 calls and Simulink had ~13500 calls/hour in the 24 hour period)

© 2009 TASC, Inc.

Status, and Method Assessment

10

CD: Question addressed

CD Status and Method Assessment Associated Tech Framework WBS

Develop modeling and simulation techniques capable of verifying that user needs are addressed.

LQN and Simulink models were implemented by creating a custom data dictionary with a GUI to automatically draw from the dictionary and create input files for the two models. This is a new, demonstrated technique for IV&V.

2.2 System architecture meets user needs

For performance requirements, establish boundary conditions for how the system performs and assess robustness i.e.: margin against those requirements.

The LQN model was used to establish boundary conditions which adequately simulated performance to provide an assessment of system capability against requirements and provided an assessment of robustness. In addition the model revealed there was a high sensitivity to changes in the boundary conditions. This is a new demonstrated technique for IV&V

5.2 Ensure design provides required capability, reliably meets user needs, sufficiently stable for implementation

Develop a simulation that operates at varying levels of abstraction to address decomposition, interfaces and integration.

Significant progress has been demonstrated towards achieving this objective. The Simulink level 1 model has been completed which represents system architecture at the highest level, derived from the driving artifacts. The model also includes a lower level of abstraction which includes additional details of the system. The level 2 model elaborates architecture, with an increased focus on processes. The level 2 model has been defined and potential Simulink tool limitations evaluated.

3.2 In-scope parent reqts. represented in appropriate child reqts., no unneeded capabilties added

3.4 Software interface reqts. adequate to meet needs of system

Define a data dictionary approach to address user scenarios that can feed into test analysis.

This has been accomplished by both the LQN and Simulink models. The LQN model can automatically vary the range of a parameter to produce a performance profile. The Simulink model can concatenate use cases to evaluate a complete scenario. This is a new demonstrated technique for IV&V.

4.1.2 Ensure planned tests sufficient to confirm integrated system complies with system software reqts.

We have drafted a method to cover the new techniques developed during this CD.