a process overview in the context of protect

PRoTECT is funded by the European Union's ISF-Police Action Grant, under grant agreement no 815356

2nd PRoTECT European Seminar 17th July 2020

Request for Information -A process overview in the context of PRoTECT

Process Aims & Objectives (1/2)

Slide 2

▪ A Request for Information (RfI) is a flexible method for

▪ collecting information about specific issues for facilitating decision making

▪ scanning of solutions to a given security problem

▪ Local Governments, in the context of protecting public spaces, can use the RfI for:

1. Identifying appropriate ideas / solutions towards:

• addressing threats

• mitigating risks

• enhancing security

2. Evaluating / prioritizing solution proposals based on predefined evaluationmechanism

3. Ranking participating solutions

4. Selecting the list of solutions to be invited for demonstration

5. Validating solutions applicability in the context of specific demo scenarios

PRoTECT – 815356 – Web seminar

Process Aims & Objectives (2/2)

Slide 3

A Request for Information could prove to be useful for ultimately:

▪ increasing security of the public

▪ increasing cost-effectiveness of security measures

▪ improving the LEAs / emergency services capability to respond to attacks

▪ selecting solutions for being demonstrated and gain operational insights

▪ ascertain if there are any solutions at all for a given vulnerability


Presentation Outline

RFI Process Characteristics

Evaluation Guidelines

Questions and Answers

Slide 4

✓


RFI Process description in PRoTECT

Slide 5

▪ A detailed process description document was drafted taking into consideration

▪ vulnerability assessments

▪ hypothetical threat scenarios dictated by partner municipalities

▪ general technology evaluation framework

▪ The RFI document (among others) clarified on:

▪ Participation / evaluation details

▪ Demonstration context

▪ Scan for high TRL solutions/ideas

▪ Proof of Concept for suitable solutions - compensation of travel costs up to amaximum*

▪ RFI disseminated, using various channels (project web site, networking, writteninvitations to networks, industry, projects etc.)

▪ Submission process facilitated though ICT means developed by KEMEA

* As per applicable legislative framework


Center for Security Studies (KEMEA)

7/17/2020Slide 6

Demonstration Case Scenarios

▪ Hypothetical demonstration scenarios, developed by the municipalities in a structured approach:

▪ Scenario Background & Prerequisites

▪ Scenario Unfolding

▪ Context of deployment of innovative solutions

▪ Abstract site characteristics used, not revealing sensitive info retaining the qualitativecharacteristics desired for the demo

1

2

3

4

5

LEGEND1. Larissa2. Eindhoven3. Malaga4. Vilnius5. Brasov



Slide 7

RFI Process Concept

RFI Document

•Published through website

•Provide context of seeked solutions

•Clarify conditions for participation

•Outline Solution Evaluation Framework

Evaluation

•Hosted by the municipalities in relevant workshops

•Investigate coverage of described needs

•Facilitate marking and ranking of solutions

Validation

•Deliver Demonstrations

•Validation of expected results

•Report on main findings / conclusions

The RFI document has been made available online as of

April 1st 2020 on https://protect.kemea-

research.gr/rfi/

WP3 WP4

Demo Scenarios

Technology Evaluation Framework

Description of Best practices

and Technologies

Remained Open until 1st of June

✓

32 Solution applications received

✓


https://protect.kemea-research.gr/rfi/

RfI process in PRoTECT ecosystem

WP2

A2.2 – A2.6

Cities Vulnerability Assessments

WP4

A4.1

Demo Cases Definition

WP3

A3.2

Technology Evaluation Framework

WP3

A3.3

RFI

WP4

A4.2

Evaluation Workshops

WP4

A4.3

Demonstration Sessions

Completed

Completed Completed

Completed Pending

Completed



Slide 9

RFI - Web SiteIndicative Screenshots

1. Registration Page

3. RFI Submission Form

3. RFI Submission Form

2. RFI Document Access – Home Page


Registered Users 86

Registered Entities 74

Submitted Proposals 35

Entities submitted 28

Submissions by country

Proposals Submitted

City Proposals

Larissa 19

Eindhoven 23

Malaga 22

Vilnius 24

Brasov 20

Unique submissions per city




Evaluation Mechanism


Slide 11

✓


Evaluation Process (General Strategy)

Slide 12

Evaluation Results

Confirmation6th General Assembly

Evaluation Workshop

Eindhoven

Evaluation Workshop

Brasov

Evaluation Workshop

Larissa

Evaluation Workshop

Malaga

Evaluation Workshop

Vilnius

Ranked Solutions Ranked Solutions Ranked Solutions Ranked Solutions Ranked Solutions

Aggregated Results / Solution Invitation


Evaluate / Rank Solutions in 3 phases

• Hold discussions among stakeholders during workshops

• Assign marks as per evaluation criteria communicated through evaluation framework

Phases

• Phase 1- Check the completeness and Compliance

• Phase 2- Check the relevance

• Phase 3- Evaluation of responses


Slide 13

Evaluation Process (Step by Step)

▪ Evaluation Workshops were held at each city (physical or virtual meetings)

▪ Evaluation process for each city comprised of the following steps / tasks

Evaluation WorkshopsPre-Workshops Actions

Form Evaluation Committees (per city)

• Decide on Size

• Invite Stakeholders

• Sign Declaration of Non-Conflict of Interest

• Assign Roles

• Provide initial training

Receive Participating Ideas

• Sort received ideaand route toapplicable cities

• ArrangeEvaluationWorkshops


Phase 1: Check the Completeness and Compliance

Slide 14

Please Select One NotesResponse date is within deadline of the RFI Yes NoStandard online application forms have beenused, fulfilment of formal requirements,completion of all required forms

Yes No

Declaration of the solutions IPR rights Yes NoCompleteness and acceptance of the necessarysupporting documents concerning Compliance.

Answers should be I AGREE and the Legalaffirmation valid (signed) and uploaded

Yes No

Response qualifies as Valid/Invalid

Failure in at least one of the above (No) rejects the response asinvalid.

▪ Horizontal assessment of all solutions against a set of eligibility / compliance criteria

▪ Non-compliant solutions to be excluded from the remaining process


Solution Name/NumberPlease Select One Notes

1st CRITERIA GROUP – Relevance of the Solution

Relevance to the scenario Yes No

Relevance to the Threat types Yes No

Relevance to capability in which thevulnerability manifests itself

Yes No

Relevance to the threat phase Yes No

Response qualifies as Valid/Invalid

Failure in at least one of the above (No) rejects the response as invalid.

Phase 2: Check the Relevance

▪ Assessment of solutions against relevance to address scenarios / threats indicated in the RFI

document


Marks are selected among scale of 0-5

Total score per solution to be used for ranking

Documentation of assigned grade

Proposed Criteria Weighting Factors

Please Select One Point Rating

NotesWeighting

factorScore

2nd CRITERIA GROUP- Technology Total 80%Accuracy 0 1 2 3 4 5 10%Compliancy 0 1 2 3 4 5 10%Reliability 0 1 2 3 4 5 10%Security 0 1 2 3 4 5 10%Ease of use 0 1 2 3 4 5 10%

Maturity 0 1 2 3 4 5 10%

Portability 0 1 2 3 4 5 10%

Maintainability 0 1 2 3 4 5 10%3rd CRITERIA GROUP- Additional information Total 20%Additional benefits 0 1 2 3 4 5 10%Sufficiently detailed and accurateinformation

0 1 2 3 4 5 10%

Total Solution Score

Phase 3: Scoring of the Responses


Center for Security Studies (KEMEA)Description of Criteria1st CRITERIA GROUP – Relevance of the solutionRelevance to the scenario This criterion examines the foreseen applicability of the solution as a response to the specific city Scenario.

Relevance to the Threat types This criterion concerns the applicability of the solution against the type(s) of threat the proposed technology could be used for.Relevance to capability inwhich the vulnerabilitymanifests itself

This criterion examines at which rate the proposed solution enhances the capability of the operators against the securityvulnerability under examination (e.g. planning and management, intelligence gathering, access control, threat deterrence, threatdetection, attack response, actuators, physical, methods and procedures etc).

Relevance to the threat phase This criterion analyses whether the proposed solution corresponds to the appropriate threat phase as indicated by the provider totheir submission (before, during, after attack).

2nd CRITERIA GROUP - TechnologyAccuracy Assess whether the result/output provided by the solution can be considered as accurate as well as the degree on which the result /

output provided may be relied upon for basing the necessary operational decision making.

Compliancy Assess whether the proposed solution, is compliant with the legal requirements, privacy protection laws, safety regulations, etc. Inthis context the capacity for parameterising the solution for quickly and easily adapting to different legal requirements are alsoexamined.

Reliability Assess the solution in terms of reliability and provisions taken to ensure its operation. In this context, the evaluators will amongothers assess aspects such as a) Will it always work? b) Is there redundancy, backup mechanisms? etc, while taking in considerationthe conditions under which the solution must work and what the council expects from it

Security Assess whether the solution Is physically and digitally secure, against sabotage, spoofing, hacking, influencing, disabling, etc.Ease of Use Assess whether the solution is easy to use for the council. In this context it will additionally be examined whether it is easy to have

installed and to have operated while also the output of the solution being easy to understand and to use in the whole securityprocess.

Maturity Assess the maturity level of the solution meets the expectation of the city in terms of availability and /or Technology ReadinessLevel (TRL).

Solution characteristics Assess the solution characteristics including size, installation specifications, performance, operability, interoperability,maintainability, deployment time etc.

Maintainability Assess aspects that refer to the solution being easily maintained or not. Moreover, the criterion should investigate whether it ismaintainable by more than one company, how long is the solution unavailable for maintenance? etc.

3rd CRITERIA GROUP - Additional informationAdditional benefits Assess whether information provided (including entity profile, previous/ ongoing (business or research) relevant projects, R&D

actions, relevant publications, involvement in projects concerning high-risk scenarios, envisaged contribution in the demonstrationsand possibilities for reckoning with the related costs) promotes the identification of additional benefits for each relevantmunicipality.

Sufficiently detailed andaccurate information

Information quality that can enhance the perception of the proposed solution such as technical characteristics brochure, additionaldescriptions etc.

Slide 17 PRoTECT – 815356 – Web seminar

Center for Security Studies (KEMEA)Point Numerical Rating ScaleNumber of

StarsScale Rating Definitions

5 points ExcellentExceptionalMasteryMuch more than acceptable

• Should ensure extremely effective performance• Significantly above criteria for successful performance• Solutions surpassed expectations• Reserved for the exemplary set of capabilities that yield a particularly sophisticated approach to handling

the situation • Meets all major/ essential/ core capability and responds to the need

4 points Very Good

Full PerformanceAbove average

• More than adequate for effective performance• Generally, exceeds expectations relative to quality and quantity capabilities required for successful

performance• Meets all the major / essential / core capabilities• No major deficiencies exist in the areas assessed• Consistently presented better than average level • Describes/ presents the full range of capabilities appropriate for handling the situation towards the desired

result and outcome needs to be obtained.

3 points GoodAcceptableSatisfactoryAverage

• Should be adequate for effective performance• Meets relative to quality and quantity capabilities required for successful performance• Meets several of the major / essential / core capabilities • Describes / presents enough range of capabilities for handling the situation and the desired outcome is

obtained. • Some deficiencies exist in the areas assessed but none of major concern.

2 points WeakLess than Acceptable

• Insufficient for performance requirements • Generally, does not meet expectations relative to quality and quantity capabilities required for successful

performance • Does not describe/ presents enough range of capabilities appropriate for handling of the situation• Describes plausible but inappropriate solutions for handling the situation or the desired result or outcome is

not obtained.

1 point PoorMuch less than acceptable

• Significantly below capabilities required for successful performance• Many deficiencies• A major problem exists • Describes/presents counter-productive solutions that have negative outcomes or consequences (make the

situation worse)

0 point Unacceptable • No answer or inappropriate answer

Slide 18 PRoTECT – 815356 – Web seminar

Timeframe

Slide 19

Any foreseeable delays will be timelycommunicated (asap) for proceeding to anyrescheduling and or alternatives in order tofollow time plan above as closely as possible

Months APR MAY JUN JUL AUG SEP OCT NOV DEC

RFI Publication to website

RFI active for submissions

Submission deadline

Evaluation Workshops at 5 cities

Announcement of invited

providers

Administrative procedures in the

5 cities

Demonstrations at 5 cities

in each one of the five cities

Milestone

Duration

Deadline 01-06-2020

Publication date 01-04-2020

17-07-2020




Evaluation Guidelines


Slide 20

✓


Summary & Lessons Learnt

Slide 21

▪ The RfI constitutes a flexible tool for scanning of solutions to a given security problem

▪ Process could facilitate decision-making towards a future strategy (not in scope of PRoTECT)

▪ No further commitment by all participants is required

▪ Limited set of prerequisites

▪ Accurate problem statement

▪ Engagement of Resources with relevant expertise

▪ Elaboration of Scenarios

▪ Elaboration of Evaluation Mechanism

▪ Evaluation / Ranking of Solutions

▪ Establishment of a detailed process risk mitigation plan

▪ Close progress monitoring for timely mitigation of risks

▪ Availability of an ICT infrastructure for process facilitation is advised

▪ Publication / Q&As

▪ Receipt of Applications

▪ Data processing / Evaluation


PRoTECT is funded by the European Union's ISF-Police Action Grant, under grant agreement no 815356

Thank you for your attention!

a process overview in the context of protect

Documents