ibrahim h. osmancss.escwa.org.lb/ictd/2094/22.pdf · personal risk 0.11 financial risk 0.09 dea -...

1

Re-assessing e-Government Services from Stakeholders'

PerspectivesIBRAHIM H. OSMANAssociate Dean & Professor,

Olayan School Business, Business Information & Decision SystemsAmerican University Of Beirut, Beirut – Lebanon

Email: [email protected]

Perspectives

1Ibrahim. H. Osman, Expert Group Meeting on the Role of ICT in Socio-Economic Development, UN House, Beirut, 9-10 April 2013

Presentation Outline Collaborating CEES & I-MEET Teams & SponsorsMotivation & ObjectivesE-government Measurement ModelsE government Measurement ModelsNew Stakeholders’ Evaluation Framework for E-Services

Stakeholders’ s Engagement ModelsCOBRA Framework to evaluate e-government servicesData Envelopment Analysis for Benchmarking

Two-case studies for Validations:T rkish data


Turkish dataUK data

Provider’s Perspective (STEPS Model)Conclusion & Further Directions

2

Collaborators of CEES & I-MEET Projects


CEES: Citizen-oriented Evaluation of E-government Systems (FP7 3-year project- 2009-2013 )

I-MEET: Integrated Model for Evaluating E-government services Transformation (QNRF 3-year project- 2010-2014)

MotivationYou cannot Manage what you cannot Measure. (Phil Murphy).

+ Without Innovative Modeling & Reporting

Value has a Value, Only If its Value is Valued, (Bryan Dyson).


( y y )

is the impact measurement methodology correct?

AUB tutorial on automated appraisal system, March 15, 2011

4

3

ObjectivesDevelop an Innovative Knowledge Management Tool using multi-dimensional indicators (economical – management-social-technological) to measure systematically e-government services value from all stakeholders’ perspectives.

T f h i t ti l t l t li t k h ldTransform research into a promotional tool to align stakeholders on common desired values;Transform Data into evidence based decisions to prioritize and rank alternative scenarios/ policies;Identify Best practices Benchmark for documenting and sharing knowledge and experience;Set targets for improvement with reference to the set of best practices;G d d i i h f bi d i bl


Generate data-driven weights for combined variable measures;Guide the trade-off process among competing indicators.

Identify a set of broad indicators beyond technology aspects.Establish National/International indices based on Input efficiency utilization and output effectiveness to monitor transformational change & impact of policy intervention over time for sustainable development of e-services

E-Government Measurement Models (1)Customer Satisfaction Models:

The Swedish Customer Satisfaction Index (Fornell et al. 1996);The American Customer Satisfaction Index;The European Customer Satisfaction Index; (Grigoroudis, Siskos 2003).

All expresses satisfaction in terms of: perceived quality, expectations and perceived value using formulas with fixed weights; But not based on direct engagement experience

E-government Success Models:system quality (DeLone and McLean (2003)

E-government Value Measurement Models:


E-government Value Measurement Models:cost/benefit, social, and political factors (Foley, 2006).

E-government Service Quality Models (Papadomichelaki and Mentzas (2009)

Fragamteded Descriptive But Not Perspective to inform the transformation of unsuccessful e-services

4

E-Government Measurement Models (2)EGDI (E-Government Development Index) is a composite indicator measuring a weighted average of three normalized indices on: scope and quality of online services; development status of telecommunicationquality of online services; development status of telecommunication infrastructure; and inherent human capital. EGDI = (0.34 × online service index) + (0.33 × telecommunication index) + (0.33 × human capital index)

Macro-level top-down Indicators rather than a holistic stakeholders’ perspectives including (bottom-up middle management and citizen centric)


EGDI & UN Human Development Index (HDI) are similarly developed.HDI is criticized for its additive aggregation using equal weights given to sub-indices & no benchmarking.HDI is re-assessed using weights derived from data, countries are relatively compared against established best practices (Despotis, 2005);

Oooh, It Is too Hard


How can an e-government service be evaluated?

5

New Stakeholders Framework for E-Service’ s Evaluation (Osman et al 2011)


e-Government StakeholdersStakeholders are “any group or individual who can affect or is affected by the achievements of the organization's objectives.” (Freeman, 1984).

Stakeholder theory has yet to capture the attention of the e-government research community as impact of stakeholders on success is poorly understood. (Flak and Rose, CAIS, 2005)

Key involved stakeholders a) Top-management; b) Local Politicians; c) Middle Managers; d) Employees; and e) Citizens. (Reinwald & Kraemmergaard, GIQ_2012)


Other e-Government stakeholders’ groups: People as service Users/Citizens; Businesses; Public Administrators; Government agencies; E-government project managers; Design and IT developers; Suppliers and IT developers; Research and Evaluators. (Jennifer Rowley, IJIM 2011)

6

Stakeholders’ Impact and ValuesLeadership and stakeholders have impact on the success of e-stamping service. (Luk, GIO_2009)

“Studies of satisfaction for each stakeholder should be done in terms of their own understanding, and, most likely, their own metric” (Tesch, Jian, Klein, DS_2003).

Coordination and collaboration increase global value chain of all stakeholders is better than combined local individual values obtained from independent acts, Jaber and Osman (2006)

E-government services need to be evaluated efficiently and


E-government services need to be evaluated efficiently and effectively to increase user satisfaction. (Millard et al. (2006), Lee et al. (2008), Osman et al (2011)

High Spending But Low Take-Up : (1% - 3.7% of GDP spending in EU/US translates into 10% - 35% engagement, EUROSAT 2009)

Stakeholders’ Engagement with E-system in the deliveryof an e-government service


7

A COBRA FRAMEWORK TO EVALUATE E-GOVERNMENT SERVICES


Osman et al. 2011: structured equation modeling & an confirmatory factor analysis for grouping indicators into each of the above categories.

Benefits Variables Cost Variables

Citizen Satisfaction

OpportunityDEA Optimization


Opportunity Variables Risk Variables Balance

8

Pictorial Data Envelopment Analysis

B

ODOPEff input =

VRS

x

B

xx

x

O

x

x

A

C

x

x

x

outp

ut

DO

P

ultip

le O

utpu

ts

P

F


x

A

Multiple Inputs

x

Input

M E

COBRA –Validation on Turkish Data 1) Focus group validation (49 out of 60 indicators were selected)2) Data Set: Responses 2785 out of 3506 (79.44%) after filtering3) Translation Validity of questionnaire

Content validity: Three workshops were conducted (Turksat and UK, Qatar)y p ( , )involving other E-government experts and professional researchers.Face validity: MBA students at AUB and Turkey to assess each question forclarity of wording; layout and style.

4) Construct validityFactor Analysis: Principle component analysis (PCA) is used to determine

construct validity. The final PCA of the four-factor solution with 49 itemsaccounted for 73 46% of the total variance


accounted for 73.46% of the total variance.Internal Consistency Reliability: Cronbach’s Alpha is computed for thequestionnaire and it is above 0.96, which indicates a high correlation betweenthe items and the questionnaire is consistently reliableStructured equation Modeling and Regression to validate relationship.

5) Structured Equation Modeling Validation

9

Users’ Satisfaction with Turkish E-Services

E-Service Service Name2005 Juridical System Portal, Body of Lawyers Entry2004 Juridical System Portal, Institutional Entry2003 (J idi l S t P t l L E t )2003 (Juridical System Portal, Lawyer Entry)2002 (Juridical System Portal, Citizen Entry)870 (Education Services, Student Information System)867 (Online Inquiry for Consumer Complains)

2000 (Consumer Portal, Application of Consumer Complaint)871 (Military Services, Deployment Place for Training)872 (I i Milit S i R l li ti f i i i


872 (Inquiry on Military Services Real person application for receiving i860 (Military Services Real person application for receiving information868 (Parliament Reservation for Meeting)

9000 (My Personal Page)9001 (Content Pages for Citizen Information)

Turkish Satisfaction Levels

Group E-serviceLocal Frontier

Output (79%) Input (78%)

Informational 9001 0.66 0.74

Interactive/Transactional

(10 e-services)

860 0.83 0.86

867 0.71 0.60

868 1.00 0.89

871 0.76 0.81

872 0.89 0.78

2000 0.81 0.80

2002 0 77 0 79


2002 0.77 0.79

2003 0.66 0.73

2004 0.81 0.78

2005 0.84 0.77

Personalized 870 0.80 0.79

9000 0.75 0.78

10

DEA Managerial Implication of 13 E-services868-Meeting MPs


867-online customer protections complaints

DEA Policy : Improvement Targets based on best practices

Policy 1: Increase users’ benefits % Change

Benefit

Service Quality +11

Information Quality +30

Opportunity

Service support +01

Technology support +24

Policy 2: Reduce users’ cost and risk

C t


Cost

Tangible cost -55

Intangible cost -45

Risk

Personal risk -44

Financial risk -43

11

DEA : Overall average of Data-Derived weights inputs and outputs

Construct Input-oriented (%) Output-oriented (%)

Benefit

S i Q lit 0 42Service Quality 0.42

Information Quality 0.15

Opportunity

Service support 0.14

Technology support 0.28

Cost


Tangible cost 0.62

Intangible cost 0.18

Risk

Personal risk 0.11

Financial risk 0.09

DEA - Classification And Regression Tree


CART is a data mining technique that allows exploration and discovery of meaningful, previously hidden information from huge databases

12

DEA + CART (1)


Importance of users based on demographical data

DEA + CART (2)

Highly Satisfied Users


13

Data Analysis + CART (3)

Dissatisfied users

Well educated users (369 users out of them 47.2%)Education level Post or Under-graduate degreeAge group Young, middle or old age; &

High frequency internet users (51 users out of them 42.2%).Frequency use Daily;


Age group Old age (more than 54 years);Ease of internet use Good or Beginner.

Validation on UK Data

E-Service Sample Size

Benefits, Retirement or Job Seekers support 310

Driving License Queries 305

Healthcare Information 310

Local Government 306

Tax Information 309


All E-service 1540

14

Data Description

Constructs Question Number

Cost- Cost time Q1 Q2 Q3 Q4 Q5 Q6 Q7 & Q19Cost time- Cost Money

Q1, Q2, Q3, Q4, Q5, Q6, Q7 & Q19Q8, Q9, Q10, Q20, Q21, & Q23

Risk- Personal Risk- Financial Risk

Q16, Q17, Q18, Q34, Q35 & Q36Q11, Q12, Q13, Q14 & Q15

Opportunity- Service Support Q22, Q37, Q38, Q39, Q40, Q41, Q42 & Q43


- Technical Support Q44, Q45, Q46, Q48 & Q49

Benefit- Benefit Information- Benefit Service

Q28, Q30, Q31, Q32, Q33, & Q47Q24, Q25, Q26, Q27 & Q29

COBRA Model


15

Model Fit Summary of All Services

Test Statistics Accepted RangeCMIN CMIN/DF 4.39 < 5.00

GFI 0.91 > 90.0Baseline Comparisons NFI 0.94 > 90.0

RFI 0.91 > 90.0IFI 0.96 > 90.0TLI 0.93 > 90.0CFI 0.96 > 90.0


RMSEA RMSEA 0.05 < 0.05 LO 90 0.04 < 0.05 HI 90 0.05 < 0.05

UK Data-Derived Weights

VRS‐Output UK

Cost_Time 0.10Cost Money 0 40

VRS-Input TR UKCost_Time 0.62 0.15Cost Money 0.18 0.25Cost_Money 0.40

Risk_Personal 0.25Risk_Financial 0.25

Opp_Service 0.23Opp_Technical 0.18

_ yRisk_Personal 0.11 0.44Risk_Financial 0.09 0.16

Opp_Service 0.25Opp_Technical 0.08Benef Info 0 38


Benef_Info 0.29Benef_Service 0.30

Benef_Info 0.38Benef_Service 0.29

Data Derived Weights Vary Across Countries & Models

16

DEA-VRS Orientation Results

E-Service Input Model Output Model

Benefits, Retirement and Financial or Job Seekers support (B)

64.4 79.7

Driving License Queries (D) 66.8 81.9

Healthcare Information (H) 65.3 81.2

Local Government (L) 63.9 79.6

Tax Information (T) 64.1 79.5

All E-service (Overall Average) 64 9 80 4


All E-service (Overall Average) 64.9 80.4

It seems that the British users are less satisfied in terms of efficient resources utilisation compared to Turkey (65% versus 79% whereas British users are slightly more satisfied with effectiveness of output than Turkish 80% versus 78%)

Identifed KPIs from Providers’s Workshops

Employee Process technology

Providers‘Perspective

onE-government

Service

Privacy User

Cost

Infrastructure Budget


Service

Data Storage

SecurityCost

PlanningData

Sharing Support

Revenue

17

STEPS Model for Measuring Providers’ Perspective

Policy Strategy / Legal

Technology Status

SESustainable

Development


Service/ CultureEmployment/Education

Social & Economic & Environment factors

Development & Impact

ConclusionIntroduced a New Framework for evaluating e-Government e-services from stakeholders’ perspective based on:

COBRA methodology validated as scale measureData Envelopment analysis is used to derive optimalData Envelopment analysis is used to derive optimal data-derived weights, aggregate scoring, establish best practice, and recommended changes with reference to benchmark.Results from field trial in Turkey & UK are very promisingData are now collected in Lebanon and Qatar for further validation on Users and Providers


Data derived weights seems to different by country & current index do not provide measure from all stakeholders, nor provide measure on efficiency of resources and effectiveness of outputs

18

Further Research & RecommendationsRe-assess various developed indicators based on clear objectives to achieve sustainable development by benchmarking the relative efficiency of transformation of inputs into effective generation of outputs and outcomes at national, regional and international levels from all stakeholders.Welcome collaborations with UN and all country representatives to work together towards the development of a better comprehensive E-government International Macro Index and National Micro Index at citizens level.Develop innovative ways to collect and process data (possibly using social media and smart-mobiles) and opinion mining tools, to capture the concerns of citizens.


Smart-Mobile platform to gather/mine opinion and concerns of citizensLeadership commitments (Singapore- Skills DF UK-public-private;Malaysia & UAE- tax exemption; Korea- infrastructure; Turkey- R&D support..)

to increase R&D support & provide taxi incentive regulations; to provide open data access;to act on evidence-based recommendations.

Questions and Suggestions


ibrahim h. osmancss.escwa.org.lb/ictd/2094/22.pdf · personal risk 0.11 financial risk 0.09 dea -...

Documents