cams user satisfaction survey report (2017) u… · q4: overall, how satisfied are you with the...

CAMS User Satisfaction

Survey Report (2017)

Issued by: Copernicus User Support, ECMWF

Date: 15th September 2017

Copernicus Atmosphere Monitoring Service

Contents

Executive Summary ............................................................................................ 1

About the survey ................................................................................................ 2

Target audience ............................................................................................... 2

The questionnaire ............................................................................................ 3

Q1: Where are you based? .................................................................................. 5

Results ........................................................................................................... 5

Recommendations ........................................................................................... 5

Q2: Which sector does your organisation belong to? ............................................... 6

Results ........................................................................................................... 6


Q3: How do you use CAMS? ................................................................................. 7

Results ........................................................................................................... 7


Q4: Overall, how satisfied are you with CAMS? ....................................................... 9

Results ........................................................................................................... 9

Recommendations ......................................................................................... 10

Q5: How useful are CAMS products and services for you, and how satisfied are you with them? ...................................................................................................... 11

Results ......................................................................................................... 11


Q6: How useful is, and how satisfied are you with non-data services? ..................... 17

Results ......................................................................................................... 17


Q7: What would you like to see in the future from CAMS? ..................................... 20

Results ......................................................................................................... 20


Q8: Do you have any other suggestions how we can improve the Copernicus Atmosphere Monitoring Service for you? .............................................................. 22


1

Executive Summary

Users of the Copernicus Atmosphere Monitoring Service (CAMS)1 are overall satisfied with the service provided by CAMS and rate the services with an overall satisfaction rating of 87.5% (3.5 out of 4).

Two thirds of CAMS users are based in Europe. About half of users are academics, the other half splits equally into commercial and public sector entities. User satisfaction is

consistent across geographies and industry sectors.

Users are mostly satisfied and find CAMS products and services useful. User uptake and satisfaction varies significantly by CAMS products and services: the most popular

services are global atmospheric composition data, European air quality data, and solar radiation data. These services have the highest numbers of users, are rated most useful,

and receive the highest satisfaction ratings. On the other end of the scale are policy support products where low user uptake and relatively low satisfaction ratings raise concerns. This is very likely due to the nature of the policy support products as related

small amount of users are targeted, as opposed to global or regional products. However, they play an important role in CAMS as the impact to social and economic development

could be high. It would be interesting to see what comes out in next surveys.

Users are also mostly satisfied with supporting services, in particular with the available data access mechanisms and with the CAMS service desk. Issues are not so much around

user satisfaction, but rather around awareness with certain supporting services like OGC (Open Geospatial Consortium) services and services communications.

The usage demographics, usage patterns and satisfaction ratings reflect the heritage of CAMS as a science programme, while also demonstrating that CAMS established a firm foothold in the commercial and public sector.

The survey identified four main areas for improvement: (a) the footprint of CAMS in Eastern Europe (b) the attractiveness of CAMS to public sector users and in particular

the utility and awareness of policy support products, (c) data access tailored to users’ specifications, and (d) more detailed scientific documentation.

This being the first CAMS user satisfaction survey, no trends can be identified.

1 http://atmosphere.copernicus.eu


2

About the survey

Gathering of user feedback is important in Copernicus and is currently under development at programme level at the European Commission (EC), with pilot survey run in partnership with ESA (for the Copernicus Open Access Hub) and EEA (for the

Copernicus Land Monitoring Service - CLMS Pan-European and Local components). Having collaborate/coordinated with the European Commission, the Copernicus User

Support Team at ECMWF (European Centre for Medium-Range Weather Forecasts) conducted the first CAMS user satisfaction survey in 2017.

Target audience

The purpose of the CAMS user satisfaction survey 2017 was to understand how satisfied CAMS users are with services provided, to collect feedback, analyse them and help

improve CAMS services.

The survey was carried out via an online questionnaire, in English only, which was open

between 13th June and 8th July 2017.

An invitation to participate in the survey was sent to 3,388 distinct email addresses (some users are registered for multiple services):

• 1,749 registered users of CAMS/MACC archive products (CAMS NRT Archive, GFAS, MACC Reanalysis)

• 1,193 registered users of CAMS solar data (CAMS_72) • 620 registered users of CAMS User Support • 114 registered users of CAMS global NRT data

• 90 registered users of CAMS/MACC GHG inversion data • 88 registered users of CAMS regional data (CAMS_50)

• 57 registered subscribers of the CAMS newsletter • 27 registered users of CAMS climate forcing data

The survey was also advertised to the public via the CAMS newsletter, the CAMS and

Copernicus websites and CAMS and Copernicus social media.

The survey targeted primarily registered users of CAMS. This is to capture the actual use

of CAMS, rather than awareness among the general public.

Some CAMS services allow anonymous access (e.g. CAMS_50, CAMS_71), users of these anonymous services are not included in the survey. Hence the actual number of CAMS

users is expected to be higher than the 3,388 users targeted in the survey.

In total 254 responses were collected (7.5% of addressees). Out of these, 246 responses

(97% of all responses) resulted from email invites and 8 responses (3%) from public advertisements).

The predecessor of CAMS, the MACC-II project, in 2013 also carried out a user

satisfaction survey. However, due to a different demographic (the MACC-II survey invited about 200 users, with 25 responses2), results are not directly comparable.

For further information please contact [email protected].

2 Results of the MACC-II User Survey, ECMWF 2013


3

The questionnaire

The following questionnaire was used:

Q1: Where are you based? (List of countries and territories)

Q2: Which sector does your organisation belong to? (Public authorities / University and Research / Commercial or private sector / International organisation, NGO / Other)

Q3: How do you use CAMS? I usually use…

• Parameters : A single parameter or species / A few at a time / Many • Time: A single date / Multiple dates / Long time series

• Vertical levels: Surface only, or a single specific model or pressure level / Multiple model or pressure levels / All model or pressure levels

Q4: Overall, how satisfied are you with the Copernicus Atmosphere Monitoring Service? (1 = not satisfied, 4 = very satisfied)

Q5: How useful are the following CAMS products and services for you, and how satisfied

are you with them?

• European air quality

• Global atmospheric composition • Policy support products • Solar radiation

• Greenhouse gas fluxes • Climate forcing

• Anthropogenic emissions • Fire emissions • Other

Q6: How useful is, and how satisfied are you with ...

• Product catalogue

• Automated data download (WebAPI, FTP, ...) • Data download portal (apps.ecmwf.int) • Remote services (e.g. WMS, CSW)

• Available data formats • Documentation

• Knowledge base • Validation reports

• Helpdesk / Support • Mailing list • Newsletter

• Other

Q7: What would you like to see in the future from CAMS?

• Interactive web maps • Subset data to a specific geographical area • Time series at custom locations

• Vertical profiles at custom locations • Epsgrams for custom locations

• Additional parameters and chemical species • More OGC services (WMS, WCS)


4

• More documentation

• Additional data formats • Additional units of measurement

• Other


5

Q1: Where are you based?

Results

Responding CAMS users are mostly based in the European Union (67% of all

responses). Outside Europe most responses come from China and the USA (4% each).

Overall and also among the EU countries Germany is leading, followed by France,

Spain, the UK and Italy (Figure 1).

Looking only at users of CAMS_50 (respondents who are based in Europe and

find the CAMS_50 service ‘very useful’ or ‘essential’), we find the same regional

distribution.

Recently CAMS targeted specifically the eastern EU countries. From there Poland is

dominant, but still with only 1% of responses (3 users).

Recommendations

The regional distribution shows that CAMS

is primarily used within in the EU, in line with its mandate.

Respondent numbers correlate with

countries’ population, however, the eastern EU countries are still underrepresented and

the efforts to increase the user uptake in Eastern Europe should continue.

Figure 1: Responses by country (where >= 1%)


6

Q2: Which sector does your organisation belong to?

Results

In terms of industry sectors, globally most users work in academia and research, followed by commercial entities and public authorities (Figure 2).

Figure 2: Responses by sector, global

Looking only at responses from EU-based users, we find the same distribution: academia

and research dominate, followed by commercial entities and public authorities (Figure 3).

Figure 3: Responses by sector, EU only

Looking only at users of CAMS_50 (respondents who are based in Europe and find the CAMS_50 service ‘very useful’ or ‘essential’), we find the same sectoral distribution.

Overall, CAMS is well known and well used in academia and research. The strong presence of the commercial sector also is encouraging and demonstrates that CAMS has

economic impact.

Recommendations

Further user uptake activities should concentrate on the public sector, where CAMS is

relatively underrepresented.


7

Q3: How do you use CAMS?

Results

To determine how CAMS should provide data products in an efficient and user friendly way, users were asked what combinations of parameters, time periods and vertical levels

they usually use.

Figure 4: Use by parameters / time periods / levels (all responses)

As illustrated in Figure 4, most notably a vast majority of users (118 of 253 respondents

to this question, or 49%) are interested in time series or multiple dates (95 respondents or 40%) rather than single dates.

The demand for time series data is present across geographies (EU as well as global), and particularly strong among research users (64% of time series users are researchers).

The strong demand for time series is present across all products: among regular users

of solar data (user who consider this data ‘essential’ or ‘very useful’ for their work), 56% usually use time series; for European air quality data the figure is 52% use time series,

and for global atmospheric data 47% use time series.

See also Section Q7 regarding demand for time series.

In terms of level types the demand for surface data is highest in all user groups.


8

Despite the strong preference for surface level data and time series, the combinations

of number of parameters, time periods, and level types required by users are diverse.

Recommendations

To respond to the diversity in user requirements, CAMS should implement mechanisms for users to tailor data access to their individual preferences.


9

Q4: Overall, how satisfied are you with CAMS?

Results

Overall satisfaction with CAMS is high, both globally and from EU respondents (Figure 5, Figure 6). The weighted average rating is 3.5 out of 4 globally, and 3.4 out of 4 among

EU-based service users.

Figure 5: Overall satisfaction, all respondents

Figure 6: Overall satisfaction, EU respondents

Sample free text comments:

“CAMS is amazing, and enables

researchers in atmospheric and earth

sciences the world over.”

“The data has really helped to conduct

some analysis that would not have

happened without it. “”

“For me it was actually rather

troublesome to obtain data.”

“The parameters I have used (aerosol)

perform very well in comparison with

other models. It is extremely useful to

have this tools available.”

Users’ free text comments emphasise the utility of the data, but are overall

inconclusive. For all comments please

see the annex.


10

However, within the EU countries the user satisfaction in the public sector is lower than

in other sectors (Figure 7).

Figure 7: Overall satisfaction, respondents from public sector users within the EU

Recommendations

Overall user satisfaction with CAMS is high and no radical change to the CAMS strategy and portfolio is required. Specific areas for improvements are identified in the more

detailed analysis below.


11

Q5: How useful are CAMS products and services for you, and how satisfied are you with them?

Results

Utility and satisfaction are highest for solar products and global atmospheric products. This pattern applies to the global user base as well as the EU user base (Figure 8).

Policy support products stand out as being relatively unknown and also as users being

least satisfied. This is very likely due to the nature of the policy support products as related small amount of users are targeted, as opposed to global or regional products.

However, they play an important role in CAMS as the impact to social and economic development could be high. It would be interesting to see what comes out in next surveys.


12

Figure 8: Utility and user satisfaction with data products and services, all respondents (168 responses)

“The products provided by

CAMS are extremely useful.”

“Some of the data I have under

rated is very difficult to get

from the site. If it was readily

available, I believe it would

have been extremely useful.”

“The ability to request more

than two download streams

would be very helpful.”


13

Stratified by sector, commercial users find the CAMS global atmospheric composition

and European air quality services most useful. Commercial users are ‘very satisfied’ or ‘extremely’ satisfied with all main services, and they are more satisfied that users from

other sectors (Figure 9).

Figure 9: Utility and user satisfaction with data products and services, commercial and private sector (30

responses)


14

In academia and research the global atmospheric composition and solar radiation

services are considered most useful. Users in this sector are generally very satisfied with the CAMS services (Figure 10).

Figure 10: Utility and user satisfaction with data products and services, academia and research (93

responses)


15

Compared to other sectors, public sector users are less aware of the product portfolio,

and they are the least satisfied (Figure 11). Where CAMS data is used in the public sector, global atmospheric composition data is considered most useful, followed by

anthropogenic and fire emissions and by European air quality data. Policy support products again stand out as being known little, and surprisingly are even less known in the public sector than in other sectors. Where policy support products are used,

satisfaction with these products is average, suggesting that they indeed have utility.

Figure 11: Utility and user satisfaction with data products and services, public sector (29 responses)


16

Recommendations

The public sector has less awareness of the CAMS portfolio than other sectors, and satisfaction is lower than in other sectors. The public sector is a key customer for CAMS,

so user uptake activities should be implemented to advertise CAMS to the public sector and its intermediaries, and in particular for policy support products.


17

Q6: How useful is, and how satisfied are you with non-data services?

Results

This question covers various aspects of data provision mechanisms, documentation, and user interaction.

Globally these services are considered useful. Notably the remote services, WMS (OGC

Web Map Service) and CSW (OGC Catalogue Service) are relatively unknown (Figure 12).


18

Figure 12: Utility and user satisfaction with non-data services, all respondents (164 responses)

The most important supporting services are automated data access, the data portals, and documentation. Available data formats are also generally considered useful, despite

“Simple download examples

with different methods are

necessary.”

“Validation reports on radiation

are not available on the site. It

very hard to find the few

available. Documentation is not

always easy to find.”

“The people at the helpdesk are

very helpful!”

“It is hard to know what is

behind the simulations (in

terms of input data, at least) in

the documentation. The

validation reports for the global

scale are of poor quality. There

is a clear lack of evaluation.”


19

anecdotal evidence that additional data formats would be useful (See Q7). There is

relatively little awareness of remote services like OGC services, but 60% of those who are aware of them rate these services ‘essential’ or ‘very useful’. The communications

services (mailing lists and newsletter) too are not well known and considered not very relevant.

This distribution is mirrored in the user satisfaction: Users are most satisfied with the

services they consider most important, plus, there is very high satisfaction with the Service Desk.

Satisfaction is lowest (users are ‘not satisfied’ or ‘somewhat satisfied’) with the CAMS product catalogue, and CAMS documentation.

Dissatisfaction with the product catalogue is reported from all industry sectors and

mostly from users of global atmospheric products. (Among users who are ‘not satisfied’ or ‘somewhat satisfied’ with the catalogue, 44% rate global atmospheric data as

‘essential’ for their work.) From the survey it is however not clear what the pain points in the use of the catalogue are.

The user group least satisfied with documentation are researchers using global

atmospheric data. (62% of users who are ‘not satisfied’ or ‘somewhat satisfied’ with documentation are researchers. 43% of users who are ‘not satisfied’ or ‘somewhat

satisfied’ with documentation rate global atmospheric data as ‘essential’.) This is confirmed by free text comments asking for more detailed scientific documentation.

Recommendations

The CAMS website and product catalogue are currently undergoing redesign, hence additional actions regarding the catalogue should be delayed until the new design is

evaluated in the next user satisfaction survey.

Consideration should be given to publication of additional scientific documentation.


20

Q7: What would you like to see in the future from CAMS?

This question aims to identify user-driven improvements to CAMS.

Results

The most requested features are the ability to customise data retrieval for specific

geographical areas, time series, and vertical profiles (Figure 13).

Differentiate by region (global vs EU respondents), these priorities persist for all user

groups.

Figure 13: The CAMS user wish list (167 responses)

Differentiated by data products and services (users rating specific data as ‘essential’ or ‘very useful’), the same pattern prevails, and in addition, users of global atmospheric data and of solar radiation data ask for additional parameters, in particular pollen.

When differentiated by sector, the same pattern prevails, and in addition, private sector users express least appetite for additional features, and where they do they ask for

improvement to data access, for example with additional file formats like JSON, while academics and public sector users ask primarily for additional and more detailed documentation.

The most requested feature are localised time series (‘Time series at custom locations’; out of 167 respondents to Q7, 126 or 77% consider this feature ‘very important’ or

‘important’). As seen in section Q3, about 50% of respondents already use time series. Respondents who would like to see localised time series mostly use the solar data service, where this feature is already available, and the global atmospheric data service

(126 respondents rate ‘time series at custom locations’ as ‘very important’ or ‘important’,

“Time-series of radiation data

are already available. I'd like to

have time-series of other

variables. WPS should be

implemented.”

“Pollen counts would be very

useful in our field of interest,

namely innovative solutions for

respiratory health”

“Aerosol Chemical species:

Nitrate. More size distribution

and better emission/

assimilation.”

“I find it a bit difficult to get a

grasp at all the services

provided despite the clear

efforts in this direction.”


21

out of which 76 also rate the solar data service ‘essential’ or ‘very useful’, and 68 rate

the global atmospheric data service ‘essential’ or ‘very useful’).

Recommendations

CAMS should consider implementing tools for customised data access, in particular time series for global atmospheric composition data.


22

Q8: Do you have any other suggestions how we can improve the Copernicus Atmosphere Monitoring Service for you?

This is an open question. Responses mostly

reiterate points already highlighted in other questions, hence no further analysis is

undertaken and no additional recommendations are identified.

“Improve the organization of API and

GRIB files documentation (parameter

list, units, description of models and

steps). Many pieces of information

are available but scattered in the

docs.”

“Please keep going - thanks for the

amazing work you do”


atmosphere.copernicus.eu copernicus.eu ecmwf.int

ECMWF Shinfield Park Reading RG2 9AX UK

Contact: [email protected]

cams user satisfaction survey report (2017) u… · q4: overall, how satisfied are you with the...

Documents