Theoretical and Methodological Concepts for Future Research and Documentation on Social Reporting in Cross-sectional Surveys

Michaela Hudler Rudolf Richter

EuReporting Working Paper No. 18

Towards a European System of Social Reporting and Welfare Measurement A TSER-Project Financed by the European Commission Subproject Stocktaking of Comparative Databases in Survey Research Paul Lazarsfeld-Gesellschaft für Sozialforschung (PLG) Vienna 2001

2

3

Content:

1 THEORETICAL AND METHODOLOGICAL CONCEPTS FOR FUTURE RESEARCH ON SOCIAL REPORTING IN CROSS-SECTIONAL SURVEYS .......... 4

1.1 Future perspectives of survey research on social welfare .................................... 4

1.2 Recent European cross-national and cross-sectional survey initiatives to measure welfare - European Welfare Surveys ................................................................................ 5

1.2.1 Euromodule - Towards a European Welfare Survey (Zapf, Habich, 1999) . 6 1.2.2 European Social Survey................................................................................ 7

1.3 Quality criteria of cross-sectional surveys............................................................ 8 1.3.1 Content.......................................................................................................... 8 1.3.2 Methodological and technical issues in executing cross-sectional surveys 12 1.3.3 Errors and biases in survey research........................................................... 15

1.4 Cross-national and cross-sectional survey research ........................................... 23 1.4.1 Some theoretical and empirical considerations towards survey equivalence . .................................................................................................................... 23 1.4.2 Developing a measurement instrument for multi- lingual and multi-cultural use .................................................................................................................... 26

1.5 Summary and Conclusions ................................................................................. 28

2 THEORETICAL AND METHODOLOGICAL CONCEPTS FOR FUTURE DOCUMENTATION ON SOCIAL REPORTING IN CROSS-SECTIONAL SURVEYS ...................................................................................................................... 30

2.1 Attempts of data archives and research institutes to publish survey data and information ..................................................................................................................... 31

2.2 Different software tools to present social sciences survey data in an user- friendly way in the world wide web............................................................................................. 34

2.2.1 NESSTAR - Networked Social Science Tools and Resources (http://www.nesstar.org, 18.06.01) ............................................................................. 35 2.2.2 ILSES - Integrated Library and Survey-data Extraction Service (http://ilses.gamma.rug.nl, 18.06.01) .......................................................................... 38

2.3 Summary and conclusion.................................................................................... 39

References.......................................................................................................................43

4

1 Theoretical and Methodological Concepts for Future Research on Social Reporting in Cross-sectional Surveys

Introduction In order to develop a social Europe and improve the welfare of the citizens there is a need for permanent monitoring of the welfare development in each region of the European Union. A growing interest in comparative data on social welfare around the world can be watched. Not only researcher but also the societies themselves want to know on which range they are situated compared to other countries. Therefore the number of countries participating in cross-national survey endeavours on social welfare is increasing. During the last decades also methodological research in social survey research has been improved. Methodological workgroups have been established to work out appropriate survey instruments and designs for gaining high quality data. A large number of influences on data quality have been analysed and there were several approaches to quantify their impact on research results. In front of this background the first part of the paper summarises research approaches in cross-sectional social survey research on social welfare on a European level to outline perspectives and strategies for future research. In the second part of the paper recent approaches towards documentation in cross-sectional surveys are discussed and again guidelines for future documentation systems are suggested.

1.1 Future perspectives of survey research on social welfare At the end of the second millennium and at the beginning of the third everybody felt the need of thinking about the future. Also social scientists, methodologists and data archivists are engaged in making predictions for the future. In the area of social indicators research Joachim Vogel (1996) outlined the future directions as following. He stated, that the major task of the social indicators movement today is to influence the political agenda and stressed the importance of § watching the changes at the international and national level, § designing a modern indicator system regarding new determinants of individual

living conditions, § reforming the methodology of surveys, social reports and other documentation

systems and

5

§ forming new networks on interested parties in order to guarantee the financing of measurement and reporting activities (Vogel, 1996).

A regular documentation has to be published and distributed especially using new media to be effective (Vogel, 1996). Furthermore he assumes, that the expansion and modernisation of the comparative research is as necessary as the development of new comparative surveys designed for social reporting. Some initiatives and networks of co-operating research teams on social welfare have developed in order to modernise and reform social indicators research and welfare measurement. One research team cares for a "European System of Social Indicators" (for details see Berger-Schmitt and Noll, 2000). The authors have brought together all important welfare concepts and designed a comprehensive framework for a "European System of Social Indicators", which can be considered as the theoretical framework for future attempts towards social reporting and welfare measurement. The major source of the social reporting activities at the individual level are survey results. Such "welfare" surveys have been executed since the beginning of the social indicators movement. As a development, differentiation and specification on the indicator level took place also the methods in survey research have specialised. The new trends in social research methodology as outlined and discussed for example at the Fifth International Conference on Logic and Methodology in Cologne 2000 are moving towards the usage of quantitative as well as qualitative research to explore the complex problems in social research. For the adaptation to the complexity of social problems adequate causal models are necessary. Especially structural equation models are considered to be appropriate to solve complex causality problems, which appear in social indicators research. To measure social welfare in a comparative way across Europe two initiatives, the Euromodule and the European Social Survey, have been launched recently, and will be described in the next section.

1.2 Recent European cross-national and cross-sectional survey initiatives to measure welfare - European Welfare Surveys

Due to an enlarging Europe and due to a lack of comparative survey data on social welfare two initiatives towards a "European Welfare Survey" developed independently from each other during the last years. The aims of both survey programmes are the regular monitoring of the social changes on the individua l and societal level and the preparation of a comparative database. Both survey initiatives are of German origin and have their roots at the one hand in the

6

German General Social Survey and at the other hand in the German Welfare Survey. Both attempts follow a bottom up approach. In the following the approaches towards "European Welfare Survey" programmes will be shortly presented.

1.2.1 Euromodule - Towards a European Welfare Survey (Zapf, Habich, 1999)

The Euromodule (http://www.wz-berlin.de/sb/fb/fp2de.htm, 21.08.01) is designed as a survey where the principles of the German Welfare Survey, replication and innovation, are combined (see Zapf, Habich, 1999). Research teams from 19 nations take part in this initiative, which is headed by the department "Sozialstruktur und Sozialberichterstattung" of the Wissenschaftszentrum Berlin and carried out in co-operation with the social indicators department at ZUMA-Mannheim. The survey is intended as a module, which should be enclosed to other national cross-sectional surveys. For each country the module has to be nationally founded. The countries, which are engaged in this research are Germany, Austria, Hungary, Sweden, Spain, Turkey, Slovenia, Belgium, Denmark, Finland, France, Great Britain, Netherlands, Norway and the Czech Republic. Up to now comparative data from Slovenia, Hungary and Germany are available. Concerning replications over time there are no intentions mentioned in the descriptions (Böhnke, Delhay, Habich 1999, Zapf, Habich 1999). In the welfare concept of the Euromodule objective and subjective indicators are employed to measure welfare. The Euromodule consists itself of two modules: a core and an optional one. The core module covers the § objective living conditions and the § subjective well-being. The optional module deals with quality of society in social and political terms. There are a minimum of common methodological and technical issues, which have to be regarded in the national Euromodule surveys to guarantee comparability of the survey results. These guidelines are outlined in the general description of the Euromodule (Zapf and Habich, 1999). The population universe of the national surveys have to be representative and cover the adult population from 18 years upwards. Random sampling is accepted as a minimum requirement. Concerning the sample size 1000 respondents are considered as minimum. The interviews have to be carried out face-to-face. More or less strict guidelines concerning demographic variables, questionnaire translation, question ordering, interviewing, documentation, data management and data dissemination are given by the principal investigators.

7

The second approach towards a "European Welfare Survey" is the concept of the European Social Survey. An overview of this research endeavour is presented below.

1.2.2 European Social Survey

The European Social Survey is basically grounded on the well documented and analysed German General Social Survey (ALLBUS). The intention is to deve lop and establish an innovative measurement instrument for monitoring changes in societies in order to enable the finding of "explanations of (changes in) social and political beliefs of Europeans form a comparative perspective" (European Science Foundation - http://www.esf.com, 21.08.01). The head of the initiative is Max Kaase. The concept of this programme is available at the homepage of the European Science Foundation. A questionnaire has not been published until now. The source of this description is a blue print published by the European Science Foundation (http://www.esf.com, 21.08.01). The countries, which will participate in this survey programme have not been fixed at the time. Only methodological, technical and conceptual information is provided. The thematical coverage of the European Social Survey is much broader than the topics investigated by the Euromodule. The concept of the survey programme informs, that replications of the surveys should take place every second year. Supranational founding is intended. "The Focus of the European Social Survey will be the systematic study of European citizens' attitudes, attributes, and behaviour relating to a core set of economically, socially and politically relevant domains. It will study distributions, differences and changes across time and space in the social, political and cultural beliefs and behaviours of Europeans" (European Science Foundation, http://www.esf.com, 21.08.01). The intended fields of interest for a European Social Survey are according to the general description of the survey programme: § individual attitudes and attributes, § social position and networks and § social context and environment. As the Euromodule also the European Social Survey is designed as a modular system and beside the above mentioned core module optional modules covering social and political structure as well as cultural issues are intended. Country specific relevant topics should also be investigated additionally. The population covered will consist of adults from 15 years and older. The persons have to be residents of the country. Citizenship, nationality or legality will not be considered as a selection criteria. Within the European Social Survey only random probability methods at every sampling stage will be accepted and the relative selection probability for each respondent will be

8

entered into the dataset. The sample size should not be lower than 1500 respondents per country. All the interviews have to be carried out face-to-face. Apart from the content driven research also methodological research and the permanent evaluation of the survey data are intended. Especially the field of translation and providing linguistic and functional equivalence is in the centre of interest. These two approaches to measure welfare show in their theoretical conception which includes the definition of the content and structure of the survey questionnaires, the definition of the population and the demographic variables as well as in the methodological design which refers to questionnaire translation, sampling, sample size, data collection, documentation, data storing and access conditions the theoretical and methodological issues which have to be regarded in future cross-sectional survey research on social welfare in order to provide reliable and valid data. After the overview of new European research initiatives towards social welfare a summary of theoretical, methodological considerations, which have to be regarded in future cross-sectional survey research on social welfare in order to provide reliable and valid data, is given. Especially problems and difficulties in cross-national surveys, which influence data quality, will be focused.

1.3 Quality criteria of cross-sectional surveys The quality of a survey and of the survey data obtained by a survey are depending on the conceptual approach towards a theme, which has to be investigated, the design of the survey instrument, the technical and methodological execution of the survey, the data processing, data and survey documentation. An ideal cross-national, cross-sectional survey should regard the international and national requirements as well as the theoretical and methodological findings dealing with survey research of all disciplines. In the following sections the single steps towards a survey concept for future cross-sectional survey research on social welfare, which will provide high quality data, will be described starting by the survey content.

1.3.1 Content

The principal issue of social surveys covering especially welfare topics is at the one hand monitoring societal changes and at the other exploring their reasons. Since social reporting and welfare measurement have been established many different theories and approaches for exploring welfare at the individual and the societal level

9

have been developed. An overview of the different approaches can be found by Berger-Schmitt and Jankowitsch (1999), by Noll (1999) and Berger-Schmitt and Noll (2000). Today there is a consensus, that objective as well as subjective indicators are applied to measure the objective level of living and the subjective quality of life (see Euromodule, German Welfare Survey). Berger-Schmitt and Noll (2000) brought together the different approaches and discussed their relevance for a European welfare concept. The result of these endeavour is a "European System of Social Indicators" (Berger-Schmitt and Noll, 2000), which is regarded to cover a comprehension of dimensions relevant for welfare measurement. The authors differentiate within their concept between life domains, goal dimensions and measurement dimensions. The measurement dimensions reveal the aspects, which have already been inquired and should be inquired in a science based cross-sectional survey for social reporting. The advantage of this system is, that although its new constitution the indicators are already well documented and data from different surveys for comparative research already exist. The indicators are already well analysed and their relevance for a European perspective has been worked out by the authors, Berger-Schmitt and Noll. The indicators capture the life domains important for social welfare on a European level. The life domains, which are necessary for a welfare concept described by the authors, are § population, § households and families, § housing, § transport, § leisure, media and culture, § social and political participation and integration, § education and vocational training, § labour marked and working conditions, § income, standard of living and consumption patterns, § health, § environment, § social security, § public safety and crime and the § total life situation. An overview of the detailed measurement dimensions is given by Berger-Schmitt and Noll (2000). For future concepts in surveys on social welfare as well as for documentation purposes of social surveys a theory driven and science based approach is necessary. The "European System of Social Indicators" seems to be an appropriate concept for future research in this area because the goal dimensions are very open, so that new developments for example in working conditions can easily be embedded within this framework. The concept is at the one side very concrete in outlining the measurement

10

dimension for a certain life domain but at the other side necessarily general, that new developments can be as well represented as old concepts. The framework allows to combine old traditions and new innovative approaches in survey research on social welfare. Especially in designing a survey questionnaire it is important to view already used questions covering a certain topic but also to investigate the questions under the aspect of comparability of results and data quality. Already used survey questions offer certain advantages for a new questionnaire like is discussed in Prüfer and Rexroth (1996). Comparability is one task, which is very important for continuing time series in welfare research but equally important is the development of innovative approaches to measure changing aspects in life domains and concepts. Innovative concepts are necessary to monitor societal progress and the increasing differentiation of facets of life domains. Sometimes also questions turn out to be not very valid and reliable and have to be replaced by more valid and reliable ones. For national surveys designing appropriate survey questions on the individual as well as on the society level covering a spectrum of topics is not very problematic from the content point of view. But for cross-national survey research the comparative cross-national issues have to be worked out. The best case as in many papers recommended is to develop a questionnaire within a multinational team of researcher, so that the supra-national equivalence, the appropriateness and importance of the issue can be guaranteed. For cross-national surveys a modular design has the advantage, that cross-national issues can be investigated in a common core and nation-specific issues can be enclosed for each country dealing with nation-specific issues as intended in the European Social Survey. Sensitive topics or methodological issues could be raised within a self completence questionnaire. In order to guarantee comparability of data the question position within the questionnaire should be kept stable. As many studies show, that different question position can cause position bias. The International Social Survey Programme practised such a modular survey design. A common programme covering issues important for each nation and this programme is enclosed to a national social survey for example the German General Social Survey (ALLBUS), which investigates the nation-specific social issues due for example to the German reunification. Such an approach serves the comparative national as well as comparative international research intentions. To sum up the quality criteria for content: A future concept on content for welfare research has to be comprehensive like the "European System for Social Indicators" (Berger-Schmitt and Noll, 2000). It has to cover indicators of all life domains which are relevant for the societies. Subjective as well as objective indicators have to be considered. It has to serve the old traditions as well as new concepts and it has to be specific but also open for new societal

11

developments, which means a permanent monitoring, reflection and evaluation of the indicators and data. Demography The content has also to cover a demographic part which describes the status of the respondent in a common accepted format. For international survey research a standardised demography, which cares at the one side for international comparison and provides at the other side for the national uniqueness. There is an initiative from EUROSTAT to harmonise demographic questions and variables especially for cross-national comparative survey research. Other initiatives, which are working on harmonising demographic variables, are for example the different approaches to standardise education for cross-national comparison (ISCED - UNESCO 1997, 1999 and CASMIN - Brauns and Scherer, 2000) as well as the occupation variables (ILO 1990). Wolf (2000) concludes from his research, that it is not so easy to ensure comparability of background variables for cross-national and cross-cultural surveys. Variables which are independent of culture are harmonisable but most demographic variables differ in meaning, categories and intensities across societies. Wolf means "the more this is the case, the less promising is a strategy trying to equalise everything from the definition of concepts to the wording of indicators. Instead, while the comparability of concepts has to be ensured by joint efforts of researchers with primary knowledge of the societies in question, the indicators will have to differ at least partly between countries to suit the respective circumstances. This is necessary to ensure functional equivalence. Then, at last, after the data have been collected the indicators will be recoded into a common measure reflecting the construct of interest" (Wolf, 2000). So even in the case of relatively objective variables you have to consider country and perhaps even regional specifities. Wolf refers to them from a rather quantitative perspective. But for a more qualitative exploration of the differences and specifities a qualitative approach is recommended to improve comparability of the results. Focus groups and expert discussions seem to be appropriate to identify the ineqivalencies across countries and regions which can allow a more detailed and country specific approach to inquire and to interpret certain topics. Beside a scientifically driven approach towards the content side for a future cross-sectional survey also new findings and deve lopments made in survey techniques and survey methodology have to be considered to guarantee good quality of data especially for cross-national comparison. Therefore an overview of some methodological and technical issues in executing cross-sectional sample surveys on social welfare is given.

12

1.3.2 Methodological and technical issues in executing cross-sectional surveys

Starting with the population concept of the survey each single criteria, which has to be regarded in developing a cross-sectional survey on social welfare will be described.

1.3.2.1 Population

There are different definitions of the population in cross-sectional social survey research. Generally the adult population is investigated. In some cases the age is limited to persons from 15 years upwards or from 18 years upwards. Sometimes also old aged persons are excluded from a survey. Depending on the survey purpose the age limits have to be defined. The European Union initiated survey programmes like Eurobarometer, the Central and Eastern Eurobarometer as well as the level of living survey of the Nordic countries generally ask the adult population from 15 years upwards, whereas the International Social Survey Programme, the World Values Survey, the New Democracies Barometer surveys, the German General Social Survey and the German Welfare Survey investigate the population from 18 years upwards. The discussion if the adult population should include the age from 15 to 18 years depends on the theoretical background one will consider for the survey. Socialisation theory started earlier (as in the concept of the European Social Survey is argued) but the voting age is 18 years in many countries. But until there is no common agreement on the voting age on a lower level the population should cover the population from 18 years upwards because the active political participation, which is considered as life domain in a "European System of Social Indicators", can be inquired by voting behaviour. Certainly there are other indicators for political participation but voting behaviour is one of the most important. The general idea should be, that all adults residing in a country should have the equal opportunity to take part in a survey on social welfare regardless of nationality. But integrating minority groups into a survey raises the language problem (European Social Survey - http://www.esf.com, 21.08.01). Should there be translations of the questionnaire into the languages of the minorities regarding the whole problems, which have to be faced when translating survey questionnaires? Should interviewer be trained or bilingual interviewers be selected? At any case several population groups are difficult to include. For example old aged persons, persons living in old aged homes, persons in educational institutions, hospitals, prisons, military services and some more are hardly to include in a survey. Scientists have to be aware of these population problems when designing a survey.

13

One possibility used in survey research for correction if the population concept has been violated is weighting. But as well as all other survey issues also the weighting procedure has to be documented very carefully so, that a possible impact on data quality due to the weighting procedures can be identified in data analyses. For future research it is recommended to base a survey on a comprehensive population concept. This means that the age of the population can start at 15 years and the whole members of a population should have an equal opportunity to take part in the survey. Special strategies for the treatment of minority groups and groups which can hardly be included have to be developed for keeping the response rate as high as possible.

1.3.2.2 Sample

In order to provide representative data for a nation or a society only randomly selected samples are appropriate. Random selection at each level of selecting respondents has to be executed. The sampling procedure has also to be documented carefully. Quota samples do not allow a generalisation for a whole population (Friedrichs, 1980) and are therefore inadequate for a cross-sectional survey on social welfare. Considering the sampling strategy is essential for interpreting the data. According to related literature there is no doubt, that only random sampling procedures will guarantee sufficient data quality for future research endeavours in welfare survey research. Important for data quality are the procedures, which will be launched in the case of refusals or non-respondents. How are refusals handled? What could the impact of refusals and non-respondents be? Is the representativeness or the population concept violated? Research results from Fitzgerald and Fuller (1982), who were investigating the effects of reluctant respondents and refusers on sample survey estimates, indicate, that refusal can cause a serious bias. They found out within their analyses of the Northern California Community Study and the analysis of other studies, that refusers come from special subgroups of a population. They conclude, that respondent and refusal groups differ demographically. These differences can lead to different estimates of dependent variables for both groups. "Although the extent of terminal substantive bias depends on a variety of factors other, that the number who refused, the refusal rate can be an important determinant for this bias" (Fitzgerald and Fuller, 1982). Interviewer can also systematically bias the population concept, for example if they do not stick to the random selection concept by excluding certain households from the sample or select the wrong respondent within a household (Kaase ,1999). Laiho and Lynn (2000) dealt with the response rate as indicator of survey quality. The response rate informs about the quality of the field performance, survey performance, and the population coverage quality.

14

As perfect sampling is never feasible the scientifically valuable studies will carefully report about the whole sampling procedure.

1.3.2.3 Sample size

1000 respondents are considered as absolute minimum by many research teams (see Euromodule) conducting national representative probability sample surveys. Ideally the size should be about 2000 respondents and over, that analyses of subgroups with enough cases for examples in regional breakdowns according to the NUTS (Nomenclature des Unitès Territoriales Statistiques) can take place. Most of the surveys which allow statements regarding the whole population are based on samples of about 1000 respondents. The standard sample size of the Eurobarometer, the International Social Survey Programme, World Values Survey and New Democracies Barometer are 1000 interviewees for each participating country.

1.3.2.4 Fieldwork

Fieldwork has to be strictly co-ordinated and standardised by responsible organisers in order to guarantee equivalence across time and also across space as far as possible. Fieldwork as well as the interviewer training and also the interviewer characteristics have to be carefully documented in order to evaluate data quality.

1.3.2.5 Periodicity

Executing a high quality cross-national survey programme is a very time, resources and money consuming undertaking. A serious estimation of time which is needed to carry out a science based well grounded multi-national survey will take at least two years according to experts which have designed the concept of the European Social Survey (http://www.esf.com, 21.08.01). From the technical side a two years periodicity is recommended for cross-national and cross-sectional surveys. The team of the International Social Survey Programme group carries out a multinational survey every year with the same questions replicated every five years. The changes are watched every five years. The World Values Study Group monitors the value changes every ten years. A compromise between the regular monitoring, the feasibility of the survey execution and the need of a periodic observations to watch the changes has to be found.

1.3.2.6 Documentation

In order to provide the scientific community and other interested users with a highly reliable and valid measurement instrument, which generates high quality data, each step and station of the survey execution has to be documented. The careful documentation is

15

a precondition to guarantee high data quality. The documentation has to follow a common standard, which has to be developed by the research group. Also the data documentation has to be well co-ordinated, that the documentation and archiving efforts can be kept to a minimum. In the chapter for documentation and archiving the problem is discussed in detail.

1.3.2.7 Funding

To guarantee a common standard in survey design and execution and to provide a high level of data quality a secure funding is necessary. Country dependent the amount of money to carry out a high quality survey is varying. Although the steps towards the design of a good cross-sectional survey seems to be clear there are many sources of errors and biases which can influence the survey and the data quality. The sources of bias and error will be discussed in the following section with special focus on cross-national surveys.

1.3.3 Errors and biases in survey research

Alwin (1991) gives an overview of the different types of survey errors which can occur. He informs about the synthesis of perspectives on survey error Groves established in 1989. Groves differentiates according to Alwin (1991) between § coverage error (referring to not included parts of the population), § sampling error (sample does not represent the population), § non-response error (data from groups of the population could not be obtained) and § measurement error (value obtained is not the true value). According to Groves (1989) the measurement error includes interviewer bias, respondent bias, instrument bias and mode bias. Alwin discusses the approach of Groves and emphasised the survey measurement error. He describes two possibilities to evaluate the quality of survey measurements: reliability and validity analyses. He worked out a theoretical concept for measurement error and considers the response error as one of the most important sources of measurement error. The author stated, that a better understanding of the various cognitive processes involved in answering survey questions across content, can lead to better survey data and increase data quality. The cognitive processes he refers to are the "comprehension and understanding of the question and information", the "access to or the ability to recall the information", the "capacities for fo rmulating a response to the question" and the "translation of that response into the response categories provided by the survey instrument" (Alwin, 1991).

16

The coverage bias, sampling bias and non-response bias can be minimised by appropriate strategies for defining the population, sampling strategies and handling non-responses and refusals. The measurement error is more crucial and will be covered by the next section. According to Groves (1989) this bias consists of respondent bias, instrument bias, interviewer bias, and mode bias.

1.3.3.1 Respondent bias

The respondent is a major source of bias. It is very important especially for comparative survey research, that the meaning of the question will be understood by the respondents (a detailed discussion of the equivalence will follow on another place in this paper). The cognitive processes, which have to be launched in answering a survey question, influence how the respondent will answer. Within prestests the cognitive processes of the respondent like the comprehension and interpretation of the question, the retrieval of relevant information from memory, the evaluation of the retrieved information and the communication of the final response (Borgers and Hox, 2000) can be observed for example using the "think aloud techniques" (Prüfer and Rexroth, 1996) and the question quality be analysed. It is helpful to clarify if the meaning of the question can be understood by the respondents. Many authors consider both respondent and question characteristics as responsible for response quality (Borgers and Hox, 2000; Alwin and Krosnick, 1991). Especially in cross-cultural surveys the cultural orientations effect survey responses. Johnson et al. (2000) analysed the underlying dimension of culture, which influence survey behaviour directly. Within their non-representative survey the authors analysed the relationship between the cultural orientations (individualism and collectivism) and the tendency for socially desirable responses of students in Germany, Turkey and the United States. The results reveal, that the culture of the respondent may influence survey related behaviour. Survey illiteracy has also to be mentioned in this context. It plays an important role for data quality if respondents are familiar with survey techniques and are used to them or if they did not ever take part in a survey. Also the public image of surveys mainly when the image towards survey research is not very high in a country or region can have an influence on the responses.

1.3.3.2 Instrument bias

The survey instrument is beside the methodological execution a very important aspect to guarantee data quality. Questionnaire design, question specific characteristics like

17

question wording, length of introductory text and information type and response scale characteristics as type and length of response scale, explicit midpoint, symmetry scale, "don't know option" and the direction of first presented category have according to the analyses of Scherpenzeel and Saris (1997) an influence on validity and reliability of survey data. Alwin and Krosnick (1991) investigated the influence of question and respondent attributes on the reliability of survey attitude measurement. They stated, that older people and persons with less schooling produce less reliable results. Concerning question characteristics they observed the effect, that questions with more response possibilities have higher reliability as well as extensive verbal labelled response categories but these results are depending on the research domain. Offering a "don't know" option does not increase reliability. Alwin (1997) analysed the effect of the number of response categories and the measurement quality in survey research in a quality of life survey. He could support the hypotheses, that more categories in the response scale are more informative and are therefore more reliable and more valid. He could show, that the systematic measurement error does not go along with an increasing number of response scale categories. Beside these question and response scale characteristics also the context characteristics (mode of data collection, the position in battery, the position in questionnaire), the research domain and the country (in cross-national survey research) have also impact on data quality (Saris et al., 1998). In order to develop a measurement instrument which will provide high quality data it is necessary to evaluate the survey questions. Prüfer and Rexroth (1996) inform about procedures, which are appropriate to evaluate the questions. These procedures can also be used in cross-national research to detect if a question has the same meaning in a translated version as in the original one. The different procedures, which are of interest, are the think-aloud techniques. Two different types can be differentiated: § the concurrent think aloud method which monitors the thoughts during answering

the question and § the retrospective think aloud method in which the respondent is asked to inform

about how the answer has been processed (Prüfer and Rexroth, 1996). Another method is probing. Probing means, that the interviewer asks several additional questions (probes) to get more information about the answer. Four versions have been differentiated: § follow-up-probing, which means, that the additional questions will be asked

immediately after the answer,

18

§ post-interview-probing which takes place after the interview, § comprehension probing to ensure, that the question has been understood and § information retrieval probing which gathers information concerning the answering

process for example the difficulty to answer the question. Confidence ratings are also mentioned by Prüfer and Rexroth (1996) as procedure to evaluate the reliability of the answer of a person as well as paraphrasing (reproducing of the question with own words after answering it), the different sorting methods (free sort, dimensional sort, vignette classifications), which shall inform about the categorisation and the concepts a respondent uses and response latency. Response latency is the time between question and response and is considered as an indicator for question quality (for details see Prüfer and Rexroth, 1996). Other procedures to evaluate survey questions suggested by Prüfer and Rexroth (1996) are focus groups and expert judgements. Focus groups can be executed in the questionnaire design stage in order to discuss and improve the questions and to discuss and interpret the pretest results. The authors mention the disadvantage of focus groups, that within a discussion social interaction will take place under the participants and will influence the result of the focus group. Expert judgements are possible and helpful at every stage of the questionnaire design. The judgements shall be coded according a well defined codesystem (Prüfer and Rexroth, 1996). The scale construction and especially the response scale construction has also to be carefully carried out because data quality is depending on the response scale and several other question characteristics (Saris et al., 1997). Prüfer and Rexroth (1996) give also valuable hints how to carry out a pretest in order to provide a well analysed and designed measurement instrument. The instrument bias can also be identified by a pretest. The authors outline the concept of a standard pretest in the following way: the survey should be executed in a sample of 20 to 50 respondents (quota or random) under conditions the survey will be executed. The task of the interviewer is to monitor and report about problems during the survey execution. Another method to examine survey question quality and to detect problems for example in cross-national questions is behaviour coding. This is a method to monitor systematically the behaviour of the respondent and the interviewer in order to gain information concerning question quality. For details see Prüfer and Rexroth (1996). Welkenhuysen-Gybels and Billiet (2000) compared different techniques for detecting cross-cultural inequivalence at the item level. They compared different detection methods (loglinear model, the logistic regression model, the signed and the unsigned area between the item characteristic curves, the sum of squares 1 and the sum of square 3 measures) to identify uniform and non-uniform item bias and pointed out, that these results only refer to a specific simulation situation and cannot be generalised.

19

To guarantee high data quality it is very essential, that the instrument is carefully designed and analysed in a pretest. Each single item or scale should be tested, if it measures what it ought to measure and if it reliably measures this construct or dimension (for constructing attitude scales for international comparison see Krebs and Schuessler, 1986). Construct equivalence is important for cross-national surveys when survey results should be compared across societies or time. The overall aim of the survey has to be to provide high quality data for well grounded indicators. Therefore a very careful quality check of each single indicator has to take place as well as the permanent observation of the data quality from country to country and wave to wave. The questionnaire needs to be pretested and data have to be analysed concerning data quality. Reliability and validity checks must be executed. Many research initiatives prefer to use already asked survey questions for comparative reasons but also because they allow quality checks so, that a pretest does not seem to be necessary. Within the EuReporting subproject 2.4 a database covering survey questions of comparative research programmes has been established. The database also informs about already executed quality analyses for those survey questions. But this does not mean, that no innovative new survey questions should be developed. Regarding the survey question construction guidelines which have been published by several authors it is a necessity to create new survey questions in order to care for the new developments in the different life domains at the individual and at the societal level. To finish the section about the instrument bias the need for qualitative research methods in cross-national and cross-sectional survey research shall be stressed. During the questionnaire construction phase, during the process of questionnaire translation and during the data interpretation phase it is very essential to reflect the meaning of the questions and responses. Qualitative approaches like already considered by Prüfer and Rexroth (1996) gain growing importance in a world of increasing complexity and differences. Group discussions, focus groups, expert groups are needed to develop cross-cultural or cross-national survey instruments. These qualitative research methods can help to identify inequivalences and inappropriateness of questions, prevent measurement errors and invalidity and provide last but not least high quality data.

1.3.3.3 Interviewer bias

The interviewer is with exception of the self completence questionnaire surveys another source which can cause a bias. Many experimental studies carried out throughout all disciplines employing sample surveys are available which have the task to analyse which effect can be watched using which type of interviewer and which type of interviewing style on which questions and research domains.

20

Dijkstra (1987) analysed the impact of personal versus formal interviewing style on social desirability and on relevant and irrelevant information. The results show, that a personal interviewing style led to a higher arousal so, that in consequence more relevant but also more irrelevant information has been produced. The tendency to conformity with the interviewer could not be observed. The interviewer bias is very closely related to the response bias and could - following the authors - be reduced by motivating the respondent to act according to the task's requirements after clarifying them (Dijkstra, 1987). The mode of conversational interviewing has been inquired by Conrad and Schober (2000) and the results show, that conversational interviewing produces more accurate measures. A negative aspect of conversational interviewing is, that clarifying questions makes an interview longer and therefore more expensive. Houtkoop-Steenstra (2000) stresses, that in personal interviews it is very important, that the interviewer behaviour is standardised in order to improve data quality. Beside the style of interviewing also the number of interviews per interviewer can have an impact on data quality (Kaase, 1999). For minimising the interviewer bias strict guidelines concerning the interviewer instruction have to be worked out and the interviewer has to be trained well (with emphasise on the treatment of refusals and non-respondents). The interviewer control is necessary for providing a high standard regarding data quality. Some initiatives (Burstein et al., 1985) suggest to enter the demographic variables as independent variables into the survey documentation and into the data set for identifying interviewer effects.

1.3.3.4 Mode bias - method of data collection

The method of data collection has an empirically proven influence on data quality and on survey results. For example Kalfs and Saris (1998) give an overview of the recent studies on mode effects and describe the different approaches of the authors working in this field of research. Although some results are contradictory the authors state, that within experimental designs the mode of data collection has no effect whereas within field experiments data collection effects do influence data. Within their study investigating the different modes of data collection Kalfs and Saris (1998) detected large differences for various modes of data collection systems. The team analysed the paper pencil mode, the computer assisted self interview and the computer assisted telephone interview. They found considerable differences due to the mode of data collection. But they also stated, that the mode of data collection itself cannot be considered to be the only variable, which is responsible for the difference obtained in the analyses.

21

Saris and Kaase (1997) published analyses of the Eurobarometer measurement instrument. On the base of the Eurobarometer 41 a panel survey has been executed on the one side via face-to-face interview and on the other side via telephone interview across the three countries France, Belgium and Spain. Different authors investigated on the base of these data sets the effects of sample design (Schubert and Greil, 1997), deviations from the population (Häder und Gabler, 1997), telephone ownership - a cause of sampling bias in Europe (Lass, 1997), mode effects (Saris and Hagenaars, 1997) especially on open ended agenda questions (Schmitt, Schrott, Thoma, 1997) and the comparability across mode and country as well as the adjustment for differences between face-to-face and telephone interviews (Saris, 1997). Saris and Hagenaars (1997) analysed panel data from the panel version of the Eurobarometer 41 concerning mode effects across countries. The two modes under investigation were telephone interview versus face-to-face interview and the countries, where the surveys have been executed were France, Belgium and Spain. The topics which have been investigated were the EU-membership, EU-benefit for the country, satisfaction, political interest and media involvement. The analysis of the questions show differences in the response probabilities in some countries for special Eurobarometer questions. Beside this result the authors could further prove, that if the response probabilities are very different, the meaning of the question is not the same. The analyses of the different authors can be summed up to the following results: the survey results are depending on the issues, on the mode of data collection, on the sample design, on the country and on the question design. Each single component has a more or less large impact on the survey results. Especially for comparative survey research it is very essential to examine the impact of each component, that comparisons can take place, that real differences can be identified and not differences due to the different survey design. That the visual design of the questionnaire executed in self administered mode and computer aided questionnaires plays an important role in effecting data has been analysed by Dillman (2000). Different data collection methods offer different possibilities for presenting questions or response scales. In face-to-face interviews for example the option to show cards is frequently used asking for income. Sensitive themes can be raised with cards. In telephone interview this mode of gathering information is not possible (see Kaase, 1999). Often missing values are the result and this means less data quality. Burstein et al. (1985) analyses the different failures which have an impact on data quality. The scientists pointed out the consequences of an imperfect data collection (e.g. measurement unreliability, incomplete or unrepresentative measurement, systematic differences between experimental and control or comparison groups) can lead to

22

overestimation of impacts and can violate the survey intentions (Burstein et al., 1985). The authors provide a list of inconsistencies in data collection which can appear. The only way to avoid these influences is according to Burstein et al. (1985) trying to prevent collection inconsistencies and to document carefully the data collection procedures by a standard reporting scheme which has to be developed regarding all sources of inconsistencies. Monitoring and documenting the data collection can help to assess the quality of data and also the potential data collection defects. The authors further suggest, that length of time of interview, number of unanswered items, dropouts from the programme,... should be enclosed as independent variables as well as the social demographic characteristics of the interviewer. The authors suggest, that it should be possible to develop an "epidemiology of the respondent burden" (Burstein et al., 1985). De Leeuw et al. (1996) investigated the impact of the data collection mode on data inquiring loneliness and general well-being. They found out an influence of the data collection methods on the relationship between variables and recommend, that for using mixed mode designs for surveys it is important to include the mode of data collection as a control variable in the data set and also in the analyses. Generally mixed mode surveys are considered to reduce non-respondents1. To guarantee, that the surveys are representative telephone interviews are not adequate. Because the probability to reach every household or person by telephone cannot be regarded as equal for the whole population. The interviews in a representative survey have to be carried out ideally face-to-face in order to reach a representative sample. Alternatively one can think of CAPI (Computer Aided Personal Interviews) interviewing but the resentments towards computers often watched by old aged persons can influence survey participation and can cause a bias. Nevertheless in times of very fast technological progress also survey research has to try to profit from the advantage and possibilities offered by for example new media and also from the different research endeavours undertaken dealing with survey research to improve the methods and techniques in survey research. Permanent methodological research analysing different effects of the various influential variables is necessary to provide high quality survey results and guarantee them also for the future. At every stage of development towards an ideal survey every single characteristic of the survey and especially the execution has to be documented to provide the information for analyses. It is important to code all information about the interview, the respondent, mode of data collection, time,... and enter the characteristics as independent variables

1 This issue has been discussed at the Fifth International Conference on Logic and Methodology in

Cologne, 2000.

23

into the data set, that occurring intervening effects can be identified and eliminated gaining a high quality data set.

The necessity of permanent methodological research for the future survey research

Several national as well as cross-national survey programme initiatives have established methodological research groups to analyse experimentally the impact of different survey characteristics and problems. The International Social Survey Programme Group has established a Mode Working Group, which has the task to analyse the mode effect on survey results. The modes, which are allowed in executing the programme surveys are the self-completence and the face-to-face mode. The principal investigators are asked to document carefully the information how adaptation across the modes took place. Also a Non-Response Working Group is working on cross-nationally valid codes. A part of the British Attitudes Surveys is carried out as a self completion module covering sensitive questions. Self completion parts in surveys are often used for methodological research reasons. Also split ballots, which are also frequently applied in the German General Social Surveys are used for methodological research. In the following chapter the issues one has to face in cross-national survey research are addressed and methodological approaches for future research outlined.

1.4 Cross-national and cross-sectional survey research The main objective in cross-national survey research is to gather comparable data across countries. In order to provide survey results for comparison equivalent survey instruments have to be designed, which were executed in an equivalent way in all countries and have to be equivalently analysed.

1.4.1 Some theoretical and empirical considerations towards survey equivalence

As already mentioned in EuReporting Working Paper 7 (Hudler, Richter, 2000) Johnson (1998) describes and discussed the various forms of equivalencies in cross-cultural and cross-national survey research. He differentiates basically between two types of equivalencies § the interpretative equivalence and § the procedural equivalence.

24

Interpretative equivalence covers the conceptual side whereas procedural equivalence deals with "the measures and procedures used to make cross-cultural comparison" (Johnson, 1998). The author subsumed the different equivalence types he considers as relevant for his two meta equivalence types and offers bibliographic hints for each single type. Furthermore Johnson proposes methods for addressing the equivalencies in cross-cultural survey research. In his report he listed and discussed the different procedures to guarantee equivalence as far as possible regarding the following phases: § the question development phase (expert consultation/collaboration, ethnographic

and other qualitative approaches, good question wording practices, good translation practices and facet analysis),

§ the questionnaire pretesting phase (cognitive interviews, structured probes, measuring response category intensity, comparative behaviour coding, compare alternative data collection modes),

§ the data collection phase (using multiple indicators, using both emic and etic questions, respondent/interviewer matching) and

§ the data analysis phase (item analysis, item response theory, generalisability theory, confirmatory factor analysis, multidimensional scaling, applying statistical controls, identity-equivalence method) (cited according to Johnson, 1998).

Johnson (1998) concludes, that in times where cultural diversity is increasing as well as the need to improve tools for developing cross-cultural equivalence for the future "theoretical and methodological assessments" of the equivalence problems in cross-cultural survey research should be encouraged (Johnson, 1998). Also Van de Vijver (1998) stated, that "bias and equivalence are integral elements of each and every cross-cultural study. ... In order to safeguard the highest possible level of equivalence, bias should be scrutinised in each and every stage of an empirical project. Hopefully, a serious concern for bias and equivalence will become a routine consideration in cross-cultural studies, in much the same way as validity and reliability have become standard concepts, that have deeply influenced our thinking about to design, administer, score, and interpret test scores. In an era, in which cross-cultural encounters are becoming more frequent and cross-cultural research is gaining monumental, it is important to design agreed-upon procedures to carry out such research" (Van de Vijver, 1998). Van der Zouwen (2000) analysed the difficulty of questions used, the clarity of wording and the comparability of the results of the ISSP questionnaires. His research based on the concept, that the variable response quality is depending on the difficulty of the question, the clarity of the task and a response biasing factor. As intervening factors he mentioned the probability, that a mistake or misunderstanding

25

will occur, the motivation, that the respondent will answer as well as the length of the questionnaire. Difficulty means in his concept, that the cognitive operations, which are sources of bias are § retrieving information from long term memory, § storing information about the question in short-term memory, § that the question relates to a hypothetical or a future situation, § a more precision of the response is required, § that the response alternatives are inadequate, and no use of responding aid devices, § the problems of self presentation for the respondent and § that the information asked for is hardly accessible (cited according to Van der

Zouwen, 2000). The clarity refers to the ambiguity of the question, unfamiliar words used in the question, absence of clear instructions according to Van der Zouwen (2000). As biasing factors he mainly considers the balance of wording. The inadequacy of the questionnaire in its entirety (logical order of the questions,...) is further mentioned as a source of bias. Referring the assessment of the comparability he analysed the equivalent wording, the wording and coding of response categories, the context of the question, and the mode of administration of the questionnaire, the similarity of the objects to be evaluated and the similarity of the survey samples. Van der Zouwen (2000) analysed theoretically all the influences and gave recommendations for the ISSP. Saris (1998) analysed the effect of measurement error in cross-cultural research following the MTMM (multi trait multi method) approach. Within the International Research Group for Methodological and Comparative Survey Research (IRMCS) quality analyses of survey instruments have been carried out. Reliability, validity and method effects for specific countries have been estimated. Saris (1998) emphasised the necessity for measurement error correction because he had proven, that survey results across countries even assessed with the same method are not comparable, if they have not been corrected for measurement error. According to the results of Saris the assumption that equal measurement procedures provide comparable results across cultures is not correct. The author describes the effect of measurement error on the correlation between variables but furthermore mentioned the other influential sources like coverage differences, fieldwork differences, mode effect,.... (for details see Saris 1998), which effect comparability of survey results in cross-cultural surveys. In cross-national survey research the equivalence of the source and the final questionnaire is a very crucial point. The present discussion on translation and

26

adaptation issues in providing questionnaires for multi-cultural and multi- lingual use are outlined in the next section.

1.4.2 Developing a measurement instrument for multi-lingual and multi-cultural use

Harkness and Shoua-Glusberg (1998) give an excellent overview of the survey translation procedures applied in cross-national survey research and emphasise the different advantages and disadvantages of each technique. In principal a translated version of a questionnaire should be equivalent with the original version. Meaning equivalence should be provided. The authors discuss the techniques "decentering", the "direct" or "one-for-one" translation, the committee and modified committee translation, the close and literal translation, the advanced translation and translation "on the fly". The approaches differ in the quality of the result as well as in time, skills, labour and costs spent. To evaluate the questionnaire translations different assessment procedures have been discussed by the authors. Back translation, committee assessment, comprehension assessment and statistical analyses (e. g. split ballot assessments, double administration tests, post hoc analyses) are mentioned (Harkness and Schoua-Glusberg, 1998). Harkness et al. (2000) discussed the three main options for designing multi- lingual questionnaires: translating an already existing questionnaire, adapting a consisting questionnaire or designing a new questionnaire. They point out different sources of translation bias: § the linguistic bias which focuses on semantics and language structure, § the functional bias which means to keep the purpose for the user, § the dominant culture bias which focuses on the majority culture and § the feminist bias which deals with rejecting or avoiding gender bias (Harkness et al.,

2000). The team suggests for future cross-cultural survey approaches to consider the basic question designing requirements like designing ask- and answerable questionnaires, work with reliably tested versions and care for data collection. Furthermore they differentiate between known hazards like the impact of the oral traditional languages, low survey literacy in some countries and cultures and unexpected hazards, which can be identified within a pretest. For designing a multi- lingual questionnaire they suggest multi-cultural and multi- lingual teams, committee translation and probes as well as think aloud tests for evaluating the questionnaire versions and a pretest. Data analysis will then show the quality of the measurement instrument.

27

Hambleton and Patsula (1998) worked out the sources of invalidity of adapting tests and discussed the influence of the cultural and language differences (test format and the familiarity with the format), the technical design and methods and here especially the selection and training of the translators as well as the factors affecting the interpretation of the results. Translation can effect various types of equivalencies and therefore effect data quality and comparability. It is essential to keep the dimensions and constructs equivalent, to provide functional equivalence, in the source and in the target questionnaires (see Braun and Scott, 1998). The meaning should be kept equivalent of the question text and response scale translations (see Mohler, Smith, Harkness, 1998). But even if the translational equivalence is safeguarded it is not guaranteed, that data from different cultures are comparable because the concepts are different. Therefore it is important, that experts from the cultures where fieldwork will take place are involved in the questionnaire design and translation as well as in the analysing and interpretation processes. Drawing a conclusion from all the biasing influences, which have an impact on data quality a replication of the survey before fieldwork will start is necessary to guarantee high data quality, to identify as much as possible intervening variables and to provide reliable and valid survey results.

28

1.5 Summary and Conclusions Future concepts and methods in survey research should be based on well designed, documented and scientifically evaluated concepts and theories. Especially in welfare research a comprehensive concept is necessary which covers all facets of life responsible for social welfare. The concept should at the one hand be based on empirical evidence but on the other hand open for new and innovative concepts regarding new societal developments. A very useful concept for welfare research seems to be the "European System for Social Indicators", developed by Berger-Schmitt and Noll (2000). Beside a theoretical concept for the thematic coverage permanent methodological research and evaluation of the concepts measured are demanded in order to guarantee high data quality and to be able to identify societal changes as well as to improve indicators and survey methodology. It has to be ensured, that a measurement instrument does not measure artefacts instead of real changes and differences. Methods and survey techniques and data quality analysing methods have improved during the last decades. The research results in survey research, the results of methodical experiments and analysis concerning survey execution, survey design, questionnaire and question design as well as documentation should be regarded in future approaches of cross-sectional survey in order to improve and increase data quality and provide comparable data across time and space. Qualitative methods like focus groups and group discussions (expert discussions) help to guarantee data validity at any stage at the survey design, translation and interpretation process. As pointed out in the prior chapters of the paper surveys have to be well designed being aware of all influences and biases, which accompany each decision on content and methodology applied in the survey. Mode effects, interviewer bias, the respondents bias, the instrument bias, the population bias due to non-respondents or refusals, and many more, which are influencing data quality as well as survey results and have especially in cross-national survey research a negative impact on data comparability. The question design is also very crucial. The reliability and validity of survey results are due to question position, context variables, type (numeral or verbal) and length response scales, type of information ask etc. (for details see Scherpenzeel and Saris, 1998). Quality criteria in survey research have recently been discussed by Kaase (1999). He gave an overview of the quality determining factors in survey research and presented the good and best practices in survey research. He emphasised, that it is necessary to

29

develop standards in survey design and execution and documentation, which have to be harmonised especially for international comparative studies (Kaase, 1999). A permanent observation, evaluation and documentation of data quality in order to provide high data quality, comparability across time and space and to minimise measurement error or invalidity. In cross-national or cross-cultural survey research the problems of equivalence have a larger impact on data quality and comparability than in national cross-sectional research. As Johnson (1998) pointed out a variety of equivalencies which he summarises to interpretative and procedural equivalence have to be regarded when designing an international comparative survey. Harkness et al. (2000) and also Harkness and Schoua-Glusberg (1998) provide useful and practical suggestions how to design international comparative surveys and how to provide equivalence of survey results as much as possible. To get hold of all these impacts which can bias survey results careful standardised documentation of all possible and up to now known influences is necessary. For the methodological evaluation of the data quality it is essential to code the intervening factors and enclose them as independent variable within the data set as several researcher have already recommended. The next section of the paper will try to outline future directions in documentation on cross-sectional surveys.

30

2 Theoretical and Methodological Concepts for Future Documentation on Social Reporting in Cross-sectional Surveys

A great deal of scientists working on social welfare issues are using already collected data for secondary analyses within their research. The advantage of sharing data is that the costs for raising data can be avoided, no duplication of effort will take place, data will be explored more extensively by various methods and analysis following different research intentions which aims also in an improvement of data quality (collecting methods, improvement of questionnaire design,...). But providing data for secondary analysis means that archives have to prepare them for user. What information about a survey is necessary for secondary analysts, that one can work scientifically with already collected data? The answer to this question is: everything. This means for elaborated secondary analysis of data and for preventing data abuse or incorrect use a comprehensive documentation of the survey including the survey intention, the theoretical concept behind, information concerning the survey design, the survey execution and the survey storing conditions have to be available as well as a well prepared data set regarding special access conditions. For the use of the dataset the questionnaire in original and the translated versions of cross-national surveys are necessary beside good codebooks. Also already published analyses are helpful for secondary analyses especially when data quality has been investigated. Information concerning data quality is very essential for secondary analysts. Data quality extremists might say that only such surveys and data are worth storing in data archives which have an excellent survey documentation describing all survey characteristics (dimensionality, sampling strategy, population, response rate, reliability, validity,...) which are considered as relevant for a high data quality. The provision of such an extensive amount of rather different information and products means a great challenge for data archives whose task is to prepare survey data and provide them for secondary analyses. The data archives offer the information they have got for archiving. But it is often very hard to get full documentation of surveys and data from the principal investigators as the archives complain. The consciousness of researcher has to be awakened that data archives shall be the place for storing survey data. Surveys have to be carefully documented and the organisation of them within a retrievable database have to be based on a scientifically well grounded concept. The job of the archives as outlined in the ICPSR-Guide to Social Science Data Preparation and Archiving - 1999 (http://www.icpsr.umich.edu/EDSEARCH/PDF/dataprep.pdf, 21.08.01) is to document the surveys by providing information about the principal investigators, the title of the data collection, the funding sources and the related acknowledgements, the persons or organisation which carried out the data collection, a project description (theoretical

31

concept of the survey), sample description and sampling procedures, the date and the place of data collection as well as the covered time period,... (for details see the extensive amount of meta- information provided by the Danish Data Archive (http://www.dda.dk, 21.08.01) and the ICPSR Guide to Social Science Data Preparation and Archiving). The archives have beside the documentation of meta- information the major task to clean the data set and provide the data in a well documented format for secondary analysts. The preparation of codebooks is also the task of the archives and this is a rather time consuming effort. The data source, the units of analysis or observation and the variables have to be reported. The documentation of constructed variables and the data collecting instruments are also important. Variables are usually described by the exact wording of the question, the item or question number, frequency distributions of the unweighted data, missing data codes, editing information, meaning of codes and location in data file (ICPSR Guide to Social Science Data Preparation and Archiving - http://www.icpsr.umich.edu/EDSEARCH/PDF/dataprep.pdf, 21.08.01). The development of the world wide web offered a new challenge for the data archives. Suddenly there was the possibility to present and to get information concerning social survey data from all over the world. Information about surveys stored in different data archives could be provided for an international user group via the new media "Internet". The data archives developed a common standard for presenting mainly meta-information in searchable catalogues at the homepages of the data archives. The extent of meta-information provided by the different archives across Europe differs quite considerably. A very exhaustive documentation on meta- information is available at the Danish Data Archive (http://www.dda.dk, 21.08.01). A common minimum standard on meta-information is provided by each of the archives but some like the Data Archive in Essex (http://www.data-archive.ac.uk, 21.08.01) or the "Zentralarchiv" in Cologne (http://www.gesis.org/en/za/index.htm, 21.08.01) offer additional facilities like downloadable questionnaires, searchable codebooks, frequencies table s, visualisations of that tables and bibliographies of data or survey related publications. The possibility to download certain data sets for secondary analysis is also often allowed by the archives.

2.1 Attempts of data archives and research institutes to publish survey data and information

The Council of European Social Science Data Archives - CESSDA (http://dasun3.essex.ac.uk/Cessda/IDC, 21.08.01) developed an integrated data

32

catalogue where the data catalogues of most of the European Data Archives and some other are online searchable by title of the study, names of related persons, content, year and by geographical coverage. It is also possible to select a single archive catalogue which is integrated into the catalogue of the CESSDA. The archives which can be separately investigated are the French, the Danish, the British, the Norwegian, the Swedish, the Israelis, the Dutch, the Hungarian, the German and a Northern American one. For establishing such an integrated catalogue a common meta-data and data structure is necessary. There are some initiatives which have developed a common format like the DDI (Data Documentation Initiative). "The Data Documentation Initiative (DDI) is an effort to establish an international criterion and methodology for the content, presentation, transport and preservation of meta data about data sets in the social and behavioural sciences. Meta data (data about data) constitute the information that enables effective, efficient and accurate use of those data sets" (Data Documentation Initiative: A project of the Social Science Community - http://www.icpsr.umich.edu/DDI/codebook.html, 21.08.01). The Document Type Definition (DTD) has been generated by the DDI consortium and applies the eXtensible Markup Language (XML) for social science codebooks. Another approach which is used for example in the LIMBER project intends to overcome language boarders by providing meta data in the RDF-framework. The DDI codebook standard will be transformed into the RDF-framework (Miller, LIMBER-workshop, 2000). "The RDF-framework (Resource Description Framework) integrates a variety of web-based meta data activities including sitemaps, content ratings, stream channel definitions, search engine data collection (web crawling), digital library collections, and distributed authoring, using XML as an interchange syntax" (W3C - Technology and Society Domain - http://www.w3.org/RDF/, 21.08.01). The research result depends on the information provided by the single archive but anyway gives an overview about content, access conditions and some technical information, but little about data quality criteria. The quality criteria of surveys have been discussed in the first section of the paper. Especially the online survey catalogues make it possible that researcher can easily search for surveys investigating certain topics in countries. The meta information concerning the survey is very essential but a researcher still has to order the data and the data documentation from the archive. The online meta- information documentation of the archives does not include links to related web sites although valuable information on some large scale surveys is although

33

available in the web. In order to provide comprehensive information of surveys the integration of links to web based sources would be very useful. Some organisations or principle investigators of survey programmes offer various information concerning their surveys at their own homepages. The Centre for the Study in Public Policy (http://www.cspp.strath.ac.uk, 21.08.01) offers at the one side theoretical and methodological information of the "New Democracies Barometer I to V" as well as bibliographic hints of survey related literature and at the other side questionnaires and a retrievable question database covering comparable survey questions which allows online analyses of data. Other research initiatives like for example the International Crime and Victimisation Survey Group or the International Social Justice Survey Committee present their survey programmes also in the world wide web but not that extensively. However, most of the principal investigators want the archives to document and store or publish their surveys and survey data. New media development like CD-ROM and Internet and different software packages offers different possibilities how to present data in an user- friendly way. The approach to publish the German National Election Studies from 1949 to 1998 on CD-ROM by the Zentralarchiv in Cologne (Zenk-Möltgen, 2000) is one efficient possibility to provide data for interested user. Especially this German approach is a very useful and user- friendly attempt to serve interested persons by offering survey data from 30 German national election studies in a SPSS portable format, the codebooks of the "Zentralarchiv" in a PDF-format, the questionnaires of the studies and the ZA-codebook explorer. An explorer allows to search through the codebooks by different selection fields (e.g. keywords). Data analyses can also be carried out and the results be visualised (Zenk-Möltgen, 2000). The "Social Indicators Department" at the "Zentrum für Umfragen, Methodik und Analysen (ZUMA) in Mannheim (http://www.gesis.org/en/social_monitoring/social_indicators/index.htm, 21.08.01) publishes a "Digital Information System for Social Indicators (DISI)" (http://www.gesis.org/en/social_monitoring/social_indicators/Data/Disi/disi.htm, 21.08.01) which provides data collected in Germany for 14 life domains relevant for welfare measurement and social reporting since the early fifties to the late nineties. About 400 indicators and 3000 time series are covered by the system which enables a user to monitor the societal and political changes and allows comparisons of subgroups. The system offers the visualisation and the exportation of the results into other systems and files. Via different search options the indicators can be viewed in German and in English language.

34

For comparative investigations of German social indicators DISI is an excellent tool. Up to now DISI is not web based. It is available in a CD-ROM version and in a downloadable version at the homepage of the "Social Indicators Department" at ZUMA. The policies from the European Community towards an e-Europe enforces the information technology sector as well as the content industry to develop web based tools and supported data archives to open their archives for online researcher. This initiative aimed in several research endeavours in the field of social sciences data presentation in the World Wide Web. User- friendly archiving and documentation services have been designed to allow uncomplicated and easy access to survey data and data services from every place in the world. The development of the world wide web makes it possible. Therefore future documentation on social surveys have to consider the new media as a main information forum for social sciences. Different initiatives from the social sciences data archives are already following this way.

2.2 Different software tools to present social sciences survey data in an user-friendly way in the world wide web

The general policy for documentation and archiving has been outlined by the head of the Zentralarchiv für Empirische Sozialforschung in Cologne, Ekkehard Mochmann at the Fifth International Conference on Logic and Methodology in Cologne 2000. Mochmann emphasised the importance of the social science data archives to provide data and documentation for secondary analyses. He informed about the attempts to offer data and meta information as well as other special services within a virtual observatory for comparative social research. Furthermore he pointed out the importance of e-services for an e-Europe in the area of social research and mentioned the achievements of the data archives in order to provide standards and tools for exchanging data, to develop tools to overcome the language barriers and from the need for the implementation of network tools, that support collaborative working in the "Social Science Data Space" (Mochmann, 2000). As soon as Internet became interesting for social research the archives started to develop appropriate software and appropriate clients for interested persons in order to provide a user-friendly data library, online access to data and data documentation. In the following the attempts of the data archives towards e-data archives are described. Two very sophisticated and elaborated software systems for social science data archives have been developed as already mentioned in Hudler and Richter (2000). Up to now one is already online available. The first system which is presented in the following is NESSTAR.

35

2.2.1 NESSTAR - Networked Social Science Tools and Resources (http://www.nesstar.org, 21.08.01)

The NESSTAR project has been launched under the Fourth Framework Programme of the European Commission. It is a joint initiative of the Norwegian Social Science Data Services (NSD), the UK Data Archive and the Danish Data Archive (DDA). "The aim of the project is to develop a common interface on the Internet to the data holdings of a large number of providers and disseminators of statistical information world wide. By means of NESSTAR, users are able to: 1. locate multiple data sources across national boundaries, 2. browse detailed meta data about these data, 3. analyse and visualise data online, and 4. download the appropriate subsets of data in one of a number of formats for local

use. The system includes advanced user authentication procedures to prevent unauthorised use of data" (Ryssevik and Musgrave, 1999). By outlining the features of a social science dream machine Ryssevik and Musgrave, (1999) had to realise that there is a gap between the ideal of a social science data infrastructure and what could be realised. The authors characterise the dream machine on the one side as follows. The ideal system should provide: "all existing empirical data available on- line; an integrated resource discovery gateway and search-system in order to identify and locate these resources; extensive amounts of metadata available (multimedia, hyperlinked and totally integrated with the data as such); the ability to browse and visualise data on- line; the ability to convert the data in one of a number of formats and copy, with the metadata, to a local machine; 'active research agents' (knownbots) mining the net and informing the user when new data within their special field of interest are made available; efficient hyperlinks from the data sources to every scientific publication ever produced on the basis of a dataset; e-mail/web addresses to all relevant researchers, departments etc. and an efficient feedback system to the body of metadata allowing the user to add to the collective memory of a dataset" (Ryssevik and Musgrave, 1999). On the other side they state that "only a fraction of the relevant data-sources are available on- line; no common gateway/search system to the available data; a lack of standardisation especially on the metadata side; poor integration between data and metadata; institutional obstacles (competition); juridical obstacles (privacy) and commercial obstacles (information as a commodity)" (Ryssevik and Musgrave, 1999). But anyway NESSTAR is one step in the right direction. NESSTAR allows online searches across archives using a search term. A list of surveys and survey description

36

are the search result. The information is organised by an explorer which also offers beside the navigation through the meta information the possibility to go through the variables. The very useful and comfortable feature is to carry out online analyses. Simple statistical procedures like frequency analyses, crosstabs, break downs, correlation and regression can be executed, the results can be visualised and exported into text files. If these analysing procedures are not satisfying for an experienced user there is the possibility to download data in different formats to serve the end user and to allow the application of advanced statistical analyses or index constructions. Certainly different access conditions have to be regarded. In a next step it is intended to link already published text or other related information to NESSTAR in order to provide comprehensive information concerning survey, data and publications. The possibilities offered by NESSTAR go very far in the directions of a social scientist's dream machine. But from the point of view of a user there are still some disadvantages which should be eliminated in order to provide a user-friendly and efficient system. § At the time it seems that there will not be a comprehensive survey documentation in

a system like NESSTAR covering all surveys available in European data archives because not each archive will join the initiative and provide the data and information in the NESSTAR system. NESSTAR seems to be only supported by a few data archives. Up to now not all studies stored even in the participating archives are online available. But agreeing on a common interface for international archive searches would make the life of scientists working with survey data much more easier.

§ Until now only a few surveys are documented in NESSTAR and not very many data sets are available for downloading.

§ Restricting the possibilities of analysing data to simple statistical procedures could also be a topic to think over.

§ Another disadvantage of searches across the NESSTAR system is that surveys are indexed. The thesaurus HASSET (http://biron.essex.ac.uk/searching/zhasset.html, 21.08.01) is used for indexing. A researcher working with social surveys especially in comparative social research is of course interested in the topic which have been investigated in the survey but is much more interested in the questions which have been used for raising a certain topic. Indexing on the question level would support researcher in order to find quickly appropriate or comparative data. Searches across survey questions which are also desirable for researcher and other interested persons are not implemented. Indexing on the question level would be very useful.

§ A researcher also wants to know how reliable and valid are the data gained by this survey. Therefore quality estimates should be offered. Information concerning data quality like reliability and validity would very important for secondary analysts and should be provided in a user- friendly way in such a system.

§ The whole surface and the content of the system is kept in English. This means that surveys stored in archives which are not available in English cannot be found or

37

viewed if there won't be a translation. But translations are only useful when they are carried out following the scientific guidelines for translations of survey questions (e. g. Harkness and Schoua-Glusberg, 1998).

§ A search option of surveys and survey data in the different national languages would be necessary if NESSTAR should be the tool for future data and survey documentation. A great achievement would be when search options are offered in each national language and also the results can be viewed in these languages. LIMBER, a project dealing with multi- lingual meta-data, will pioneer this field.

§ It would also be very helpful if comparative survey questions and survey data could be easily identified, time series (numeral and graphical) could be created and exported into other systems in different formats.

NESSTAR as well as other web and archive initiatives have not realised an approach which indexes survey questions and also provides essential information concerning data quality for interested persons and researcher. That is why the EuReporting subproject 2.4 started such an approach in order to provide an information centre for social welfare surveys which includes searches across survey questions guided by a science based indexing system and offers quality criteria if available. A follow up project of NESSTAR is FASTER (Flexible Access to Statistics, Tables and Electronic Recources). The basic intention of this project is to bring together official statistics and data archives to "develop advanced meta data models and systems" (Musgrave and Ryssevik, 2000). FASTER shall cover time-series data, aggregate data and all types of survey data in a flexible user- friendly environment including as much information (e. g. data related articles in journals, links and other multimedia objects) as possible. Active agents and the concept of bookmarks shall allow an efficient use of data stocks (Musgrave and Ryssevik, 2000). "In this environment the user will be able to create immediate access to any number of electronic resources, such as their favourite data sites, data sets, tables, search strings, user guides and wider resources. Not only will this assist the user in bringing together multiple data sources, but it will create an environment in which the goal of flexible user driven tabulations and visualisations become a reality" (Musgrave and Ryssevik, 2000). NESSTAR and the follow up project FASTER can be considered as very efficient and user friendly interfaces which are tending towards the ideal social sciences service machine which everyone working with social science's data desires. Searching for surveys, data, carefully documented meta information, related literature, related information as well as the possibility to carry out online analyses, visualise the results and download data for advanced analyses are features which would be advantageous for every user. The second initiative which develops a rather literature related tool (NESSTAR is data oriented) is ILSES.

38

2.2.2 ILSES - Integrated Library and Survey-data Extraction Service (http://ilses.gamma.rug.nl, 21.08.01)

ILSES (Integrated Library and Survey-data Extraction Service) - is a project of the iec ProGAMMA, the Zentralarchiv in Cologne, the University of Amsterdam, the NIWI-Archive, the Centre d' Informatisation des Donnees Socio-Politiques, and the Trinity College at the University of Dublin. This project has also been founded under the Fourth Framework Programme (Telematics Applications of Common Interest, Telematics for Libraries). The task of ILSES is "to develop a service that enables individual users to access and retrieve documentary information and empirical data related to large-scale surveys such as the biannual Eurobaromenter surveys" (http://ilses.gamma.rug.nl, 21.08.01). ILSES is designed as a tool for end-user and content provider. The designers emphasise that ILSES is an open system which can be used for libraries and data-holdings. The open system allows also the use of ILSES via Internet. In the following the different tools of ILSES will be described shortly. § Administrator-ILSES stores meta data and bibliographic data as well as the access

interfaces. § LIB-ILSES is applied for content-providers and end-users to establish and access

bibliographic data. § DAT-ILSES is used for content-providers and end-users to establish and access

meta data. § E-ILSES provides the electronic interface for end-users to get access to services

offered by LIB-ILSES and DAT-ILSES. § NET-ILSES is the WWW-based version of E-ILSES (http://ilses.gamma.rug.nl,

21.08.01). Although ILSES allows navigation through data and related literature, the possibility to analyse data is not provided as well as online visualisation. Up to now ILSES is not freely accessible via WWW, which means that one cannot search online for certain surveys and literature. Only in several lectures at conferences ILSES has been presented therefore the discussion of the different features cannot really take place at this stage.

39

2.3 Summary and conclusion Future knowledge discovery and retrieval systems for use in www have to regard according to Maurer (2000) certain requirements. To minimise network and server loads Maurer suggests distributed information sources. The systems have to be designed as open at the one hand for integrating information into its system and at the other hand the system itself should be exportable into other systems. It has also to be open for new developments. Filtering and ranking functions as well as intelligent agents are of use for future service systems. The integration of parts of the applications within the search system could also be helpful for expanding the system. Distributed and co-operating broker structures would allow searches across different sources at the same time. The results would be collected and summarised. Another requirement is that the system has to provide comprehensive information which has also to include external data bases, information services and links. The links have to be consistent and up to date. Via special programme routines the links can be checked, frequently updated and completed. Furthermore the system has to provide quality criteria and reliable results. Expert experience, user experience and additional information should be able to be integrated into such future systems. An effective and efficient indexing system is also necessary (Maurer, 2000). The aim of future documentation in social welfare survey research should be over all to be of service for user. Customer orientation is the keyword of the future. In order to fulfil this requirement the use of www, modern information technology and content industry have to be applied. Best practice would be that each information about and product of a survey to a certain topic can be identified and is available via user- friendly systems. Data and the access to information systems for secondary analyses should be free and easy. Active clients which inform about new data or research initiatives in an area of interest would be useful. Especially in times where the amount of information is growing and growing a clearing system which only informs about topics of interest are very wishful for social scientists working with already collected data. The ideal case would be that an user has not to leave the work place in order to get for example the information about the attitudes of women towards family in the Czech Republic in 1995. Even the visualisation of the relation between the attitudes of men and women towards this issue should be possible online. Fact is that there are some approaches of different data archives to present social sciences data. These approaches consider the www as new information platform, as new distribution media and provide a core part of information concerning surveys like meta information, questionnaires etc. and additional parts and tools related to the system (e. g. ILSES and NESSTAR) in the web. English has developed to the language which is mostly used in information platforms in the WWW.

40

Unfortunately a comprehensive system which provides all information (survey concept, theoretical design and methodological issues) also the information concerning surveys executed by official statistics, the data, online data analysing and visualisation tools, related literature, related information concerning data quality, related links,...) is not available although it would be very desirable. Future documentation should consider the wish of researcher and other interested user for comprehensive information. A standardised format of an elaborated meta information concept which informs about the survey concept and execution (theoretical and methodological approach) should allow an estimation of the data quality. Meta information has to include estimates concerning reliability and validity of the data gained by the surveys. As in the FASTER concept is discussed the introduction of a data quality index regarding well defined quality criteria of surveys (population concept, questionnaire design, questionnaire translation process, translation control, sampling procedures, data collecting methods, pretest results, validity and reliability estimates, ...) could be very useful for interested persons to identify scientifically valuable data sets. The criteria which should be considered for a quality index have to be well defined and provided together with the data set or even in the data set. Within expert groups such a comparative quality index could be developed. A quality index would allow a rather inexperienced user to select data sets of high quality. Certainly one should not totally rely on this quality categorisation and check critically how the quality categorisation has been executed but it would be a first selection step in order to examine data sets concerning data quality. Searching for reliability and validity analyses of survey data one has to realise that there is not very much information available respectively published concerning these quality estimates. Only a few surveys as the International Social Survey Programme and the German General Social Survey offer analyses dealing with data quality and methodological research results have been published frequently. But for providing data for scientific secondary analysis and data of high quality the consciousness for quality analyses of survey data should be enforced. One could argue that only surveys where reliability and validity analyses have been carried out within a pretest are worth storing in data archives. Data respectively survey selection regarding data quality is one way to guide user to an appropriate data set. But one can think of defining different user profiles to serve different user types like it is also discussed within the FASTER concept. Following such an approach the problem who defines the profiles and what would such an approach help or make the survey selection process more complicated. Targeted searches through data sets in order to identify comparable data should be enabled. This feature would allow to produce time series and find comparable data across time and space.

41

The future documentation systems have to be multi-lingual. Information and data have to be provided in a multi- lingual system. Researcher should be able to search for information concerning surveys and survey data in their national language and get the appropriate response in the language which has been chosen. Multi- lingual thesauri are necessary. A science based concept for translation purposes have to be worked out to guarantee high quality trans lations. A well designed science based thesaurus is essential for indexing surveys but even more important is an appropriate thesaurus for indexing survey questions. Indexing survey questions is absolutely necessary in order to provide and develop a real service for user of social science data. The indexing of surveys means if an interested person who finds a survey covering the topic of interest has to go through the questionnaire or codebook searching for the question text she or he is looking for. Searching across questions provides immediately the question of interest and comparable ones. These searches can be designed as a simple text search across question text and labels (like some codebook searches which are already available at the homepages of the "Zentralarchiv für empirische Sozialforschung" in Cologne) or the questions have to be reliably indexed using a well defined and appropriate scientifically based categorisation system as thesaurus. A combination of free text search and a well defined indexing system for categorisation would be very useful for a researcher to identify questions of interest. A scientific thesaurus has to be based on a scientific concept, providing enough specified keywords for indexing so that an user can easily find appropriate hits. Within an international thesaurus the keywords have to regard the national characteristics as well as the international ones. The keywords have to be cross referenced with related terms for comprehensive searches. The thesaurus has to be specific and well defined for welfare topics but also open for new concepts and developments. Unfortunately the thesaurus HASSET which is one of the most employed thesauri in the social sciences does not provide sufficient keywords and terms for different topics to index certain life domains in comparative welfare research in detail. For example the benefits an unemployed person gets differ across the countries as well as the levels of education. HASSET does not provide enough specified keywords for these life domains. Therefore it is recommended to use the different specific elaborated thesauri for indexing questions of certain life domains (ILO (1990) thesaurus for labour concerns and CASMIN - (Brauns and Scherer, 2000) for educational topics). The generation of a meta-thesaurus which brings together the rather general thesauri as well as the topic or life domain specific ones would be very useful for future documentation on social welfare. As the thesauri have already mutated into dynamic data base oriented ones it would make sense to generate a meta-thesaurus summarising all relevant and well defined thesauri which cover the life domains related to social welfare.

42

To summarise the requirements for future documentation on social reporting in cross-sectional surveys: § Future research and documentation on social welfare surveys has to be first of all

science based regarding old concepts and open for new developments. The indicators should be imbedded in social and economic theory.

§ Beside the reporting activities in reports, articles and CD-ROMs the world wide

web is the new platform where information has to be published. Information and data on social welfare have to be world wide easily accessible.

§ The new information systems in the www have to be designed as open systems

which allow the integration of already existing information as well as the development of new concepts. The system itself should also be able to get integrated into other systems.

§ New systems should offer comprehensive information on social reporting this

means information on official statistics as well as on academic surveys. § Internet based information systems have to provide elaborated and standardised

meta-information documentation and as much interactive services new information technology and content industry allow to serve user.

§ Online question databases, links to related web sites, links to data related articles,

links to other sources like related web pages, e-mail addresses and data bases, data quality estimates, codebook searches, downloadable data, online data analysis (even advanced analysis) and visualisation of results have to be a minimum requirement for these future systems.

§ The information system has to be user-friendly, provide simple and efficient

interfaces and work quickly. Therefore distributed information sources as well as distributed and co-operating broker structures are necessary to guarantee that systems can reply quickly and no troublesome network and server loads will occur.

§ The integration of expert and user experience shall be possible in an open

information system. § To regard the preferences of each user active intelligent clients shall help to select

information to a certain topic. Filtering and ranking possibilities of the search result are additional services which reduce server loads and help to achieve appropriate search results.

43

§ An important criterion for selection should be data quality. Even an inexperienced user should be able to identify high quality data sets. A quality index should allow to identify high quality surveys and survey data.

§ Qualitative high levelled documentation in multi-lingual systems with multi-

lingual comprehensive science based categorisation systems have to be offered by future documentation systems. Searches and results have to be optional multi-lingual in order to allow national surveys to be found by any interested person regardless of language concerns. This means of course scientifically high levelled translations considering the whole translation problematic occurring.

§ A multi-lingual categorisation system for social surveys covering welfare topics

has to be science based, detailed enough that appropriate keywords can be found for certain questions and open enough for new concepts and developments. For an international reporting system national characteristics as well as international comparative topics have to be regarded.

§ The system and the content of the data bases have to be kept up to date. This means

a permanent enlarging, updating the content and checking of information sources, especially of links which usually change the address over time or disappear is necessary as well as the frequent adaptation of the system to new developments of the information technology and content industry.

The future initiatives towards documentation on social reporting in cross-sectional surveys have to be guided towards the outlined directions in order to provide user-friendly, comprehensive services for social scientists, politicians and other interested users.

44

References: Alwin, Duane F.: Research on Survey Quality. In: Sociological Methods & Research, Vol. 20, Sage Publications, 1991. Alwin, Duane F.: Feeling Thermometers Versus 7-Point Scales. In: Sociological Methods & Research, Vol. 25, Sage Publications, 1997. Alwin, Duane F.; Krosnick, Jon A.: The Reliability of Survey Attitude Measurement. In: Sociological Methods & Research, Vol. 20, Sage Publications, 1991. Berger Schmitt, Regina; Jankowitsch, Beate: Systems of Social Indicators and Social Reporting. The State of the Art. EuReporting Working Paper No. 1, Mannheim 2000. Berger-Schmitt, Regina and Noll, Heinz-Herbert: Conceptual Framework and Structure of a European System of Social Indicators. EuReporting Working Paper No. 9, Mannheim 2000. Böhnke, Petra; Dehhay, Jan; Habich Roland: Das Euromodul - ein neues Instrument für die europäische Wohlfahrtsforschung. ISI 24, ZUMA-Mannheim 2000. Borgers, Natascha; Hox, J. J.: Reliability of Responses in Questionnaire Research with Children. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000. Braun, Michael; Scott, Jaqueline: Multidimensional Scaling and Equivalence: Is having a job the same as working? In: Cross-Cultural Survey Equivalence. ZUMA-Spezial Nachrichten Nr. 3, Mannheim 1998. Brauns, Hildegard; Scherer, Stefanie; Steinmann, Susanne: The CASMIN Educational Classification in International Comparative Research. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000. Conrad, Frederick G.; Schober, Michael F.: Conversational Interviewing and Data Quality. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000. De Leeuw, Edith D.; Mellengergh, Gideon J.; Hox Joop J.: The Influence of Data Collection Method on Structural Models. In: Sociological Methods & Research, Vol. 24, Sage Publications, 1996. Dijkstra, Wil: Interviewing Style and Respondent Behaviour. In: Sociological Methods & Research, Vol. 16, Sage Publications, 1987.

45

Fitzgerals, Robert; Fuller, Linda: I Hear You Knocking But You Can't Come In. In: Sociological Methods & Research, Vol. 11, Sage Publications 1982, 2-32. Friedrichs, Jürgen: Methoden empirischer Sozialforschung. Westdeutscher Verlag, Opladen 1980. Groves, R.M.: Survey Errors and Survey Costs. New York: Wiley 1989. Häder, Sabine; Gabler, Siegfried: Deviations from the population and optimal weights. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Harkness, Janet A., Schoua-Glusberg, Alicia: Questionnaires in Translation. In: Cross-Cultural Survey Equivalence. ZUMA-Spezial Nachrichten Nr. 3, Mannheim 1998. Hambleton, Rondals K. and Patsula, Liane: Adapting Tests for Use in multiple Languages and Cultures. In: Social Indicators Research, Vol 45, Kluvers 1998. Houtkoop-Steenstra, Hanneke: Interaction and the Standardized Survey Interview. The Living Questionnaire, Cambridge University Press, Chambridge 2000. ILO - International Labour Office: International Standard Classivication of Occupations: ISCO 88. Geneva 1990. Kaase, Max: Quality Criteria for Survey Research. Akademie Verlag, Berlin 1999. Kaase, Max; Saris, Willem E.: The Eurobarometer - A tool for comparative survey research. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Krebs, Dagmar; Schuessler, Karl F.: Zur Konstruktion von Einstellungsskalen im internationalen Vergleich. ZUMA-Arbeitsbericht Nr. 86/01, Mannheim 1986. Johnson, Thimothy P.: Approaches to Equivalence in Cross-Cultural and Cross-National Survey Research. In: Cross-Cultural Survey Equivalence. ZUMA-Spezial Nachrichten Nr. 3, Mannheim 1998. Johnson, Timothy; van de Vijver, Fons; Harkness, Janet; Mohler, Peter Ph.; Ozcan, Yusuf Ziya: The Effects of Cultural Orientations on Survey Response: The Case of Individualism and Collectivism. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000.

46

Laiho, Johanna; Lynn, Peter: Response Rates as a Measure of Survey Quality. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000. Lass, Jürgen: Telephone ownership - a cause of sampling bias in Europe? In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Lass, Jürgen, Saris, Willem E.; Kaase, Max: Size of the different effects: Coverage, mode and nonresponse. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Maurer, Hermann: Intelligente Wissenserfassung und Wiederauffindung in künftigen WWW-Systemen. Bundesministerium für Bildung, Wissenschaft und Kunst. Wien 2000. Mochmann, Ekkehard: Towards the virtual observatory for Comparative Social Research. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000. Mohler, Peter Ph.; Smith, Tom W.; Harkness, Janet A.: Respondents' Ratings of Expressions from Response Scales: A Two Country, Two-Language Investigation on Equivalence and Translation. In: Cross-Cultural Survey Equivalence. ZUMA-Spezial Nachrichten Nr. 3, Mannheim 1998. Musgrave, Simon; Ryssevik, Jostein: Beyond NESSTAR: FASTER access to data. Paper presented at IASSIST 2000. Prüfer, Peter; Rexroth, Margit: Verfahren zur Evaluation von Survey-Fragen: Ein Überblick. ZUMA-Arbeitsbericht Nr. 95/5, Mannheim, 1996. Ryssevik, Jostein; Musgrave, Simon: The Social Science Dream Machine: Resource discovery, analysis and delivery on the Web. Paper presented at the IASSIST Conference, Toronto 1999. Saris, Willem E.: Comparability across mode and country. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Saris, Willem E.: Adjustment for differences between face-to-face and telephone interviews. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997.

47

Saris, Willem E.; Hagenaars, Jacques A.: Mode effects in the standard Eurobarometer questions. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Saris, Willem E.: The Effects of Measurement Error in Cross-Cultural Research. In: Cross-Cultural Survey Equivalence. ZUMA-Spezial Nachrichten Nr. 3, Mannheim 1998. Schmitt, Hermann; Schrott, Peter; Thoma, Michaela: Mode effect on open-ended agenda questions. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Instrument for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. Schubert, Peter; Greil Angelika: Sample design and consequences. In: Kaase, Max; Saris, Willem E. (Eds.): Eurobarometer: Measurement Insturment for Opinion in Europe, ZUMA Nachrichten Spezial Band 2, Mannheim 1997. UNESCO: International Standard Classification of Education ISCED 1997. UNECO 1997. UNESCO: Operational Manual for ISCED 1997. UNESCO 1999. Van der Zouwen, Johannes: An assessment of the difficulty of questions used in the ISSP-questionnaires, the clarity of their wording, and the comparability of the responses. In: ZA-Information Nr. 46, Köln 2000. Van de Vijver, Fons J.R.: Towards a Theory of Bias and Equivalence. In: Cross-Cultural Survey Equivalence. ZUMA-Spezial Nachrichten Nr. 3, Mannheim 1998. Vogel, Joachim: The future direction of social indicators research. In: Social Indicators Research 42, Kluwers Academic Publishers, 1997, 103-116. Wolf, Christof: The ISCO88 International Standard Classification of Occupations in Cross-National Survey Research. In: Bulletin de Methodologie Sociologique, 54, 1997. Wolf, Christof: Comparative measurement of demographic variables. Paper presented at the Fifth International Conference on Logic and Methodology in Cologne 2000. Zapf, Wolfgang; Habich, Roland: Euromodule - Towards a European Welfare Survey. Wisseschaftszentrum Berlin 1999.

48

Zenk-Möltgen, Wolfgang: Deutsche nationale Wahlstudien 1949-1998 auf CD-ROM verfügbar. In: ZA-Information 47, Köln 2000.

49

References in the World Wide Web Centre for the Study in Public Policy - http://www.cspp.strath.ac.uk Danish Data Archive - http://www.dda.dk Data Archive in Essex - http://www.data-archive.ac.uk Data Documentation Initiative: A project of the Social Science Community - http://www.icpsr.umich.edu/DDI/codebook.html Digital Information System for Social Indicators (DISI) - http://www.gesis.org/en/social_monitoring/social_indicators/Data/Disi/disi.htm, European Science Foundation (ESF): The European Social Survey (ESS) - a research instrument for the social sciences in Europe - http://www.esf.com HASSET - http://biron.essex.ac.uk/searching/zhasset.html ICPSR Guide to Social Science Data Preparation and Archiving (1999) - http://www.icpsr.umich.edu/EDSEARCH/PDF/dataprep.pdf ILSES - Integrated Library and Survey-data Extraction Service - http://ilses.gamma.rug.nl NESSTAR - Networked Social Science Tools and Resources - http://www.nesstar.org Social Indicators Department at Zentrum für Umfragen, Methodik und Analysen (ZUMA) in Mannheim - http://www.gesis.org/en/social_monitoring/social_indicators/index.htm The Council of European Social Science Data Archives (CESSDA) - http://dasun3.essex.ac.uk/Cessda/IDC W3C - Technology and Society domain - http://www.w3.org/RDF/ "Zentralarchiv für empirische Sozialforschung" in Cologne - http://www.gesis.org/en/za/index.htm

50

List of EuReporting Working Papers Berger-Schmitt, Regina; Jankowitsch, Beate: Systems of Social Indicators and Social Reporting: The State of the Art. EuReporting Working Paper No. 1, Subproject "European System of Social Indicators". Mannheim: Centre for Survey Research and Methodology (ZUMA), Social Indicators Department, 1999 Adamski, Wladyslaw; Pelczynska-Nalecz, Katarzyna; Zabowrowski, Wojciecch: System of Social Indicators, Social Reporting and Polish Society Transformation. State of the Art Report. EuReporting Working Paper No. 2, Subproject "European System of Social Indicators". Warsaw: The Polish Academy of Science, Institute of Philosophy and Sociology, 1999 Noll, Heinz-Herbert: Konzepte der Wohlfahrtsentwicklung: Lebensqualität und "neue" Wohlfahrtskonzepte. EuReporting Working Paper No. 3, Subproject "European System of Social Indicators". Mannheim: Centre for Survey Research and Methodology (ZUMA), Socia l Indicators Department, 1999 Harcsa, István; Spéder, Zsolt: Social Reporting and Social Indicators Movement in Hungary before and after the Transformation. EuReporting Working Paper No. 4, Subproject "European System of Social Indicators". Budapest: Central Statistical Office; Budapest University of Economic Sciences, 1999 Niklowitz, Matthias; Suter, Christian; Budowski, Monica; Meyer, Peter C.: Summary Health Indicators in Social Survey Research: A Useful Tool to Cover Health Status? Gesundheitsindikatoren in der Sozialforschung - ein Vergleich. EuReporting Working Paper No. 5, Subproject "European System of Social Indicators". Zurich: Swiss Federal Institute of Technology (ETH-Centre), Department of Sociology; Neuchatel: Swiss Household Panel; Zurich: Psychatric University Hospital, 1999 Suter, Christian; Niklowitz, Matthias: Social Reporting in Switzerland. The Hidden Roots and the Present State of the Art. EuReporting Working Paper No. 6, Zurich: Swiss Federal Institute of Technology (ETH), Department of Sociology, 1999 Hudler, Michaela; Richter, Rudolf: State of the Art of Surveys on Social Reporting in Western and Eastern Europe. EuReporting Working Paper No. 7, Subproject "Stocktaking of Comparative Databases in Survey Research". Vienna: Paul Lazarsfeld Gesellschaft für Sozialforschung (PLG), 2000 Maratou-Alipranti, Laura: Greece: Contributions to Social Reporting: Institutions, Activities, Publications. EuReporting Working Paper No. 8, Subproject "European System of Social Indicators". Athens: National Centre for Social Research (EKKE), 1999

51

Berger-Schmitt, Regina; Noll, Heinz-Herbert: Conceptual Framework and Structure of an European System of Social Indicators. EuReporting Working Paper No. 9, Subproject "European System of Social Indicators". Mannheim: Centre for Survey Research and Methodology (ZUMA), Social Indicators Department, 2000 Schulz, Wolfgang: Explaining Quality of Life - The Controversy between Objective and Subjective Variables. EuReporting Working Paper No. 10, Subproject "Stocktaking of Comparative Databases in Survey Research". Vienna: Paul Lazarsfeld Gesellschaft für Sozialforschung (PLG), 2000 Veèerník, Jiøí: Social Reporting in the Czech Republic since 1989: The Present State of the Art. EuReporting Working Paper No. 11, Subproject "Stocktaking of Comparative Databases in Survey Research". Prague: Czech Academy of Sciences, Institute of Sociology, 2000 Martinelli, Alberto: social Reporting at the Local Level: The Milan Metropolitan Area. EuReporting Working Paper No. 12, Subproject "European System of Social Indicators". Mannheim: Centre for Survey Research and Methodology (ZUMA), Social Indicators Department, 2000 Hudler, Michaela; Richter, Rudolf: Source-book about Questions on Social Reporting in Cross-national and Cross-sectional Surveys - An Example on Questions Covering the Life Domain Education. EuReporting Working Paper No. 13, Subproject "Stocktaking of Comparative Databases in Survey Research". Vienna: Paul Lazarsfeld Gesellschaft für Sozialforschung (PLG), 2000 Berger-Schmitt, Regina: Social Cohesion as an Aspect of the Quality of Societies: Concept and Measurement. EuReporting Working Paper No. 14, Subproject "European System of Social Indicators". Mannheim: Centre for Survey Research and Methodology (ZUMA), Social Indicators Department, 2000 Del Campo, Salustiano; Camacho, Juan Manuel: Social Reporting in Spain. A Recent Tradition. EuReporting Working Paper No. 15, Subproject "European System of Social Indicators". Madrid: Universidad Complutense, 2000 Berger-Schmitt, Regina: Dimensions, Indicators and Time Series in a European System of Social Indicators by Example. EuReporting Working Paper No. 16, Subproject "European System of Social Indicators". Mannheim: Centre for Survey Research and Methodology (ZUMA), Social Indicators Department, 2001

52

Hudler, Michaela; Richter, Rudolf: Source-book about Questions on Social Reporting in National and Cross-sectional Surveys - An Example on Questions Covering the Life Domain Education. EuReporting Working Paper No. 17, Subproject "Stocktaking of Comparative Databases in Survey Research". Vienna: Paul Lazarsfeld Gesellschaft für Sozialforschung (PLG), 2001