PIAAC Cycle 2 – Validation of selected items from the draft PIAAC Background Questionnaire: Findings from the US, Germany, France, Poland, and Japan - Mixed-Methods Cognitive Online-Pretest
Abstract:
This report contains findings of the questionnaire pretest that GESIS - Leibniz Institute for the Social Sciences (Germany), member of the international consortium, conducted for the OECD study PIAAC – Programme for the International Assessment of Adult Competencies.
The subject of this pretest are new and/or revised items of the draft PIAAC backround questionnaire (BQ) which were tested in five participating countries (and languages), namely the United States, Germany, Poland, France, and Japan.
The questions were tested in a mixed-methods cognitive online pretest that combined quantitative (response distributions, split-ballot experiments, response times analysis) and qualitative testing (web probing).
The results of this pretest are documented in the PDF file (see Downloads).
Citation:
Meitinger, K., Neuert, C., Hadler, P., Lenzner, T., Behr, D. & Menold, N. (2018): PIAAC Cycle 2 – Validation of selected items from the draft PIAAC Background Questionnaire: Findings from the US, Germany, France, Poland, and Japan. Mixed-Methods Cognitive Online-Pretest. GESIS - Leibniz Institute for the Social Sciences
DOI:
Number of Participants:
2584
Selection of Target Population/Quotas:
This online pretest was fielded in five countries: the US, Germany, France, Poland, and Japan. The target population was employees and self-employed persons. Unemployed or non-working respondents were not included in the sample because the majority of items in the pretest were not relevant for this group. The question regarding employment that was used for quota setting was very “inclusive”. It asked for a minimum of 1 hour of paid work (item from PIAAC cycle 1). Moreover, all educational levels were covered in the pretest.
The selection of the target population was based on the quota scheme presented in table 1 in the report:
Age (18 to 65 years in three groups: 18 - 30; 31 - 50; 51 - 65); gender (male; female); education (low; medium; high, based on PIAAC cycle 1 education items).
In addition, 20 percent of respondents in each country were ex-pected to be self-employed and 80 percent of respondents in each country were expected to have taken part in any training or course in the past 12 months, whether job-related or for their personal interest. With the latter quota question, we aimed at getting a sufficient number of respondents for testing the training questions. 10 percent oversampling was allowed free of charge by the panel provider.
The panel provider was positive towards our quota settings when receiving the specifications of the study so that we proceeded along these lines when setting up the study; during field work, however, we had to relax our specifications - regarding lower levels of education (issue particular prevalent in Poland and the US); percentages of self-employed respondents and respondents with prior training experience in the last 12 months - due to problems of meeting the quota. Online access panels are recruited according to non-random criteria, which is why a representative coverage of all population groups unfortunately cannot be guaranteed. Nevertheless, our sample is sufficiently diverse and our data, both quantitative and qualitative, allows us to produce meaningful pretest results.
We reacted to the problem of meeting the education quota for lower education by increasing the number of respondents in the other two educational categories. This way, we were at least able to meet our goal of approximately 500 respondents per country. Furthermore, we reacted to the problem of getting self-employed respondents (20%) and those with training experience (80%) into the sample, by relaxing our quota to some extent. These solutions apply to all countries.
Test Date:
08/2018
Survey mode:
CAWI
Pretest mode:
Web Probing
Procedure:
Usage of an online questionnaire with additional open-ended and closed-ended probing questions (Web Probing)
Incentive for respondents:
Incentivization according to the guidelines of the panel provider
Project Description:
PIAAC – The Programme for the International Assessment of Adult Competencies – is an international comparative study initiated by the Organisation for Economic Co-operation and Development (OECD) with the goal to assess key adult competencies. PIAAC is implemented simultaneously in several countries and is repeated at regular intervals. The study assesses key adult competencies necessary to successfully navigate demands in everyday life and in the workplace. These competencies include literacy, numeracy, and adaptive problem solving. They are believed to be central to information processing in modern society and they constitute the foundation for the development of many other, more specific competencies.
The goal of PIAAC is to provide a comprehensive overview of the human capital of the participating countries, both at national and international levels. PIAAC allows to compare key competencies across participating countries and to identify factors that play a crucial role in their acquisition, retention, and maintenance. In addition, PIAAC sheds light on how these competencies are used in everyday life and in the workplace and how they have developed in the past ten years. Finally, international comparison and exchange allow the participating countries to learn from each other and to develop measures to promote the acquisition of key competencies in their adult populations.
The background information collected in PIAAC, such as comprehensive information on formal education, continuing education, and employment, allows for the investigation of factors that influence the development and retention of competencies. In addition to the assessment of competencies, PIAAC also captures whether adults use certain competencies in their everyday life and in the workplace and whether these competencies differ from their subjective assessment of workplace requirements. Furthermore, PIAAC provides important information on whether the school and education systems of the participating countries are able to promote the acquisition and retention of the competencies necessary for successful participation in life in modern society.
Website:
https://www.gesis.org/en/piaac/piaac-home
Client:
OECD
Pretest Report (English):
