IFF Research chooses the Stats People to redesign the sampling and weighting methodology for the Employer Skills Survey (ESS)
The brief for redesigning the Employer Skills Survey (ESS)
We were commissioned as partners by IFF Research, the Department for Education’s appointed survey research agency, to redesign the sampling and weighting methodology for the Employer Skills Survey (ESS) and to integrate the survey with the parallel Employer Perspective Survey (EPS), a smaller scale survey with a different design. The ESS is the largest B2B CATI survey in the UK, surveying businesses from two to 250+ employees. It provides a rich body of evidence used widely for policy development around the extent and nature of skills needs and challenges faced by employers, their (training) response and engagement with the skills system.
Our approach
The work initially involved a full review of the sampling and weighting designs for the existing survey design, highlighting strengths and weaknesses and investigating sources of non-response bias. Following our recommendation to move the survey to a random probability survey (RPS) design, we advised and implemented the pilot sampling plan. We consulted with IFF and DfE on migrating a large-scale telephone survey to a practical random probability framework. After the successful pilot implementation, we worked on fully migrating the survey to an RPS design.
The brief
To redesign the sampling and weighting methodology for the Employer Skills Survey (ESS) and to integrate the survey with the parallel Employer Perspective Survey (EPS), a smaller scale survey with a different design.
Approach
Following our recommendation to move the survey to a random probability survey (RPS) design, we advised and implemented the pilot sampling plan.
The step-by-step process
- Meetings, fact-finding calls and emails with the IFF team
- A phase of exploratory modelling of different options
- Drafting appropriate sampling and interviewing rules and procedures for managing the amount of sample we issued
- We undertook modelling throughout the first phase of fieldwork to adjust the amount of sample released in each stratum
- The full rollout incorporated lessons from the pilot phase to optimise the design.
The challenges we faced
There were no published examples of large-scale CATI RPS Surveys to base our recommendations on, and so the new survey design was built up from ground zero. Initially this meant making some strong assumptions that we had to update on an ongoing basis as fieldwork progressed. At the beginning, we had to simulate what random probability sample would look like.
Example: Our Random Probability Sampling simulator for the ESS CATI Survey:
Outcome
Despite reservations about the large-scale change to sampling, the pilot was a great success. It was much more practical and cost-effective than initially anticipated, to the extent that it is now being rolled out in the next wave to other regions.
Challenges
There were no published examples of large-scale CATI RPS Surveys to base our recommendations on, and so the new survey design was built up from ground zero.
Outcome
It is now being rolled out in the next wave to other regions.