Ever since the world witnessed the rise of COVID-19, society had to adapt to the new rules and health guidelines. We shifted from the office setting to working from home. We started to avoid plane trips and big gatherings. Businesses had to reinvent themselves to keep alive. In the same way, the humanitarian and development sector were also affected. The projects’ operations in the field had to readjust to keep running. Different stages of the project cycle management were hit, including the monitoring process.
Usually aid agencies do the monitoring in person, visiting households to collect the survey data, for example. However, where access is restricted, remote data collection methods have long been utilized. They were a useful solution during the Ebola outbreak in 2014. Methods of remote data collection gained new relevance as COVID-19 imposed limitations to the gathering of people and close contact, thus to in-person data collection. To overcome this challenge, there are different types of remote data collection, but here we will focus on SMS surveys, computer assisted telephone interviewing (CATI) and interactive voice response (IVR). The aim is to provide an overview of those methods, presenting the pros, cons and possible biases of each.
SMS Survey
SMS Survey is a survey administrated through the use of small messaging service (SMS) to collect data from respondents. Its usage is more appropriate the collection of quantitative data. Compared to the other two methods, SMS surveys is the cheapest one to implement and run. It requires less staff, less field trips or material. As soon as the survey is completed, the results are instantly registered. This reduces significantly the reporting time and allows faster feedback and analysis. As the SMS removes the human element on the other side, it increases the sense of anonymity, which grants more honest answers to sensitive questions.
However, SMS surveys present certain challenges, as the response rate. This kind of survey must focus on the most essential questions, since the amount of questions is inversely proportional to the completion rate. In addition, SMS usage requires the implementing parts to adopt clear and simple language to avoid any kind of confusion and ensure a higher rate of response. To make sure that the language used in the surveys is appropriate and objective, it is important to hold focus discussion groups. This will render enough feedback analyze the understanding of the questions and restructure if necessary.
To ensure the reliability and validity of the data, it is important to consider possible data biases. As SMS surveys requires a cellphone to work, the data can present socio-economic bias. This means that the surveys will only be able to reach people who own a cellphone, excluding beneficiaries who cannot afford one. In addition, SMS surveys only allow participation if the selected beneficiaries are able to read, excluding the answers of who is illiterate.
Computer Assisted Telephone Interviewing (CATI)
CATI is an interview technique that is done over the phone and utilizes an interview guide provided by a software. This is the most expensive option among the three presented here. It requires investment in material, in building a team of local enumerators, in training them and it takes longer to structure the operations. In addition, the data gathering process is the slowest of the three.
In opposition to SMS, phone interviews allow the collection of qualitative data (as well as quantitative), as the individuals can better elaborate and structure their answers. When a local enumerator administers the survey over the phone, it is possible to include the answers of illiterate individuals in the data collection process. Another benefit of using an enumerator is that they can provide clarification to the questions the interviewees do not understand. This helps increase the validity of the data. On the other side, this alternative still limits the target audience to only interviewing who owns a phone. Yet, this is an issue that all of the three remote solutions present.
Interactive Voice Response
IVR is an automated phone system that is structured with pre-recorded voice messages that will apply the questionnaire to the respondent. It’s a middle term solution that gather pros and cons from both of the previous options. It requires moderate investment, since the implementing agency must have a certain level of infrastructure as a software for the IVR. However, costs with training and staffing are much lower here. To give an overview, Foley (2017) reports that “face-to-face surveys cost $20-40 per questionnaire to implement. By contrast, SMS costs $5-6 and live calls and IVR cost $7-9 per questionnaire”. Eckman and Himelein (et al, 2020) point out that the cost can vary depending on the location and that CATI is generally more expensive than IRV per completed questionnaire.
The collection time and processing of the data is in a middle range as well. It is faster than phone interviews as it relies less in human capacity, but it takes longer than SMS as it is less objective. Just like CATI, IRV solves the illiteracy bias. However, it requires a certain level of familiarity with the technology – from the part that is taking it – to properly go through the survey. This is particularly relevant if you are trying to reach older members of the community. On the other hand, IRV brings the sense of anonymity present in the SMS surveys, making people more comfortable to talk about sensitive issues.
Final considerations
As previously mentioned, the socio-economic bias – that excludes the people who cannot afford a cellphone – is a common issue present in all of those solutions. However, due to the advance in the technology, basic phones with call and text function have become more affordable (and accessible) in emerging markets. In addition, when dealing with data, we must be aware of data privacy and security. It is crucial to have mechanisms in place to ensure those to the interviewees, as data breach could put them at risk of being targeted.
It is important to emphasize that there is no better solution [from the three presented here] that stands out from the rest; there is only the most appropriate one, taking under consideration your specific project and it’s needs. To choose, it is important to consider your target audience; the time you have available to collect and process the data; what kind of data you need and your budget. Eckman and Himelein (et al, 2020) state that they can also be used complementarily as a way to triangulate the data.
About the Author
Erika Rizzo is a Junior Officer in the TPM and Research department at Trust. She has a master degree in International Development Studies from the Norwegian University of Life Sciences. Her main areas of interest are monitoring & evaluation, peacebuilding and food security.
Read more about Erika on LinkedIn.
Sources:
https://www.elrha.org/wp-content/uploads/2017/09/Case-Study-WFP-mVAM.pdf
https://www.geopoll.com/blog/pros-and-cons-of-interactive-voice-response-surveys/
