Trial and error with digital technology in Sri Lanka

By Ryotaro Hayashi

blogs.adb.org: The success of ADB’s relatively new results-based lending instrument hinges on accurate validation of outcome and output indicators. But this process can delay the program if it is not conducted expediently.

Maybe digital technology can help to accelerate verification, I surmised when I was working on the Skills Sector Enhancement Program (SSEP) in Sri Lanka.

The goal was to improve the employability of Sri Lankan youth that graduated from Government technical and vocational education (TVET) programs supported by ADB. More specifically, the target was to raise the employment rate of TVET graduates to 52% by 2016. A tracer study is the conventional approach to assess its effectiveness, but the employment rate might be inflated if teachers ask students about their job status.

To prevent systematic positive reporting bias, I conducted interactive voice response (IVR) surveys. I was particularly intrigued about this digital technology, which uses automated phone calls to reach out to people. IVR technology is incredibly fast and affordable compared to manual telephone interviews.

Digital technology faces with social norms

Another advantage of IVR is its simplicity. The process requires questionnaire recording and its uploading to a special website with TVET graduates’ mobile phone numbers. In this case, I am interested in the employment rate, so I record relevant questions, such as ‘Are you currently employed? If yes, press 1; if no, press 2’ in the country’s two official languages, Sinhala and Tamil. SMS was also used for prior notification so that respondents knew when the IVR surveys were initiated. The IVR surveys were rolled out typically on evenings or weekends to improve the response rate.

It seemed like all the ideal conditions were in place for piloting IVR surveys under the SSEP. Mobile penetration in Sri Lanka reached 115% in 2015, and young people are more adept at using mobile phones than senior citizens. The IVR surveys had to be short (3-5 minutes) to obtain a good response rate, but asking about employment status did not require many questions I was very excited to find out if this digital technology would work for us.

Unfortunately, it did not. The IVR surveys were not a complete fiasco, but certainly not up to the mark to make a good judgment on outcome validation. Why? Among several reasons, let me share three social barriers that caught my attention.

First, family members often share the same mobile phone number, so the target respondent is not necessarily the one to pick up. Second, respondents fear they will be charged if they make key presses with a call from an unknown number, so many do not pick up. Third, some respondents shared a social preference to use landlines for official surveys. 

Yet another challenge was that many mobile phone numbers registered in the database were not necessarily correct, in some cases the person who picked up the call was not even a family member but the trusted village leader.

Trial and error works

I addressed the shortcomings of IVR surveys by deploying another form of digital data collection: the computer-assisted telephone interview (CATI). CATI is a manual telephone interview conducted by human enumerators with data input digitised through a tablet. CATI surveys were carried out in parallel with IVR surveys to also cover other topics, such as the socioeconomic background of graduates and quality of courses, to better understand the broader TVET system in Sri Lanka.

This technology is expensive, and you have to train enumerators and procure tablets. The upside is that enumerators can ask longer questions, and responses are recorded automatically when the enumerators input them. After each survey is completed, the responses are instantly transmitted to a cloud server, significantly reducing the risk of data transcription error encountered in paper-based surveys. CATI can also solve the social barriers encountered through IVR surveys by flexibly addressing the respondents’ concerns.

Without CATI, I could not have completed the verification on schedule.

Fortunately, the assessment confirmed that employment rate for TVET graduates surpassed the 52% target, so the Government received $5 million through the SSEP. Trial and error, I learned, is an inevitable process to harness the power of the digital revolution.

(The writer is Social Sector Specialist, South Asia Regional Department, ADB.)