Introduction

In standard face-to-face surveys, some individuals may be reluctant to reveal personal information about their experiences and attitudes, for fear of a lack of confidentiality or to conform to the prevailing social norm. Sensitive questions can also create embarrassment or stress when they concern painful experiences or controversial subjects.

Faced with these possible measurement biases, the survey, here called "LE Insecurity", combined two different methods: direct questioning and a list experiment. The latter avoids respondents directly disclosing information about themselves, and thus, by avoiding any form of intrusion, enables prevalence rates to be measured free from social desirability biases.

The list experiment (LE, also known as item-count technique) has been mobilized in a variety of contexts to document social desirability biases on a wide variety of behaviors, such as voting (Holbrook & Krosnick, 2010), loan use (Karlan & Zinman, 2012), having experienced physical violence (Porter et al., 2021), sexual behaviors (Jamison et al., 2013; Coffmanet al., 2017), condom use among female sex workers (Treibich & Lépine, 2019; Lépine et al., 2020) and domestic or spousal violence (see for example Joseph et al., 2017; Agüero & Frisancho, 2022; Cullen, 2023). LEs have also been conducted to measure response biases concerning opinions on child marriage (Asadullah et al., 2021) or female circumcision (De Cao & Lutz, 2018; Gibson et al., 2018).

To our knowledge, the effectiveness of this method, compared with direct questioning, for measuring experiences and attitudes towards insecurity in a conflict context has not yet been examined in the literature. This is the objective of the "LE Insecurity" survey, carried out in Mali in July 2021[1]. More specifically, the survey aimed to measure the prevalence of five experiences and attitudes related to insecurity and conflict among the Malian population, as well as the social desirability biases that can characterize these sensitive questions when asked directly. This article provides feedback on this survey, with the aim of assessing both the feasibility of an LE survey technique and its effectiveness compared with the standard direct question survey method, in the context of a fragile state and around sensitive issues related to insecurity.

The national sampling frame from which the sample was selected was provided by the Institut national de la statistique du Mali (INSTAT). The survey was coordinated by the Groupe de Recherche en Economie Théorique et Appliquée (GREAT) in Bamako, and conducted from July 6 to 20, 2021, by telephone, by a team of 12 interviewers recruited and trained by GREAT, in collaboration with the authors. The choice of conducting the survey by telephone was dictated by the health context linked to the Covid-19 epidemic. Despite the risks of a low response rate that generally characterize telephone surveys, this strategy enabled us to interview a sample of individuals from all regions of Mali, at a relatively low cost. Having a sample covering all regions of the country strengthens the validity of our results at national level.

The main results show that the implementation of the phone survey was a success. Of the 1,719 telephone numbers dialed, over 91% were found to be functional. This made it possible to contact 1,569 people nationwide, of whom only 60 (3.8%) did not consent to answer. The final sample of 1,509 individuals is made up in equal parts of adult men and women, with an average age of 42, around 40% of whom have completed elementary school, and around 80% of whom claim to be married.

The results of the experimental survey show substantial social desirability biases on political attitudes. These biases are more marginal for experiences linked to insecurity. In particular, respondents over-reported their support for military rule in Mali by around 11 percentage points, and under-reported their confidence in foreign armed forces by the same proportion. Thus, 63% of respondents think that today's Mali should be run by the army (as opposed to 74% who report it with the direct question technique), and 36% (as opposed to 24.7%) trust foreign armed forces. These response biases are quite similar to those identified in the political science literature concerning support for current regimes in China and Russia (Blair et al., 2020).

In the rest of this article, we first present the survey method and then the main results, reviewing the survey process and the specific features of a telephone and item-count survey. We then propose some elements for discussion and conclude.

Survey method

The sample

INSTAT provided us with the 2018 Harmonized Household Living Conditions Survey (EHCVM) sample, representative at national, regional and urban/rural levels. This survey frame contains 6,602 households with 8,000 individuals. The cell phone number of at least one member is available for 92% of these households.

To create our sample, we selected 2,000 telephone numbers (one per household) by a random draw stratified by region and gender. This sample was divided into two lists, a main list (1,400 individuals) and a replacement list (600). Individuals on the main list were contacted first. If contact was unsuccessful, another individual living in the same region was selected from the replacement list. Further details of the replacement protocol are provided in Appendix A.

The item-count method involves separating the sample into a control group and a treatment group at random. This method is described in detail in the following section. The two groups were formed on the basis of a random draw stratified by region and gender. The observable characteristics of the two groups, for which we test the validity of the random assignment, are described in the paragraph "Assigning respondents to treatment and control groups". Finally, 1,719 people were contacted (874 in the control group, 845 in the treatment group), of whom 1,509 answered the phone and agreed to answer the questionnaire (764 in the control group, 745 in the treatment group).

The List Experiment (LE)

The List Experiment method involves reading lists of statements to the respondent (in our case, three or four per list, depending on treatment status), without the respondent reacting to each statement, but asking them to state the total number of statements with which they agree. The treatment group is presented with lists containing four statements, including one sensitive statement. The control group is presented with the same lists, containing only the three non-sensitive statements. This survey comprises five lists, with the aim of measuring the prevalence and social desirability bias characterizing five experiences or attitudes related to insecurity.

For each list, a comparison between the average response given by the treatment group, who responded to the version of the list with the sensitive statement, and the average response given by the control group, who responded to the version without the sensitive statement, provides the prevalence of the sensitive statement. In addition, we questioned the control group directly on the five sensitive questions covered by the survey. Comparing the responses to these direct questions with the prevalence rates measured from the LE enables us to estimate the size of the social desirability biases.

Familiarization with the LE method was an important part of interviewer training. To ensure that respondents fully understood the method, interviewers were asked, at the start of the LE module of the questionnaire and after explaining the specific format of the questions to come, to make it clear that the respondent was not to react to each statement, but only to give, at the end of the list, the number of statements he/she approved of.

To facilitate the respondent's understanding of how LE works, the interviewers relied on the following test list, then discussed the various response options with the respondent:

 a) I like chatting with my neighbors

 b) Yesterday, I went to the market

 c) Yesterday, I called my sister

 d) I've been sick recently

In particular, they clarified that a person who, for example, likes to chat with his neighbors, didn't go to the market yesterday, didn't call his sister, but has recently been ill, should answer: "2" (because statements a and d are true in his case). A person who doesn't like chatting to his neighbors and hasn't been ill for a long time, but who went to the market yesterday and called his sister, should also answer: "2" (statements b and c are true in his case).

For the quality of the data collected, it was essential that the respondents understood how the list questions worked, and that the interviewers took the time to explain in the most educational way possible. To help the respondent keep track of which statements he/she agreed with, the interviewer advised him/her to start each list with a closed fist, and to raise a finger at each statement with which he/she agreed. At the end of the list, the respondent simply indicated how many fingers were raised. The trainers stressed the importance of reading the statements calmly and pausing between each one, to give the respondent time to understand and decide whether to raise a finger.

Each interviewer had to administer both types of questionnaire, the order of which was randomly drawn.

The questionnaire

The questionnaire comprises three modules common to all respondents, and one module in two versions depending on whether the respondent is assigned to the treatment or control group. It is important to mention that respondents could interrupt the questionnaire at any time.

The introductory module is common to all. It is used to make contact, present the purpose of the survey, and identify whether the person contacted is eligible - i.e. at least 18 years of age - or, failing that, whether another member of his or her household can be surveyed. If the person contacted (or another member of his or her household) is eligible but unavailable at the time of the call, the introduction module also enables an appointment to be made to continue the survey at another time. Finally, consent to participate in the survey is collected here.

Once consent has been obtained, the actual questionnaire begins. The second module (general characteristics module), also common to all, collects information on age, level of education, marital status, mother tongue and commune of residence. Individuals are also asked since when they have been living in their commune.

The third module concerns our study object: the five experiences and attitudes linked to insecurity. It is divided into two versions, one for the treatment group, the other for the control group.

The treatment group is asked about the five insecurity-related experiences and attitudes in the form of five lists. Each list contains four statements, including one sensitive statement - the prevalence of which is to be measured. The interviewer reads each list to the respondent and asks him/her to indicate, at the end of each list, the total number of statements with which he/she agrees. The respondent therefore does not name the statements with which he/she agrees, but only gives the number of statements with which he/she agrees. Since the lists contain four statements, the respondent's answer to each list is a number between 0 and 4.

The control group is questioned on the five sensitive experiences and attitudes covered by the Insecurity module in the traditional way: the interviewer asks the respondents to answer these five questions directly with Yes or No. The control group is also mobilized for the list experiment: the interviewer reads to the control group the five lists from which the sensitive statements have been removed, and asks them to indicate, at the end of each list, the total number of statements with which they agree. This time, the lists contained three statements, and the responses to each list ranged from 0 to 3.

As explained in the paragraph "The list experiment (LE", the comparison between the average response observed for each list in the treatment group and in the control group will provide, for the analysis, a measure of the prevalence of each sensitive statement free from social desirability biases.

The lists are shown in Table 1, where sensitive statements - read only to the treatment group and asked as direct questions to the control group - appear in italics. Direct questions to the control group are shown in Table 2. Respondents were given the option of not answering/passing all the direct questions, both the sensitive ones and the questions asked in the General Characteristics module.

Finally, the questionnaire contains a final module, common to all respondents. This concluding module thanks the respondent, informs him/her of his/her right to withdraw, and provides a telephone number if he/she wishes to be put in touch with organizations specializing in supporting victims of violence. It is also in this final module that the interviewer indicates the language used for the interview.

The introductory, consent and general characteristics modules are shown in Appendix B.

Table 1: Lists of the Insecurity module

Table 2: Direct questions asked to the control group

The results

Assigning respondents to treatment and control groups

The validity of the LE is based on the random assignment of respondents to treatment and control groups. In order to verify that the two subgroups are not statistically different from each other, Table 3 reports the mean characteristics of respondents according to whether they belong to the treatment or control groups, as well as the p-value associated with the difference in means between the two groups (t-test on averages)[2]. The variables Age and Date of residence in current commune are continuous variables. All the other variables in the table are dichotomous, taking the value 1 when the characteristic is observed in the individual. Their averages can therefore be interpreted as proportions. The two groups appear to be statistically similar.

None of the differences between the two groups is significant, with the exception of the proportion of people who have lived elsewhere than in their current commune of residence, where the difference is significant at the 10% threshold. There were no differences in region of residence, proportion of men, age, level of education, married status or mother tongue.

The sample is 54% male, with an average age of around 42. Less than 50% have completed primary education, 88% are married, half speak Bambara as their mother tongue, and less than 20% Songhai. On average, they have lived in their current commune since 1987, and 30% had previously lived in another commune.

Table 3: Average characteristics, treatment and control groups

* p-value < 10%

Survey procedure and specificities

The survey was conducted by telephone and Android tablet, using ODK Collect software. Telephone interviews took place at GREAT's premises. The questionnaire on the tablets was written in French, with paper translations available in the five other languages offered to respondents: Bambara, Soninke, Tamasheq, Peulh and Songhai.

The interviewee chose the language of the interview. In the event that the interviewer did not master the desired language, the supervisor designated an interviewer within the team who was able to conduct the interview in the language chosen by the respondent.

The survey was conducted exclusively by telephone - a format that can involve specific difficulties. Compared with a face-to-face interview, a telephone interview can make mutual understanding more difficult, for example because face expressions are not visible and sound can be altered; and involve more sources of distraction (network cuts, surrounding noise, difficulty for the respondent to isolate himself/herself to answer, etc.). The respondent's trust and patience may also be harder to win, and he or she may fear being approached by lure agents or crooks.

To ensure the success of the survey, questionnaire training placed particular emphasis on the particularities of conducting a telephone survey, so that interviewers were as careful as possible. In particular, special attention was paid to the clear presentation of the home institution at the very start of the interview. Interviewers were also asked to pause regularly to ensure that they were heard and understood. More practically, the battery and credit levels of tablets and phones were systematically checked before any call. In addition, we designed the questionnaire to be as concise as possible, so as to limit the length of the interviews as much as possible. In the end, an average of twenty minutes was enough to complete a full questionnaire.

Maintaining the respondent's attention throughout the telephone interview was a challenge for the interviewers as they entered responses on the tablet at the same time. In this respect, the time devoted to familiarizing interviewers with the tools and practising on the questionnaire during interviewer training proved particularly useful.

Supervisors also played a key role, shaped by the special nature of telephone interviews. As well as ensuring that the equipment was working properly, they were responsible for allocating respondents to interviewers and reassigning them in the event of language incompatibility, monitoring appointments and mobilizing the replacement list when necessary. The supervisors also ensured regular monitoring and control of the work carried out by the interviewers, as well as checking completed and incomplete questionnaires, and transmitting the collected data to the storage server.

Respondents were compensated for their participation with 1,000 FCFA, in the form of telephone air-time credited to the number contacted once the questionnaire had been completed.

Non-response

Of the 1,719 telephone numbers dialed, only 41 did not work, and 109 never answered (after several attempts over several days). Over 91% of the numbers were found to be functional. This represents a success, given that these numbers had been collected three years before. Another success was the rate of consent. In strict compliance with current ethical protocols, once contact had been made, the interviewer had to explain the purpose of the survey and ask for consent to conduct it. Of the 1,569 people contacted, only 60 (3.8%) did not consent to answer.

A total of 210 people could not be surveyed, corresponding to a non-response rate of around 12% (table 4). Interestingly and reassuringly, the probability of non-response at the individual level was not a function of the assignment group (treatment or control) of those contacted.

However, a number of members of the control group did not wish to answer some of the sensitive questions asked directly: only 0.3% for the first two questions, 3.3% (25 observations) for the third question (“Are you, or a member of your household, prepared to take up arms to defend your community?”), 13% (99 observations) for the fourth question (“Should today's Mali be run by the army?”) and 9.3% (71 observations) for the fifth question (“Do you trust foreign armed forces in Mali?”). These refusals to answer might suggest a higher degree of sensitivity of the questions raised.

Table 4: Number of non-respondents and non-response rate, total and by group

Measured prevalences

Using the LE method, we calculate prevalence rates free of social desirability bias for five experiences and attitudes related to insecurity in Mali. Specifically, for each of the five LE questions, we compare the average response obtained in the treatment and control groups, using regression controlling for stratification variables (region and gender), as well as for interviewer fixed effects. This ensures the validity of the random assignment, and corrects for any systematic differences between interviewers. The estimated difference between the two average responses thus represents the prevalence rate of the sensitive statement.

Table 5 shows the results. The first row of the table shows the levels as measured with direct questioning; the second row shows the differences between the prevalence rates derived from direct questioning and those in LE. Finally, the last row shows the unbiased prevalence rates estimated using the LE method. Based on the responses obtained with LE, it appears that 17.1% of respondents, or a member of their household, have suffered a physical assault in the last twelve months (Q1). 16.9% of respondents' households own a firearm (Q2). In a country that has been in conflict for almost a decade, 30.4% of respondents have a member of their household ready to take up arms to defend their community (Q3). Finally, a few weeks after the coup d'état in May 2021 and after eight years of foreign military presence, 63.2% of respondents believe that the army should be in charge of the country (Q4), and 36.3% trust the foreign forces present in Mali (Q5).

Furthermore, the protocol reveals that social desirability biases are significant with the direct method for responses concerning political attitudes, but less so for those concerning experiences of insecurity. In particular, respondents over-reported their support for military rule by around 11 percentage points, and under-reported their confidence in foreign forces by the same proportion. These results are similar to those reported in a meta-analysis by Blair et al. (2020). The 21 listed studies carried out in authoritarian regimes, mainly China and Russia, also estimate biases in responses to questions asked directly about support for or opposition to the regime in power. The sizes of the estimated biases are very consistent with our results, averaging 14 and 8 percentage points depending on whether support or opposition to the regime is measured.

Table 5: Prevalence rate of sensitive questions by survey technique and estimated response bias

Conclusion

Several lessons can be drawn from the "LE Insecurity" survey conducted in July 2021 in partnership with GREAT, both in terms of data collection and analysis.

As far as data collection is concerned, the list experiment survey protocol appears to be a very rich tool, but also complex to implement. In particular, the experience of the "LE Insecurity" project shows that the success of an experimental survey with LE depends largely on the skills of the interviewers recruited and on the quality of the questionnaire training. On the other hand, a telephone survey requires special adjustments. In the present case, the contact rate was ultimately high and non-response rare - despite the fact that the use of mobile telephony in a country like Mali, and in particular the habit of regularly changing telephone chips, raised fears of major difficulties in contacting numbers collected three years earlier. This high contact rate was probably due, at least in part, to (i) flexible sampling, which did not require contact to be made with the person who had declared the telephone number in question in the sampling frame used; (ii) particular rigor on the part of the interviewers and supervisors in following up attempted calls and call-backs; and (iii) a replacement protocol to compensate for the few contact difficulties encountered.

As far as data analysis is concerned, the LE method offers vast possibilities for producing unbiased measures of potentially sensitive experiences and preferences, but also for quantifying the response biases that can characterize them when measured by direct questioning. The results of the analysis of the data collected by the LE Insecurity survey are promising in both these directions. Data exploration will also highlight possible heterogeneities in measurement bias, and thus identify segments of the population whose experiences and attitudes may be particularly difficult to measure in a standard way. This could prove particularly important for the targeting and evaluation of public policies related to insecurity, for example in a victim care or disarmament perspective.

Appendices

Appendix A: Questionnaire modules common to all respondents

Introduction module

Replacement block

 Block Following appointment booking

Consent module

General characteristics Module 

Appendix B: Replacement protocol

As explained in the "Sampling" paragraph, the sample of 2,000 telephone numbers was randomly divided into a main list of 1,400 numbers, and a replacement list of 600 numbers. The replacement list was mobilized when a number on the main list could not be surveyed, in particular in the event of an unassigned number, refusal to participate, or absence of an eligible person in the household.

Specifically, the 1,400 telephone numbers on the main list were allocated to the 12 interviewers on the team. In the event of an unsuccessful call, the interviewer was asked to indicate whether the number was unassigned or whether the phone had not been picked up. In the first case, the replacement list was mobilized by the supervisor to provide the interviewer with an alternative telephone number with the same treatment status, taking care to respect the balance of the draw by region. In the second case, the interviewer called back the number three times a day, for three days, before requesting a replacement number.

It should be noted that, while the EHCVM sampling frame provided the identity of the owner of each telephone number in 2018, as well as a set of its characteristics, the survey protocol did not require that the person who responded to the EHCVM be traced. In particular, the "LE Insecurity" survey was offered, and administered if consent was given, to any eligible person (i.e. of age) who responded to a sampled telephone number.

In the event of telephone contact with an ineligible person (aged under 18), the interviewer sought to establish contact with an eligible member of the same household - either directly during this first call, or by calling back on the same number or on an alternative number indicated by the caller, at the appropriate time.

The contact and replacement protocol is summarized in figure 1.

The GREAT's 12 interviewers finally surveyed a sample of 1,509 people: 764 in the control group and 745 in the treatment group.

Figure 1: Contact and replacement protocol