Every year, thousands of products are distributed to communities in low-resource settings. Unfortunately, many of these products are underused or nonfunctional, with little follow-up to determine why. As development practitioners, we have an ethical responsibility to evaluate our products and practices, especially when working with some of the poorest and most marginalized communities in the world. Understanding your organization’s impact (and continuously working to improve it) advances practice, increases funding opportunities, and brings you closer to really making a difference.
Proper evaluation of your organization’s product or service will pay off long-term.
Product evaluation methods can be broken down into two categories: quantitative and qualitative. To truly understand a product or service’s impact, you’ll need both. Quantitative methods are most helpful when gathering technical operating data and determining rates of adoption. Measuring performance parameters is essential to understanding whether a product is meeting its technological objectives, such as the thermal efficiency and fuel consumption of an improved cookstove.
To measure acceptance rates, practitioners have traditionally relied on surveys, a cost-effective way to collect large amounts of data from the targeted population, however, respondents may not always provide the most accurate or honest answers. This is particularly true in the context of international development where power relationships exist. Here, a respondent (poor individual) is likely to answer questions in ways that please the surveyor (employed individual).
Sensor-based data collection examine user adoption rates and other technical parameters.
To mitigate this, practitioners are increasingly moving to sensor-based data collection to examine user adoption rates and other technical parameters. This is especially true within the context of improved cookstove distribution where many sensors have been developed; including Stove Use Monitoring System, StoveTrace by Nexleaf, FUEL by Collaborative Sense, and SWEETSense. To assess water pump usage and monitor maintenance requirements, consider deploying the AFD-1 Water Flow Sensor. For uses that are not restricted to water technologies, more flexible sensor-data-collection platforms have been developed such as the MoMo and Sensen. These platforms work with the specific sensors you need, whether it be water flow meters or electricity power meters, to better evaluate how often and when your product or service is being used.
While quantitative methods can tell you if a product is not being used, they do not provide much information about why that might be the case. Qualitative methods, such as interviews, observations, and other ethnographic methods aim to better understand human behavior and individual preferences, particularly those that have been unintentionally unconsidered. It is important to note that there is no “one-size-fits-all” method and practitioners should always select context-specific and appropriate evaluation methods. Reverting to general survey questions will not provide your organization with comprehensive user-experience data as these only facilitate information that survey designers already know exists.
A well-conducted semi-structured interview can be an effective technique to better evaluate how a person really uses a product.
Using an ethnographic approach can lead to comprehensive user data within a specific context. A well-conducted semi-structured interview can be an effective technique to better evaluate how a person really uses a product and what they like and do not like about it. In a semi-structured interview, the interviewer follows a flexible guideline, refrains from asking leading questions, and encourages the respondent to share detailed stories. Participant observation is another ethnographic tool to better understand how and when users are really using your product. Practitioners gain insight through observation of the handling of the product, nonverbal expression of feelings, and occurrence of events that the respondent may have not considered sharing.
Digital tools such as the platforms included in E4C’s Mobile Data Collection Webinar Series can automate surveys and speed the gathering of data: How to use mobile data collection tools to improve global development work.
While it may seem time-consuming at first, proper evaluation of your organization’s product or service will pay off long-term. Using a combination of qualitative and quantitative evaluation methods provides information of who, what, and why people are (or are not) benefiting. As practitioners who really aim to make a difference in the lives of the poor and marginalized, we have the ethical responsibility to evaluate the full implications of our products and continue to work to develop improved solutions.
