Are one-size-fits-all metrics for global WASH really appropriate?

If you’ve ever wondered where global WASH statistics come from, you’re not alone.

For example: In 2015, 844 million people around the world lacked access to an improved drinking water service. How can we know that? Or this: Since 1970, the number of people using an improved drinking water source increased from 2.4 billion to over 6.2 billion (64 to 89 percent coverage). That seems like good progress, but I see those numbers and can’t help but raise an eyebrow. Who’s tracking this stuff and how are they so sure?

What we track and how we track it makes a huge difference in this sector. Tracking affects how much money the world spends on infrastructure, which technologies win funding, and what problems the designers of new products attempt to solve. With so much on the line, we should take the time now and then to consider if our tracking is really bringing us closer to our goals.

I was surprised to learn recently that water and sanitation monitoring has been going on for almost 90 years. It started sometime after World War I with the formation of the League of Nations.

Back then, country health departments did their own monitoring and simply reported their stats to international entities like the League of Nations—and later to the UN or WHO—that then compiled the data. Now, the Joint Monitoring Programme (JMP) is a combined effort of the WHO and UNICEF and has become the international authority for collecting water and sanitation data. They’ve also developed more sophisticated statistical methods of sampling a country’s population and extrapolating estimates for coverage, something called Multiple Indicator Cluster Surveys, but a lot of data was still reliant on individual governments reporting coverage up until as recently as 2000.

The problem with international monitoring programs is they require standardization to be meaningful. That has led to one-size-fits all understandings of “safe” and “improved.” The reality is that these definitions don’t always fit the situation.

Collecting data on such a massive scale not only presents the problem of consistency and reliability of methods across languages, cultures, and vastly different geographies, but somehow normalizing an understanding of access. When we started trying to quantify how many people had access to water and sanitation services around the globe, we also needed to come up with a standard definition of what “access” really is. By default, then, we also started setting goals for what it should be.

That definition has been a moving target over the years, making goal-setting and evaluation of progress a frustrating endeavor.

For example, what really is the definition of an “improved” water source? Is it a spring source? What if that spring is unprotected and contaminated with fecal coliforms? Is it a piped supply? What if those pipes aren’t maintained and leach lead? (Or what if that pipes come from the contaminated spring?)

How might our sense of innovation change if we let our definitions of access and success evolve?

And what is “improved” sanitation anyway? A pit in the backyard? Depends on how often it is cleaned out, and how safely that waste is managed. Does a bucket that gets collected count as “improved”? How about a citywide sewerage system (the “gold standard” for separating a user from waste) that dumps that sewage straight into the lake outside of town?

Unsurprisingly, country health departments have tended to report very differently on these questions over the last 90 years depending on their differing interpretations, making early statistical work on global progress more like sausage-making than an exact science. Just smash all those numbers together and see what comes out. So it is that the JMP, since its inception in 1990, has struggled to standardize these definitions and develop a common understanding of these goals.

Cue the Millenium Development Goals and their progeny, the Sustainable Development Goals—UN-sponsored efforts to rally the world around a host of lofty rights- and health-based goals, including water and sanitation. As we march forward into the 21^st century, it seems we are closer and closer to converging on a solid, universal understanding of access and progress; an idea of where we need to go and the commitment to get there.

Right?

Internationally sanctioned goals turn public health into a global Whac-a-Mole game—smashing endlessly at “unsafe” drinking water and sanitation indicators.

I’m not convinced. The problem with international goals and monitoring programs is that they require normalization and standardization if they are to be meaningful at all. They require that we conceive of everything as a ladder to climb—the “drinking water ladder” and the “sanitation ladder”—so that we can place people and countries along a spectrum and measure them. That has led to rigid, one-size-fits all understandings of “safe” and “improved.” The reality is that these definitions don’t always fit the situation.

Take water delivery, for example. Solutions considered “unsafe” or “unimproved” may actually be the best practice in some situations, according to a review by Dr. Jaime Bartram, Director of the Water Institute at the University of North Carolina Chapel Hill, and a team of researchers who advise the JMP. They wrote in the August 2014 edition of the International Journal of Environmental Research and Public Health:

Water delivered by tanker trucks may in fact serve as a source of sufficient safe water in some circumstances. This is reflected in the classification of such services as improved in the United Arab Emirates, Oman and Kuwait, where this is a regulated service provided by the drinking water authorities to nomadic populations and communities not connected to a piped network. Similarly, in some circumstances, sanitation services involving cartage… are managed hygienically… Changes in the definitions of these categories would have large impacts.

It’s like letter grades in school. They measure a student’s ability to perform by certain metrics but do not measure her true intelligence or ability to succeed. Some practices and technologies may be the best solution in certain contexts, even though by most international frameworks they aren’t considered acceptable.

Some approaches in sanitation, such as pathogen flow modeling, adopted by the organization Water and Sanitation for the Urban Poor, have begun to recognize the need for more nuanced metrics that inform decision making based on the maximization of health benefits (and the corollary, the minimization of pathogen exposure) at any point, not only for the first “user.” Approaches such as these are much needed innovations that engage the complexity of the whole system and may lead us to make unconventional decisions based on what actually works best—i.e., what most protects public health, not only the user’s health—in a specific context.

Plenty of homes in Bolivia have access to an “improved” tap on their property that is completely unsafe. I know other families who have to ride bikes several kilometers to collect water, but at least that water is chemically and microbiologically safe.

Let’s take a real-world example. I spent the last couple years living and working in the highlands of Bolivia which, according to the JMP, has still only achieved a little over half coverage of basic sanitation access and only 27 percent in rural areas. As many as 40 percent of the people still practice open defecation. By the numbers, there’s a long way to go. But the numbers don’t say much about the reality in the communities I worked with. The latrines I encountered in rural communities were never safely managed. Many had fallen into disrepair and disuse, were overflowing or leaking waste, and attracted clouds of flies. Instead of protecting health, they became a significant health hazard for the community. The concentration of unsafe fecal waste in latrines in some communities seemed to cause more gastrointestinal health problems, especially among children, than open defecation did in other communities. How would that change Bolivia’s rating?

Almost all conventional knowledge in WASH forbids me from saying that open defecation could ever be safer than latrines—but perhaps we can use this as a thought experiment at the very least. Is it worth reshaping culture around open defecation with practices like Community Led Total Sanitation (CLTS) if the alternatives are unlikely to be adopted and not truly safer? It forces us to remember that our end goal is more than climbing a prescribed ladder.

The same thinking applies to drinking water. Plenty of homes in Bolivia have access to an “improved” tap on their property that is completely unsafe. I know other families who have to ride bikes several kilometers to collect water, but at least that water is chemically and microbiologically safe. As Bartram and colleagues explain: “If a measure of the quality of the water were incorporated into the definition of an improved drinking water facility, the drinking water component of the MDG target would be badly off track.”

Perhaps we should allow ourselves just a hint of dissent when it comes to the global ladders and be willing to rewrite the metrics from the ground up

Indeed, many WASH metrics would off track if we changed the definitions. The question is, what track? The track toward “access” or the track toward health? By our current metrics, they are not necessarily the same track.

These questions shed light on our priorities as WASH practitioners and researchers. Internationally sanctioned goals, definitions, and ladders turn international development and public health into a global Whac-a-Mole game—smashing endlessly at “unsafe” drinking water and sanitation indicators, watching the points go up, but never slowing to consider, is this really appropriate?

Are we letting ourselves think boldly about what works and what doesn’t, and letting the answer be different for different contexts? How might our sense of innovation change if we let our definitions of access and success evolve?

Perhaps we should allow ourselves just a hint of dissent when it comes to the global ladders and be willing to rewrite the metrics from the ground up, informed by the places where they will be put in place. Often, the result will look a lot like what we’ve already got. Sometimes, it won’t.