Dear Kris,
Thanks for telling us about this article (title: The Problem With Evidence-Based Policies) which explains nicely the problems with randomized controlled trials (RCTs)!

I bumped into this problem last year while working on the Wikipedia article on "mass deworming" (en.wikipedia.org/wiki/Mass_deworming). You can see here on the talk page (en.wikipedia.org/wiki/Talk:Mass_deworming) how we (in particular Joe Turner) were trying to make sense of new studies (Cochrane Review) that showed no evidence of school based deworming programs on child health, and struggeling to present everything correctly in the article. There were several factors involved but one was that RCTs were regarded by other Wikipedia editors as the "gold standard" of research and if it wasn't an RCT then it was deemed as irrelevant.

In this context, I want to point out to you the famous parachute paper which is pretty revealing and thought provoking (and funny!):
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials




Regards,
Elisabeth

P.S. the person who sent me this parachute paper also wrote the following per e-mail (he used to work for WHO but is now retired; he wants to stay anonymous):

++++++++

I have checked the web page and have a comment regarding evidence on helminthiasis control. The papers you quote are systematic reviews of randomized controlled trials (RCT). Observational studies are, by the Cochrane group, not considered appropriate and are disregarded for decision making and their “golden standard” is that ONLY systematic reviews of RCT should allow for decision making for what works in health and also in public health (PH). Their patronizing message, that they include in most if not all their published papers, is: In modern medicine, the efficacy of an intervention should be investigated in well-designed randomized trials. Results from the trials should be collected in a high-quality systematic review, if possible with a meta-analysis. And finally, the evidence should have its repercussions on practice guidelines.

This in is a wrong starting point. It is wrong as many PH decision have been made by observational studies or simple “expert opinion”, have worked and continue to work for the benefit of millions of people around the world. Vaccination of small pox, “discovered” in 1798, is a typical example that has allowed us to achieve the only eradication yet of a disease. Similarly the evidence for the feasibility of the eradication of guinea worm has never been proved by any RCT. I could go and on with both examples in PH and general medicine. I am attaching a comic paper (parachutes) that was published time ago on the BMJ on parachutes……….the recommendation for the RCTs fundamentalist stands today as it did when it was published!

Regarding worm infections they just do not know what they are speaking about. The issue is due to the “over-dispersed distribution” of worms in infected populations and the difference between macro and micro parasites. As these macro parasites (worms) do not replicate (at least most of them and surely the one we are discussing) in the human final host it takes time to build a worm load and those people that have high worm loads (that are few in an infected population) are those that have the worse effects of worm infections, with associated stunting, malnutrition, anaemia in case of hookworms and schistosomes, etc. Those with few worms have more subtle pathology and morbidity.

As a consequence those that have a high worm load will benefit more from intervention and those that have a few worms will of course benefit less. You do not need a rocket scientist to understand that!!! In an RCT you need to take this into account and to take this into account you need huge sample sizes and compare effects in different groups based on their original worm load. So the design of the study must be based on the host parasite relationship and dynamic of transmission and ecology of the infections. Very few RCT on helminthiasis have done this as it would cost a fortune. Furthermore different parasites have different physio-pathological effects on their hosts. Hookworm will mainly cause anaemia (and this is directly related to the number of hookworms in the host and the species (Necator americanus or Anchilostoma duodenale with A.duodenale having a more important impact on anaemia per worm as they cause more blood loss). Ascaris lumbricoides more an effect on growth, etc.

I am afraid the various Cochrane reviews on helminths did not take into account studies conducted in areas with mainly one parasite transmitted and mixed effects of one species with that of another (stunting for hookworm, etc. when hookworms will have effect on anaemia etc.). They even quoted papers I was involved in areas of hookworm transmission looking at effects on growth! They just have no knowledge on the epidemiology of helminthic infections and reach conclusions based on such biased views as they do not look at papers and read them as parasitologists but as “systematic reviwers” with no biological knowledge.

Furthermore all the present discussion on the relevance of impact large scale chemotherapy against worm infection regards more “subtle” effects such as growth, school performance, work performance and even increased wages of people treated versus workers not treated. This impact is important but forgets another element: large scale interventions prevent also deaths and severe morbidity of a small but important proportion of the population. That is per se sufficient to justify large scale chemotherapy against helminthic infections that WHO defines Preventive Chemotherapy. We could say that similarly the proportion of people with paralytic polio is “small” and that this per se should justify spending a billion USD a year to eradicate this horrid disease. If not one scientist dares to, most appropriately, argue this general point of view why does a small group of “systematic reviewers inquisition fundamentalists” argue so much on large scale Preventive Chemotherapy? What are their real objectives?

In the 19th century, in one of the most elegant studies on Ancylostoma duodenale ever performed, Italian parasitologists observed (observational studies again I am afraid!!) that the Italian workers that were drilling the St Gottard tunnel in Switzerland were becoming anaemic. Coming from the south of Italy most of them were infected with hookworms and due to the conditions of work in the tunnel and complete lack of sanitation they became severely infected with hookworms transmitted in the tunnel itself, consequently becoming very anaemic and many of them eventually dying. In 1880, Edoardo Perroncito first noted the correlation between hookworms and anaemia among miners digging the St. Gottard tunnel in the Alps. They also correlated worm loads with severity of anaemia and Grassi was the first to mention that counting worms was critical to understand the relationship with morbidity. Grassi in 1878 had looked at the importance of the worm load 40 years after Dubini and stated in Italian “la diagnosi viene agevolata dallo studio delle uova…..con queste ricerche arriviamo ad affermare la presenza dell’elminto; ma siccome dal suo numero dipende il grado della malattia, cosi’ fa profitto anche lo stabilire la diagnosi quantitativa. La quale é possibile all’occhio esercitato in base ad un fatto da me ripetutamente certificato, e cioè che la quantità delle uova è direttamente proprozionale al numero degli elminti” (reported by Ivo de Carneri page 253, Parassitologia Generale e Umana 11ª Edizione, 1989). 137 years later do we still need to have proofs that hookworm infection has an impact on anaemia based on intensity of infection and to justify large scale chemotherapy with the safest drugs ever used in medicine like albendazole and mebendazole?

I think the Cochrane group has a mentality similar to that of the 15th century inquisition and is unable to think beyond dogmas: i.e. that if you do not go through a RCTs (and especially their own review, their own “holy inquisition” to get their imprimatur) you will not be allowed to become PH policy. This is deeply and dangerously wrong and the visibility given by the press has been dangerous. Anthelminthic treatment on a large scale is only one of the PH interventions they have reviewed and “dismissed” as not evidence base. Their influence is dangerous and we need to combat it for the benefits of millions of people. For the time being I will continue to be with the heretics, the Giordano Bruno’s of today, and continue to promote large scale preventive chemotherapy, up to the time the Liverpool fundamentalists will get me and try to burn me!

+++++++++++

PH = public health

Posted this elswhere already, but this is actually the thread I was searching for to "bump" with this interesting article/opinion piece:
www.project-syndicate.org/commentary/evi...rdo-hausmann-2016-02

This thread which Patrick started a year ago, is still as topic as ever.

Right now we have a big debate going on about mass deworming and its long term health benefits (also difficult to prove with randomized controlled trials, which are supposedly - but maybe not - the gold standard in evaluating health benefits). This is the topic of this thread here:
forum.susana.org/forum/categories/159-in...as-much-of-an-impact

Coming back to the more general question, I wanted to point out a keynote presentation by Peter Harvey (UNICEF) at the recent WEDC conference which was a real conference highlight for me (and - going by the number of tweets - also a highlight for many others at the conference).

Invited Speaker: Peter Harvey
WASH, Nutrition and Health: A futile quest for evidence?

Peter is Regional Adviser - Water, Sanitation and Hygiene
UNICEF, Eastern and Southern Africa Regional Office
Nairobi, Kenya


Peter has kindly made his presentation available for sharing here:




These two slides caught my particular attention:

Very interesting account of UNCIEF's path with regards to WASH since 1946:



Clearer image:


Limitations to randomized controlled trials (slide 11):



The references that he mentioned on this slide are:

Robert William Sanson-Fisher, Billie Bonevski, Lawrence W. Green, Cate D’Este, Limitations of the Randomized Controlled Trial in Evaluating Population-Based Health Interventions. American Journal of Preventive Medicine Volume 33, Issue 2 , Pages 155-161 , August 2007

Porzsolt, F., Kliemt, H., Ethical and empirical limitations of randomized controlled trials Med Klin (Munich). 2008 Dec 15;103(12):836-42. doi: 10.1007/s00063-008-1132-x

++++++++

Some points from my notes:

1. At UNICEF, WASH has always been looked at from a child health perspective... We know that health has an important connection with WASH but there is more to it. It's not (only) about health, it's about burden and convenience as well (Peter asked: "would you give up your toilet tomorrow if someone told you that it had no health benefits?" No!). There are also the human rights aspects.
2. In the 2014-2017 Strategic Plan, WASH is included and one new element is the link with nutrition
3. How is evidence applied: Donor priorities, organizational priorities, implementation strategies, geographical targeting

Some tweets that were made py participants during his presentation:

Stephen Jones ‏@stephen_djones 30. Juli
#WASH & nut evidence has implications for donor priorities, geog targeting, units of intervention - Harvey #wedc38

Susan Davis ‏@improve_intl 30. Juli
Harvey:Why do we need to talk about links of #WASH to health? #wedc38

Stephen Jones ‏@stephen_djones 30. Juli
Need to distinguish btwn evidence 4 benefits of #Wash on health & evidence for diff approaches to achieve #WASH + health #wedc38

Stephen Jones ‏@stephen_djones 30. Juli
Harvey: non-health #WASH benefits can be at least as important, or more, but hard to quantify - Harvey #wedc38

Susan Davis ‏@improve_intl 30. Juli
Harvey: #water is a fundamental right, nothing about health, about burden & convenience #wedc38

Cheryl ‏@Cheryl_McD 30. Juli
Peter Harvey @UNICEF - tackling SDG 6 on #WASH will help achieve the other SDGs #WEDC38

Susan Davis ‏@improve_intl 30. Juli
Harvey: sustainability of #water & #sanitation services is an important part of rights based approach #wedc38

++++++++

If you have comments or questions about Peter's presentation, please put them here. I will alert him to them.


Regards,
Elisabeth

Hi. I was asked to comment on the Fit for School Health Outcome Study. As far as i can see from this preliminary report, this study had similar problems as most of us have when we research complex interventions. Study design was largely driven by operational constraints which are perhaps greater in our than in any other field. Hence the poor evidence base despite clear biological plausibility.
On the plus side:
-objective outcomes
-a credible intervention with fairly clear pathways to changing the outcomes, especially dental health
On the minus side
-the biggest issue is that the number of clusters is very small. As far as I could tell there were only 4 schools in the intervention and 3 schools in the control. In such cases, direct comparison between intervention and control become impossible. All that one can do is test whether the intervention changed the outcomes in the intervention arm, with the control arm serving only to demonstrate absence of secular trends. But there were secular trends, for example for worm infections.
-i cannot see a very consistent intervention effect across the different outcomes. Some show promise others not. If a primary outcome has not been specified, then its not easy to make much of it.

Dear Cor,
unfortunately I am not able to share the deskstudy in public due to some internal information included. It had not produced additional or surprising information to what Schmidt had already elaborated in his article.
Cheers
Christian

[Start of Page 2 of this discussion thread; access Page 1 with the page button above or below]

This certainly is proving to be quite an interesting discussion, with equally interesting tangential issues coming in with every post.

Firstly, with regards to the "downgrading" of potential financial returns on the health benefits of improved WASH services, I can understand the argumentation very well. The (increased) degree of uncertainty of the health benefits makes a mockery of any claims to exact calculations. But rather than simply downgrading, I imagine a fuzzy logic approach to estimates could prove more useful. I'm not sure at all how this would work in practice but do see it as offering a way out of definitive, carved in stone declarations. This would not necessarily imply a downgrading, but more a blaring of the edges of what the likely benefits could be within a framework of possibilities.

Secondly, having read the GIZ supported report from the Philippines I have to say, I think in a way it highlighted exactly the issues Wolf-Peter Schmidt was talking about in his editorial. Not being an expert in clinical methodology I confess to not having understood all the fine points of the report, however it did strike me as strange that the experimental group was entirely on a small island, whilst the control group was on a different, much larger island - meaning the external boundary conditions were likely to be quite different.
Also, as far as I could see, in essence from a hygiene point of view, the only main difference between the experimental group presented and the external concurrent control group was (apart from the toothbrushing) that they washed their hands once a day with soap and clean water as a supervised activity in the school. Both groups were biannually dewormed. The noted improvements in the experimental group, particular with regards to improved mean Body Mass Index didn't seem to me to be particular dramatic (from 14.73 as baseline to 14.88 after 1 year) as the children remained well in the "underweight" category (< 18.5 according to the hospital authority of Hong Kong). Given that the only difference really seems to be the handwashing once a day and that children's daily lives includes a lot more than that one handwashing moment within school, I personally think its stretching more than a little bit to draw a direct causation between the EHCP and the slight improvement of mean BMI after one year.

And thirdly, I'm not at all convinced about a causal link between the quality of social services and peace. I would rather imagine peace is more directly related to the equity of access to essential resources to enable households / social groups to fulfill their basic needs (this would thus include land, water, other natural resources). And whilst access to SOME of these may be improved through social services, social services in themselves are not a determining factor, but rather can serve to buffer temporary variations in access to essential resources.

all the best,
patrick

If return of investment figures from health interventions are based on flawed and overly optimistic studies (which most likely they are) then they certainly need to be downgraded.

However one maybe almost too common-sense fact (but seemingly still overlooked) is that these interventions are always to be seen in a complex setting. There can not be a direct causal relation in the sense that there are so many other factors that can effect the outcome. For example, even the best household/personal hygiene practises and WASH infrastructure will not show much effect if people do unprotected manual work in rice paddies flooded with highly polluted river water.
In most cases WASH services are also not the last but one of the first of many interventions to allow good development and thus returns from later interventions (like electrification) are probably overstated comparatively.

dietvorst wrote: Similarly we like to think that there is a causal link between improved social services, including WASH, and peace. The millions invested in social services in South Sudan up to the end of 2013 have not brought peace because of underinvestment in security, good governance and employment for young men/former liberation fighters.

Anecdotal evidence actually points in the opposite direction, e.g. that improved social services by "outsiders" lower the cost of conflict and often prolong it (Edit: article link ).


[End of Page 1 of this discussion thread; access Page 2 with the page button above or below]

Christian wrote

At GIZ we conducted a deskstudy and had many discussions with reserach partners about this topic

Dear Christian, could you please share this study on WASH impact with us? I could not find it immediately on the GIZ website.

Bracken wrote:

I would argue that downgrading the figures is justified when you consider the limited resources available for investing in development interventions. Otherwise you raise the false expectation that investing in a single intervention like WASH will automatically lead to large economic benefits in the form of improved productivity.

Similarly we like to think that there is a causal link between improved social services, including WASH, and peace. The millions invested in social services in South Sudan up to the end of 2013 have not brought peace because of underinvestment in security, good governance and employment for young men/former liberation fighters.

I'm not sure if there is necessarily a justification for downgrading the return on investment figures simply because they are based in part on health care saving.
These health care savings will almost certainly occur, even if the necessary monitoring methodologies and structures do not currently exist to exactly quantify them.

I suppose the challenge lies in identifying suitable indicators and proxies that can be used to measure other noticeable improvements in living standards as a result of WASH interventions in the field, and which are not necessarily reliant on self-assessment of households or of the programme implementers. It seems to me that much of the problem comes directly from the bias in this method of assessement and the rose coloured glasses of all involved. This is then further complicated by the difficulty in carrying out trials monitoring health indicators

The Philippines study from GIZ certainly looks impressive at first glance (particularly to me, as I am not a health professional) and surely merits a closer look. Would of course be interesting to hear Wolf-Peter Schmidt's opinion on it.

Great discussion, I will ask Wolf-Peter Schmidt, the author of the "The elusive effect of water and sanitation on the global burden of disease", if he wants to join in.

As Schmidt mentions in his editorial, it would be much better to focus on the socioeconomic benfits of WASH. That being said, the current figure "For every $1 invested in water and sanitation, $4 is returned in economic returns (Hutton, 2012)" would need to be "downgraded" as the calculations are partly based (10-25%) on assumed health care savings ( WHO/HSE/WSH/12.01 ).

Interesting to note, is that the much discussed CLTS evaluation study commissioned by Plan Australia, actually found that health was the most cited motivator for initially building a latrine in both ODF and OD households and for maintaining a latrine in ODF households.

Maybe we should move away from "demand creation" to asking people (men, women, children, disabled etc) what their priorities are. If that's some form of WASH (and often it is), half our work is done. Beware the tyranny of experts .

Very interesting discussion. I have also read the article and got renewed confirmation that health impacts are difficult to measure for practical reasons. Nicely summarised by Schmidt. Nevertheless as the Schmidt says rightly and what also Patrick says is that non-health impacts are equally if not more important depending on the context and aim of intervention. So justifying WASH interventions soley by health impacts are not the best way and anyway shortsighted.

At GIZ we conducted a deskstudy and had many discussions with reserach partners about this topic. There is surely potential to make WASH project more effective, also with regard to health impact. If there are proper indictors on adequate use of toilets and handwashing devices (at userinterface level) and also a functioning sanitation chain with safe disposal or reuse at the end, it should be a good proxy for assumed health impacts.

In the scenario of schools there is actually good evidence on health impacts. For example the longitudinal study of the Fit for School programme in the Philippines by GIZ has shown impressive improvements of BMI, dental health and school attendance of primary school children. The simple interventions of daily handwashing, toothbrushing and bi-annual deworming have proofed to be effective measures for the Philippines. see the study here www.biomedcentral.com/1471-2458/13/256

Cheers

bracken wrote: I'd agree with you Florian. The Cost / Benefit analysis of rural roads (and even electrification) is probably relatively easy to perform simply by looking at rural economies, access to markets for agricultural goods and so on, and its easy to demonstrate the "bankability" of infrastructure projects in general.
Its much harder to show the effects of behavioural changes in society and even harder when there is a subjective eye on the monitoring of results (as needs to be done for sanitation and hygiene). The bias in reporting impacts seems at the moment somehow inevitable.

I totally agree (and never meant to say anything else) that rural roads and electrification probably have a big impact (although I think railways are more cost effective).

What I mean is that those investment areas never had the kind of scrutiny applied to it. It is just considered to be working and some rather simplified economic analysis probably also shows that they have the intended impact.

However I am sure if one would start doing these kind of double bind etc. studies focussing on a single factor over a short period of time, the same kind of results as in WASH would come up, i.e. either badly designed and overly optimistic studies, or ones that fail to show a significant impact.
Take for example rural electrification: initially all it does is to allow some richer people to have a fridge and a TV... yet no-one really denies that the overall long term effects of it can probably lift people out of poverty.

The issue is really summed up by the original author's comment that a certain approach "may not be the right tool to prove" the impact of infrastructure improvements (or hygiene behaviour change interventions).

That said, I think it is good that these questions a raised and the fact of failure to show impact with such test-tools is thought provoking and should not just lead to a discarding of such tools without an alternative.