What this thread about the methodology of the Plan Australia study indirectly reveals is the very limited research into the long term outcomes of CLTS and how important this is. My own research with Anggun Susilo on Indonesia, which there's a link too on the thread on CLTS and human rights (see: forum.susana.org/forum/categories/5-clts...mit=12&start=24#7279), led us to believe that sustainability could be a concern in a number of fronts, the lack of focus on emptying services being one. However our main focus was in the use of shaming, fines and sanctions mostly employed against the poorest in the communities we studied. Our analysis suggested that this approach was unlikely to produce long term behaviour change as similar approaches used in colonial times in Indonesia had not. Equally to link to the human rights issue raised earlier in this thread, we see that the shaming techniques impinged on the rights of villagers. In terms of practical suggestions, how about ending the use of fines, taunting and sanctions against poor and often vulnerable households. It's no more than school yard bullying legitimised by program goals.

Susan Engel
University of Wollongong

A couple of points in response:

Firstly, the discussion of sustainability in this thread came not from the factor analysis part of the study but the (purely observational) data from the re-verification phase.

Secondly, on the study methodology:

The list of factors was not presented to households during the interview process (as stated in the post below) - households never saw the list of factors. The study used a latrine timeline tool in which household members in a group were asked to develop a timeline for their latrine, from initial construction to the present. The timeline consisted of a line on the ground (or sim) with major milestones identified - latrine construction/re-construction, maintenance, collapse, pit filling up etc, as well as any significant events such as wet/dry seasons, harvest, ODF celebrations etc. The enumerators then focused the discussion on a significant event such as the final latrine collapse (for OD households) or repair (for ODF households) and were trained to facilitate in a non-leading way, listen to (and record) the response and then classify it according to the pre-determined factor list. The analysis used both the quantitative data (number of factors mentioned) as well as the qualitative data (the recorded comments). Further details of the methodology can be seen in Annex D of the report (enumerators guide).

We acknowledge the shortcomings in the methodology - particularly the facilitation skills of the enumerators (and the variation between enumerators) - and the difficulties of capturing behavioural information through observations and interviews. I think anyone doing M&E in WASH faces the same challenge as the resources required to do longitudinal studies or conduct more intense processes are not available in most implementing organisations. This is one of the reasons why the study stops short of making definitive statements or conclusions and in fact raises as many questions as it answers. We see it as the beginning of a process to look at sustainability of CLTS rather than providing final answers.

As someone who is going through IRC's WASH life-cycle costing course, I have to say that I find IRC's evaluation of sustainable sanitation very compelling albeit more suitable to issues of service rather than behavioural change.

While I'm not saying that CLTS is a definitive solution to WASH behaviour change, I do think that the way the Plan study was conducted generated a misrepresentation of what the actual effects of CLTS are. The study acknowledges its shortcomings, but I don't think they're sufficiently taken into consideration when understanding their impact on the study's conclusions.

The list of motivational and facilitative behavioural factors that was generated a priori to the fieldwork, and then presented to households biased the way that households responded - the fact that only these factors were mentioned by the households is indicative of the way that the presence of these predetermined factors crowded out alternative ideas. We know that when faced with a researcher, households tend to give answers that they think the researcher wants to hear. We also know that people are not always fully conscious of what factors are predisposing us towards daily behaviours, which we mostly perform "mindlessly". To ask people to identify "rational" causes for their behaviour is to ask them to haphazardly guess and make up answers that will satisfy the researcher.

In short, I think that the focused discussions and surveys generated distorted data. It would help if the design of discussions and surveys were more sensitive to unintentional behavioural cues offered by the facilitator and the setting by using opposing behavioural triggers to neutralize existing unavoidable and biasing triggers.

I also wonder about the reliability of performing one-off observations to determine use of latrine and handwashing facilities, which likely changes over time, and fail to capture frequency of use. These "snapshot" assessments must be taken with a large margin of error.

My foremost concern is the continued use of survey methods to analyze behaviour change, prioritizing quantity of information over accuracy. Behaviour is ephemeral and very different from the kinds of hardware interventions that survey methods were originally designed to assess. We have to come up with better methods for capturing and representing behaviour change, because survey methods do not adequately address temporality. We need to learn from performance-based fields like behavioural economics and ethnography, which have developed tools for studying human behaviour.

Just a quick reply to Marielle - would it be possible for you to state what IRC's position is on how to create demand for sanitation, especially in the dominantly rural-poor context where most open defecation takes place? Other than questioning CLTS, it would be helpful to know what approach IRC promotes or advocates for. Thanks!

The Plan's report is endorsing some of our questions around CLTS and indeed we find that the results they report underpin our understanding that this may not be the only approach or stand alone approach. However it is one of the key approaches that is being used and taken up at national level in various countries in Africa and Asia. We remain open to reflect on this approach, but maintaining questions around the sustainability aspect. What is a key importance is that open dialogue and research continue to reflect on this approach in line with the whole sustainable sanitation debate which is currently taking place.

Marielle Snel
IRC- International Water and Sanitation Centre

Dear All

I've just come across this thread and want to add our perspective. Firstly it is great that the report is being circulated and read, and the results discussed. For us (and I believe Plan) the point of the study was to examine their approach and learn from what has worked and what hasn't, and to contribute to the evidence base for CLTS.

I think it is a mis-representation to state that the study shows CLTS is unsustainable - our conclusion was actually the opposite: 87% of households continuing to maintain their latrines is actually a very positive result.

However each of the 4 study countries had their own set of criteria for ODF verification/certification as applied by the respective governments, most of which are indicators of improved hygiene behaviour, and it was when these were applied that the slippage rates increased. Therefore rejecting CLTS because the hygiene promotion component of a sanitation program is not as effective as it could be is like throwing the baby out with the bath water (to use an apt analogy). A more constructive conclusion would be to look at how hygiene behaviour change can be improved (something I would argue every WASH program struggles with - CLTS or not) to build on the success of household uptake and retention of latrines.

This is what the study was trying to say, and I commend Plan for investing resources in objectively critiquing it's approach in an attempt to improve sustainability, and for making the results public so others can benefit also. I think it would be an unfortunate outcome if selective bits of the study were used (without basing them in the overall study findings) to promote a particular viewpoint.

To this end I think the title of this thread is unhelpful as it links Plan Australia to what I believe is an incorrect conclusion (although I agree with the Elisabeth's comment that provocative titles do create more of a stir).

Glad to see some strong reactions to this report, though unfortunately it doesn't amount to much more than the same, tired old lines from the "CLTS Detractors" (and then the replies). Are we really still having this conversation?

As mentioned previously by other commenters - it would be much more helpful to focus on the important findings of the report (as darrens pointed out) and how to improve the strength and longevity of not only CLTS, but ALL community sanitation program outcomes. CLTS has been demonstrated effective at changing community behaviour regarding toilet use but is still challenged by hygiene and latrine up-gradation issues. The Plan Australia study is another important data point in this regard.

Personally, I have grown quite tired of the CLTS-basheing. As other commenters said - rather than do that, comemnters please provide constructive suggestions about alternative or innovative approaches that can match, or improve upon, open defecation drives carried out by CLTS programs (and the evidence to support it would also be much appreciated).

This work is hard enough - there's no need for those of us who practice these arts to be taking pointless potshots at other peoples' work.

Dear Darren,

First of all, Plan should be complimented for commissioning one of the most comprehensive independent evaluations of CLTS sustainability to date. Hopefully the study will convince other sector organisations how essential sustainability evaluation is.

That said, the most striking finding remains the fact that 92% of the CLTS programme households surveyed failed to meet Plan's own ODF standards 2 or more years after latrine construction. This answers the most important question, did the CLTS approach used lead to a sustained ODF ststus. That Plan is a learning organisation and is willing to take up the challenge to make its sanitation programmes more sustainable is praiseworthy. I have highlighted some of the nuances of the report in other posts to this discussion.

I think the study confirms what we know. That ODF criteria vary considerably in country and that the more you load criteria into ODF status, the less likely you are to achieve a sustainable outcome. By drawing a partial conclusion from the study, you fail to communicate the broader story - the richness is in the detail, and shouldn't be ignored.

The Plan Australia commissioned study itself includes an analysis of factors that lead households to revert to OD and factors that motivate people to remain ODF + six focus areas for improving CLTS approaches. The fact remains that the Plan study, based one of the largest field surveys to date, reveals that standard CLTS approaches so far have not proved sustainable.

Good! if this question mark has been introduced earlier, it would have saved us much stress, however, the stress worth while, it has opened eyes and minds. It has increased scope of thoughts.
Now the "question mark" has made it more of a forum where response is either YES or NO with your line of reasoning. It is now more of opinion oriented than argument.
That makes me appreciate the comment of JKMakowka, asking for more enlightenment. I would like to share my view with you. or mail me personally on This email address is being protected from spambots. You need JavaScript enabled to view it. because in most time i respond quickly to mails than chatting. However whatever we discussed would be posted here later.
Regards to all

Oh, OK. Actually, I didn't make up the thread title in the first place, it was Cor Dietforst. As a moderator, we sometimes modify the thread title after the discussion has developed in a certain direction (so that the thread title represents well the thread content) - in agreement with the person who proposed the thread title in the first place. So I will consult with Cor.

We should keep in mind here that sometimes a thread title that is "provocative" leads to more of a discussion than a thread title that only reads "New study released by Plan Australia", which nobody even clicks on...

So perhaps we can find a better thread title that is content-wise "correct" but still doesn't do injustice to the original idea of the creator of the thread.
I do think that Cor's idea of discussing one main issue that he found in the study is a valid one - but it may need to be modified if you think it is totally misleading to say it like this...

Might be nice if the authors of the study were also involved in this discussion.
According to the document they are Paul Tyndale-Biscoe, Matthew Bond and Ross Kidd by a company called FH Designs: fhdesigns.com.au/consultants.html
I will alert them to this discussion thread.

Regards,
Elisabeth