Note from Moderator: This post is the cut-off point of what will be included in the Thematic Discussion summaries and synthesis. New posts will not be included in the summaries or synthesis, but we welcome new contributions and further discussion in these threads!


Dear Krishan

To follow up our week 1 discussion (sorry was on holidays).
Yes we may loose something with coming up with a multi-criteria decision approach instead of a ladder concept for the JMP monitoring, but I still believe a "scorecard" can be very useful in strategic sanitation planning by supporting frameworks like CLUES, Sanitation 21 etc.
The discussion here on the forum as moreover generated many valuable inputs for the development of a criteria hierarchy that could be applied in a multi-dimensional scorecard approach (e.g. interdependence of different criteria, data collection, human rights approach, etc.).

I also believe it could be tested in Nepal within my research. If you are still interested, please drop me a line via email (see below).

Dorothee

Regarding my earlier post (#12050 dated 13 Feb 2015) regarding costs of collecting data, I had based it on the assessment paper by Morten Jerven titled ‘Benefits and Costs of the Data for Development Targets for the Post-2015 Agenda’. Please find attached notes from Morten Jerven’s assessment paper referred above. A brief summary is given below:

1. The UN High Level Panel has called for a data revolution. Addressing inequality / inequity issues is clearly present. The data revolution will have a considerable cost - yet the cost of data has so far gone missing in the MDG debates. If the call for a ‘data revolution’ is met, it has to be accompanied by a realistic assessment of the costs and benefits of providing the data.

2. Cost estimate: The best possible cost estimates indicate that if the previous MDG agenda would have been measured it would have cost about $28 billion. Yet, as we know there were gaps in the data and many indicators were never properly measured between 1990 and 2015. The $28 billion cost for 18 MDG targets computes to around $1.5 billion per target. Therefore, at $1.5 billion per target, the 169 SDG goals will have an estimated cost of $254 billion + opportunity costs of data not collected --> (i) this is almost two times the total annual spend on official development assistance (ODA) globally and (ii) 12.5 percent of the total ODA over the post-2015 period is spent on statistics.

3. Possibility for under- or over- estimation: The lower end of all guesses was chosen, but there is also evidence that costs might be much higher. There are several reasons why the actual costs could have been higher (listed in the attached document).

4. Cost benchmark: Information on budgets was unavailable for the statistical offices in different countries, but for a comparison Statistics Norway had a budget of 733 million NOK in 2013. This compares to a total government budget of 324 billion NOK – or about 0.2%. Thus, if the post-2015 measurement agenda is about as willing as the Norwegian state to spend on statistics, it should recommend and prioritize 3 or 4 targets, not 169.

5. Avoid bad data: Increase in demand for data can be met with inferior data. During the MDG period, both demand > supply of reliable data and incentivizing (rewards/punishment) data provision resulted in generation of inferior data. There is evidence showing that “results-based financing can have undesirable effects, including motivating unintended behaviours, distortions (ignoring important tasks that are not rewarded with incentives), gaming (improving or cheating on reporting rather than improving performance), widening the resource gap between rich and poor, and dependency on financial incentives”. Therefore, much care should be given in responding to calls for increased ‘accountability’ in measurement and ‘paying for results’ to achieve the MDGs.

6. Data sources and reporting frequencies: Reporting on MDG indicators have consisted of an update of the baseline (a census) every ten years, an annual smaller survey, and more sizeable surveys every five years or so to get reasonably accurate reporting on MDG progress. The most commonly used standardized surveys around the world are (i) population census, (ii) Living Standards Measurement Study (LSMS), (iii) Demographic and Health Surveys (DHS), (iv) Core Welfare Indicator Questionnaire (CWIQ), and (v) Multiple Indicator Cluster Surveys (MICS). Larger countries, with the exception of India and China, do seem to have more expensive censuses (per capita) than the smaller ones. Presumably this is because in larger countries some parts on the population are harder to reach.

7. Administrative and survey data: Administrative data is defined as readily accessible information which are regularly collected by the governments due to its day to day operations – the cost burden is born solely by the governments’ existing mechanisms, but collection, aggregation, reporting and dissemination is still resource demanding. Whether the data is primarily or typically collected from administrative or survey systems does vary from country to country, and as a general rule, in countries with weaker capacity in state administration, data are necessarily drawn from survey sources rather than administrative sources. The objectivity of the data is generally believed to be higher in survey data. It has been well documented that in poor countries data on improvements in—for instance-agricultural production, health and education tend to be overstated in the administrative data. In 2008, there were 60 MDG indicators in effect, the majority of them being survey data. Of the 60 indicators, 21 required administrative data and 49 required survey data.

The other papers in the series are:
1. Post-2015 Consensus: Data for Development Perspective, by Johnston ;
2. Post-2015 Consensus: Data for Development Perspective, by Demombynes Sandefur ; and,
3. Post-2015 Consensus: Science and Technology Assessment, by Maskus .

Warm regards,

Sowmya

@Elisabeth, thank you for the examples and explanation from the SuSanA case studies! Such a system is what I had in mind, with the scoring perhaps being a more complicated index which relates directly to the sanitation ladder rungs.

In answer to your questions:

However, I would like to understand better what you have in mind, Roslyn. Could you give a concrete example of how such a scoring system for health systems is used by our colleagues in the health sector (with whom we should collaborate much more)?

Here are a couple of examples from the health field (although I don't believe they use a numerical ranking system):

European Portal for Action on Health Inequalities (European Commission Project)
www.health-inequalities.eu/HEALTHEQUITY/...d_practice_database/

Projekte der Gesundheitsförderung: Good Practice (German, in German only)
www.gesundheitliche-chancengleichheit.de...rweiterte-recherche/

Qualitätssystem quint-essenz, for evaluation and quality-assurance, with project examples (Swiss, in German only)
gesundheitsfoerderung.ch/public-health/g...et/quint-essenz.html

And, not directly health-related but one I have come across is:
UN-HABITAT Best Practices Database
mirror.unhabitat.org/content.asp?typeid=19&catid=34&cid=10256
ex. at the level Award Winners: mirror.unhabitat.org/bp/bp.list.aspx
and their "ranking" system is described here: mirror.unhabitat.org/bp/UsingDatabase.aspx

Would people appreciate it if their project or programme is being ranked compared to another project or programme?

I am not sure the answer to this one, but I think that the focus is less on the ranking and competing, and more on the reasons it is in a certain category, but it would of course be potential for an issue (and as you mentioned, as is the criteria to maintain subjectivity). I will keep my eye out for good practice databases which use a ranking system, I don't have an example off-hand.

Roslyn

Dear all,

This is my first post in the TDS on the Sanitation Ladder and SDGs (Thematic Discussion Series), and I would like to congratulate the organisers, experts and participants for a very interesting discussion so far! Thank you.

I would like to come back to Roslyn's post from 14 Feb (scrol up 4 posts in this thread).

She said:

Using the idea from Krischan/ Florian of an index, individual case studies of sanitation projects are ranked based on a range of factors (towards an index) which reflect at which “rung” that project would be on the sanitation ladder.

Patrick then mentioned the schematic on Page 1 of the SuSanA case studies.
I actually want to highlight the table that is included in each of the SuSanA case studies under Section 12, see for example this one from a case study in Benin :



We have this kind of table for all 84 (!) SuSanA case studies (www.susana.org/en/resources/case-studies)

During my time at GIZ I worked hard on ensuring the quality of these case studies, and this table was always something that I felt strongly about.

Instead of using the +, - and o score, you could use a 1, 2, 3 score in which case you could add it all up and arrive at an overal numerical score.

I once did such an exercise where I plotted these assessment scores of all case study together in a 3-dimensional graph in Excel. One thing that became clear was that the majority of case studies (most of them being pilot projects only) scored well in the indicators technology and environment but quite low on the indicators financial sustainability and institutional sustainability (reasons: no longer-term financing concept, not embedded in institutional mechanisms).

Of course the scores are somewhat subjective (often times I corrected them downwards from when I first received a draft case study because people scored their own project overly positive), but as someone who knew the entire set of case studies at the time I could ensure that the scores made some sense for all case studies when comparing one to the other.

So this data exists if someone wants to take it further and work with it.

However, I would like to understand better what you have in mind, Roslyn. Could you give a concrete example of how such a scoring system for health systems is used by our colleagues in the health sector (with whom we should collaborate much more)? Would people appreciate it if their project or programme is being ranked compared to another project or programme? Actually, why not, it could be quite interesting and lead to more learning from each other.

Regards,
Elisabeth

Krischan wrote:

What is rather needed is a function based system that is thus flexible enough to account for systems appropriate for poor people's living conditions without the label "appropriate" being used as an euphemism for badly thought through government programmes.

I think that's a very appropriate way to wind this first week up actually. For me it brings in one of the trickiest things about using a function based ladder as a basis for monitoring, particularly in the Post-2015 landscape - how can we adequately bring in aspects of equity into sanitation implementation and monitoring?

Roslyn - I was thinking along the same lines as you with regard to the Case Studies publication of SuSanA. For each of the 25 case studies on the first page (top right corner) there is a small diagram which shows what particular aspects of sanitation are covered by the system described. I think this does fall short of what you were saying, but not by very far. I mean, it wouldn't be too difficult to "upgrade" these case studies to add such a feature if that ever would be needed.

F H Mughal wrote: My experience shows that the only sanitation ladder rung that matters to marginalized communities is the cost factor.

This is indeed bringing us to the 2nd weeks topic, but I disagree that a separate ladder for marginalised communities is needed.

I think the point that needs to come out of this and the general SDG equity discussion is exactly the opposite.
What I mean is that there is / should be a realisation that sanitation is a public good, and that marginalised populations have to be somehow financially supported to reach this public good. However they should not be further marginalised by supporting only special (or more often uninspired and badly implemented) systems "just" for the poor.

Therefore having a separate sanitation ladder would be counter productive in my mind. What is rather needed is a function based system that is thus flexible enough to account for systems appropriate for poor people's living conditions without the label "appropriate" being used as an euphemism for badly thought through government programmes.

Sanitation Ladder for the Marginalized Populations

Dear Roslyn,

An overview of the discussions shows that, it is the ‘advantaged’ populations that we have been talking about, when it comes to the sanitation ladder. We have, in large parts, omitted the sanitation ladder for the disadvantaged (vulnerable, marginalized) populations.

A fairly significant population in Pakistan and India consists of marginalized communities. They have no say in the government, political circles, and in the active communities and NGOs.

So, the question is: What should be the design of sanitation ladder for such populations; And can the sanitation ladder reduce inequalities?

Since the use of sanitation ladder is associated with sustainability concept, a target (in the attachment) reads: “Inequalities in sustainable use of water, sanitation and hygiene are eliminated or reduced.”

My experience shows that the only sanitation ladder rung that matters to marginalized communities is the cost factor.

This would be a good question for the discussions in the upcoming week. The worldwide concern of right to water and sanitation, and Catarina de Albuquerque’s Handbook on Human Right to Water and Sanitation, triggered this thought.

F H Mughal

As I’ve been putting together the summaries for the discussions here for Week 1, I have had a particular question that has come to mind (to me personally).

I have come to feel that there is somehow a triple role of the sanitation ladder:

• Monitoring aspects
• Setting goals for sanitation
• Providing recommendations on pragmatic implementation

The shift to the functional sanitation ladder from a monitoring and goal setting perspective has the potential to include the interconnectedness of sanitation to health, the environment, ecosystems, human rights and equity. But from a pragmatic level, what seems to be missing is pragmatic guidance on a decision-making level of “what do we implement in our own situation”. It is clear that, from the technology-prescriptive sanitation ladder, selecting certain technologies and recommending them as improved or not, is also not a way forward to provide pragmatic guidance.

Good practice database
So this brought me to thinking (particularly as I come from a health background) of good practice databases which are in the process of being developed at an EU level for health prevention/ promotion (which also has a multitude of factors which influence “good” implementation, and shares with sanitation a difficulty of monitoring and cause-effect indicators) for example, the aim to develop a good practice database from the Joint Action for Chronic Disease (www.chrodis.eu/our-work/05-health-promotion/).

Is there opportunity for a good-practice database which complements the sanitation ladder, and thus provides a practical implementation aspect to the monitoring aspect (and perhaps would also influence monitoring)?

My idea is to provide an accessible "good practice database" as a guideline of sanitation options for decision makers and implementers of projects (available online, book form, regularly updated). Using the idea from Krischan/ Florian of an index, individual case studies of sanitation projects are ranked based on a range of factors (towards an index) which reflect at which “rung” that project would be on the sanitation ladder. This good practice database would then not reflect how well certain technologies function overall, or whether they would always be a good option to improve the sanitation system, but rather, how, in a specific project circumstance (ex. a desert climate with a certain economic, political, social situation) that project reached the aims of the rungs of the sanitation ladder.

In some ways, I come to think of the case studies that SuSanA has on their website already (www.susana.org/en/resources/library) but a good practice database would rather be a systematic system which provides an organised (and thus standardized) scale of comparison to provide concrete examples towards achieving the rungs of the sanitation ladder.

Perhaps such a thing already exists?

I think this will be a question for Week 3 of the discussion (with a focus on “The way forward…adaptation of the sanitation ladder to post-2015 period”), but I wanted to include it here, because I think it has a relevance to the evolution of the sanitation ladder, and broadening the evolution of the sanitation ladder to see it as potentially evolving not just as a stand-alone framework, but as a framework which is used in combination with other tools to meet the needs of monitoring, goal setting, and pragmatic implementation.

-Roslyn

bracken wrote: As Elisabeth said, the ladder was deliberately chosen as the metaphor of choice because a monitoring ladder is still a broadly accepted concept, and we hoped and still hope to have an impact on the existing monitoring system. I don't think the scorecard approach would enable us to do this.

Ultimately a scorecard or index really is a different thing to the ladder, and I have to agree that building on top of the ladder metaphor does have advantages in the sense that it makes it easier to "implant" the idea of a function based rating of a sanitation system.

I think the main concern with the ladder is its strict linearity. This of course has some advantages too, as mentioned by Patrick, but maybe introducing at least 2 dimensions like done in the graphic by Florian make the functions based approach more worthwhile.

Of course 3 or even 5 dimensions / sustainability criteria would be ultimately the best, but for the sake of keeping it as a simple visual concept, stairs that follow the 45degree axis within a diagram like Florian's would be a suitable evolution of the ladder (ladder - stairs)?

What I mean is that while we as a informal network will probably not have much impact on JMP monitoring and the like, further promoting the focus on functionality and using the "meme" of the ladder for it might indeed turn out to be worthwhile and realistic as a goal.

Interesting discussion … thanks for all your contributions …

Two minor comments in relation to the discussion about similarities / differences between a functional-based ladder and the index approach suggested by JKM and Florian:

I agree with Patrick and Elisabeth about the issue of presenting / disseminating the data … a two or even three-dimension graph approach probably gives more information than a one-dimension graph. However, there is in my opinion another difference. The index approach (score card or other similar approaches) allows various dimensions of sanitation to be combined (weighted and aggregated). In the example provided by Florian, there are four different dimensions that can be aggregated or presented separately … In a ladder, the issue is that all the dimensions are somehow aggregated using a multiplicative function. That is, one cannot climb one ladder rung unless all dimensions are fulfilled. I am not very sure about this, but my feeling is that for simple approaches (one or two dimensions, i.e. monitoring only one or two functions), a ladder may work very well. But if one aims to monitor different aspects of sanitation, I am not sure if a ladder is best solution. I agree that one can focus on the final number of households in each of the rungs … and this information is valuable! But in this process you may lose a lot of information (for instance, you are not showing those households located in rung one that are fulfilling some criteria of rung 2) … Probably, this information is useful for planning purposes …

Another issue I would like to raise is if one single ladder could cope with the sanitation challenge all over the world, including both developed and low-income countries … In my opinion, I would say that these approaches are very useful for monitoring basic sanitation … I am not sure about the utility of the ladder in Europe … Therefore, one alternative could be to simplify the ladder by directly including those aspects that are major concerns in those settings where sanitation is elusive. One conceptual framework that is very relevant when defining these aspects / dimensions is the rights approach, thus including the issues of availability, physical accessibility, safety, acceptability, affordability …

Not sure if this helps the discussion …

Best regards,

Also it strikes me that moving away from technological steps on the ladder will make analysis more difficult. Whilst it might be relatively easy to tell whether or not improved sanitation technology is provided (can we can give diagrams of the tech we mean for each step), how will anyone tell if these functional steps have been achieved?

As the higher steps are reached, an increased amount of information about achieving the prior steps will be needed - to prevent trade-offs happening between the different requirements. What happens if an intervention is proposed which does not address the prior steps? Would that not count at all?

Sowmya makes some very good points there - especially given what we discussed about the World Bank report on sanitation the other day: forum.susana.org/forum/categories/164-fi...ing-in-a-supermarket

As the complexity of measuring sanitation indicators increases, the cost/complexity of collecting data in a way that is comparable across nations is going to inevitably increase, one would think.