How do you evaluate the long-term impact of school-based HIV prevention interventions that target children before they are sexually active? Long term impact evaluations of school-based HIV programmes are rare due to several methodological challenges:
- The expense of long-term cohort studies that track school children after school
- The difficulty of tracing young people after they leave school
- The challenge of defining an appropriate counterfactual to test intervention impact
- The difficulty of matching case and control groups
In 2017, Social Surveys Africa and Genesis Analytics conducted an Impact Evaluation for the Soul City Institute which used an innovative methodology to measure the long-term impact of having been a member of a Soul Buddyz Club in 2004-2008 among youth who are now 18-24 years old. By recruiting ‘ex-Buddyz’ in Gauteng and KZN and then identifying a matched control group of youth who had not been members, it was possible to show that being a member of a Soul Buddyz club significantly reduced the probability of having multiple concurrent sexual partners in the past year and, most importantly, increased the probability of being HIV negative by a factor of 2.92.
Social Surveys Africa Research Director Dr Tara Polzer Ngwato presented methodological lessons from this study at the 8th South African AIDS Conference. Download poster here. The study showed that it is methodologically possible to construct a reliable retrospective treatment and control cohort, even over a long time period like 10 years. This is a much more affordable and logistically feasible model for conducting long-term impact studies than tracking participants and controls over long periods of time.
These insights were also presented (together with Genesis Analytics) at a workshop on Impact Evaluations at the 6th SAMEA Conference in Johannesburg on 23 October 2017. Entitled “Impact Evaluations: Designing and Implementing Comparable Case and Control Samples. Practical Methods in Context”, the discussion focussed on the practical challenges of designing and implementing a reliable control sample for a retrospective cohort study. Workshop slides can be downloaded here.