Understanding Learning Experience at IkamvaYouth

By Masha Lisak and Carmel Farzaneh, Learning & Impact at Omidyar Network

IkamvaYouth staff were stumped. Why were high numbers of learners unable to meet the minimum 75% attendance requirement of their after-school tutoring programs? It was a recurring topic at team meetings, and they were struggling to understand the situation. The group had brainstormed possible reasons, and had two hypotheses: first, that learners didn’t find the program valuable and therefore stopped attending, and second, that compulsory after-school sessions were causing schedule clashes, but they really had no way of confirming whether these were correct, or the extent to which they impacted learner attendance.

The South African nonprofit was trying to solve a big problem through its after-school tutoring programs: the country consistently ranks low in educational outcomes and only 10% of youth in South Africa go on to tertiary education. Omidyar Network has supported IkamvaYouth since 2013 and has seen how its programs strive to provide learners from the most disadvantaged South African communities with the knowledge, skills, networks, and resources to access tertiary education or employment opportunities.

The organization’s staff were eager to figure out how to retain these learners, and we had an idea of how they could find out. We connected IkamvaYouth with Acumen’s Lean Data team, which applies experimentation principles to use low-cost technology to gather social performance data directly from customers and aims to create value for a company and its customers. Patrick Mashanda, the Programmes Manager of IkamvaYouth, applauded Acumen’s consumer research, noting that it “came at just the right time.”

Within a week of their initial conversation with IkamvaYouth, the Lean Data team was off and running with the project. They first defined the research questions: why do learners sign up? What impact does IkamvaYouth have once learners start attending? What keeps learners coming back? They also made a novel suggestion: in addition to surveying the students in the program, the team could reach out to learners who dropped out and their parents to understand what was driving dropouts. IkamvaYouth was intrigued. What would those who had left the program say?

With research questions clearly defined, the team created and carried out an online survey with tutors, in-depth phone interview with learners, dropouts, tutors, and parents, and IVR (automated phone) temp-check trials which allowed learners to give feedback within 24 hours of their tutoring sessions. Through these methods, they directly connected to 531 unique voices that could share insights about their experience with IkamvaYouth.

After analyzing the data, the Lean Data team shared findings with IkamvaYouth. They found that word of mouth from friends was the most important driver for acquisition, responsible for nearly half of those enrolled. Teachers were also important advocates of the program, encouraging about a third of the learners to join IkamvaYouth.

Interestingly, both current students and dropouts rated the program highly. Improved grades were cited by nearly half of students as the main outcome. For parents, observing that their children were more committed to their education was also important. These findings held across both active learners and dropout learners. The results surprised the IkamvaYouth team, who had assumed that dropouts did not find value in the program.

With their original hypothesis debunked, the team dug deeper into identifying why learners drop out. A requirement to participating in the program had been to attend 75% of their tutoring sessions, and when learners didn’t meet this bar, staff assumed this was mainly due to a lack of interest. However, surveys found that many learners were actually missing the requirement due to legitimate conflicts rather than a lack of interest. More specifically, a quarter of those who stopped attending had commitments at home or caretaking responsibilities for a relative which made their attendance impossible. For others, the issue was simply a conflict with other after-school programs or difficulties traveling to the tutoring site. One such learner said, “The problem is I live far from Ikamva, so I did not have transportation to attend all the time, and during winter it was too cold to walk…”

Inspired by these findings, IkamvaYouth is now taking two important steps. First, they are carefully considering distance from the centers when enrolling new students. During intake, they match students to sites that are more accessible, and within a 20-minute walking distance. Second, they’ve developed an “Ikamva Lite” model, a less intensive program with fewer attendance requirements that can serve learners with scheduling conflicts and are already piloting this in Diepsloot, a township in the north of Johannesburg. Going forward, IkamvaYouth plans to also track why students drop out so they can continue to address the changing needs of their learners.

The Lean Data team also suggested evaluating tutor quality to understand how the volunteer tutors might be affecting student learning. To assess this important piece of their program, they piloted a phone survey in which they asked learners to evaluate their last session. While many responses were positive, some showed that learners were disappointed by the sessions, leading IkamvaYouth to more actively evaluate their tutors.

The findings of the Lean Data project are opening conversations about how to engage tutors in development and how to ensure they are well equipped. IkamvaYouth has started a practice at some of their branches where learners give feedback to tutors right after each session, and they’re finding that this simple reflection creates accountability and strengthens the relationship between tutors and learners.

Bolstered by the findings of the Lean Data project, IkamvaYouth is now themselves learning better ways to support and encourage the next generation of South Africans.

Designed specifically for social enterprises, Lean DataSM helps build more impactful businesses by providing them with data on their social performance, customer feedback, and behavior. Lean Data uses low-cost technology and methods to gather high-quality data quickly and efficiently. Find out more here and take their free +Acumen Lean Data online course.