Get the biggest bang for the tribal buck
Most of the money that Iwi and Māori organisations spend is directed towards programmes that are trying to change the world. Whether it’s environmental advocacy, health services, education scholarships or postage stamps on the tribal pānui, tribal programmes are kaupapa driven and attempt to create a better lived reality for current and future generations.
It goes with the territory that it’s hard to know whether, and if so, by how much the organisation’s work is actually creating change. Performance measurement frameworks usually involve setting indicators of success and tracking performance against those indicators. However performance measurement systems are designed for measuring the speed of production and sale of widgets. They are not designed to measure quantity of social, cultural, environmental, political or identity change.
The kaupapa indicator bank is intended to help measure and track the performance of what actually matters – the kaupapa.
By improving the rigour of assessment of which projects and programmes contribute to the kaupapa, it will help with getting better at doing what matters. That is it will improve decisions around resource allocation (what to fund and how much), and ultimately assist in getting the biggest bang for limited tribal bucks.
The kaupapa indicator bank was developed on the basis of two key assumptions:
- That the real reason for the work of Iwi and Māori organisations is to contribute to the retention and strengthening of hapū and Iwi (or Māori) identity, and all of the work we do should contribute directly or indirectly to this mission
- That all the work of Iwi and Māori organisations should create benefit for whānau and hapū
The kaupapa indicator bank was developed to respond to these assumptions through one overarching question:
What does that (work/project/programme) mean for our people?
The rational for developing the kaupapa indicator bank is that many organisations appear to be focusing on articulating and measuring the scale of activities (e.g we will create 3 hapū strategic development plans), but are not doing particularly well at articulating or measuring the actual impact or benefit being created for the Iwi, hapū or whānau. e.g. those strategic plans will contribute to the strengthening of ahi kaa, measured in terms of x,y or z.
While there might be a honeymoon period where members are satisfied projects are being completed, perhaps with increasing scale, there will inevitably be a point where people start asking – so what? Over time, there will be increasing expectations that Iwi and Māori organisations are able to demonstrate that they are contributing to kaupapa that matter, and creating real change and benefit for real people. The kaupapa indicator bank is intended to be a modest step in being able to demonstrate benefit and meaningful impact.
The kaupapa indicator bank therefore seeks to:
- Create indicators that represent the ultimate kaupapa driving the work of Iwi and Māori organisations. These indicators endeavour to represent the successful outcome or impact that the tribal programmes are seeking to contribute to or generate
- Identify data sources so that organisations can track their performance over time against the kaupapa indicators
It is a reasonably comprehensive performance management system for the work of Iwi and Māori organisations, but it will require some efforts to implement and most likely, will also require some degree of adaptation so that it is consistent with the unique traditions and values of each Iwi and Māori collective.
The tool endeavours to help with connecting, judging and measuring our work against the things that matter to the perpetuation of Iwi identity and the lived realities of whānau and hapū: the ultimate kaupapa that drive our work. Each indicator is a result of the ‘so-what test’, which is an irreverent and blunt approach to asking the question ‘so what if we do x, y or z, how does that contribute fundamental parts to our collective identify and how does that contribute to the things that are important to whānau and hapū?
The ‘so-what test’ helps to ensure there is a clear connection between projects undertaken and the kaupapa that they are designed to serve. The model develops four sets of indicators that endeavour to connect the many busy parts of our work to the kaupapa that matter, as shown below:
Steps to Implementing the Kaupapa Indicator Bank
- Review and adapt the indicators contained in the Kaupapa Indicator Bank to fit the traditions, values and priorities of your organisation
- Correlate the indicators to projects, initiatives and activities your organisation undertakes
- Integrate the Kaupapa Indicator Bank into the strategic and annual planning processes
- Develop an IT system to record data collected to show progress toward the Kaupapa Indicators for your organisation (this could be as simple as an excel spreadsheet or there are a range of other programmes that could be used such as [insert])
- Conduct surveys as necessary to collect data for some indicators (See Kaupapa Indicator Surveys for support material and survey tools) to Measure Success against the Kaupapa Indicator Bank
- Report on your progress against kaupapa values and priorities either in a separate impact report or in the annual report produced by your organisation
Some Advice on Developing Your Own Indicators
There are two main frameworks for creating indicators, but the most important add-on is the ‘so what’ test. The ‘so what’ test is simply to keep asking ‘so what’. So what if we create 25 scholarships, how does that contribute to the actual kaupapa of deepening understanding of Iwitanga/marae development? Keep asking ‘so what’ until there is a connection with the ultimate kaupapa.
The two approaches to defining indicators are:
- SMART – indicators should be Specific, Measurable, Achievable, Relevant, Time-bound
- SPICED – which was developed for indicators designed to measure the nature and extent of change created, and focuses on involving people in developing indicators.
|Subjective||Informants have a special position or experience that gives them unique insights which may yield a very high return on the investigators time. In this sense, what may be seen by others as ‘anecdotal’ becomes critical data because of the source’s value.|
|Participatory||Indicators should be developed together with those best placed to assess them. This means involving a project’s ultimate beneficiaries, but it can also mean involving local staff and other stakeholders.|
|Interpreted and communicable||Locally defined indicators may not mean much to other stakeholders, so they often need to be explained.|
|Cross-checked and compared||The validity of assessment needs to be cross-checked, by comparing different indicators and progress, and by using different informants, methods, and researchers.|
|Empowering||The process of setting and assessing indicators should be empowering in itself and allow groups and individuals to reflect critically on their changing situation.|
|Diverse and disaggregated||There should be a deliberate effort to seek out different indicators from a range of groups, especially men and women. This information needs to be recorded in such a way that these differences can be assessed over time.|
Integrating the Kaupapa Indicator Bank into Planning Processes
Integrating the Kaupapa Indicator Bank into the strategic and annual planning process should help both frame and express strategic objectives that seek to give effect to kaupapa, in the way depicted below (which is a generic planning process):
Indicators Don’t Replace Strategy
Indicators should inform strategy, but not replace it. The ‘So What’ test might also help develop organisational strategy by helping explore whether current programmes are the optimal solution for achieving the desired outcome. For example, they might help with the strategic conversation about whether open scholarships for university or targeted scholarships in particular fields are more likely to lead to an outcome of marae development.
Don’t Get Hung Up On Causation
Tribal programmes won’t be the only contributors to the ultimate outcome: government, whānau, NGOs, business and the like will also contribute to realising the desired outcomes. It will not be possible to prove more than a contribution to the ultimate outcome, and proving that contribution might require some theory to be put forward and some extrapolation.
Measurement Takes Effort
There is a pithy saying that measurement isn’t an activity, it’s a culture. Collecting data needs to be hardwired into the organisation.
Background Information on Performance Management Systems
Performance measurement is based on the philosophy that organisations should ‘measure what they treasure’, because measuring is a way to inform how to get better at what is being done.
What to Measure
There are three types of things that are typically measured; inputs, outputs and outcomes.
- Inputs – are the amount of time, money and other resources that are put into trying to create an outcome. For example, the amount of capital, staff time, advertising and administration costs that go with creating a scholarship.
- Outputs – are the tangible products that are produced, usually in the short or immediate term. For example, the number of scholarships that are awarded.
- Outcomes – are the actual impacts or change or benefits created. Outcomes are the hardest to measure because they are typically created over the long term and may not be readily crunched into ‘units that can be counted’. But, outcomes are the things that are actually important and the real test of the value of tribal programmes because they demonstrate progress toward the kaupapa.
How To Measure The Different Types of Measures
- Inputs – measure in monetary terms e.g. it costs a total of $X (input). This data can be consolidated through internal accounts. To get the most robust figure, include the staff and overhead costs. To calculate the staff cost, assign a proportion of the person’s salary according to how much time they contribute to the project/programme.
- Outputs – typically ‘counted’ to form some kind of quantitative data. The figures should be sourced in the records kept by the organisation. For example, the number of scholarships, or number of submissions presented to Select Committees and the like.
- Outcomes – are more likely to be measured through qualitative data (peoples impressions). Collecting data about how people perceive the benefits or impacts created requires surveys and evaluations about the programmes and projects of the organisation.
Using The Data – measuring outputs, efficiencies and impact should help with:
- Resource Allocation – every organisation will have to make decisions about which programmes are funded and which are put on probation. The indicator bank should help in two key ways:
- Efficiency Measure – the Cost Efficiency Analysis (CEA) can help compare and contrast the relative costs for different programmes, and is likely to be useful in making a decision between two different approaches to the same goal. There are two other methods that can be used to assess efficiency; Cost-Benefit Analysis (CBA) and Social Return on Investment (SROI). Both these methods involve monetising (put a monetary value on) the social and cultural benefits through creating financial proxies. Creating robust financial proxies is ideologically fraught and complex, so they aren’t used in the indicator bank, but there is more information on them in Commercialism – Trade Offs Modelling section.
- Outcome Indicators – the outcome/impact indicators should help frame the ultimate benefits that will be delivered by a particular programme, and provide a way of talking about relative priorities
- Performance Improvement – all three types of measures will help form baselines and a way to track the organisation’s performance over time, and naturally, inform ways to make improvements