Tuesday, 21 November 2017

Two, four, six, eight, let’s all inn-o-vate!

The final evaluation report on the Children’s Social Care Innovation Programme [1] seemed to me to raise more questions than it answered.

While the authors seemed generally upbeat about the impact of ‘Wave 1’ of the programme, to my mind their report seemed to qualify many of its conclusions on the service quality improvements which the programme may have spawned. I found myself wrestling with phrases such as “The quality of services increased in 42 of the 45 projects that reported outcomes in Wave 1, in so far as these outcomes reflected the aims, or service users reported improvements” (page 70). I am still debating with myself exactly what that sentence means.

Many of the ‘hard’ indicators used to evaluate the projects (such things as reducing the number of children looked after, the number of children assessed as being in need, the number of re-referrals etc. etc.) also seemed to me to suffer from being what I call the ‘usual suspects’ – data that is collected centrally in the belief that it somehow relates to service quality, but no-one is exactly sure how.

And some of the ‘soft indicators’ seemed very soft indeed – e.g. ‘improving the quality of relationships between young people and their peers’ and ‘improving young people’s and families’ resilience’. I can’t think how I would measure either of those.

I also wasn’t convinced by the section of the report dealing with the value for money of the projects. While there seems to be evidence that there were some savings as a result of the projects, the report gives little information on the methodology used, except to say that not all the projects used the same methodology to monitor costs and benefits. There is also no discussion of the considerable difficulties in measuring unit costs in organisations which have large overheads and in assigning indirect costs to particular activities. [2] And I could find no discussion of whether the issue of whether local authority costs might be reduced, not because of greater efficiencies but as a result of work being picked up by other agencies.

My final reservation about the qualified optimism of this report concerns what is known as the Hawthorne effect [3]. In the 1920s an Australian psychologist, Elton Mayo, conducted research at a factory in Illinois. The aim of the study was to see if workers would become more productive in improved lighting conditions. At first productivity appeared to improve when changes to the lighting were made. But the changes were not sustained and productivity dived when the study ended. Mayo and others hypothesised that the productivity gains occurred because of the effect on workers’ motivation as a result of the interest being shown in them by the researchers. Subsequent research confirmed the existence of such an ‘observer effect’.

Armed with this piece of knowledge from what used to be called ‘industrial psychology’, it does not take a great deal of imagination to see how many of the perceived improvements witnessed in the innovation projects may be as a result of workers and managers experiencing improved morale and motivation as a result of the interest shown in them by the project’s funders and by evaluation researchers. It follows that a true test of the effectiveness of the innovation can only be made some time after the first evaluation has taken place. Are the changes sustained or do they quickly erode after all the fuss has died down?

A lot of what we know about innovation in organisations suggests that it is a fact of life that innovations can make considerable initial impact, only to be followed by a period of sustained retrenchment. That thought brings me to developments in theory and practice that took place in Japan in the second half of the twentieth century [4] and that will be the subject of my next post.


[1] Sebba, J. Luke, N. McNeish, D. and Rees, A. Children’s Social Care Innovation Programme - final evaluation report, Children’s Social Care Innovation Programme Evaluation Report 58, November 2017, London, Department for Education

[2] For a brief account of some of these issues see the following article in The Economist: “Activity-based costing” 29th June 2009

[4] Imai, M. Kaizen, the Key to Japan’s Competitive Success (McGraw-Hill, New York, 1986).