BTSR guidebook ~ Training & skills self-evaluation

What provision at this level looks like and feels like

For the organisation

In 2008, 26% of broadcasters, evaluated at this level. 62% were lower and 12% were higher.

At this level, individual training interventions are routinely evaluated for their impact on learners; in terms of the Kirkpatrick model, the organisation is at the 3^rd level.

The results for a given intervention are reported in a consistent way. They might or might not be comparable across departments, subject matter and types of intervention (for example, on-job and off-job training in the same subject might be evaluated in different ways).

At this level, the evaluation systems stop short of evaluating the impact on the organisation.

For the training function

In many ways this is the cosiest level for competent trainers. Successes are recognised. Other outcomes are quickly detected, giving you as much time as possible to turn them into successes… whereupon the turn-round will count as a further success. And all of this takes place within the familiar cocoon of the organisation.

Benefits & risks of provision at this level

Benefits

Benefits of provision at this level can include:

• Learners, trainers and (maybe) learners’ line managers understand that the organisation cares about the quality of training & development provision, and therefore invest it with a degree of care and respect
• The organisation has relevant information that helps it make sound decisions about training & development. It can mine its evaluation data to answer questions such as:
  • was this year’s new approach better or worse than the one it replaced?
  • how does class size affect outcomes?
  • do any instructors particularly stand out?
  • if learners do this practical experience before attending that course, does it affect the amount they learn?
• Those sound decisions can be used to raise the organisation’s game in every aspect of its training provision (and therefore in all strands of the self-evaluation).

Risks

Risks of provision at this level include:

• When evaluation reports are not comparable and not aggregated, the picture they paint could be deceptive
• Since evaluation doesn’t extend to business impact, bad decisions can be made unknowingly
• In particular, if the organisation is providing wonderful training that everybody admires, but business outcomes are suffering, there might be nothing in the data to say so.

Processes & benchmarks for evaluating provision at this level

Distinguishing from «base« provision

The differences between base and medium provision are mainly to do with how regularly and systematically training & development interventions are evaluated. At this level there is an evaluation process that is all or most of:

• Routine – it happens for all learning interventions
• Inquisitive – it gathers enough data to provide insights into why Reaction, Learning & Behaviour outcomes are the way they are
• Consistent – at least some of the data is comparable across different training interventions and over time
• Coherent – it is feasible to aggregate at least some of the individual evaluations into a bigger picture.

Distinguishing from »high» provision

The differences from high provision are mainly to do with the scope of evaluation and the degree of integration with other business systems. At this level:

• Evaluation stops short of the 4^th level of the Kirkpatrick model
• While the evaluation of training is routine, it might not be systematic (but, rather, a series of separate tasks or projects)
• Evaluation processes are not fully integrated into other business systems.

Processes

Here are some diagnostic questions:

• Does pretty much every course and other training intervention get evaluated at each of the first three levels of the Kirkpatrick model?
• For each intervention, does the data collected amount to at least 10 variables & comments per participant?
• When an intervention is not evaluated, is that because of a conscious decision (rather than, say, an oversight)?
• Can recent evaluation data be meaningfully compared to data for the same or similar interventions over the last 12 months (or more)? Even between departments?
• Do learners’ line managers contribute to evaluations of business impact?
• Does senior management receive regular reports of evaluation results?

If the answers are mostly Yes, that’s a fairly strong indication that the organisation is providing at this level.

Reference material relevant to this strand and level

Need it meets	Title	Words	Link

Suggestion? Please leave a comment below

Examples of provision at this level

Here are some examples of evidence used by broadcasters, in previous years, to demonstrate provision at this level:

• Documentation of evaluation methods and results
• Staff/employees are able to articulate how job has been affected
• Business impact of learning and development discussed in team meetings
• Learners formally present to fellow workers the benefits of the training they have taken
• Individual staff members are able to describe evaluation activities
• Individuals are able to articulate impact of training on own development
• Training Event Evaluation Form.

The wording is broadcasters’ own, and does not necessarily follow our usage preferences

What it takes to get to the next level

It’s quite a reach from here to the next level, and the first thing you need is senior management buy-in to the effort and short-term disruption.

Assuming you get that, here are some steps you can take:

• Work with those responsible for business planning processes and for the organisation’s management information systems (or MIS or management accounts or whatever it’s called in your organisation) to learn how they approach their work… and how they can help you
• Pilot a Results evaluation (ie Kirkpatrick 4) on an important and frequent training intervention
• Once you can do that successfully, design an integrated process for Kirkpatrick 1-4 for that intervention. It’s likely to include all or most of:
  • modifying the performance review system to provide information on course quality, learning achieved, increase in job skills and impact upon business (including inputs from learners’ line managers)
  • holding structured discussions with staff immediately following training and/or after a period of 2-3 months to capture thoughts on activities undertaken and resulting impact
  • gathering organisation-level information including: attendance statistics, success rates, overall feedback on tutors and content, matching of course objectives to the organisation’s key performance indicators
• Implement the new integrated process, again on a pilot basis. Ensure evaluation data is fed through – in a regular, orderly and standardised way – to those who can use it. Use it yourself when decisions affecting the subject matter used for the pilot need to be made
• Once you can do that successfully, extend the approach as quickly as possible to the remainder of your training interventions
• While the data is accumulating during the pilot, start developing new systems for making data-driven decisions (such as approving investments in training and selecting and developing trainers).

There’s going to be a frustrating period between implementing the new system organisation-wide and starting to reap the benefits, especially those that relate to staff performance reviews and to business planning processes. They often follow an annual cycle, and the real insights will only come once you have a full year’s data in the can.

But you should still be able to start making better-informed training decisions straight away.

Level D evaluation is another area where the experience needed to make the leap might not reside in the organisation, and will need to be brought in from outside.