BTSR guidebook ~ Training & skills self-evaluation

What provision at this level looks like and feels like

For the organisation

In 2008, 12% of broadcasters evaluated at this level.

At this level, all training investments are evaluated systematically. The evaluation uses a variety of approaches and measurements that together build into a clear overall picture of where training & development is working… and where it isn’t. This information is reported to senior management – who take an interest – and shared openly with line managers and others.

Evaluation pervades thinking about training. Questions such as “have we achieved the best possible results?”, “was that the most cost-effective approach?” and “how will we know whether this is a success?” are routine. Evaluation is not just a reflective activity: how did we do? It is built into the system from the start: what business impact are we seeking and how will we measure it?

As a result, the whole organisation’s approach to providing training – and employees’ approaches to receiving it – are positive, engaged and inquisitive. This in turn promotes innovative solutions and team cohesion.

For the training function

All this might seem intimidating but – assuming you know your business – it shouldn’t be. A trainer should welcome sophisticated evaluation in the same way that a top sprinter welcomes the stopwatch.

Benefits & risks of provision at this level

Benefits

In many ways, attaining Level D in this strand pays back better than in any other. Benefits include:

• Abundant relevant information enables the organisation to make smart decisions about training & development, and to monitor their effects rapidly – the flight deck is equipped with a full array of instruments
• Those smart decisions can be applied to every aspect of the organisation’s training provision (and thereby maximise provision in all strands of the self-evaluation)
• When reports are systematic, frequent and rich you can have confidence in the accuracy of the picture they paint
• Because training evaluations measure business impact (4^th level of the Kirkpatrick model), training is only judged a success if it has a positive impact on achieving business goals
• The very fact of being seen to measure outcomes makes success more likely – “what gets measured gets done”¹
• If you also measure RoI, investments in training can be appraised alongside alternative uses for the same capital.

Risks

There’s a flipside to “what gets measured gets done”¹: measuring the wrong things will send you hurtling way off course.

Processes & benchmarks for evaluating provision at this level

Distinguishing from «medium« provision

The differences from medium provision are mainly to do with the scope of evaluation and the degree of integration with other business systems. At this level:

• Evaluation extends to the 4^th level of the Kirkpatrick model, and maybe to selective RoI evaluations as well
• The evaluation of training is conducted through a system (rather than as a series of disconnected tasks or projects)
• That system is embedded into the systems for:
  • setting the organisation’s objectives and measuring whether they have been achieved
  • evaluating the performance of individuals and departments
  • identifying learning needs, and planning and approving training & development interventions.

Processes

Here are some diagnostic questions:

• Are evaluations of training & development presented in a standard, organisation-wide format?
• Are evaluations of training & development summarised and presented regularly to senior management, who take a close interest?
• Are unexpected evaluation results – be they positive, negative or mixed – acted upon by management?
• Before an investment in training & development is authorised, does the proposer have to articulate the business objectives that it will address? And how success will be measured?
• Is it the case that well-received training that produces measurable improvements in performance is nevertheless considered unsuccessful, unless it also achieves the intended business impact?
• Are employees’ assessments of the effectiveness of the training they receive taken into account in evaluating training? Their line managers’ assessments likewise?
• Do decisions about selecting trainers – and about any subject matter or training skills training they might require – take into account the results of recent evaluations?
• Do decisions about selecting external training providers take into account the results of recent evaluations?
• Do decisions about selecting training venues take into account the results of recent evaluations?

If the answers are mostly Yes, that’s a fairly strong indication that the organisation is providing at this level.

Reference material relevant to this strand and level

Examples of provision at this level

Here are some examples of evidence used by broadcasters, in previous years, to demonstrate provision at this level:

• Evaluation is systematic and links personal objectives to the organisation’s Key Performance Indicators
• Evaluation includes metrics on organisational impact
• Overall holistic evaluation of all factors feeds into L&D outputs and into organisational objectives
• Defined evaluation process applies to every training intervention. It evaluates courses, trainers and individual performance
• Changes in evaluation results feed into next year/courses/team/individuals
• Revisions to training plan/delivery as a result of evaluation are evidenced in course material or course design
• Staff articulate specific outputs/value for money as a result of training
• Regular team meetings discuss training & development and its impact
• Management information summaries include evaluation records.

The wording is broadcasters’ own, and does not necessarily follow our usage preferences

Case studies that illustrate high provision in this strand

• GMG Radio Holdings - Induction Programme
• KM Radio - Evaluation System for Training Provided to Staff