BTSR guidebook ~ Training & skills self-evaluation

What provision at this level looks like and feels like

For the organisation

In 2008, 59% of broadcasters, evaluated at this level. 3% were lower and 38% were higher.

There are two main reasons for being at this level:

• The organisation doesn’t fully understand what evaluation entails¹ – or can entail – and has therefore marked itself down in the self-evaluation
• The organisation does fully understand what evaluation entails, and its provision properly belongs at this level.

In an organisation that belongs at this level, the evaluation that does take place might be sporadic, reactive, superficial, inconsistent or disconnected… or several or all of these. Attentive managers will probably wonder why they are getting data on some training interventions and not on others, and why the data sets are not comparable.

For employees, provision at this level will probably seem unremarkable. They get the message that the organisation cares about the training it provides – at least a little bit – and might feel slightly encouraged to speak up if they see something seriously awry. But mainly they won’t notice.

¹ During 2007/2008 validation visits, poor understanding applied to this strand more than to any other.

For the training function

How provision at this level feels for the trainer depends on why evaluation is so minimal:

• If you are pushing to do more and meeting resistance, that’s an awkward and rather exposed position to be in; you need to understand why it’s happening and what that means for your – and the training function’s – prospects
• If others are pushing for more and you don’t know how to deliver, getting help should be a priority
• If everybody is happy with the status quo, see Level A.

Benefits & risks of provision at this level

Benefits

The data that the organisation chooses to collect presumably has some value.

Risks

The main risks of provision at this level include:

• Investments in training & development are ineffective, or even counter-productive
• Substandard courses, presenters and training venues fail to be identified
• Employees feel that training is being done to them, not for them
• The organisation avoids decisions about training & development because it’s unsure what’s happening – if you can only hear occasional notes, it’s hard to tell whether you are singing in tune
• The organisation makes faulty decisions based on extrapolating sparse, unrepresentative data – “can you see what it is yet?” is a risky game to play with organisations and their people
• Lack of data pulls down the quality of training provision in all strands of the self-evaluation, not just this one.

Processes & benchmarks for evaluating provision at this level

Distinguishing from »medium» provision

The differences between base and medium provision are mainly to do with how regularly and systematically training & development interventions are evaluated. At this level evaluation is some or all of:

• Sporadic – it happens when somebody remembers or has a bit of spare time
• Reactive – it happens when management starts questioning a particular training & development intervention
• Superficial – it doesn’t go much beyond “course OK then?”
• Inconsistent – different approaches are applied to different training interventions, or to the same interventions at different times
• Disconnected – there is no process for aggregating individual evaluations into a bigger picture.

Processes

Here are some diagnostic questions:

• Can a course, or other training intervention, go unevaluated?
• When evaluation does occur, is it mainly informal?
• Can similar training interventions be evaluated against different criteria, or with differing degrees of formality, thoroughness and regularity?
• Other than as a result of systematically improving evaluation practices, can the same training intervention be evaluated in different ways in different parts of the organisation? Or at different times in the same part of the organisation?
• Is an intervention more likely to be evaluated if management perceives it to be problematic?
• When an intervention is evaluated, does the data collected amount to fewer than 5 variables & comments per participant?
• Is it difficult or impossible to compare evaluation data from department to department? From period to period?
• Is it difficult or impossible to aggregate evaluation data from different training & development interventions into an organisation-wide picture?

If the answers are mostly Yes, that’s a fairly strong indication that the organisation is providing at this level (or below).

Reference material relevant to this strand and level

Need it meets	Title	Words	Link
Introduce a range of approaches to evaluating training	Evaluate your training	2,054	www.businesslink.gov.uk
Explain practical steps for applying the Kirkpatrick model	Kirkpatrick’s learning and training evaluation theory	2,058	www.businessballs.com

Examples of provision at this level

Here are some examples of evidence used by broadcasters, in previous years, to demonstrate provision at this level:

• Feedback to managers on training undertaken is oral only
• Discussions of the effectiveness of training between staff and managers are informal
• Staff can make comments on training effectiveness during their performance reviews
• Training evaluation forms exist
• Evaluations from a number of sources are aggregated in an ad hoc way.

The wording is broadcasters’ own, and does not necessarily follow our usage preferences

What it takes to get to the next level

Good news: you’re already doing most of what you need to get to Level C. You just need to do it more systematically and consistently.

While you’re working on this strand, it makes sense to keep Strand 1 firmly in view as well. Both strands relate to the organisation’s goals, and it might be possible to move them forward in tandem.

Probably the biggest stretch is going to be the 2^nd & 3^rd levels of the Kirkpatrick model:

• 2^nd level:   Learning – the change in knowledge & skills resulting from the training
  The classic approach to this is to pre-test and post-test. In principle this is dead easy: you know the knowledge and skills that the intervention is designed to address, so you just measure them before and after and note the change.
  
  In practice it can be tricky because it requires you to be very specific in the way you define the desired level of knowledge and skills. And precise in the way you test for the knowledge and skills. Good!
  
  
• 3^rd level:   Behaviour – the change in workplace performance
  The issues here are similar to the 2^nd level, except that you are now looking at what actually happens in the workplace rather than at what employees are capable of. A good understanding of this is important because it can flag up barriers to performance that stem not from employees’ abilities but from other factors (such as the reward & punishment system, or the tools/information/support provided to employees). A pre- and post-test approach is still relevant – because you still want to measure a change – but in this case you’ll involve the people who are accountable for employees’ performance: their line managers.
  
  Clearly it’s essential for trainers and line managers to reach an agreed definition of what success will look like before the training takes place. Again, good!