Designing Simulations

The observation of real work involving high-level competencies is not always a practical option. For example, it is difficult for assessors to directly observe candidates generating designs and ideas, analysing and solving problems, leading teams and developing long term plans. In other cases, the presence of assessors may be intrusive (such as in counseling).

Other critical competencies, such as responding to unforeseen problems and emergencies, are even harder to assess because they are seldom used. Most candidates who work in high-level roles are not closely supervised. Therefore, there is also a question of who can know enough about what they do in order to assess them.

For institutional RTOs, the observation of candidates performing real work is even more problematic since many of their students have no access to a workplace.

Despite the practical difficulties involved, there are some circumstances where it is highly desirable for assessors to observe key aspects of high-level performance. To address this need, the following case study is used to illustrate the design of effective simulations.

The case study is workplace-based and involves assessing personnel who manage responses to major events that seldom occur, or where it is unsafe or too costly to conduct assessment in the real work environment. It describes how to design and write a simulation for the Unit of Competency PUASES006A: Manage emergency operations, from the Public Safety Training Package. It should be noted that while emergency management in the mainstream public safety sector has been chosen as an example, most managers and team leaders may be called upon to manage serious and unforeseen emergencies. For example:

• A retail store manager may have to evacuate customers and staff in the event of a fire
• A supervisor in a warehouse may have to implement emergency procedures after a container of dangerous chemicals has been ruptured by a forklift.

The use of simulations in high-risk situations arises out of the reluctance of assessors to rely on other forms of past and present evidence (such as third party reports, interviews and workplace documents). The simulations are designed to predict the ability of candidates to apply critical skills in pressure situations and in a real workplace environment. Prior to assessing their overall performance, assessors should be satisfied that candidates already possess the skills and knowledge to perform the individual tasks involved.

Simulating emergency management

Effective emergency management does not simply rely on the application of technical expertise and well–defined operating procedures. It also requires candidates to demonstrate non–technical skills, such as efficient communication and decision-making, sometimes under considerable pressure. Without adequate development of these non–technical skills, problems can arise during crisis management such as:

• Poor appreciation of an emerging crisis
• Inadequate situation analysis
• Weak leadership
• Faulty decision making
• Blind adherence to procedures
• Role ambiguity
• Lack of explicit coordination and communication.

Assessment strategies must be focused towards confirming these skills as well as an all round capability to deal effectively with emergency situations. Emergency management requires professionals who are able to combine a range of technical, human and logistical skills to ensure appropriate responses before, during and after an emergency occurs.

A simulation exercise provides a controlled opportunity to observe an emergency management performance. It also enables assessors to confirm the ability of candidates to respond to unfamiliar and unforeseen contingencies and manage emergency responses without close supervision.

The skill of assessors is crucial to the success of conducting large-scale simulations. The general difficulty of observing management roles together with the issue of subjectivity when assessing candidate responses to problems can lead to unreliability in assessment. One way to increase reliability is to employ subject matter experts to assist with the design and conduct of the exercise. Another way is to design tools that provide a consistent basis for professional judgment of competence.

What is a simulation?

Exercises, or simulations, fall into three categories.

1. Hypotheticals – where a facilitator asks participants a series of ‘what if….?’ questions as a way of eliciting ideas or developing a novel solution to a problem.
2. Table top exercises – where the participants work together to consider a problem and produce one solution (from a group of known solutions). While all the decision makers can be involved, the artificial format makes it difficult to represent what would happen in the real world.
3. Simulated or ‘live’ exercises – where the participants respond to a scenario using the resources provided. These exercises can be highly sophisticated, as in the case of war fighting, and involve the deployment of personnel and equipment. They may also involve the development of sophisticated computer programs. These large-scale exercises are prohibitively expensive for enterprises outside aviation and the defence, fire and police services. However, simulations can be successfully designed on a much more modest scale.

Next we provide information on how to design a large-scale exercise and relevant assessment tools to collect the evidence produced. A small-scale exercise will involve exactly the same approach. The challenge for assessors in all simulations is to design and document a scenario that is realistic in terms of the activities and resources required.

Managing a simulated flood event

The following case study describes the assessment of senior State Emergency Service (SES) personnel during the management of an emergency operation.

The relevant Unit of Competency is PUASES006A: Manage emergency operations.

The unit applies to the following personnel:

SES agencies throughout Australia are typically responsible for the combat of natural hazards such as floods and storms. They may also provide response teams for a range of other emergency activities, such as search and rescue exercises. SES agencies are highly decentralised with the establishment of units reflecting local government areas and the locations of greatest threat. It is common for the local Operations Controllers and all of their team members to be volunteers. The high incidence of volunteer members poses special problems for training, including issues relating to their motivation and availability.

The Unit of Competency PUASES006A covers the management of emergency operations, usually within a single agency command. The information within the unit provides valuable advice on relevant sources of evidence for assessment. A sample of this information has been extracted in the following table.

Identifying sources of evidence for the unit PUASES006A (only elements 1 and 2 are considered for this example)

 The Table above highlights the importance of simulated exercises in providing evidence of competence for this unit.

Writing a simulation

A typical simulation has five elements.

1. Creating a scenario. This is called the general idea. This should set the scope and context for the exercise and outline the types of resources that are available to the team or individual.
2. Designing additional information that participants must react to and complicated problems that they need to solve. These are called the special ideas. They must be realistic, but not ‘over the top’. They should also have a logical sequence, mirror the likely occurrence of various unforeseen problems and be consistent with the overall time constraints of the exercise.
3. Providing instructions for the exercise manager(s). Their role is to:
4. rotate candidates through OC and OCM roles at simulated change of shifts
5. monitor the decisions that candidates make as the exercise unfolds
6. second guess consequences for the actions and decisions that candidates take and modify the exercise to suit
7. decide whether to intervene to keep the exercise safe
8. decide how much pressure candidates can take and whether to push them beyond their comfort zone
9. decide whether to rescue candidates (as in the real world) to prevent catastrophic failure and to keep the exercise flowing
10. decide when to introduce special ideas to increase complexity and pressure
11. role play interactions with the candidate(s) to draw out their reasoning behind key decisions and actions.

Decisions will also need to be made on the duration of the exercise, the location, the number of candidates and whether they will rotate through various roles. The manager must know what resources are available. For example, will all information be provided in print or can the exercise manager contact the group via the phone, fax or email to increase realism? The information could also be assembled in a kit.

1. Providing the assessors with tools that they can use to record the performance of candidates during the exercise. Depending on the budget, there may be more than one assessor if there are six or more candidates. In the SES case, a District Controller or the State Training Manager will be used as the second assessor to enhance quality assurance. Assessors could be allowed to ask questions during the simulation, although questioning may distract the candidates. In any event, there should be a ‘post mortem’ of the exercise and questions relating to the process can be left to this debrief.
2. Providing the participants with the information they need.

Case study: The flooding of the Retnuh Valley

 The following special ideas will be introduced during the exercise.

Preparing assessment tools for the simulation

The design of an effective simulation should include preparation of relevant assessment tools to record the evidence that is gathered during the exercise.

As the simulation is intended to be a practical demonstration of a management task

The following table has been adapted to provide a checklist for assessors to use as they observe the simulation.