Department of Information Communication Technology

Unit of Competence:
Plan and Monitor System Pilot
Module Title:
Planning and Monitoring System Pilot
Module Code: ICT ITM5 M04 1115

Prepared by

Getachew G/Hana (Instructor, Msc)

1
LO1 Prepare for pilot system

1. Procedures and techniques in conducting pilot system

Pilotmeans:-A system for evaluating new procedures for handling data in which a sample that is
representative of the data to be handled is processed.

A pilot is a way of testing a theoretical model on a small scale level, in order to discover potential
problems that otherwise would not be detected until full-scale deployment. If these potential problems
are not detected on time, it can cost a lot of money and time to introduce changes once the solution has
been deployed to more schools. Pilots’ are very different from the first stage of a progressive
deployment and demand a specific type of planning.

'Piloting' of an ICT project is defined as the implementation of an ICT technology, software, or

related project on a small controlled scale to allow for its full impact, benefits and weaknesses to be
evaluated before implementation on a regional or nationwide basis.

The primary purposes of a pilot are to demonstrate that your design works in the production
environment as you expected and that it meets your organization’s business requirements. Conducting
a pilot reduces your organization’s risk of encountering problems during full-scale deployment. To
further minimize your risk during deployment, you might want to conduct multiple pilots consisting of
separate pilots for different technologies or operating systems, or you might want to conduct a full-
scale pilot in phases.
Depending on the context of the project, the pilot release can take the following forms:
 In a general business enterprise, a pilot can be conducted with a group of users on a set of servers
in a datacenter.

 In a Web development enterprise, a pilot can be conducted by hosting site files on a staging
server or servers or in folders that are live on the Internet, using a test Web address.
 Commercial software vendors, such as Microsoft, often conduct a pilot with a special group of
early adopters.
2. Why are pilots important?

 Before investing in a large-scale project, testing its assumptions on a smaller scale can
leave us better equipped to plan and execute for the larger scale deployment.
2
  We can reduce the risk of propagating mistakes by detecting errors at the pilot stage.
 Pilots can be used to assess the impact of the technology on the schools, the people, the
community, and whether equipment is used effectively by students and teachers, etc.
 It is easier to secure funding for a pilot than a large-scale deployment.
 The project team members can gain more experience before engaging in a more
demanding project.
 And also used to compare two or more similar solutions in order to find out which one
works best in the field.

Why Did We Conduct a Pilot Study?

Piloting reduces the risk of rolling out a flawed process, procedure or other solution component to
broad multi project environments.
The idea behind a pilot is to testthe solution component within a bounded and controlled environment
before the component is sanctioned for broader use.
During a pilot study, the usability of the solution component is evaluated in a near real-world project
setting.
Experience demonstrates that such a test always exposes improvement opportunities that can be
exploited to hone and refine the solution component before broader dissemination.

Defining the Pilot Scope and Objectives

The first step in planning your pilot is to define what you want to accomplish (the objectives or goals)
and what you plan to include and exclude (the scope). Be sure to align the pilot objectives and scope
with those for the deployment project as a whole, as defined in the master project plan. Ensure that the
pilot plan includes an opportunity for the team to evaluate features identified in the project plan as
high priority, to ascertain that they successfully meet all of your business objectives.
If you plan to conduct multiple pilots, define the scope and objectives of each by using these
guidelines.

Defining Pilot Objectives

Explicitly state the objectives of the pilot. Use the objectives to identify criteria for measuring the
success of your pilot.
Many organizations have primary pilot objectives, such as:

 Ensure that the system works properly in the business environment.

3
  Ensure that the design meets the business requirements.
 Build user support for the Windows Server 2003 deployment project.
Many organizations have additional objectives, such as:

 Test the deployment process.

 Train the installation team.
 Create documentation for the full deployment.
 Train the support and Help desk teams.
 Train the administrative teams.
 Gather information for estimating future support requirements.
 Gather information for estimating actual hardware requirements.
Pilot Types
The Project Evaluation Guidelines will focus in providing support for conducting the following three
pilot types:
 Proof-of-Concept Pilot:
Artificial setup and not production ready. Mainly for learning and feasibility study on one or
more issues on one of the EIF 2.0 interoperability levels.
 Test Pilot:
Artificial setup, but in principle production ready. Mainly for addressing issues on several EIF
2.0 interoperability levels.
 Production Pilot:
Real life set up, but not rolled out (into a production environment). Mainly for addressing
issues on all EIF 2.0 interoperability levels.
Pilot types, details
1. Proof-of-Concept Pilot
 Narrow scope - only one part of an interoperability level in EIF 2.0 is addressed
(technical, semantic, process).
 Artificial setup - there’s no impact on organizations and its production system when
running the interoperability model.
 Still old processes in production
 Interchange of information is one to one
 Small number of participants
 No focus on quality
 Short lifetime of pilot (less than one month)
4
2. Test Pilot
 Medium scope - Not all interoperability levels in EIF 2.0 is addressed (technical,
semantic, process)
 Artificial setup there’s no impact on organizations and its production systems when
running the interoperability model.
 Still old processes in production
 Interchange of information is one-to-many (or many to one)
 Small to medium number of participants
 High focus on quality (testing)
 Medium lifetime of pilot (1-3 months)
3. Production Pilot
 Large scope - All interoperability levels in EIF is addressed (technical, semantic,
process, organizational, legal)
 Real life setup - impact on organizations and its production systems when running the
interoperability model.
 No parallel (old) production processes
 Interchange of information is many to many
 Medium to large number of participants
 High focus on quality (testing)
 Long lifetime (3-12 months)

3. Develop Pilot Study Success Criteria

Prerequisites for a successful pilot

- Define and document all key applications
- Establish clear cut performance objectives and evaluation criteria
- Perform detailed records analysis
- Review existing retention schedules and amend, as appropriate
- Develop data requirements from required outputs and from source materials
- Consider security concerns and access permissions/restrictions
- Perform prototype work sessions with package before customizing
- Finalize system design
- Test the efficacy of the enterprise-wide taxonomy
- Complete conversion specifications and procedures
5
 - Complete preliminary user documentation
- Develop quality acceptance methodology for conversion data
- Develop work package for critical implementation tasks
- Perform risk analysis
- Build sufficient time into schedule to fully test the system
- Test everything

You will want to know:

Did the solution component generate the outcome that it was intended to achieve?
• What are you hoping for in terms of performance change when using the solution component?
• Try to define performance standards that will help you determine this explicitly. Are there any
historical data that can be used to baseline the status quo?
Did the users experience difficulty in its use?
• What are your expectations in terms of the solution component’s impact on changing people’s
attitudes and behaviors?
“Success” of a pilot can be defined as the smooth running of the pilot as planned in an adequately
selected sample of schools/students, producing results that can be trusted for the variables/indicators
selected in the Monitoring Plan. Some of the things that can be done to ensure that the pilot runs
smoothly and yields valid results are:
- Carefully select the sample schools, classes and/or students in such a way that it represents
different situations found in the field.
- The monitoring plan is in place and the indicators to be measured have to be properly defined.
- Do not disregard the importance of the human factor.
- Do not rush (move or act): give pilots the time that they deserve. Give pilots enough time.
- Involve the community.

Executing the pilot

Once the pilot planning document is ready, the resources are in place, and some candidate
schools/classes have been selected, it is time to actually run the pilot.
The execution will usually include these stages:
• Pre-analysis: final selection of the sample schools where the pilot will run. Analysis of the school
infrastructure that is in place and execution of required adaptations if necessary (i.e. buildings,
classroom infrastructure, specific furniture, electricity provision).
• Set-up: acquisition, transportation, installation and configuration of the equipment and/or software.
• Project presentation: several informative talks with people directly and indirectly involved like
teachers, headmasters, parents, community members and students about the objectives of the pilot,
how long it will last and how to record the experience, etc.
• User training: Training users to use the specific tools.
• Execution:
• End of pilot:

Pilot projects are valuable at three stages during implementation.

6
  During the earliest stages (Assessment or Pre-Assessment), to deliver proof of concept. At this
stage you are not answering “How does KM work for us”, rather seeking to answer “Would
KM work for us at all?”
 During the Selection phase, in order to test specific KM tools and techniques, to answer the
question “Would this KM tool or process form part of our KM framework? If so, what
modification would it need?”
 In order to validate and perfect the KM framework. At this stage you are seeking to ask the
question “What changes do we need to make to our KM framework before finalising it for roll-
out?”

The four main purposes of running pilot projects are;

 To act as a learning field, where you can try knowledge management tools processes and
techniques, finalise the knowledge management model, and answer the questions above
 To act as a seed for the knowledge management culture and behaviours, which can then spread
into neighbouring areas of the business, and
 To deliver demonstrable value to the organisation, and so to prove the in-house business value
of knowledge management, and
 To create stories for internal marketing purposes

Planning system pilot

During the planning phase of the deployment project, product management, program management, and
release management teams collaborate to create the pilot plan. The pilot plan defines the scope and
objectives of the pilot and identifies pilot participants and where the pilot will be conducted. It
includes a schedule for deploying and conducting the pilot and plans for training and communicating
with pilot participants, evaluating the pilot, identifying risks and contingencies, and other key
activities that occur during a pilot deployment.

When the pilot plan is ready for review, have project team members, necessary support personnel, and
management representatives read and approve the plan. Be sure that the supervisors of everyone
directly affected by the pilot have a chance to review the plan. For example, if the schedule allots time
for a particular user group to participate in the pilot, have the supervisor of that group review the
schedule.

Defining the Pilot Scope and Objectives

The first step in planning your pilot is to define what you want to accomplish (the objectives, or goals)
and what you plan to include and exclude (the scope). Be sure to align the pilot objectives and scope
with those for the deployment project as a whole, as defined in the master project plan. Ensure that the
pilot plan includes an opportunity for the team to evaluate features identified in the project plan as
high priority, to ascertain that they successfully meet all of your business objectives.

Defining Pilot Objectives

Explicitly state the objectives of the pilot. Use the objectives to identify criteria for measuring the
success of your pilot. Many organizations have primary pilot objectives, such as:

7
  Ensure that the system works properly in the business environment.

 Ensure that the design meets the business requirements.

 Build user support for the Windows Server 2003 deployment project.

Many organizations have additional objectives, such as:

 Test the deployment process.

 Train the installation team.

 Create documentation for the full deployment.

 Train the support and Help desk teams.

 Train the administrative teams.

 Gather information for estimating future support requirements.

 Gather information for estimating actual hardware requirements.

 Meet the baseline requirements for functionality that were established in testing.

 Develop and test end-user training materials.

You might need to define objectives that are related to specific technologies, such as Active Directory,
or to your network infrastructure. If you plan to conduct multiple pilots, define the objectives for each.

Defining Pilot Scope

Define the scope of the pilot by clearly stating which services and features will be included and which
will not. When you list the services and features you plan to include in the pilot, also state how you
expect them to perform. Describe the areas of functionality that the pilot implementation affects, and
note to what extent they are affected, and in which situations they are affected.

Do not expect to test every feature or service during the pilot. Focus on processes that present the
greatest risk and events that are most likely to occur. Prioritizing the features to be tested in the pilot is
particularly important if your team plans to conduct multiple pilots. When multiple pilots are planned,
start small and gradually increase the scope of successive pilots.

If certain aspects of your design cannot be covered by the pilot, describe them. For example, if your
organization plans to upgrade your domain using your existing architecture without any restructuring,
and then restructure the architecture later, you might choose to exclude the restructuring process from
the first pilot.

Also specify the duration of the pilot, in terms of either time or of the criteria to be met.

Be sure to describe how you expect to proceed after the pilot is complete. If you plan to keep some
functions in place and remove others for the full production rollout, identify the features that will be
8
removed. For example, if you are redesigning your namespace, you might want the option of changing
it after the pilot concludes. Indicate how to back out features that will be removed.

Pilot selection
A pilot project should be selected around an area of business need. The business need leads the way;
the KM pilot provides one or more possible solutions which can be tested. These are some of the areas
where you might consider suggesting a knowledge management pilot.
 If there is the business critical activity that is new to the organization, then rapid learning will
deliver business benefits. If it is new to only one part of the organization, then transferring
learning from where it has been done before, will give huge benefits.
 If there is repetitive activity, where continuous improvement is needed, then knowledge
management can help drive down the learning curve
 If there is activity that is carried out in several locations, where performance level varies, then
knowledge management can help exchange knowledge from the good performers, to improve
the poor performers.
 Finally if there is an area of the business which is stuck due to lack of knowledge, then
knowledge management can help develop the knowledge needed to get unstuck.

Pilot Ranking
If you have a selection of pilot areas, you need some way of ranking these opportunities so that you
select the best one. It is very unlikely that you will be constrained by a lack of opportunity, it is far
more likely that you will be constrained by a lack of resources to deliver the opportunities. So you
need some form of ranking criteria.
These are some suggested criteria which we often use;
o if the project is successful, can we measure the difference, or the value generated?
o Is there is strong support from management?
o If we create knowledge, is it purely for the pilot team or can others use it?
o Finally, can we practically do it in the timeframe with the resources available?

You can imagine that, if you can answer yes to although these questions, you have chosen a good pilot
project.

Organization of the Pilot Project

The pilot project should be owned by the business, with KM providing support rather than leadership.
There are three main roles in the pilot project
1. A business sponsor, who provides resources and agrees the goals

2. A leader for the pilot project, who should be someone from the relevant business unit or function

3. A KM professional, to provide support, KM expertise, and facilitation

The pilot project itself

Once you have decided on a pilot opportunity, then you need to think about the different phases of the
pilot project.
1. Initially you need to raise awareness in the target area, and may need to do some “selling” of the
concept to get people on board.

9
2. Then you need to scope the project to determine what time and resources are needed,

3. You need to tailor a local knowledge management system (a combination of roles, technologies,
processes, activities and governance) that will fit the working patterns of the project team,

4. You need to embed the knowledge management processes and activities into the process of the
business,

5. And then you need to measure and deliver the results.

Creating the story

If the pilot project has been a success, then the exchange of knowledge will have led to a reduction in
cost, a reduction in time, an increase in profitability, or some other business measure. This gives you a
success story, which you can use for further marketing of the benefits of change throughout the
organization. This story will be best received, if it is told in the words of the business people involved.
Ask the business manager, or one of the key players on the pilot project team, to define these benefits.
If you can get a quote from them, or a video from them, this will be very powerful for you.

How to collect and measure data

As we mentioned before it is critical to design a proper Monitoring and Evaluation Plan and use the
pilot for the collection of relevant data that will allow for the verification and validation of the original
ideas proposed by the project.
Below various collection methods are outlined:
- Automatic: it is possible to have automatic tools that will collect some numeric data like number of
hours a system was used, exactly when and by whom. This way data does not have to be collected
from users directly.
- External: One or several observers from the team can be present during teaching hours or at the
students’ homes. This can be done with or without interfering with the normal teaching process.
- Subjective: Participants can document their impressions, experiences and ideas through several tools
like journals or diaries, questionnaires, tables or forms.
- Environmental: Data can also be gathered by questioning teachers, students, headmasters and also
the parents and the community about the pilot results and their perceptions.

Outline of the structure of a Pilot Planning document

- Projects name
- Introduction
-Project History: how this Pilot originated
- Project’s objective/s: Normally educational objectives and mention of how the
project specific objectives will support them.
- Scope: number of institutions, location, classes involved, number of teachers and
students involved.
- Planning
- Detailed description of the IT solution characteristics
- Detailed deployment plan
- Team members profiles, roles and responsibilities
- Budget and funding sources
- Timeline (preferably using some planning technique like GANTT)
- Execution
10
- Installation procedure and standards
- Support and maintenance, replacement plan
- Implementation details
- Training Plan
- Logistics
- Monitoring and Evaluation Plan

Activities related to pilot projects can be divided into three distinct phases:

Preliminary -

 Define the purpose, goals, objectives, and scope of the pilot/proof of concept demonstration
project

 Establish the success criteria for the pilot, with input from all stakeholders, technical staff,
records management staff, and users

 Outline the benefits of conducting a pilot and risks of not doing so

 Establish an administrative infrastructure to support and guide pilot project activities.

Conduct of the pilot -

 Determine whether preliminary decisions and assumptions made regarding hardware and
software performance, as well as service level required by technical staff, were accurate

 Develop and use tools facilitating documentation, communication/knowledge transfer, and

metadata processes.

Test and evaluation -

 Assess hardware and software, system and database design, and procedures (for training,
scheduling, system management, and maintenance)

 Test product(s) in dissimilar locations (e.g., in terms of RM and IT support delivery) for
functionality, usability, and benefits derived from using ERM

 Validate requirements and performance expectations.

Criteria for selecting a pilot area

 Mission critical functions with strong leadership

 Range of job types, differing degrees/level of comfort using
computers, and both active and infrequent users of records (of
the type included in the project) in routine daily tasks
 Level of files and types of records, representative of all formats
that will require that all applications be run/thoroughly tested
 Reasonably large data base to adequately test the ability of the
system to manage the records management functions required
11
 by the agency
 Moderate activity, so as not to overload the system during this
testing phase
 Headquarter and other office locations (with differing
mechanisms/approaches for RM and technical support)
introduced to the pilot in a phased approach

Minimizing risks associated with pilot projects

An agency should acknowledge that some prudent(acting) risk-taking is necessary when it comes to
adopting new technology and changing business processes. To minimize the risks associated with a
pilot launch, the project team should:

 Establish clear performance objectives and evaluation criteria

 Involve and continually encourage pilot project participants to use the system

 Perform prototype work sessions with the software before customizing it

 Finalize system design

 Develop quality acceptance methodology

 Expand the pilot through incremental rollout to other areas of the agency and inclusion of other
record formats

 Assure that pilot's requirements are measurable and clearly understood by participants.

Conducting the Pilot

Certain critical decisions need to be made and documented before the pilot begins. This can only be
accomplished by reviewing similar projects, determining whether any additional data is required
before proceeding, and considering which performance data need to be collected through the pilot to
enable meaningful evaluation.

Specific elements necessary for the conduct of a pilot project include:

 A pilot monitoring system that consists of service level requirements for the pilot (e.g., data
load, update, refresh) and a problem log to note any disruptions in service that occur during the
conduct of the pilot that includes what was done to address each situation.

 A determination as to whether significant changes to the agency IT infrastructure will be

required to execute the pilot, including the acquisition and installation of new hardware or
modifications to the network.

 Availability of knowledge application developers (programmers) and system analysts to deal

with the project. Once you have assessed the capacity of technical support staff to monitor

12
 performance and troubleshoot during the pilot project, you will know how much outside
support you will need

 Availability of analysts to identify and test potential business process improvements and
measure their impact on the agency as well as budget analysts to accurately assess pilot costs
and adjust predicted estimates for full-scale implementation.

 Tools facilitating documentation, communication/knowledge transfer, and metadata processes

(and automated categorization) should be established for your pilot

 Training is essential for all involved in the pilot project. You may need to reinforce agency
staff's understanding of basic records management by:

Evaluating the Pilot

Evaluation is perhaps the most important part of the pilot project. A carefully constructed pilot project
will make provision for objective analysis of the results and an assessment as to how to proceed with
full deployment. The evaluation team for one agency pilot project identified five categories of
performance measures:

 Installation: Time to install on the network, test, and install on user workstations

 Training: Ready availability of training; keeping users well informed about training
opportunities; providing assistance in registering for training; conducting well-organized and
understandable training sessions; follow-up after training

 Usage: Streamlined procedures and the use of templates; meetings to increase comfort levels of
users and to develop work-specific file plans; application effectiveness/user satisfaction;
privacy/security issues adequately addressed

 Knowledge: Increased level of knowledge of RM after pilot

 Communication: Sharing lessons learned beyond the pilot.

The mechanisms designed into the project to monitor the progress of the pilot will inform the
evaluation. These include:

 Communications/knowledge transfer mechanisms that you have set up for your pilot project,
serving as a source for valuable feedback necessary for adequate analysis.

 Questions posed to help desks.

 Minutes of telephone and Web-based conferences with pilot participants, as well as technical
team meetings, providing additional input for the evaluation.

A formal approach to quantitative and qualitative analysis of the pilot project should be built into the
pilot project plan. The methodologies employed can include a mix of surveys and interviews with
participants conducted periodically, including:

13
 An initial baseline analysis will help you to understand the concerns of participants, giving you
an opportunity to address them through pilot trainings and any communications mechanisms
you established for the pilot.

 Interim assessments can evaluate the effectiveness of particular aspects of the pilot (e.g.,
training workshops). These can gauge changes in usage of the system (increasingly frequent
usage with less time required per session) and user satisfaction (as the pilot team responds to
requests from participants to modify the system/procedures).

 A final evaluation that demonstrates the effects of the business process and indicates changes
to be made before the projec is deployed agency-wide.

14