Program Evaluation Plan: Nursing Self-Leadership Program

Jennifer Bay
Walden University

Dr. Mitchell Pratt

Program Evaluation EIDT 6130
October 18, 2015

Table of Contents
Program Analysis .....................................................................................................................1
Evaluation Model Table ............................................................................................................6
Evaluative Criteria .....................................................................................................................8
Data Collection Design and Sampling ...................................................................................11
Reporting Strategy Table........................................................................................................15
Appendix A ..............................................................................................................................17
References ..............................................................................................................................18

Program Analysis
The Nursing Self-Leadership Program (NSLP) is a program offered at Kids Healthcare of
The City (Kids HTC), a not-for-profit pediatric health care system with three hospitals, multiple
community locations, and over 8,000 employees. The target audience for the program are Kids
HTCs bedside nurses with at least one year of nursing experience.
The idea of a NSLP began in 2009, when, during strategic planning sessions with nurses
at all levels of the organization, it was identified that even though Kids HTC provided leadership
development opportunities for those in supervisory or executive positions, there was a lack of
leadership development opportunities for the bedside nurse. Schools of nursing also do not
provide leadership development as part of their curriculum. A guiding team was assembled, and
based on their discussions on how to provide leadership development for the bedside nurse, a set
of desired leadership behaviors for the bedside nurse at Kids HTC was developed. The idea
became a reality when a donor specifically requested that funds be used for professional
development of the bedside nurse, which allowed Kids HTC to partner with a globally
recognized leadership development expert (Partner Company), who had content already
developed related to the desired leadership behaviors outlined by the guiding team. The NSLP
was trialed with a group of nursing managers, assistant managers, and educators in 2013, then
offered to the first cohort of bedside nurses in 2014; the second cohort is scheduled to complete
the program in two weeks. Due to several contextual factors related to the roll out of the
program, which will be described in a later paragraph, current perception of the NSLP is very
mixed, with current plans to complete one more cohort in 2015 and then to let the program fade
away.

Page | 1

The goal of the NSLP is to provide the bedside nurse with the self-leadership knowledge
and skills that correlate to the desired leadership behaviors, and to have the nurse apply this
knowledge and skills. The NSLP is a hybrid program consisting of seven asynchronous online
modules, which participants access through the Partner Companys website, and face-to-face
debrief sessions associated with each module, designed to discuss how the module content
applies to the bedside nurse. Originally, participants were given four weeks to complete each
module, followed by a two-week period to attend a debrief session, either facilitated by the unit
educator, or the centralized education department coordinator, which, in total, made the NSLP a
42-week program. The timeline and delivery of the program have been adjusted twice since the
initial rollout it is now a 22-week program, with participants completing lessons 1-3 within a 6
week period, followed by a 4 week debrief window for the lessons, then 8 weeks to complete
lessons 4-7, again with 4 weeks to complete the debrief. The NSLP was initially a requirement
for all bedside nurses with at least one year of experience, and in order to provide the program to
the 2000 nurses in the system while keeping budgets and staffing in mind, the program was
rolled out in cohorts. Originally, there were going to be a total of four cohorts, but this has since
been adjusted, and there will have been a total of three cohorts once the program finishes in
2016. To date, we have had two cohorts in the program.
A coordinator from the Centralized Education Department is responsible for coordinating
all aspects of the program, which includes, but is not limited to, working with the Partner
Company to ensure access to the PCs website, communicating with all participants throughout
the program, facilitating debrief sessions, and working with the technology department to resolve
any participant issues. To date, there have been 700 nurses in the program. The completion
percentage of the online module portion of the course is on track to be higher in cohort 2 than in
Page | 2

cohort 1. Since the program was first rolled out to the initial cohort of managers, ANMs, and
educators, several program changes have been made, based on feedback from the initial manager
cohort, the NSLP guiding team, and participants of cohort 1. Several of the changes have been
related to adjusting the course timeline, especially the time commitment required for the face-toface debriefs.
The bedside nurse is the target audience of the program, and should be considered the key
stakeholder. However, the program was initiated by Senior Nursing Leadership, with funds
provided by a donor, and both of these groups are therefore also key stakeholders. The donor will
want to be provided with updates on how the funds were utilized, as well as on the success of the
program. Senior Nursing Leadership is interested in program success if the nurses apply the
knowledge and skills from the program, and demonstrate the desired leadership behaviors, then
Senior Nursing Leadership can use program data to promote nursing at Kids HTC, for example.
Another stakeholder is the Partner Company (PC). Even though the PC has done business with
Kids HTC in the past, this is the first time the curriculum is being targeted to a nursing audience.
The PC is interested in program success not only to continue the partnership with Kids HTC, but
they can also use program data to showcase the program to other health care systems and nursing
groups. The Centralized Education Department is another key stakeholder group, as they are the
ones responsible for coordinating the program, but also responsible for incorporating key
concepts from the program into existing and future education offerings. Lastly, frontline nursing
leadership (manager, ANM, educator) is the stakeholder group most closely connected to the
target audience, as they are the ones who are able to observe directly if course participants are
applying the knowledge and skills from the program and are ultimately demonstrating the
desired leadership behaviors.
Page | 3

In the paragraph that gave a brief history of the program, it was mentioned that the
current perception of the program is very mixed, which is largely due to contextual factors within
the organization. One of the biggest issues with the program is that it did not get off on the right
foot, as several key stakeholders were not involved until the program was ready to be rolled out
to the initial frontline leadership group. Yes, there was a guiding team and pilot group, and yes,
they were nurses, but these groups did not really represent the bedside nurse. The guiding team
and pilot group nurses were nurses that already were in leadership positions, or already
demonstrated several of the key leadership behaviors. The unit educators and coordinators from
the Centralized Education Department were not involved in the planning of the program either.
Additionally, the Centralized Education Department was delegated the task of coordinating the
entire NSLP without having been involved in any of the planning they only became involved
when it was time to start working on the implementation of the program, and being the last to
become involved has produced lack of trust and lack of support for the program. Another issue is
that the program was rolled out as mandatory. Because the program requires the nurses to work
online on the modules, it is usually not possible to complete it during their work shift, which
means they are either having to stay over their shift, or work on the modules at home, which
causes work/life balance issues. There are also other competing education requirements for the
nurses time. All of this has affected the overall perception of the program at Kids HTC.
The biggest potential ethical challenge with an evaluation of this program is how the term
complete is defined when reporting completion status. There is a potential to play with the
numbers if the definition of complete is changed to make the program appear more successful
than it was. Is program completion considered 100% completion of all seven modules and
attendance to all required debrief sessions, or is it 80% of the modules and a portion of the
Page | 4

debrief sessions? If this definition is not agreed upon at the very beginning of planning, then it
can lead to an ethical issue for the evaluator, especially as it relates to program reporting.

Page | 5

Evaluation Model Table

Evaluation Model
EXPERTISE AND CONSUMER-ORIENTED
APPROACHES

PROGRAM-ORIENTED EVALUATION
APPROACHES

DECISION-ORIENTED EVALUATION
APPROACHES

Advantages
Approach focused on providing
information for the general public.
Used by various organizations and
industries.
Experts, such as those involved in
accreditation, have inside knowledge of
the program/process, as they are, or have
been, members of that specific field.
(Fitzpatrick, Sanders, & Worthen, 2010)
Easy to use.
Objectives-oriented approach combined
with use of a logic model useful in
determining if and how objectives of the
program were met. (Fitzpatrick, Sanders,
& Worthen, 2010)
Useful when goal of program is to make a
decision. (Fitzpatrick, et al., 2010)

Disadvantages
Those performing the evaluation may be
content experts, but are not experts in the
evaluation process itself (Fitzpatrick, et
al., 2010, p. 142).
Evaluation literature not currently focusing
on this approach. (Fitzpatrick, Sanders, &
Worthen, 2010)

There are several factors that may impact

a program, which may be missed if the
evaluator is only looking at the objectives.

This approach involves only managers or

those that are part of the decision-making
process.
This type of model does not take into
consideration social equity and equality
(Fitzpatrick, et al., p. 185).
Stakeholders may try to influence the
evaluation process potential for bias
(Fitzpatrick, et al., p. 224).
Credibility of results may be questioned by
certain stakeholder groups if they were not
involved in the process (Fitzpatrick, et
al.,2010)

PARTICIPANT-ORIENTED EVALUATION
APPROACHES

Places emphasis on input from

stakeholders throughout evaluation
process. Stakeholders have knowledge
that we do not (Fitzpatrick, et al., p. 223).
Stakeholder involvement increases the
stakeholders understanding of the
evaluation process, and provides the
evaluator with an increased understanding
of the program being evaluated.

Page | 6

Explain your choice of model for your program evaluation: The goal of the NSLP is to provide the bedside nurse with the self-leadership
knowledge and skills that correlate to the desired leadership behaviors, and to have the nurse apply this knowledge and skills (Bay, 2015). In
order to measure if the goal of the program has been met, the evaluation will need to focus on the program objectives, but will also require
stakeholder involvement throughout the evaluation process. To achieve this, a combination of program-oriented and participant-oriented
evaluation approaches will be utilized for the NSLP.

Page | 7

Evaluative Criteria
As previously stated in the program analysis of the nurse self-leadership program
(NSLP), the goal of the NSLP is to provide the bedside nurse with the self-leadership
knowledge and skills that correlate to the desired leadership behaviors, and to have the nurse
apply this knowledge and skills (Bay, 2015). This program involves various stakeholders, from
the donor that provided funds, to the senior nursing leadership team, who all have an interest, not
only in whether the overall program goal has been achieved, but also how it was achieved.
Conducting a program evaluation will provide the stakeholders with the data that will show if,
and to what extent, the program goal was achieved, and with data on the program objectives, all
of which can be used to make decisions about the program.
The evaluation model for the NSLP is a mix of the participant-oriented approach and
program-oriented approach, using a logic model. Based on the logic model developed for the
NSLP, I propose five evaluation questions for the NSLP. The questions, along with the rationale
for each, are listed below.
1. What is the participants perception of the support provided to participate in the NSLP?
Rationale: The program requires participants to complete online modules, which require
time to enable focus on the content as well as face-to-face debrief sessions. The
program is therefore difficult to complete alongside the participants shift work and
patient care. Indicators used to measure this question can provide information related to
staffing time and staffing budget related to the program (logic model inputs).
2. What is the participants satisfaction with the delivery method used for the program?
Rationale: This is the first time the organization has used a hybrid (online content
Page | 8

modules paired with face-to-face content debrief sessions) course delivery method with
the nursing staff. Indicators used to measure this question can provide information
related to the program objectives, as well as if the delivery method chosen for the
program is effective (Logic model outputs).
3. Was there an increase in the participants self-knowledge and communication skills?
Rationale: This question is directly related to the two major program objectives (logic
model outcomes/short).
4. Did the participants apply the program knowledge and skills at the bedside? Rationale:
This questions ties directly to the overall goal of the program, and indicators used to
measure this question can provide information from the participants themselves, as well
as from their colleagues and unit leadership, including any barriers (logic model outcomes/medium).
5. To what extent did the participants demonstrate the desired leadership behaviors of the
bedside nurse? Rationale: This question is linked to question 4, and is intended to
answer if the overall program purpose of the program was met. Being able to apply the
knowledge and skills should lead to the desired leadership behaviors (logic model
outcomes/medium).
The NSLP content was developed by the Partner Company, and therefore the content is
proprietary to the Partner Company. Because of this, components related to/ built-in to the online
module platform, such as module sequence, and learning activities within the modules, will not
be evaluated, as Kids HTC does not intend to make changes to these items.
As previously stated, the NSLP evaluation method combines program-oriented and
participant-oriented approaches. The participant oriented-approach relies on the various NSLP
Page | 9

stakeholders to be involved throughout the evaluation process, including determining the

evaluation questions and criteria. According to Fitzpatrick, et al., we should be sure to consider
all stakeholders and should listen to what each group and individual has to say even during
informal conservations (Fitzpatrick, Sanders, & Worthen, 2010, p. 322). Senior Nursing
Leadership, unit leadership, and the centralized education department are the stakeholders that
will need to provide input to all of the evaluation questions and evaluation criteria. Senior
Nursing Leadership are the ones sponsoring the program, so they need to be included; the
centralized education department and unit leadership need to be involved because they are
directly involved in implementing the program, and/or directly observing if the skills and
knowledge are applied. The bedside nurse, the target audience of the program, needs to be
involved in the planning stages, to get their input and perceptions of the program. The Partner
Company will need to be involved to provide input related to the program content, and the
foundation/donor will also be involved in the planning stage.
The program evaluation standards reflected in the proposed evaluation questions are as
follows: attention to stakeholders (U2); relevant information (U5); practical procedures (F2);
responsive and inclusive orientation (P1); transparency and disclosure (P5); valid information
(A2); and sound designs and analysis (A6) (Fitzpatrick, Sanders, & Worthen, 2010, Appendix A,
pp. 499-500).

Page | 10

Data Collection Design and Sampling Strategy

The program evaluation of the Nursing Self-Leadership Program (NSLP) will utilize a
combination of a participant-oriented approach and program-oriented approach. The rationale
for the mixed-method approach is that the evaluation will need to focus on the program
objectives, and have stakeholder involvement, in order to measure if the goal of the program has
been met.
As part of the program-oriented approach, a logic model for the NSLP was developed,
which was used as a guide to develop the program evaluation questions, and identify the
indicators that assist in answering the evaluation questions. Therefore, the type of information
that needs to be collected stems from the indicators identified for each of the evaluation
questions. The following is a description of the type of information needed, the choice of data
collection methods, and how the data collection method will be administered for each of the five
evaluation questions:
Question 1: What is the participants perception of the support provided to participate in
the NSLP? In the logic model, this question is associated with the inputs of the program,
and the indicators used to measure this question provide information related to staff and
time. Information data sources to answer this question includes participant input,
obtained through an online survey, and user time logs, obtained through a review of
records of the Partner Company.
Question 2: What is the participants satisfaction with the delivery method used for the
program? This question is associated with the program outputs, and the indicators will
help answer questions related to program delivery. The primary data source for this
Page | 11

question will be input from the participant, obtained through an online survey. Data on
participant use of technical support will also be collected through a review of records of
the in-house technical support team, and from the Partner Companys records.
Question 3: Was there an increase in the participants self-knowledge and
communication skills? This question relates to the program objectives and is depicted as
a short-term outcome on the logic model. To answer this question, data will be collected
from the program participants through an online survey.
Question 4: Did the participants apply the program knowledge and skills at the bedside?
This question, depicted as a medium-term outcome on the logic model, ties into the
overall goal of the program, and requires data to be collected from the participants and
unit leadership. Input from the participants will be collected through an online selfassessment survey. Unit leadership will be provided with an observation questionnaire,
which will be distributed via email, and collected in person by the evaluator. Data from
the participants annual review, which include competencies related to the desired
leadership behaviors, will be obtained through a review of records.
Question 5: To what extent did the participants demonstrate the desired leadership
behaviors of the bedside nurse? This question, a medium-term outcome on the logic
model, is directly related to question four. The data obtained from the unit leadership in
question 4 will be utilized to answer this question.
The evaluation design focuses on obtaining information from two of the stakeholder
groups, the program participants and unit leadership, but the overall purpose of the data
collection is to answer the evaluation questions, which takes into account the interests of all
Page | 12

stakeholder groups. The Partner Company and the Centralized Education Department will be
interested in data related to the delivery method of the program, as it will provide information
that can be used for program improvements. Data collected for questions four and five, which
relate to the overall goal of the program, takes into account the interests of all of the stakeholder
groups. For example, if based on the evaluation, the NSLP is judged to be a program that does
increase the participants self-knowledge and communication skills, it provides the donor with
validation that the funds provided were well invested. For Senior and Unit Leadership, it
provides information that can be used to further promote the program.
The possible limitations involved in the data collection design for the NSLP includes
participant availability, limited resources, and potential bias related to the program. Data on the
application of the program knowledge and skills at work will be obtained through observation,
which depends on participant availability and resources - the unit leader needs to take the time to
complete the observation questionnaire for each of their staff nurses who participated in the
program. If a particular unit does not have many nurses in the program, this may not be an issue,
but units with a large group of nurses may have trouble collecting this data. The potential for bias
is associated with the program itself. Nurses are required to complete a wide range of training
throughout the year, most of which are required for regulatory purposes. There may be staff that
think the NSLP takes up time they do not have, or, due to the nature of the program delivery
method, they may feel that it disrupts their work/life balance.
In order to collect the data for the NSLP to answer the evaluation questions, the online
survey will be distributed to all participants enrolled in the NSLP program. The observation
questionnaire will be provided to all assistant nurse managers (ANMs) whose direct reports are

Page | 13

enrolled in the program, as well as the unit educators and managers for each of the departments
whose staff are enrolled in the NSLP.
The participant online survey items includes rating scales (Likert scales), yes/no
questions, and open-ended questions, asking participants to provide specific examples related to
the program content and application of the content. These questions will elicit the data, as
previously described, from the perspective of the participant. The item types are varied to keep
the time participants need to spend on the survey to a minimum. The observation questionnaire
will include rating scales associated with each of the desired leadership behaviors. Since unit
leadership will have to complete one survey per course participant, utilizing a rating scale also
helps to keep the time needed to spend on each questionnaire to a minimum.

Page | 14

Reporting Strategy Table

Stakeholder
Foundation/Donor

Senior Nursing Leadership

Partner Company

Unit Leadership

Reporting Strategy
Written interim report at end of
each participant cohort and final
program report.

Interim report at program midpoint and end for each cohort and
final report.

Email communication, phone

discussions, and final report.

Email communication at program

mid-point and end of each cohort,
verbal presentation at unit
leadership meetings, and final
report.

Implications
The information provided to the
donor can be used to make
decision on whether or not to fund
future nursing programs like
NSLP.
The NSLP is a considered a key
nursing initiative, therefore
communication to the senior
nursing leadership needs to occur
throughout the evaluation
process. They are key in
providing support needed to
continue the program as is, or to
promote any program
enhancements made from
evaluation findings.
This is the first time the Partner
Companys program has been
used in healthcare. The Partner
Company can utilize the findings
to enhance their program (make it
more healthcare focused), and
also to promote the program to
other health care companies.
Unit leadership is a key
stakeholder, and vital to the data
collection process. They are also
key in providing the work
environment where the nurses
apply the program knowledge and
skills. Evaluation data that shows
that program objectives are being
met, and that participants are
able to apply on the job, will show
the value of their participation in
the process.

Stakeholder Involvement
Donor involvement in the
reporting process is minimal. The
foundation may require a specific
report, in addition to the final
report.

Senior Nursing leadership will

need to provide input on the
criteria and standards.

The Partner Company has data

that will be obtained through
document reviews. They also
developed the content for the
program and can provide input
into development of the
participant online survey.
This group is key to the data
collection process, as they are
involved in completing the
observation questionnaire.
Because of this, they also need to
be involved in the development of
the questionnaire.

Page | 15

Centralized Education
Department

Email communication at program

mid-point and end of each cohort,
working sessions, and personal
discussions.

The beside nurse

Email communication to cohort at

end of program, NSLP website,
personal discussions

This group will use all evaluation

data, both positive and negative
findings, to make program
improvements.

The bedside nurse who

participated in the program can
use the information on the report
to validate their own program
experience. Those that have not
participated in the program can
use the findings to decide if they
would like to participate (program
is recommended, not mandatory).

This group is involved in all of the

components of the program, as
they are responsible for
coordinating the NSLP program
and for facilitating the face-to-face
debrief sessions. Members from
this group will need to be involved
in the entire evaluation process.
The bedside nurses involvement
will be through their participation
in the online surveys (feedback
included).

Values, Standards, and Criteria:

The goal of the NSLP is to provide the bedside nurse with the self-leadership knowledge and skills that correlate to the desired leadership
behaviors, and to have the nurse apply this knowledge and skills. To obtain the necessary data to determine if this goal has been met, the data
collection design previously outlined will be utilized. To ensure openness and credibility of the data, the values, standards, and criteria that will
be used for data interpretation are listed below for each questions. Criteria is listed followed by standard in parentheses.
Question 1: # reporting that yes, they were provided dedicated time to complete modules at work (80% yes responses), # reporting that yes,
they were able to schedule debrief sessions without issues (80% yes responses), # hours per module/participant, # reporting yes, they were
able to maintain work/life balance during program (80% yes responses).
Question 2: # reporting yes, modules were easy to navigate, # reporting yes, modules were easy to access, # reporting yes, module content
applicable, # reporting yes, modules and face-to-face debriefs met objectives (80% yes answers for all criteria).
Question 3: # self-reporting yes, there was an increase in self-knowledge, # self-reporting yes, there was an increase in communication skills
(80% of yes responses); # of participants observed by ANM, educator, and manager demonstrating increased knowledge and # demonstrating
increased skill (80% of respondents for knowledge and for skills, each).
Question 4: # of participants self-reporting applying knowledge and skills 3 or more times per shift (80% of respondents); # of times participants
observed applying skills and knowledge 3 or more times per shift during the observation period (80% of respondents).
Question 5: # of participants observed demonstrating desired leadership behaviors (80% of participants); # of participants whose yearly
evaluation indicate leadership behavior competencies.
Potential ethical issues:
A potential ethical issue is that the NSLP coordinator, from the Centralized Education Department, is also the evaluator, and facilitates the faceto-face debrief sessions.

Page | 16

Appendix A
Nurse Self-Leadership Program (NSLP) Logic Model

Logic Model template adapted from the Program Development and Evaluation unit of the University of Wisconsin-Extension

Page | 17

References
Bay, J. (2105). NSLP logic model. Unpublished manuscript. Walden University.
Bay, J. (2105). Program analysis: Nursing self-leadership program. Unpublished manuscript. Walden
University.
Fitzpatrick, J., Sanders, J., & Worthen, B. (2010) Program evaluation: Alternative approaches and practical
guidelines (4th ed.). Boston, MA: Pearson.
The National Academy of Sciences (2010). Summary of The future of nursing: Leading change, advancing
health. Retrieved from http://www.nap.edu/catalog/12956.html
University of Wisconsin-Extension (2008). Developing a logic model [PowerPoint slides]. Retrieved from
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Page | 18