Measuring the Impact of Knowledge Management

MEASURING THE IMPACT

OF KNOWLEDGE
MANAGEMENT

NOTE: PARTICIPATION IN THE MEASURING THE IMPACT OF KM COURSE ENTITLES

EACH PARTICIPANT ACCESS TO ONE COPY OF THE COURSE MATERIALS IN
ELECTRONIC (PDF) FORMAT. THE COURSE MATERIALS ARE FOR USE INTERNALLY
WITHIN PARTICIPANT’S ORGANIZATION ONLY AND MAY NOT BE MARKETED,
DISTRIBUTED, OR SOLD TO THIRD PARTIES, WHETHER OR NOT FOR PROFIT. EXCEPT
AS MAY BE PROVIDED BY SPECIAL AGREEMENT OR LICENSE BETWEEN PARTICIPANT
AND APQC, PARTICIPANT MAY NOT DISTRIBUTE, DISPLAY, PERFORM, REPRODUCE,
PUBLISH, LICENSE, CREATE DERIVATIVE WORKS FROM, TRANSFER, OR SELL THE
COURSE MATERIALS.

©2014 APQC.  ALL RIGHTS RESERVED.
1
 Measuring the Impact of Knowledge Management

WHO WE ARE

o APQC is a member-based, 501(c)3 nonprofit specializing in

benchmarking, knowledge management, measurement, and
process improvement.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

3

ADVISORY & CUSTOM SOLUTIONS

o We work with you to assess and improve your internal

processes by helping you identify the strategies, tactics, and
measures appropriate to your organization to drive business
results.
o Services available:
 knowledge management,
 business excellence,
 quality management,
 measurement,
 and custom benchmarking

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

4

©2014 APQC.  ALL RIGHTS RESERVED.
2
 Measuring the Impact of Knowledge Management

LOGISTICS & GROUND RULES

o Logistics
 Review safety/emergency guidelines
 Breaks/Lunch – Start on time
 Class etiquette:
• Turn cell phones on vibrate
• For Internet access, click on “APQC Guest,” and enter guest code
123789123
o Ground rules:
 Ask questions at any time
 Share your experiences
 Work together and have fun

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

5

COURSE LEARNING OBJECTIVES

o Understand how to use APQC’s Knowledge Management

Value Creation Framework™.
o Identify the common knowledge flow process steps and KM
approaches.
o Identify and align standard measures that can be used to
track the impact and success of knowledge management
activities on business outcomes.
o Apply APQC’s performance measurement development
process and tools to KM processes and approaches.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

6

©2014 APQC.  ALL RIGHTS RESERVED.
3
 Measuring the Impact of Knowledge Management

AGENDA

Time Topics

8:00 AM Training registration and continental breakfast
8:30 AM • Introductions and Expectations
• Fundamentals for Measuring KM
• Frameworks and Processes for Aligning KM 
to Business Value

12:00  Networking Lunch
PM
1:00 PM • Developing Useful KM Measures
• Processes and Tools for Creating a KM 
Measurement System
• Building Buy‐in for KM Measurement

4:30 PM Wrap up and evaluation

5:00 PM Adjourn
www.apqc.org

EXERCISE

o Break into small groups.

o Give name, organization, and role.
o Collectively identify the top 5 things you want to learn today.
o Out-brief to the group.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

8

©2014 APQC.  ALL RIGHTS RESERVED.
4
 Measuring the Impact of Knowledge Management

YOUR INPUT COUNTS!

o Everyone needs an internet capable device (i.e., phone, tablet.

or pc).
o Go to POLLEV.com
o You will be presented with an empty text box to enter an
answer code as your response.
o The answer code (e.g., 553484) to enter will be presented on
the screen.
o After entering the answer code, submit response. See picture
on bottom right.
o Your answer will be acknowledged
as received.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

9

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

10

©2014 APQC.  ALL RIGHTS RESERVED.
5
 Measuring the Impact of Knowledge Management

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

11

3 REASONS TO MEASURE KM

o Demonstrate value
o Know where to invest more or less in KM
o Drive behavior (knowledge sharing, learning, improvement)

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

12

©2014 APQC.  ALL RIGHTS RESERVED.
6
 Measuring the Impact of Knowledge Management

FUNDAMENTALS FOR MEASURING KM

MODULE LEARNING OBJECTIVES

o Identify common knowledge management processes,

capabilities and approaches.
o Identify key measurement activities across APQC’s Levels of
KM Maturity™.
o Understand the investment and financial impact of knowledge
management.
o Discuss the capabilities required for measuring the impact of
knowledge management.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

14

©2014 APQC.  ALL RIGHTS RESERVED.
7
 Measuring the Impact of Knowledge Management

KNOWLEDGE IS INFORMATION…

. . . IN ACTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

15

DRIVING KNOWLEDGE FLOW

o “Because we have to.” o “Because we can.”
o Need for speed and cycle o Understanding of tacit and
time reduction explicit knowledge
o Revenue growth o Infrastructure capabilities
o Competition for customer  Rise of powerful network,
relationships communication, database
and collaborative
o Lost knowledge from technologies
turnover, down-sizing, and o Change management and
restructuring process skills
o Knowledge has a higher o Social computing
margin than product
o Globalization
o Generational perspectives

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

16

©2014 APQC.  ALL RIGHTS RESERVED.
8
 Measuring the Impact of Knowledge Management

KNOWLEDGE MANAGEMENT DEFINED

Systematic approaches
to enable information and knowledge
to grow, flow, and create value.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

17

KNOWLEDGE IN ACTION CREATES VALUE

APQC’s Knowledge Flow Process

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

18

©2014 APQC.  ALL RIGHTS RESERVED.
9
 Measuring the Impact of Knowledge Management

KNOWLEDGE MANAGEMENT APPROACHES

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

19

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

20

©2014 APQC.  ALL RIGHTS RESERVED.
10
 Measuring the Impact of Knowledge Management

LEVELS OF MATURITY

Level 5 Continuously
Innovate Improving practices
Dynamic
Level 4
Driving knowledge
organizational
Optimize outcomes
Leveraged
Level 3 Aligning processes knowledge
Standardize and approaches
Applied
Level 2 Growing knowledge
Develop involvement

Ad Hoc
Level 1 Growing knowledge
Initiate awareness

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

21

MEASURE AT EACH LEVEL OF KM MATURITY

 Little or no internal  Continue collecting and

measurement sharing stories
 Use external  Assess the potential
benchmarking to learn payoff (tangible and
from others intangible) for KM
 Begin collecting stories approaches in order to
select initiatives
 Assess readiness and
capabilities for
knowledge
management

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

22

©2014 APQC.  ALL RIGHTS RESERVED.
11
 Measuring the Impact of Knowledge Management

MEASURE WITH STORIES

Stories serve to establish the causal

linkage between inputs, behaviors and
outcomes for the business case and
measurement reporting process

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

23

EXAMPLE: KM QUALITATIVE SUCCESS STORIES

“The chief engineer had the foresight to have me research on a project for an
innovation grant proposal … it was a complicated project … it took time to
gather all the right people, but when the sponsor called to ask us ‘what we
knew’ about the project, we had ‘everything in place,’ which had a big impact
on the customer.”
APQC KM Measures That Matter Peer to Peer Study, 2010

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

24

©2014 APQC.  ALL RIGHTS RESERVED.
12
 Measuring the Impact of Knowledge Management

LEVELS OF KM MEASUREMENT MATURITY

 Build measures into pilot  Consistently correlate KM

project design measures with business
 Leading indicators (input, performance measures
activity) (outcomes)
 aligned to  Extrapolate gains from KM
process/output measures; projects to underwrite
 Qualitative indicators to expansion
supplement metrics  of the KM program
 Assess maturity and
robustness of KM
capabilities
 (people, process
and technology)
and costs to scale up

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

25

RESOURCE COMMITMENT OVER TIME

Shared Accountability for Implementing and Sustaining KM
100%

Business
Units

KM Core
Group

TIME
www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

26

©2014 APQC.  ALL RIGHTS RESERVED.
13
 Measuring the Impact of Knowledge Management

LEVELS OF KM MEASUREMENT MATURITY

 Fewer new KM-specific measures

• Continue to track behaviors and outcomes
• Surveys (‘I can’t work without it.”)
• Usage and Participation
 Business performance measures reflect value of KM
capabilities across the enterprise

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

27

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

28

©2014 APQC.  ALL RIGHTS RESERVED.
14
 Measuring the Impact of Knowledge Management

EXERCISE

o Complete the Measurement Section of APQC’s KM

Capability Assessment Tool™ (See appendix A)

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

29

IN SUMMARY…

o We’ve discussed:
o The Knowledge Flow process and KM activities alignment
with business inputs, processes, and outcomes
o The potential benefits gained from applying KM to achieve
business outcomes
o The difference in KM measurement activities across the
various stages of implementation
o APQC’s Levels of Knowledge Management Maturity™

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

30

©2014 APQC.  ALL RIGHTS RESERVED.
15
 Measuring the Impact of Knowledge Management

FRAMEWORKS AND PROCESSES FOR ALIGNING

KM TO BUSINESS VALUE

MODULE LEARNING OBJECTIVES

o Understand the goals of a KM measurement approach

o Understand the scope of measurement
o Learn how to align KM activities to business processes and
outputs
o Conduct a Value Path Analysis

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

32

©2014 APQC.  ALL RIGHTS RESERVED.
16
 Measuring the Impact of Knowledge Management

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

33

A KM MEASUREMENT APPROACH PROVIDES

o A target for knowledge management performance that aligns

to and impacts the performance of business processes
o A diagnostic to identify areas for improvement and learning
o A basis for communication using agreed upon and consistent
definitions

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

34

©2014 APQC.  ALL RIGHTS RESERVED.
17
 Measuring the Impact of Knowledge Management

GETTING STARTED

Aligned Measures Tell A Meaningful Story

Start with the business measures of

outcomes and then work backwards to
create measures and indicators of KM
activity focused on that outcome.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

35

GROUP DISCUSSION
What area of your business can be improved by applying KM?

o What knowledge needs to be put into action to positively impact

business outcomes and financials?

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

36

©2014 APQC.  ALL RIGHTS RESERVED.
18
 Measuring the Impact of Knowledge Management

EMBED KNOWLEDGE FLOW INTO THE BUSINESS FLOW

KM Investment
(Input)

KM Activity
(Process)
Knowledge Management
approaches enable
KM Use knowledge flow
(Output)

Input Process Output Outcome

KM Costs and Activities Impact on Business Revenues/Profits/
Infrastructure Associated with KM Objectives Customer Satisfaction/
Approaches Loyalty

The ‘Value Path’  Links KM Activities and Business Outcomes
Source:  APQC’s Input – Outcome Measurement Methodology, 2003

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

37

VALUE CREATION
Key steps

 Business Goals   Core Business 
 Business Outputs  Processes
(Outcomes)
(Critical Success Factors) (Business performance and 
leading indicators)

 KM Investments  KM Approaches
(KM core group and business  (KM activity measures and 
processes) performance indicators)

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

38

©2014 APQC.  ALL RIGHTS RESERVED.
19
 Measuring the Impact of Knowledge Management

VALUE PATH ANALYSIS

Identify The “Why”

o Understanding the link (i.e., cause and effect) provides the "Why”
for KM
o Ensuring business value requires you to make the desired business
outcomes explicit
o Having relevant KM methods/tools/processes
require an understanding of the linkage between the KM activities
and the business processes
o Demonstrating the impact – find measures that are relevant to the
business

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

39

VALUE PATH EXAMPLE

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

40

©2014 APQC.  ALL RIGHTS RESERVED.
20
 Measuring the Impact of Knowledge Management

MEASUREMENT ALIGNMENT
APQC’s Measurement Alignment Worksheet™

Microsoft Office
Excel Worksheet

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

41

EXERCISE

o Conduct a value path analysis:

 Review your process scenario.
 Identify at least two business goals and associated critical success
factors.
 Assume at least two of the following KM approaches will
improve/impact knowledge flow of the business process:
• Community of Practice
• Transfer of Knowledge & Best Practices
• Lessons Learned
• Portal Tools
 Trace and analyze the value path from the KM activity to the business
goal using your measurement alignment worksheet template.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

42

©2014 APQC.  ALL RIGHTS RESERVED.
21
 Measuring the Impact of Knowledge Management

IN SUMMARY…

o Measuring KM has to have context.

o A robust framework and tools for alignment is necessary.
o Regarding KM measurement, determine if there is a data
problem or thinking problem.
o Need guiding principles for measuring KM and its impact.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

43

DEVELOPING USEFUL KM MEASURES

©2014 APQC.  ALL RIGHTS RESERVED.
22
 Measuring the Impact of Knowledge Management

MODULE LEARNING OBJECTIVES

o Understand the difference between hard and soft business

measures and their value
o Define and measure business outcomes and outputs
o Identify the common measures used for the knowledge flow
process and KM activities
o State requirements for designing useful measures

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

45

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

46

©2014 APQC.  ALL RIGHTS RESERVED.
23
 Measuring the Impact of Knowledge Management

HARD & SOFT MEASURES

o Hard Measures o Soft Measures

o Cost reduction o Customer satisfaction
o Quality improvement o Employee satisfaction
o Productivity o Anecdotal stories /
enhancement testimonials
o Sales growth
o Cycle time reduction

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

47

SOFTER MEASURES & INTANGIBLES

o Opinion-based
o Harder to draw a direct line to tangible results
o Calls for interpretation
o However…
o Provide the “human side” to the story
o Provide context to hard metrics
o Gives a more complete picture of KM activities

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

48

©2014 APQC.  ALL RIGHTS RESERVED.
24
 Measuring the Impact of Knowledge Management

MEASURES TIED TO BUSINESS OUTCOMES

o Does the organization have measures tied to desired

business outcomes (goals)?
o Can KM get access to data for these measures?
o In what situations are explicit business outcomes identified
before you start a KM activity?
o Examples of outcomes:
 Increase market share
 High product quality
 High customer satisfaction

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

49

OUTPUTS ARE CRITICAL TO SUCCESS

o The vital areas essential to achieve business outcomes

include:
 Increased productivity
 Increased sales
 Reduced rework
 Improved cycle time
o Necessary, but not sufficient to
ensure business goals are achieved

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

50

©2014 APQC.  ALL RIGHTS RESERVED.
25
 Measuring the Impact of Knowledge Management

MEASURING BUSINESS PROCESSES

o MEASUREMENT o BUSINESS PROCESSES

CATEGORIES

o Cost Effectiveness
o Staff Productivity
o Process Efficiency
o Cycle Time

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

51

MEASUREMENT DATA
for KM Programs & Approaches

KM Program CoPs
 Number of communities  Frequency of visits to a CoP site
 Number of CoPs launched by year  Number of reads per month
 Number of participants  Number of postings
 Number of participants as a % of   Number of downloads
potential total participants  Member satisfaction ratings
INPUTS
•FTE
•Direct labor cost
•Contract labor expense
•Technology expense
•Materials and supplies
•Travel

Transfer of Knowledge and Expertise Transfer of Best Practices
 Number of active participants  Number of best practices identified
 Number of staff achieving a target   Number of best practices submitted
competency rating  % of submitted practices validated and published
 Reduced time to target competency  Number of best practices downloaded/accessed
 Projects on time/under budget  Number of active participants
 % of project teams fully staffed  Number of replications
www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

52

©2014 APQC.  ALL RIGHTS RESERVED.
26
 Measuring the Impact of Knowledge Management

KM MEASUREMENT SCORECARD
Measure Our Progress Above the Flow
Knowledge Management Scorecard
Service Activity Measure
% Complete

Start by measuring KM activities above

Engineer of the Year
Customer Satisfaction
o Programs SEPP
% Complete
Customer Satisfaction

the flow of our process – show

% Complete
Engineering Summit
Customer Satisfaction
Customer Satisfaction ILT
School of Engineering:

relevancy, impact and value.

Customer Satisfaction CBT
% Complete
Development
New Requests
Completed ILT

o Include hard measures such as number Offerings

Completed CBT
New Requests
# Classes Backlog Demand

of participants, knowledge articles,

Learning ILT Customer Satisfaction
CBT Customer Satisfaction
Budget Variance to Forecast

courses offered and requests for

Participation level
Academies
Courses created/modified
Dual Degree Customer Satisfaction
Sessions

workplace coaching. New Hire Orientation

KM Maturity
% Participation
% Customer Satisfaction

KM CAT Level

Report customer satisfaction and

Participation

o CoPs Customer Satisfaction

Value

successes through storytelling, surveys,

Number of Visits
Avg Monthly Unique Users
Epedia Knowledge Base
Number of Articles
Knowledge Customer Satisfaction

and interviews.
Sharing Lessons Learned Contributions
Views
Research Center
Requests
Utilization

Quantify soft measures.

eSearch
Customer Satisfaction
o Workplace Coaching
Requests Received
Requests Fulfilled
~ Number of Hours

Source: 2011 APQC KM Conference Breakout Session

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

53

EXERCISE

o Identify measures for your team’s value path components.

These measures should reflect the following:
 Outputs for KM approaches
 Performance indicators for business processes
 Business outputs
o Use the Measurement Alignment Worksheet to capture these
measures.

Tool:
APQC’s List of Standard KM Activity Measures (See appendix B
in this book)

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

54

©2014 APQC.  ALL RIGHTS RESERVED.
27
 Measuring the Impact of Knowledge Management

IN SUMMARY…

o KM activity measures become the KM Program measures.

o CoPs and Facilitated Best Practice Transfer are activities that
can be easily tracked and have hard dollar impacts if
implemented and measured correctly.
o Best practice organizations have
 been able to tie KM activities to individual performance measures.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

55

CREATING A KM MEASUREMENT SYSTEM

©2014 APQC.  ALL RIGHTS RESERVED.
28
 Measuring the Impact of Knowledge Management

MODULE LEARNING OBJECTIVES

o Learn what key activities support the KM measurement

process
o Apply tools that enable the KM measurement process and
system
o Use in-depth analysis to show the linkages between inputs,
process changes and outcomes
o Understand the guidelines and process for creating a
measurement system that works

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

57

KM MEASUREMENT FRAMEWORK

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

58

©2014 APQC.  ALL RIGHTS RESERVED.
29
 Measuring the Impact of Knowledge Management

POLLING QUESTIONS

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

59

KM MEASUREMENT ACTIVITIES
The KM program takes an active role in the measurement process.

100.0%

Reporting 68.4%

100.0%
Calculating
26.3%
benefit
87.5%

Analysis 57.9%

100.0%
Tracking 68.4%

75.0%
Collection 63.2%

87.5%
Development 52.6%

0% 20% 40% 60% 80% 100%

Blue bar = best practice organizations Gray bar = non-best practices organizations

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

60

©2014 APQC.  ALL RIGHTS RESERVED.
30
 Measuring the Impact of Knowledge Management

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

61

DATA NORMALIZATION & VALIDATION

o Data normalization - puts data on a common basis (e.g.,

downloads per employee) .
 Mitigates issues of organizational scale
 Supports inferences about relative performance
o Data validation - Ensures numeric data provided by
respondents are accurate and consistent with standard
definitions and scope used to collect information.
o If validation does not resolve data issues, the information
should not be included in analysis.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

62

©2014 APQC.  ALL RIGHTS RESERVED.
31
 Measuring the Impact of Knowledge Management

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

63

ANALYSIS & REPORTING

Measurement Analysis and Distribution
22. How will the data be How will the analysis be performed and by
analyzed? whom?
23. How will the data be Who gets the data; does it get reported to
distributed and to whom? different audiences in different ways?
24. When is the best time to When should the metric data be reported?
distribute the data? Is one time better than another?
25. When will it be reported to When does the process owner get to see it?
the Process Owner?
26. When will adjustments be When can corrective actions be expected
made to improve results? to take place?

Tool:  APQC’s Measurement Information Worksheet™

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

64

©2014 APQC.  ALL RIGHTS RESERVED.
32
 Measuring the Impact of Knowledge Management

ANALYSIS TYPES

o Correlations:
 Shows relations between two variables (e.g., KM outputs and process
performance)
 Note: Correlation does not necessarily equal causation
o Comparative analysis:
 Examines performance relative to some external benchmarks,
targets, or prior period.
o Trend analysis:
 Determines the degree variables move in tandem over time.
o Qualitative analysis:
 Examines KM impact from the personal perspective of business
process staff.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

65

LINK KM ACTIVITIES TO BUSINESS PROCESS

o Create hypotheses for KM measures to business process

performance and business process performance to business
outputs.
o Use correlation matrix to validate the value path.
KM Outputs
Correlation Matrix Communities of Practice Self‐Service

Process Performance
Higher use of KM self‐
Higher CoP participation 
service tools results in 
Output per Employee results in higher output 
higher output per 
per employee
employee
Higher use of KM self‐
Higher CoP participation 
service tools  results in a 
Production Error Rate results in a lower 
lower production error 
production error rate
rate
www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

66

©2014 APQC.  ALL RIGHTS RESERVED.
33
 Measuring the Impact of Knowledge Management

DISCOVER THE IMPACT

o Use statistical analysis to uncover KM impacts on process

performance. KM Outputs
CoP participants at meetings 
Downloads from KM portal 
as a % of required 
per production employee
participants 
Process Performance
An increase of 10 
5% increase in CoP 
downloads per production 
participation 2 months ago 
employee last month 
Output per Employee results in a 4 units increase 
results in a 5 unit increase 
in output per employee 
in output per employee 
this month.
this month.

An increase of 10 
5% increase in CoP 
downloads per production 
participation last month 
employee 3 months ago 
Production Error Rate results in an 1% decrease 
results in a 2% decrease in 
in production error rate 
production error rate this 
this month.
month.

Mock data for illustration purposes.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

67

ANALYSIS: CORRELATION
Time Period  CoP  Output per 
(Quarter) Participation  Employee KM Measurement 
Rate
Correlation Scatterplot
Q110 25.8% 18.0 80.0
Q210 36.6% 16.0
70.0
Q310 27.7% 15.0
Output per Employee

Q410 58.0% 23.0 60.0

Q111 58.1% 35.0 50.0
Q211 61.7% 24.0
40.0
Q311 63.0% 27.0
Q411 65.0% 22.0 30.0
Q112 70.2% 35.0 20.0
Q212 71.9% 49.0 10.0
Q312 81.0% 40.0
0.0
Q412 83.5% 47.0
0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
Q113 84.4% 53.0
CoP Participation  Rate
Q213 86.4% 56.0
Q313 87.0% 63.0

Determine relationship Strong correlation ‐ As participation increases, output increases.

Q413
Q114
92.6%
93.9%
57.0
68.0
between variables (e.g., KM outputs and business
Note: Correlation does not equal causation. Other factors could influence output.
performance).
Mock data for illustration purposes.

Determine strength and direction of relationship.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

68

©2014 APQC.  ALL RIGHTS RESERVED.
34
 Measuring the Impact of Knowledge Management

ANALYSIS: CORRELATION

ROCKWELL COLLINS
PROGRESS KM MEASUREMENT

Approach

o Will continue to collect

data to show correlations
of KM activities to real
business measures.
o Incorporate into
Engineering Scorecard.
Source: 2011 APQC KM Conference Breakout Session

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

69

QUALITATIVE INFORMATION

o Qualitative information can be collected through interview

or survey.
o Stories about the impact of KM keep people involved and are
a great way to provide context for the data being reported.
o Stories should explain how knowledge is flowing as a result
of KM.
o Qualitative analysis should focus on how KM has impacted
process performance. The information should be as specific
as possible in terms of impact.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

70

©2014 APQC.  ALL RIGHTS RESERVED.
35
 Measuring the Impact of Knowledge Management

EXAMPLE OF KM QUALITATIVE SURVEY QUESTIONS

Survey Questions (Likert scale) Survey Questions (open response)

o I am encouraged to share my knowledge o What impediments, if any, impact your
and experience with others in the ability to find and share information and
organization. collaborate with others in the organization
o I routinely share what I discover, create, and beyond your current department and
produce. project(s)?
o We create new knowledge and use it o If you could make one recommendation
effectively in our work programs. for an investment that the organization
o Demonstrates a strong knowledge-sharing could make that would improve our ability
culture and practice. to bring all of the organization to bear, what
would it be?
o Please rate how much the ability to find
information and tap what the organization o Of the tools available to you on the
knows has helped you or your team over organization’s intranet, which are the most
the last year (i.e., saved time getting work useful to you for sharing knowledge and
done, improved the quality of work, etc). getting access to the knowledge of others?
o Do you have a story about knowledge
sharing in the organization – either positive
or negative – that you would like to share it
with the survey team?...

APQC KM Measures That Matter Peer to Peer Study, 2010

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

71

EXAMPLE: QUALITATIVE SURVEY RESULTS

Social Networking enabled me to:

90% 78%
80% 75%
64% 67% 63% 64%
70% 61%
60%
50% 42%
40%
30% 24%
20%
10%
0%
Save time Solve work Identify sales Find expertise Share and Find Get an Find relevant Stay
problems opportunities disseminate information immediate learning connected,
information answer to a feel like a part
Survey results indicate clear business value from Social Networking, with 82% of  question of the
organization
respondents intending to use the tools on an ongoing basis

APQC KM Measures That Matter Peer to Peer Study, 2010

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

72

©2014 APQC.  ALL RIGHTS RESERVED.
36
 Measuring the Impact of Knowledge Management

EXAMPLE: KM QUALITATIVE TESTIMONIALS

Leveraging Social Networking capabilities is helping Area 3…
…assimilate new practitioners … drive sales
“I joined recently from another large consulting firm that didn't have "I receive many BI health check queries that I
the depth and capabilities of the organization's social networking have guided to the community to post questions
tools. The tools and their usability have made it very easy for me to and leverage content. We've sold 20 health check
get connected and stay on top of ABC information. I was only two engagements to date (at about $50K each). I
weeks with the organization and already using the Portal and attribute a lot of this to the collaboration in the
community"
communities to extend my network as well as use sales and
delivery content to help with client opportunity pursuits. These tools
have made it very easy to find, review and become quickly BA Business Development Executive, Atlanta
proficient in my immediate sales and delivery responsibilities.”
Associate Partner, Area 3 Strategy, Philadelphia
…drive Service Line maturity
“As a recent joiner to Area 3, the organization's social networking
“[Social Networking] is the most important vehicle
tools have been critical in me connecting with the right people in
for us to drive teamwork as we are spread out
'real time' to resolve a challenge we've only just that minute across the world. It is crucial in the forming of
faced. This is the power of these tools - channeling the collective the Area 3 Service Line, bringing the Service
knowledge base of the organization and making it available to Line together as one team.”
everyone.”
Area 3 Consultant, Johannesburg Area 3 ECM Practice Leader, Copenhagen

APQC KM Measures That Matter Peer to Peer Study, 2010

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

73

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

74

©2014 APQC.  ALL RIGHTS RESERVED.
37
 Measuring the Impact of Knowledge Management

MEASUREMENT RESEARCH
Best Practice Attributes

o Ensure senior managers meet on a routine basis to analyze metrics

and make decisions based on the data
o Use the measurement
system as a tool to
understand performance
drivers (e.g., KM outputs)

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

75

EXERCISE

o Using the set of measures developed for a KM activity in

Section 4, review and ratify the value path created in Section
3.
 Develop the correlation matrix for your measures.
 What additional relationships can be uncovered by using the
information from your measures?
 What intangible benefits (e.g., social cohesion, job satisfaction, and
time-to-competency) will provide a broader perspective of successful
knowledge sharing activities?
 Identify the key trends you would like to track over time related to
KM impacts.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

76

©2014 APQC.  ALL RIGHTS RESERVED.
38
 Measuring the Impact of Knowledge Management

IN SUMMARY…

o KM core teams and programs support the measurement

process.
o Identify the KM measurement reporting requirements before
creating the processes for providing information.
o Calculating benefits and value of KM is on-going and needed
to continue to secure funding and investments for the KM
program/efforts.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

77

BUILDING BUY-IN FOR KM MEASUREMENT

©2014 APQC.  ALL RIGHTS RESERVED.
39
 Measuring the Impact of Knowledge Management

MODULE OBJECTIVES

o Learn key features needed to build buy-in for KM

measurement.
o Discuss factors to ensure ownership of the KM
measurement system.
o Identify ways to overcome resistance to KM measurement.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

79

KEY FINDINGS: ENSURING BUY-IN FOR MEASUREMENT

o Use the following change

management techniques
to secure buy-in for
measurement:
 Create a sense of
ownership by involving
staff early in developing
the KM measurement
system.
 Provide a documented
roadmap for the
measurement process
and outcomes.
Source: Using Metrics That Drive Bottom Line Value study. APQC 2012.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

80

©2014 APQC.  ALL RIGHTS RESERVED.
40
 Measuring the Impact of Knowledge Management

KEY FINDINGS: ENSURING BUY-IN FOR MEASUREMENT

o Use a variety of
techniques to foster
buy-in and
accountability within
the organization:
 Have an owner of the
measurement system.
 Include measures as
part of performance
appraisals.
 Connect measures with
compensation.

Source: Using Metrics That Drive Bottom Line Value study. APQC 2012.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

81

POLLING QUESTION

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

82

©2014 APQC.  ALL RIGHTS RESERVED.
41
 Measuring the Impact of Knowledge Management

ACCOUNTABILITY
Corrective Action
27. Who takes action Who is responsible for fixing the problem
to make the that has led to sub-standard performance?
adjustments if analysis
shows that the
process is below the
required standard
performance? What does it take to turn performance
28. What actions are around?
necessary to improve
performance?
What happens if nothing is done? Not a
29. What are the worst case scenario, but the probable
consequences of situation that would reflect the typical
delaying action? consequences for delaying action.
30. What are the Measurement drives behavior. How are
rewards for meeting people incented to improve/maintain
performance goals? performance?

Tool:  APQC’s Measurement Information Worksheet™

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

83

ASSIGN AN OWNER

o To ensure data accuracy o In addition, ensure

and instill “one version of automated data
the truth” around collection systems are
organizational measures, properly supported by IT,
those responsible for when needed.
performance
measurement need to:
 define measures
consistently.
 use data in a consistent
manner.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

84

©2014 APQC.  ALL RIGHTS RESERVED.
42
 Measuring the Impact of Knowledge Management

EXERCISE

o Group discussion:
o As a group, identify key issues related to building buy-in for
KM measurement.
o Rank the top three most important factors for sustaining
buy-in for KM measurement.
o Identify ways to overcome resistance to KM measurement.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

85

IN SUMMARY…

o KM core teams have the responsibility to build buy-in for

measurement by showing business process owners the value
that comes from this information.
o Involvement of business process staff is essential for KM
measurement to be successful.
o Communication should emphasize that sustainability of the
KM program is tied to the ability to measure its impact.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

86

©2014 APQC.  ALL RIGHTS RESERVED.
43
 Measuring the Impact of Knowledge Management

RECAP AND LESSONS LEARNED

ENDURING LESSONS

o Measurement of KM activities starts Day 1

o KM activities must be aligned to the core business goals and
strategies
o Qualitative measures help, but quantitative metrics are
critical in building support
o Tie new KM measures to already accepted process measures
and metrics
o Different stakeholders need different measures

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

88

©2014 APQC.  ALL RIGHTS RESERVED.
44
 Measuring the Impact of Knowledge Management

FINAL THOUGHTS

o Yes, you can measure KM, and what's more, you had better if
you want results and sustainability
o The financial ROI is surprisingly high
o Conversations and communities are
the platform for results in all the
best practice firms
 Big gains come from the
KM conversations on
customer-facing topics
o Knowledge at rest is just potential;
knowledge in action is where the value emerges.

www.apqc.org

©2014 APQC. ALL RIGHTS RESERVED.

89

IN SUMMARY…
o Focus on what the business cares about
 Productivity
 Revenue
 Profits
o Use methods that work (and show tangible
benefit)
 Design teams and Communities
 Facilitated Best Practice Transfer
o Partner with IT
o Measure and demonstrate results

www.apqc.org

90

©2014 APQC.  ALL RIGHTS RESERVED.
45
Measuring the Impact of Knowledge Management

QUESTIONS

123 N. Post Oak | Houston, TX | 77024

www.apqc.org

©2014 APQC.  ALL RIGHTS RESERVED.
46