International Journal of Project Management 20 (2002) 165177

www.elsevier.com/locate/ijproman

Performance measurement of R&D projects in a multi-project,

concurrent engineering environment
A. Sivathanu Pillai a,*, A. Joshi b,c, K. Srinivasa Rao d
a

Defence Research & Development Organisation, 117C, B Wing, Sena Bhavan, New Delhi - 110 0011, India
b
Management Faculty, University of Pune, India
c
Indian Institute of Cost and Management Studies & Research, 85/1, Chiplunkar Road, Erandavane, Pune - 411 004, India
d
Defence Research & Development Organisation, 117D, B Wing, Sena Bhavan, New Delhi - 110 0011, India

Abstract
An R&D Project can be characterized by its life cycle with three phases of evolution, viz., Project Selection Phase (Screening,
Evaluation, Selection), Project Execution Phase (Technology Development, Product Development, Performance Demonstration)
and Implementation Phase (Production, Marketing, Sales). The traditional approach of performance measurement deals with each
of these phases in isolation. As a result, the evaluation models and performance measurement criteria are separate for each phase.
Once a project is selected, all attention is focused on its completion within the stipulated time and cost, without much consideration
to either the assumptions made at the time of project selection or the requirements of the implementation phase. As a result, performance measurement system for project execution phase is totally independent of other phases. In an R&D environment with
high uncertainty and complexity, coupled with multiple projects competing for limited common resources, use of dierent models
of evaluation at dierent phases may lead to incorrect assessment and poor overall performance. This paper addresses this important issue and suggests a framework for an Integrated Performance Index encompassing the entire lifecycle of R&D projects. The
framework identies the key factors in each phase of the project lifecycle and integrates them through a formula to derive an
Integrated Performance Index that can be used to measure the overall performance of a project at any point of time during its life
cycle. # 2001 Elsevier Science Ltd. All rights reserved.
Keywords: Integrated performance measurement; Multi-project environment; Concurrent engineering; Project management

1. Introduction
Performance measurement plays an important role in
ensuring the project success and its subsequent usefulness to the sponsoring organization. In a controlled
environment, the organizational and project performance is known to be sensitive to the metrics of measurement. Hence it is very important to devise
appropriate performance measurement system to suite
the project and organizational environment. The notion
of performance measurement generally implies identication of certain performance metrics and criteria for
their computation. Several metrics have been developed
to evaluate R&D projects during the selection phase
[123]. Similarly there are well developed metrics for

* Corresponding author. Tel.: +91-11-301-3220; fax: +91-11-3017882.

0263-7863/01/$22.00 # 2001 Elsevier Science Ltd. All rights reserved.
PII: S0263-7863(00)00056-9

project performance measurement during the execution

phase, mostly built around PERT/CPM and the earned
value system. But there is no link between the performance metrics of the project selection phase and the
project execution phase. Also, there is no explicit link
between the performance factors measured during the
project execution phase and the factors that may determine the project performance during the implementation phase. These missing linkages may lead to poor
overall performance of the project. It is essential to
bridge these gaps through an integrated performance
measurement system that could be used for all phases of
the project life cycle. This paper addresses this important issue and proposes a framework for integrated
performance measurement.
The following paragraphs explain the motivation for
the proposed model, problem denition, proposed
approach for integrated performance measurement,
conceptual model, identication of key factors and their

166

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

integration, and the advantages of using an Integrated

Performance Index.

framework for the integrated performance measurement

of R&D projects.

2. Motivation for the proposed model

3. The problem of performance measurement in R&D

projects

The proposed model is motivated by the real-life

experiences of the authors and the problems faced in
dealing with the high technology aerospace R&D projects. The rst author had the opportunity to work on
the management system for the Integrated Guided Missile Development Programme (IGMDP) of India. The
Programme is characterized by a multi-project environment comprising of ve large R&D Projects involving
the development of technologies and systems that would
be contemporary at the time of their deployment 10
years later. Due to the lack of adequate technology base
within the country and the restrictions on the ow of
technology from outside, all the required technologies
had to be developed within the country. The Programme adopted concurrent engineering philosophy to
reduce the cycle time from design to deployment. In spite
of this high complexity and large magnitude of task, the
programme succeeded in the development of several
critical technologies indigenously through a partnership
network of R&D organisations, academic institutions,
public and private sector industries and user services,
using certain unique management practices. One of the
important factors that contributed for the success of this
Programme is the integrated way of looking at the project performance and the associated tools, techniques
and methodologies evolved and used in the Programme.
This experience led to the development of the proposed

R&D managers throughout the world are continually

faced with a series of decisions as to, how to select the
most appropriate projects from several competing proposals, how best to evaluate a project during its execution, how best to make use of available knowledge
about the project to forecast a project failure and initiate its early closure to prevent further drain of resources, etc. Due to the inherent complexity and uncertainty,
R&D projects are not easily amenable for performance
measurement. This situation is compounded further by
the multi-project and concurrent engineering environments. Under these conditions, the approach of dealing
with each phase of the project life cycle entirely independent of other phases will lead to poor overall performance. Too often, at the completion of a project,
management may realise that the market no longer
exists, or the technology is obsolete, or the original
purpose no longer t the current business strategy. How
can we remedy this situation? This is the main problem
addressed by this paper.
Fig. 1 shows the typical life cycle of an R&D project.
Project proposals are initiated based on stated or perceived customer requirements. These proposals are then
screened, evaluated and selected with the help of some
project selection methods using certain criteria. The
selected projects will then enter the Project Execution

Fig. 1. Life cycle of a typical R&D project.

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

phase where the required technologies and products are

developed and the product performance is demonstrated through various trials. In a concurrent engineering environment, the designs and technologies
developed are concurrently transferred to the production
agencies for the production of systems for development
trials and performance demonstration trials. Upon successful completion of the design and development
phase, production, marketing and sales will commence
to recover the investment made and to realize the other
intangible benets envisaged at the time of project
selection. Thus, a project will become successful only
after meeting the objectives and expectations envisaged
at the time of its selection. In this process, the performance in each phase is an essential but not a sucient
condition for project success. Therefore, an eective
performance measurement system should be able to
measure the overall performance of the project in each
phase that will lead to project success. Realisation of
such a system is the problem addressed by this paper.
A literature survey to identify similar models revealed
that while there is a substantial amount of literature on
the metrics for measurement of R&D performance in
the project selection and project execution phases, comparatively few eorts are reported on the overall performance measurement. Meyer et al. [4] gives a summary of
28 references of R&D eectiveness measures proposed
in the literature, classied based on the method of measurement and the area of measurement. However, under
the full life cycle performance they could cite only seven
references in all. Among them, Cordero [5] proposes a
model of performance measurement in terms of outputs
and resources to be measured at dierent levels. Outputs are measured to determine whether they help
accomplish objectives (eectiveness) and resources are
measured to determine whether minimum amount of
resources are used in the production of outputs (eciency). The model proposes measurement of outputs
and resources at three levels, for technical performance
at the level of R&D units, for commercial performance
at the level of marketing, manufacturing etc., and for
the overall performance at the level of rm or strategic
business unit. The model further proposes the performance measurement at two stages viz., planning stage
and control stage. During the planning stage, these
measures are used to evaluate alternatives and to select
those that help the rm to accomplish its strategic
objectives. During the control stage, these measures are
used to compare the output and resources utilised with
those estimated during the planning stage. Cordero [5]
also discusses several possible measures that could be used
to evaluate technical, commercial and overall performance. The Integrated Performance Measurement framework proposed in this paper follows a similar approach
but diers in terms of the main focus being on the project
life cycle performance rather than the organizational

167

performance. The main advantage of the proposed

model is its ability to indicate the project performance
at any point during its lifecycle.
Foster et al. [6] presents the results of survey of R&D
directors and identies 13 activities that are believed to
have the greatest impact on R&D performance and
summarises 42 suggestions for improving the contribution of the technology function to corporate protability. Patterson [7] proposed a method of evaluating
R&D performance through auditing the economic contribution of completed developments. The contribution
of R&D projects was assessed in terms of cost reduction, sales advantage, sustaining of business activities,
technology sales, capital avoidance, capital expansion,
energy/raw materials, knowledge/feasibility of new concepts, environmental and safety related matters. The
results of this evaluation are used for setting the project
selection criteria and project priorities.
Porter [8] also proposes the post audit of the completed projects to estimate their contributions through
an economic indicator called the research index ratio.
Batson [9] proposes that information processing for
external communication, internal control and decision
support is an important determinant of R&D managerial performance and recommends to employ the
recently developed technology of decision support systems to match the information processing needs of R&D
Managers. In this model, the R&D process itself is
viewed as an information conversion process in which
the organization attempts to move from uncertainty in
technology and requirements to the certainty of an inproduction product. Krogh et al. [10] describes how 3M
evaluates its R&D programmes. The method essentially
involves regular audits of the programmes in terms of
technical factors such as overall technology strength, personnel, competitive factors, remaining R&D investment,
manufacturing implementation, probability of technical
success, business factors such as nancial potential,
competitive position and probability of marketing success and overall factors such as organization/planning,
stang, programme balance and coordination/interaction. Meyer et al. [4] addressed the important problem
of dening metrics for the product family in terms of
platform eciency, the degree to which a platform allows
economical generation of derivative products, and platform eectiveness, a degree to which the products based
on a product platform produce revenue for the rm
relative to the cost of developing these products.
Many more research eorts to dene various metrics
for the R&D performance measurement can be found in
the literature, however, the authors did not come across
any single model that can be used to measure the performance of the R&D project during all the phases of project lifecycle by linking the key factors from each phase.
The Integrated Performance Measurement framework
proposed in this paper attempts to bridge this gap.

168

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

4. What needs to be done to integrate various phases

of the project life cycle?
Probably the rst step towards integration of dierent
phases of the project lifecycle would be to broaden the
focus of project management to incorporate organisational perspectives and the dynamic nature of R&D
projects. The need for broadening of project management has been stressed by many authors in the project
management literature. From this the following three
important directions for broadening of project management can be observed.
4.1. Broadening of project management to encompass
the life cycle environment
Fangel [11] in his comment on the broadening of
project management highlighted the need for broadening of applications ``from a focus on the period from
contracting up until commissioning, towards handling
the entire lifecycle from concept to the full-scale utilisation of the project outcome'', broadening of concept
``from concentrating on managing quality, time and
resources within well dened frame, towards focusing on
the integration of the project performance, including
interaction with its environment'' and broadening of
method ``from the promotion of predetermined models
for project development and implementation, towards a
focus on how to select appropriate management methods, and how to design procedures that are appropriate
to the project situation''.
4.2. Broadening of project management to encompass
organisational environment and stakeholder needs and
expectations
Turner [12], while reviewing the project management
future developments for the short and medium term,
indicated the broadening of project management from
its traditional areas, as a medium term development. It
may be relevant to quote two of his important observations, viz., ``The majority of project managers nd that
what determines the success of their projects is not the
sequencing of a critical series of activities, but the
prioritising and sharing of resources across a portfolio
of multiprojects. . .'', and ``. . .the shareholders of a
company judge the success of a project by its ability to
create future cash ows. Time, budget and functionality
are key determinants of this, but not the main criteria''.
The implications of these observations are very
important. The rst observation shows that in a multiproject environment, project performance does not
depend on the project management alone. The second
observation shows that project success no longer
depends on completing it within the specied time and
cost. Then what is the role of project management? The

answer is given by the Project Management Body of

Knowledge guide [13], which denes project management
as ``the application of knowledge, skills, tools, and
techniques to project activities in order to meet or exceed
stakeholder needs and expectations from a project''.
4.3. Broadening of project management from being
static to a dynamic process
Gaynor [14] stressed the need to consider a project as
a living organism and suggests that the original
assumptions on which the project was approved must be
reviewed as a continual process. del Cano [15] stressed
the need to convert the present static feasibility study
concept, into a dynamic instrument so that it will be
useful as a tool for the correct assessment of the project
risks.
5. Requirements of a performance measurement system
on a broadened project management perspective
Traditionally, project performance was viewed as
synonymous with its ability to complete the project
within the specied time and cost. This resulted in the
use of PERT/CPM and `earned value' based performance measurement systems. However, with the broadening of project management, this view will change and
the new requirements of a performance measurement
system may be summarised as follows:
1. Reect the needs and expectations of all the stakeholders.
2. Continuously revalidate the assumptions made
during the past, in the light of the knowledge
gained about the project during the course of its
execution and consider the requirements for the
future success. This means an integrated approach
linking various phases of the project lifecycle.
3. Predict the project success/failure well in advance
to prevent further drain of resources.
The performance measurement system must be able
to provide some kind of a quantitative measure reecting the above requirements.
6. The proposed approach for Integrated Performance
Measurement
This paper proposes an approach for evolving an
integrated performance index that could adequately
reect the performance of the R&D project at any point
during its life cycle, by integrating the key factors from
each phase of the project life cycle.
The integrated performance index is derived in the
following three steps:

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

1. identication of important phases of project life

cycle;
2. identication of key factors in each phase; and
3. integrating of all the key factors into an integrated
performance index.
Thus the integrated performance index may be dened
as:
Zi fX1i . . . Xni
where Zi=integrated performance index of the ith project; X1i. . .Xni=key factors that dominate various phases of the project life cycle for ith project.
7. Conceptual model of the Integrated Performance
Measurement System
Fig. 2 shows the conceptual model of an R&D environment with an Integrated Performance Measurement
System. The organisation under consideration is an
R&D organisation operating in a multi-project and
concurrent engineering environment. From the organisational point of view, projects are considered as the
means for implementing organisational strategy for
growth. The three important phases of the projects are
identied as the project selection phase, project execution phase and the implementation phase. The project
proposals are generated on a continual basis, based on
stated or perceived customer needs, and projects are
selected from several competing proposals based on
screening and evaluation using well laid out criteria.
During execution, projects are managed by project

169

management teams and will share common resources

based on the priorities assigned by the organisation in a
dynamic manner. Completed projects will enter the
implementation phase for production, marketing and
sales and successful projects will enable the sponsor to
realize the benets expected from the project. The integrated performance measurement system will measure the
performance of the project in each phase in an integrated
manner to ensure the realization of benets expected
from the project at the time of its selection. Thus, the
integrated performance measurement system forms the
core of the management control, as shown in Fig. 2.
Based on this scenario we need to identify the key
factors that inuence the project performance and integrate them to derive the integrated performance index.
In real life there are innumerable factors that inuence a
project success/failure and it is very dicult to identify a
few key factors among them. Hence the model proposes
the identication of key factors by analysing the needs
and expectation of the stakeholders, as described in
succeeding paragraphs.
8. Needs and expectations of stakeholders
PMBOK Guide denes stakeholder as ``Individuals
and organisations who are involved in or may be aected by project activities''. From this point of view, we
can identify three important stakeholders, viz., the
sponsoring organisation, project management team and
the customer. What are the needs and expectations of
these stakeholders?

Fig. 2. Conceptual model of an R&D environment with integrated performance measurement system.

170

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

8.1. Needs and expectations of sponsoring organisation

The sponsoring organization views the project as an
investment opportunity and expects certain tangible and
intangible benets from the project. We may call this
expectation as ``merit''. The sponsoring organisation is
concerned about the possibility of a project being not
able to deliver the expected ``merit''. We may call this
the ``risk'' of the project. Further, in a multi-project
environment many types of projects may coexist. The
sponsoring organisation may have preference to certain
types of projects that may be more suitable to the organisational environment. This factor may be represented
by ``category bias''. Hence as a whole, we may represent
the needs and expectations of the sponsoring organisation
through three factors viz `merit', `risk' and `category bias'.
8.2. Needs and expectations of project management team
The project management team will be responsible for
attaining the project objectives within the specied time
and cost. This task is usually carried out by a project
planning and control mechanism which involves setting
a baseline plan, measuring the project status using some
metrics, comparison of the plan and the actual status,
and controlling the project activities to minimise the
variance between the planned and the actual status.
Hence, the primary concern of the project management
team would be to accurately measure the project `status'
and to be able to eectively control the deviations.
Project management models assume that once the
variance is known, it can be controlled through a managerial action such as additional resources, higher priorities, etc. This may not be true in the case of many
R&D projects due to the need for specialised resources/
equipment which are scarce. Further, in a multi-project
environment, resource management becomes very complex due to the sharing of common resources by several
projects. Here the control of resources and priorities
become more important and the performance depends
more on the concepts of portfolio thinking [16], programme management [17,18] and matrix management
[19]. As a result, the number of factors outside the control of project management increases. Under such conditions, the performance of a project depends on the
existence of appropriate management system at the
organisation level. We may represent this eect through
a `decision eectiveness' factor. Decision eectiveness
represents the capability of the management to take
right decisions at the right time in the face of multiple
options, the ability to anticipate problems well in
advance and prevent their occurrence, contingency
planning and problem solving, optimum deployment of
resources matching the requirements of the task, and
providing leadership and direction to all the eorts
towards the project/organisational goals.

Hence, the needs and expectations of the project

management team may be represented by two factors
`status' and `decision eectiveness'.
8.3. Needs and expectations of the customer
The customer, at least for the present and the time to
come, will remain the focus of all business activities, and
is the most important stakeholder. The needs and
expectations of the customer may be summarised by the
following from Robins [19] ``The customer is very interested in the company's reputation for technical excellence.
However, the customer is particularly uninterested in how
that technical excellence is achieved. Frankly, the customer could not care less which CAD system is used to
produce the drawings, or the perfection of the regression
analysis behind the statistics etc., ad innitum. The customer is concerned about three things and three things
only: Product - Will it work, will it be reliable, and will it
be ecient to use and maintain?, Schedule - When will
the delivery be made, and how condent can one be in
that date?, and Cost''. To be more precise, the customer
would be ``committed'' to a product only if it meets his/
her needs and expectations, is available in time, and its
performance is ``worth its cost''. To cater for these needs
and expectations, it is essential to integrate the customer
into the project right from the beginning stage and use
concurrent engineering philosophy to reduce the cost
and cycle time from design to marketing. The use of
concurrent engineering also strengthens the project management by shifting more activities towards the initial
phase of the project and introduction of production
preparedness into the product development phase. This
would eliminate design iterations by ensuring that the
product design is producible. We may measure this
through a factor called `production preparedness'. Hence
the needs and expectations of the customer may be
represented by three factors, viz., `customer commitment', `cost eectiveness' and `production preparedness'.
Thus as a whole, we can identify eight factors to be
considered for evolving the integrated performance
index for R&D projects in a multi-project concurrent
engineering environment, viz., merit, risk, category bias,
status, decision eectiveness, customer commitment,
cost eectiveness, and production preparedness.
The next step would be to identify the criteria to
compute each of these factors and their functional relationship with the integrated performance index.
9. Identication of criteria and functional relationships
9.1. Merit and the project performance
Merit of a project represents the expected benet of
the project to the organization. It is the very purpose of

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

the project. Hence, the performance should be zero

when the merit is zero. The expected benet from the
project as envisaged at the time of its sanction should
remain the same or preferably increase as we move from
the inception to the completion. This will give a relationship in such a way that as the merit of the project
grows or declines it would be directly reected in its
overall performance, as follows:
Z Merita  fother factors
Traditionally merit is represented through various
economic indices such as the net present value (NPV),
internal rate of return (IRR), ratio of maximum expenditure justied to the development cost, etc. However,
an R&D project merit cannot be represented by any
single measure. The merit of a project is usually evaluated with the help of a set of criteria and these criteria
are combined in variety of ways to obtain the overall
merit rank. Some of the possible project evaluation criteria for merit ranking of the projects can be project
attributes such as project desirability, expected utility,
strategic need, development time and cost, stage of
innovation, product life before obsolescence, potential
technical interactions with existing products, potential
market interactions with existing products, technology
spin-os and other non-tangible benets, Organizational Attributes such as competence and experience on
similar projects, knowledge/skills availability, research
sta availability, raw materials/components availability,
facilities availability, technology availability, degree of
internal competition for resources, etc., Environmental
attributes such as potential market size, expected market share, degree of competition, competitors eort in
similar area, Government policies, economic regulations, social ambiance, safety considerations, environmental considerations, etc.
During evaluation, projects will be given a score
against each criteria and the overall merit rank will be a
weighted sum of all these scores.
Merit rank for the project 1=n

n
X
Wi Si
i1

where n is the number of criteria used, Wi is the weight

of ith criteria and Si is the score of ith criteria.
9.2. Risk and the project performance
Risk is a quantitative representation of the uncertainty associated with the expected `merit' from the
project. As risk increases the project performance
decreases and when risk is 100% then the performance
will become zero. Based on this, risk may be related to
the project performance as follows:

Z 1

171

riskb  fother factors

Several methods of risk assessment have been reported in the literature [20,21]. As in the case of merit
ranking, for the risk ranking also there is no universally
acceptable measure evolved so far. Traditionally the risk
is expressed with the help of probability measures such
as the probability of technical success, probability of
commercial success, etc., and the overall risk is computed
through some formula. In most cases the risk assessment is not easily amenable for quantication. Keeping
these factors in view the new model proposes a risk
ranking methodology based on subjective estimates.
The sponsors are expected to provide a list of risk
parameters q1, q2,. . .,qn which in their view represents
the risk at dierent stages of the project. These parameters are selected in such a way that they are independent of each other and occur in a sequential manner
during the life cycle of the project. For example the
technical risk (the probability of not being able to meet
the technical specications), economic risk (probability
of not being able to produce the required quantity at the
required cost), commercial risk (probability of not being
able to attain the required sales volume) fall into this
category. Based on the perceived risk of each factor, a
score is assigned against each risk factor and the overall
risk rank of the project is computed as a product of
these scores. The overall risk of the project can be the
product of these individual risk factors.
n

Overall risk for a project  Si

i1

where Si is the score of ith risk factor.

9.3. Category and project performance
Project category has been identied as the key factor
based on the intuition that certain projects may be more
important than other projects and hence needs more
attention. This requires that a relationship be devised in
such a way that certain projects need to perform better
to give the same level of performance index. The relationship between the project category and the performance may be taken as follows:
Z 1 category Biasc  fother factors
Numerous attempts have been reported in the literature on the classication of the projects. The prominent
classication schemes of the projects include oensive and
defensive research, fundamental research, exploratory
development, advanced development, engineering development, operational systems development, management
and support related projects. A useful classication
system found in literature is the goals-and-methods

172

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

matrix proposed by Turner and Cochrane [22] which is

a 22 matrix classifying projects into four categories
namely: projects with goals and means clearly dened,
projects with goals clearly dened but means not clearly
dened, projects with goals not clearly dened but
means clearly dened and projects with goals and means
not clearly dened. The common practice the authors
have come across during their experience is to classify
the projects into two categories viz: product development projects originating from the requirements specied by the customers, usually from outside the
organization, and the competence build-up projects originating from the perceived needs of the organization to
develop core competence to meet the future challenges.
Projects may be categorised based on any of these
methods, and the sponsor may assign suitable value for
the `category bias' depending upon his/her perception of
importance of a particular category.

9.4. Status and the project performance

Traditionally the project status has been taken as
synonymous with the project performance. Numerous
ways of indicating a project status can be found in literature but the earned value approach has been the
most popular so far. However, a more suitable and
eective method has been suggested by Pillai and Rao
[23] based on a simple graphic tool to determine the
progress deviation (dp) and cost deviation (dc), as an
improvement over the earned value system. Fig. 3
depicts this graphic tool.
The graphic tool shown in Fig. 3 is a four-quadrant
graph with cost vs. time graphs (planned and actual)
plotted in quadrant 1 and progress vs. time graphs
(planned and actual) plotted in quadrant 3. Quadrant 2
represents the ow of time through the project lifecycle.
The cost vs. progress curves (planned and actual) in

Fig. 3. Graphic tool to determine progress and cost deviation of an R&D project (From Pillai and Rao [23]).

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

Quadrant 4 can be obtained by transforming corresponding points from Quadrants 1 and 3 (by projecting
the time intercepts of the progress and cost graphs into
Quadrant 4). From the cost vs. progress graphs, the deviations of the cost and progress can be directly measured.
The underlying principle of the four-quadrant graph
is better than the earned value analysis and provides an
integrated costprogresstime analysis. The earned value
analysis uses three elements, viz., the budgeted cost of
work scheduled (BCWS), the actual cost of work performed (ACWP), and the budgeted cost of work performed (BCWP) or the earned value. The cost variance
is given by BCWP
ACWP, which gives a true variance in cost. But the schedule variance is given by
BCWS BCWP which is an interpretation of the schedule variance in terms of cost, which roughly means
that a behind schedule condition requires a variance
amount of cost to get back to schedule. Accurate analysis of schedule condition in terms of time requires utilisation of network or milestone schedule.
In Fig. 3, P gives the ACWP and Q gives the BCWP.
Hence the cost variance computation using the four
quadrant graph is the same as with earned value analysis. R in Fig. 3 represents progress that should have
been made for the actual cost, that is the work scheduled for actual cost (WSAC), while the x co-ordinate of
point P gives the actual progress for actual cost. The
dierence between the x co-ordinates of R and P will
directly give the progress deviation. Further to this, additional schedule/cost information can be directly derived
from Quadrants 3 and 1 and the progresscost interrelationship is more clearly understood from this graph.
It is known that the delays and cost overruns have
adverse eect on the project and for the success of a
project both the progress and cost deviations should
remain as close to the zero as possible throughout the
project execution phase. Keeping this in view, the status
parameters may be related to the project performance in
the following manner:
Z 1=1 pd 1 ce  fother factors
9.5. Decision eectiveness and the project performance
Decision eectiveness represents the eectiveness of
the existing project management system in ensuring the
success of the project. The decision eectiveness will
have a positive inuence on the project performance
and the relationship with the project performance may
be specied as follows.
Z 1 decision effectivenessf  fother factors
Decision eectiveness can be computed as a weighted
sum of number of factors that contribute towards a

173

more eective decision making system.

Decision effectiveness 1=n

n
X
Wi Si
i1

where n is the number of factors used, Wi is the weight

of ith factor and Si is the score of ith factor.
The PMBOK guide provides nine ``Knowledge areas''
and associated ``project management processes''. The
existence of these processes and their eective use could
be a measure of decision eectiveness. In addition the
managerial system for resource sharing and priority
control in a multi-project environment and mechanisms
to cope with complexity of concurrency may also be
considered a measure of decision eectiveness. The
other factors may include leadership at various levels,
goal clarity, technical and managerial review systems,
partnership in decision making, innovative management
practices and special procedures.
9.6. Customer commitment and the project performance
Customer commitment is vital for the success of any
type of project. In the case of R&D projects where customer is known in advance, the customer should be
completely integrated into the project right from the
inception and the continued commitment of the customer must be assessed periodically. For market oriented
products, the customer commitment should be assessed
periodically through the market survey. All the projects
start with certain amount of stated or perceived customer commitment. For the success of the project, the
customer commitment should grow with time. This
relationships may be represented as follows.
Z 1 customer commitmentg  fother factors
The customer commitment varies for dierent projects from a mere awareness of the project to a total
commitment by the customer through placement of
bulk production orders. The authors came across a
number of projects where the end products could not be
marketed in spite of their technical success mainly due
to the lack of customer commitment. At the same time
authors are involved in at least two major projects where
the customer expressed total commitment by placement
of production orders during the development phase and
became full-edged partner in the whole development
process. In these two projects the co-operation between
the customer and the development organization extended to the extent of joint nalization of requirements,
positioning of customers' teams to assist the development team, customer associated development trials,
joint nalization and conduct of trials, joint evaluation
of system performance, customer training during the

174

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

development phase, preparation of operational and

maintenance documentation jointly by the customer and
development agencies, issue of bulk production clearance
concurrent to the customer evaluation trials, etc. This
approach not only helped in elding of the systems at the
shortest possible time but also improved the cost eectiveness of the system by minimizing the additional
design cycles and resulting wastage of resources.
From the above experience it is possible to identify
some of the factors that represent the customer commitment for the success of the project. These factors
may include origination of project requirement from the
customer, agreement on the product requirements
between the customer and the development agency,
involvement of customer in project decision-making,
transparency and the exchange of information between
the customer and development agency, joint funding of
the project by the customer, placement of production
orders by the customer during development phase,
availability of the customer teams at development
agency, joint preparation of customer documentation,
joint training of operational and maintenance crew, etc.
Based on the identication of factors as given earlier,
it is possible to estimate the customer commitment as a
weighted sum using the following formula.

The cost eectiveness undergoes drastic changes during

the life cycle of an R&D project due to many factors.
Hence, there is a need to continuously monitor this factor.
This is possible only through continuous monitoring of
the product's cost, development of equivalent systems by
competitors, their comparative performance and cost.
The factors that may be considered while assessing the
cost eectiveness may include product's technical performance, development cost and product cost, life cycle
cost including training, operation and maintenance cost,
cost advantage for the derivative systems, operational
aspects, strategic advantage, multiple applications, etc.
The cost eectiveness can be computed as a weighted
sum of the above factors.
Index of cost effectiveness 1=n

n
X
Wi S i
i1

where n is the number of factors used, Wi is the weight

of ith factor and Si is the score of ith factor.
9.8. Production preparedness and project performance

where n is the number of factors used, Wi is the weight

of ith factor and Si is the score of ith factor.

The projects in a concurrent engineering environment

dier from the conventional projects mainly in terms of
the growth in the production preparedness during the
product development phase of the project. Production
preparedness has a positive inuence on the project
performance and the relationship between the project
performance and production preparedness can be
derived as follows:

9.7. Cost eectiveness and project performance

Z 1 production preparednessJ  fother factors

Index of customer commitment 1=n

n
X
Wi S i
i1

Cost eectiveness is another vital characteristic for

the success of any project. The product cost eectiveness depends on a series of technical decisions taken
during the project execution. It is surprising to see that,
to date there is no model that can monitor and integrate
the product cost eectiveness into the project performance in an explicit manner. The PMBOK Knowledge
area on ``Project cost management'' states that ``project
cost management is primarily concerned with the cost of
the resources needed to complete project activities.
However, project cost management should also consider
the eect of project decisions on the cost of the project's
product''. Hence, the present paper proposes the concept of ``concurrent costing'' wherein the product cost is
continuously estimated throughout the project phase
and with the help of this, the product cost eectiveness
to meet the intended purpose is estimated. The relationship between the cost eectiveness and the project
performance may be dened as follows;
Z 1 cost effectivenessh  fother factors

The production preparedness can be computed as a

weighted sum of number of factors that indicate the
preparedness for production of systems from the
designs and technologies developed.
Production preparedness 1=n

n
X
Wi S i
i1

where n is the number of factors used, Wi is the weight

of ith factor and Si is the score of ith factor.
The factors to be considered may include identication
of production agencies, formation of multifunctional
teams, institution of mechanisms for technology transfer, development of common documentation formats
for ow of designs and technology, preparation of production agencies for faster technology absorption, planning and setting up of production facilities, establishment
of quality assurance and quality control systems,
mechanisms for pre-production reviews, organizational

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

couplings for faster decision making, cost control

mechanisms, etc.
10. Computation of integrated performance index
Based on the above relationships the integrated performance index may be computed as follows:
x1i a 1 x2i b 1 x3i c 1 x6i f 
1 x7i g 1 x8i h 1 x9i J
Zi
2cfghj 1 x4i d 1 x5i e
where Zi=integrated Performance Index of project i;
x1i=normalized variable representing the merit of project i; x2i=normalized variable representing the risk of
project i; x3i=normalized variable representing the
category bias of project i; x4i=normalized variable
representing the progress deviation of project i;
x5i=normalized variable representing the cost deviation
of project i; x6i=normalized variable representing the
decision eectiveness of project i; x7i=normalized variable representing the customer commitment of project i;
x8i=normalized variable representing the cost eectiveness of project i; x9i=normalized variable representing
the production preparedness of project i.
The values of x1 to x9 may vary between 0 and 1. The
coecients a,b,c,d,e,f,g,h,j represents the relative

175

importance of the various factors and are assigned by

the sponsor based on his/her perceived importance of
various factors. The value of these coecients may vary
between 0 and 1. The factor 2(c+f+g+h+j) is introduced
in the denominator to normalise the value of Z between
0 to 1. When a factor is not applicable, the corresponding coecient is made zero in both the numerator and
denominator. In practice the relative importance of
various factors change during various phases of the
project life cycle. To cater for this requirement the value
of coecients may be varied as a function of time.
11. Application of integrated performance index
Fig. 4 summarizes the application of IPI during the
life cycle of an R&D project. The integrated performance index can be applied in all the phases of the
project lifecycle by adjusting the coecients of dierent
factors. During project selection phase, the coecients
d,e,f and j may be set to 0, and the IPI can be used to
rank the projects. During the execution phase, the IPI
can be used to set a performance base line for the projects and to compare the performance of the projects
from the base line. The projects under execution can be
compared with the proposals waiting for investment. At
the completion of the project, the IPI data can be used
as an input for the management of future projects. The

Fig. 4. Validation of integrated performance index.

176

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

Fig. 5. Application of integrated performance index.

IPI ensures that the focus of performance measurement

is on the overall performance of the project as a whole,
keeping the business objectives in view.
12. Validation of the integrated performance index
The IPI model has been validated using data from
two real life projects, the rst one after its completion,
and the second one during its execution. The trends
observed in the values of IPI during these project lifecycles
are shown in Fig. 5. Project A was a high technology
project involving development of many unknown technologies. The project started with an optimistic estimate
and the actual execution took more time and cost. This
is reected in the continuous reduction of the IPI during
the project lifecycle. The objective of the project was the
development and demonstration of certain technologies
for the future projects. This objective was accomplished.
Project B is an operational system developed against
specic requirements of the customer. Right from its
inception, the project was managed using an integrated
performance evaluation philosophy. This is one of the
reasons for its better performance. The drop in IPI
during the lifecycle is due to the time and cost overruns.
It can be seen from Fig. 5 that IPI gives an indication
of project performance during its lifecycle. Using this
information, it is possible to nd the exact problem by
going through the behaviour of various factors. Thus,
IPI alerts the management about any problem in the
project and triggers an in depth analysis of the problem.
13. Advantages of integrated performance index
The integrated performance index links the project
selection, execution and implementation phases in a
unique manner. The use of integrated performance
index for the project selection as well as its performance
measurement during execution phase will revalidate the
assumptions made during the selection phase in the light
of knowledge gained during the project execution.

At the time of project selection the risk estimated will

be based on a number of assumptions about the technological and managerial inputs to the project during its
execution. These inputs may not materialise during the
actual execution. Incorporation of risk factor into the
integrated performance index ensures continuous evaluation of risk and thus `assumption surfacing', which
greatly helps the timely decision making. Similarly the
integration of the decision eectiveness factor into the
integrated performance index helps in surfacing the
hitherto unknown managerial problems that have crept
in during the project execution and provides an early
warning about the project failure, so that the project
can be terminated early to prevent the further drain of
resources.
One of the greatest problems of the management
models is the poor acceptance by the practising managers, due to the practical diculties in using these
models. The IPI is very easy to use and close to the
intuition and mental model of the practising managers.
By linking the individual project performance to the
organisational goals, the IPI enables the sponsors to
articulate their vision and goals in a more precise manner. The IPI ensures a balance between dierent types of
projects through the category bias.
R&D projects always suer due to creeping scope and
multiple design iterations. Concurrent engineering was
evolved to reduce the design iterations arising out of the
productionisation requirements. However, there is no
such solution for user originated design iterations. The
IPI provides for continuous and simultaneous monitoring of the production preparedness as well as the user
commitment which will drastically reduce the design
iterations. This is a unique feature of the IPI, where the
usually neglected vital factors are integrated into the
performance indicator.
Cost eectiveness is a much talked about concept in
the modern literature. However, the authors did not
come across any eort to monitor the product cost
growth and cost eectiveness by any of the available
performance metrics. The focus has always been on the
development cost and in the process the more important
product cost growth is neglected. The new model recties this situation by proposing the concept of `concurrent costing' and by integrating the cost eectiveness
into the integrated performance index.
14. Conclusion
The concept of IPI has evolved based on real life
experience of the authors with number of large R&D
projects. Even though a number of publications appear
on the critical success factors for project success, this is
probably the rst time that an attempt has been made to
quantify these factors and integrate them into a project

A.S. Pillai et al. / International Journal of Project Management 20 (2002) 165177

performance indicator. The authors believe that the

application of IPI will help the R&D organizations to
monitor the performance of R&D projects in a manner
that ensures better resource utilization and greater contribution of the projects towards organizational goals.

[11]
[12]
[13]

References
[1] Chang Zeph Yun, Yong Kit Chin. Dimensions and indices for
performance evaluation of a product development project. International Journal of Technology Management 1991;6(1/2):15567.
[2] Schmidt RL, Freeland JR. Recent progress in modelling R&D
project selection process. IEEE Transactions on Engineering
Management 1992;39(2):189201.
[3] Martino JP. R&D project selection. USA: John Wiley, 1995.
[4] Meyer MH, Tertzakian P, Utterback JM. Metrics for managing
research and development in the context of the product family.
Management Science 1997;43(1):88111.
[5] Cordero R. The measurement of innovation performance in the
rm: an overview. Research Policy 1990;19:18592.
[6] Foster RN, Linden LH, Whiteley RL, Kantrow AM. Improving
the return on R&DII. Research Technology Management
1985;MarchApril:1322.
[7] Patterson WC. Evaluating R&D performance at Alcoa Laboratories. Research Management 1983;MarchApril:237.
[8] Porter Jr. JG. Post audits an aid to research planning.
Research Management:2830
[9] Batson RG. Characteristics of R&D management which inuence information needs. IEEE Transactions on Engineering
Management 1987;EM-34(3):17883.
[10] Krogh LC, Prager JH, Sorensen DP, Tomlinson JD. How 3M

[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]

177

evaluates its R&D programs. Research Technology Management

1988;NovemberDecember:1013.
Fangel M. The broadening of project management. International
Journal of Project Management 1993;11(9):72.
Turner JR. Project management: future developments for the
short and medium term. International Journal of Project Management 1994;12(1):34.
A guide to The Project Management Body of Knowledge, Project
Management Institute, USA, 1996.
Gaynor GH. Monitoring projects It's more than reading reports.
Research Technology Management 1996;MarchApril:457.
del Cano A. Continuous project feasibility study and continuous
project risk assessment. International Journal of Project Management 1992;10(3):16570.
Platie A, Seidel H. Breakthrough in multi project management:
how to escape the vicious circle of planning and control. International Journal of Project Management 1993;11(4):20913.
Turner JR, Speiser A. Programme management and its information systems requirements. International Journal of Project
Management 1992;10(4):196206.
Payne JH. Management of multiple simultaneous projects: a
state-of-the-art review. International Journal of Project Management 1995;13(3):1638.
Robins MJ. Eective project management in a matrix-management. 1993;11,1:1114.
Berry J, Townsend PRF. Risk management. International Journal of Project Management 1993;11(4):2018.
Halman J, Keizer JA. Diagnosing risks in product-innovation projects. International Journal of Project Management 1994;12(2):7580.
Turner JR, Cochrane RA. Goals-and-methods matrix: coping with
projects with ill dened goals and/or methods of achieving them.
International Journal of Project Management 1993;11(2):93102.
Pillai AS, Rao KS. Performance monitoring in R&D projects.
R&D Management 1996;26(1):5766.