sustainability

Article
Identification of Key Performance Indicators in
Project-Based Organisations through the
Lean Approach
Carolina Cruz Villazón , Leonardo Sastoque Pinilla * , José Ramón Otegi Olaso ,
Nerea Toledo Gandarias and Norberto López de Lacalle
Faculty of Engineering, University of the Basque Country, 48940 Bilbao, Spain; [email protected] (C.C.V.);
[email protected] (J.R.O.O.); [email protected] (N.T.G.); [email protected] (N.L.d.L.)
* Correspondence: [email protected]




Received: 30 June 2020; Accepted: 22 July 2020; Published: 24 July 2020


Abstract: For the time being, companies and organisations are being forced to compete in utterly
complex and globalised environments, facing massive natural, economic, and technological challenges
on a daily basis. Addressing these challenges would be impossible without a proper approach that
helps them identify, measure, understand, and control the performance of their organisations.
Lean principles and techniques rise as a solution. This paper justifies and proposes the use of lean
principles and techniques to identify key performance indicators (KPIs) in project-based organisations
based on their organisational and operational needs. The research focuses mainly on the identification
and categorisation of KPIs through a qualitative approach, based on systematic literature review (SLR)
of performance indicators, project management, and project success. As a case study, an analysis of
relevant information of an R&D and innovation project-based organisation, such as quality manuals,
a benchmarking process, internal studies, and surveys regarding what success means for different
kinds of stakeholders and for the organisation itself was conducted. As a result, this research is of a
high value for project-based organisations, especially those that are not apprised of how to correctly
formulate a series of KPIs, or whose path to it is still not clear.

Keywords: lean; key performance indicators; DMAIC; CTQ; project success; project-based organisations;
technology readiness level

1. Introduction
Nowadays, organisations are forced to compete in utterly complex and globalised environments,
facing massive natural, economic, and technological challenges on a daily basis. Addressing these
challenges would be impossible without noticeable management of the performance of their business.
It is essential to every business, no matter the size, scope, or resources to identify, measure,
understand, and control the progress of that performance [1,2]. Measurement systems are needed to set
organisational goals and to control the improvements by monitoring the effectiveness and efficiency [3].
A common approach to carry out these measurements is through the use of KPIs metrics. KPIs provide
an objective criterion for measuring business activities and project success [4] and are a remarkably
important part of corporate strategy for forecasting, measuring and planning business [2].
However, it should be noted that performance metrics vary in their purpose, definition, and content.
Therefore, different methodologies are used to define and select the business KPI’s, to make sure that
they match the competitive environment and strategy. According to Iuga et al. [2], three important
criteria need to be taken into account for the optimal selection: Validity, helpfulness, and relevance;
and to ensure that the measurement system is under control a disciplined methodology is needed.

Sustainability 2020, 12, 5977; doi:10.3390/su12155977 www.mdpi.com/journal/sustainability

Sustainability 2020, 12, 5977 2 of 18

The lean approach has been explored for defining, measuring, and monitoring performance.
The methodology focuses on continuous process performance improvement and enables decision
making based on real facts and data analysis, information and objective evidence gathered through
quantification and estimation methods [2,5]. The lean concept is characterised by managing the
efficiency and effectiveness of the organisation, by putting the emphasis on customer value and
waste reduction.
So, lean consists of a set of principles and tools that have been implemented in a broader range of
industries [6,7]. Although its origin was within manufacturing, it has also been applied in non-industrial
organisations such as services [8], banking, or healthcare [9]. Also, lean focus has expanded in a broad
range of disciplines like sales, product development [8], project management (PM), and Research and
Development (R&D) [6,10]. Although R&D is a new concept, some studies on lean R&D have been
published, especially in the field of healthcare [6,7,11].
Despite the fact that the mentioned concepts have been studied, a lack of homogeneous process
for identification and measure of KPIs was detected. Although the adoption of KPIs in order to achieve
objectives within the production environment is widely spread [8], there is scarce research published
on developing a model to identify the KPIs in project-based companies.
Literature concerning KPI’s in a project-based organization context is generally related to project
success [4,12,13]. The measures on which the success or failure of a project is judged are the success
criteria and the KPIs are the factors that constitute those success criteria [4]. Project management KPIs
are crucial as they enable the progress of projects to be monitored. It must be ensured that KPIs are
aligned with the organisation’s strategies, that the perspectives of all stakeholders are considered
and that short- and long-term benefits are covered [12]. Regarding the relationship between project
management performance and project success in project-based organisations the KPI’s of the project
management are the most significant variables for the success of a project [12].
Measuring R&D and innovation (R&D&I) project performance and if a project is successful or not
in project-based companies has become a fundamental concern for managers and executives in the last
decades. As a result, the issue has been extensively debated in the literature. However, determining
whether an R&D&I project is successful is a subtle matter [14] and a challenging task. If someone is
able to reach the top performance on your project, it does not guarantee that particular project will
be successful.
R&D&I projects are complex per se, with several dependent phases that makes it even harder
to determine project success factors criteria. It is clear that projects usually have multi-dimensions,
and that different people involved in the project have different priorities. Therefore, not only should
several dimensions for assessing the project success be considered, but also the fact that project
success may vary over time based on different people’s interest [15]. Davis [16] stated that different
project success dimensions (PSDs), such as time, mission and objective, project manager competencies,
strategic benefits, and top management support, have different importance for different stakeholders.
This paper justifies and proposes the use of lean principles and techniques to identify KPIs in
project-based organisations. A qualitative approach, based on SRL, was adopted, which was used
to analyse and compare research about project success and the use of lean for the identification of
performance metrics. Using the SLR output, a lean-based KPI model is proposed with the aim of being a
guide for the identification of performance indicators in project-based organisations. For the elaboration
of this model, Lean Six Sigma process so-called DMAIC (define–measure–analyse–improve–control)
as a cycle-based approach [17] was followed and adapted. Later, the model was validated through a
case study in a R&D project-based organisation. Since the focus of this paper is on the identification
and categorisation of KPIs, the validation, adequacy, and control of KPIs are recommended as
future research.
This research will answer practical questions about how lean tools and principles lead to KPIs
identification and which are the benefits from it. Additionally, it will be shown how a model for the
identification of KPIs developed with lean will benefit project-based organisations.
Sustainability 2020, 12, 5977 3 of 18

In Section 2, we will expand the research methodology and how it leads us to the Literature
Review (Section 3). Next, in Section 4, the definition and description of the model will be discussed,
followed by the description of the Case study in Section 5. The results of the utilisation of the model
will be shown in Section 6, and general conclusions and future research will be given in Section 7.

2. Research Methodology
To identify and examine the current state of the art of the research topic an SLR was followed.
The process starts with the description of the research questions stated in the Introduction. Then,
databases to be used to search the publications were chosen. To fill the keywords in the databases,
Boolean “AND” operator was used to combine the keywords and to focus the results of the search.
The range of years was not specified in order not to limit the number of publications. Nevertheless,
it was observed that most of the research done on the subjects in question was published during the
previous 20 years.
In the third step of the SLR, the documents identified were selected according to the exclusion
and inclusion criteria. The central focus of the study was the relationship between project success, lean,
and performance indicators in project-based context. In the scope, it was included subjects such as
the use of lean for the identification of performance indicators; the use of lean tools and principles
for KPI development; project success criteria; and KPIs in project-based companies. This last subject
was selected to establish a context for the case study. Different types of documents were sought
for distinct parts of the investigation. The SLR process continued with the fourth step of analysing
the publications through thematic analysis and synthesising the information gathered. The last step
consisted of reporting and using the findings, which will be discussed in the Literature Review section.

3. Literature Review Discussion

Lean is defined in many ways, some authors call it methodology, others philosophy [18], yet there
is a common consensus on its approach. In the literature, lean is explored from two broad ways:
Strategic and operational. The first is associated with lean principles and goals, while the second refers
to practical aspects related to its implementation, tools and techniques [5].
Lean is based on five fundamental principles: Defining value from the customer perspective,
identifying the value streams, making the value flow, implementing pull-based production, and striving
for perfection continuously [5]. It also focuses on waste elimination by improving process performances
and value creation [5,8,18]. The term waste in lean context is defined as an event or process that
does not reflect customers value or generate any added value to the final output [2]. Seven types of
waste are usually described in lean literature: Transport, inventory, motion, waiting, over-processing,
overproduction, and defects [19]. Other researchers include an eighth kind of waste, the unused
people’s creativity [2]. Cherrafi et al. [19] highlighted another concept to be considered, which is
sustainability (economic, social, and environmental concepts) due to an increase in the environmental
and social awareness.
The lean approach to customer value and the elimination of waste leads to the identification
of an organisation performance indicators. Waste has a direct impact on performance [19]; hence,
some authors have researched on the definition of KPIs based on these types of waste. Iuga et al. [2]
explored KPIs selection criteria based on the same waste categories. The link between KPIs and the
lean waste concept enables a wider perspective on the performance assessment analysis [18].

3.1. Lean in Project-Based Organisations/R&D

Even though lean in R&D is a rather new approach [6], there are studies from years ago that
have explored the use of lean for performance improvement within this discipline. Marti [11] explains
how Lean Six Sigma leads to a better understanding of crucial customer requirements providing
more value to services and to focus on improving critical areas of the R&D process. On the other
hand, recent studies have also shown a positive impact in the implementation of lean within this
Sustainability 2020, 12, 5977 4 of 18

area. Foruhi et al. [10] demonstrate how lean principles and tools in R&D organisations can improve
their key skills by increasing efficiency and reducing waste and therefore costs. Al et al. [6] developed
a model using lean to map and improve the functions of R&D project activities. Foruhi et al. [10]
identified and determined the customer value as the main focus of lean concepts and how can be
applied to all disciplines including R&D. Hence, Panat et al. [20], through a case study, demonstrated
the benefits of using lean combined with Six Sigma methodology in the infrastructure and operations
of the R&D organisation.
Lean can be used in conjunction with other improvement methodologies such as Six Sigma.
Lean Six Sigma has been studied as a business strategy and methodology to measure and improve
operational performance [11]. By integrating the tools, techniques, and principles of both methodologies,
it achieves to eliminate defects in processes and improves process performance focusing on customer
value [11,19]. Lean Six Sigma is also used on projects with the aim of improving the process through
workflow creation and elimination of variation [21].

3.2. The DMAIC (Define, Measure, Analyse, Improve, Control) Methodology

The DMAIC is a Lean Six Sigma method consisted of five process phases: Define, measure, analyse,
improve, control. In this study, the central idea behind DMAIC process is used for the creation of a
lean KPI model by remaining focused on customer value.

3.2.1. Define Phase

In this phase, the scope of the project is aligned with the organisation strategy to detect in which
aspects the performance meets or not the customer needs [11]. One of the lean tools for identifying the
needs and requirements is the voice of the customer (VoC) [22]. The information captured with the
VoC can be used to identify performance indicators.
Through VoC, critical requirements of the client and what he considers as value can be identified.
By collecting the customer’s needs, the information can be structured in a hierarchical way prioritized
in terms of relative importance and customer perceptions of performance [22,23]. In addition to the
VoC, there is the voice of the business (VoB), which concerns what the organisation aims to achieve [17].

3.2.2. Measure and Analyse Phase

During the measure phase, the VoC and VoB specifications are translated into measurable and
controllable factors (quantitative data) through the critical to quality (CTQ) tool. The CTQs are specific
quantifiable metrics that are linked to the organisation objectives [24]. This tool displays the customer’s
expectations towards the quality of a product. The CTQs are represented with a flow-down tree and
when applied for performance indicators, some authors refer to them as CTQ [25]. Figure 1 shows the
generic representation methodology or a project where there is an input (e.g., requirements, statements)
provided by a stakeholder (e.g., supplier, customer), where there is a process to deliver an output [13].
In Lean Six Sigma, there is a similar mapping of a flow process known as SIPOC (supplier, input,
process, output, customer) [20]. This diagram also works as a guide to identify metrics as KPIs. In CTQ
context, the outputs are represented by Ys and the factors that impact them, and the inputs by Xs [21].
The CTQs are considered as performance indicators [21]. KPIs, within the VoC, correspond to
the CTQ characteristics, which are a set of indicators with clear targets and specified limits [17].
Yang et al. [25] specified the CTQ-Y as the KPIs of the CTQ. The specifications of the CTQs are the
measures of the dependent variable (Y) and the Xs are the key variables or drivers. The factors affecting
the CTQ can be represented by the Equation (1).

Y = f(X) (1)

The current state of the CTQ is specified and the performance measures or key variables (Xs) are
searched. In this phase, the potential Xs can be identified through a value stream map (VSM) [21].
Sustainability 2020, 12, 5977 5 of 18

VSM is a process flow chart that identifies the added-value and non-value-added activities in a stream
product transformation process [5]. The VSM reveals hidden issues in the process, brings options to
the surface, and enables the potential to maximise performance by eliminating the waste.
After defining and understanding the process, and having specified and documented the
performance measurements, we proceed to the analysis phase. In this phase, the critical factors directly
related to the Ys are established. The analyses of the data and process activities allow the detection of
the main factors
Sustainability that
2020, 12, have
x FOR anREVIEW
PEER impact on quality from the customer’s perspective (value-added) [11].
5 of 19

Figure 1. Supplier, input, process, output, customer (SIPOC)—process representation (based on [21]).
Figure 1. Supplier, input, process, output, customer (SIPOC)—process representation (based on
3.2.3. Improve and Control Phase [21]).

The
Theimprove
CTQs are phase is for implementing
considered as performance the proposed
indicatorsmethods andwithin
[21]. KPIs, improvements
the VoC,[18]. To control
correspond to
and monitor the progress, Cortes et al. [18] recommended the use of a web application (e.g., customised
the CTQ characteristics, which are a set of indicators with clear targets and specified limits [17]. Yang
dashboard) in orderthe
et al. [25] specified to access
CTQ-Ythe required
as the KPIs ofstandards
the CTQ.and
Theproject management
specifications of thetools.
CTQsSince
are thethemeasures
focus of
this paper is on the identification and categorisation of KPIs, the validation, adequacy,
of the dependent variable (Y) and the Xs are the key variables or drivers. The factors affecting theand control of
KPIs are recommended as future research
CTQ can be represented by the Equation (1). in the Conclusions section.

3.3. Key Performance Indicators (KPIs) Y = f(X) (1)

KPIs are metrics
The current state ofused
the by
CTQ some organisations
is specified and theto track the success
performance measures and or guide their progress
key variables (Xs) are
towards specific strategic objectives. In addition to business strategy, it is important
searched. In this phase, the potential Xs can be identified through a value stream map (VSM) [21]. to consider the
corporate
VSM is aculture
process since
flowthere is athat
chart direct connection
identifies thebetween the organizational
added-value and non-value-addedculture andactivities
performance.
in a
A solid corporate culture drives the performance and each specific feature
stream product transformation process [5]. The VSM reveals hidden issues in the process, brings impacts the strategy’s
implementation [26]. Popa
options to the surface, [26] highlights
and enables that intoorder
the potential to develop
maximise KPIs, four
performance main factors
by eliminating themust be
waste.
takenAfter
into account.
definingFirst,
and ensuring that activities
understanding are in line
the process, andwith the objectives;
having specified and second, collecting the
documented the
required information to improve the activities; third, controlling and monitoring
performance measurements, we proceed to the analysis phase. In this phase, the critical factors the activities and the
people
directlyinvolved;
related to supporting
the Ys arethe reports forThe
established. the stakeholders.
analyses of the data and process activities allow the
In a project
detection of thecontext, the KPIs
main factors are
that linked
have an to the project
impact success
on quality or failure
from [13]. Theseperspective
the customer’s indicators have to
(value-
be measurable
added) [11]. and controllable therefore must be quantitative and qualitative [4,13]. All organisations
are different and have their own needs and objectives, so KPIs must be tailored to each one; nevertheless,
a3.2.3. Improve
general and Control
framework Phase as a guide [4]. Kerzner [13] stated that a KPI’s most crucial attribute
can be useful
is that it is actionable, meaning that actions can be taken to correct any unfavourable trends.
The improve phase is for implementing the proposed methods and improvements [18]. To
Within the literature review, several descriptions of KPIs categories were detected. Cortes et al. [18]
control and monitor the progress, Cortes et al. [18] recommended the use of a web application (e.g.,
cite five strategic KPIs categories: Cost, quality, flexibility, stock, and lead time. With these categories,
customised dashboard) in order to access the required standards and project management tools. Since
they aimed to capture the company strategic goals and enable the alignment of strategic, tactical,
the focus of this paper is on the identification and categorisation of KPIs, the validation, adequacy,
and operational performance. Ogunlana [4] identified several authors that included other performance
and control of KPIs are recommended as future research in the Conclusions section.
measurements in addition to the classic iron triangle (time, cost, and quality): Customer satisfaction
and
3.3. overall satisfaction
Key Performance of stakeholders.
Indicators (KPIs)
Within projects, Kerzner [13] identifies time, cost, resources, scope, quality, and actions as
KPIs arefor
core metrics metrics
projectused by some organisations
management to track
KPIs. Additionally, the success
Ogunlana [4] and guide
delves intotheir progress
performance
towards specific strategic objectives. In addition to business strategy, it is important
measurements adding the capacity of the project team to manage project risks and solve problems found to consider the
corporate culture since there is a direct connection between the organizational culture and
performance. A solid corporate culture drives the performance and each specific feature impacts the
strategy’s implementation [26]. Popa [26] highlights that in order to develop KPIs, four main factors
must be taken into account. First, ensuring that activities are in line with the objectives; second,
collecting the required information to improve the activities; third, controlling and monitoring the
Sustainability 2020, 12, 5977 6 of 18

in projects in order to evaluate the success of the project. The authors also mentioned other research that
suggested measuring project success by the technical performance efficiency of execution, managerial
and organisational implications, personal growth, manufacturers’ ability, and business performance.
Furthermore, España et al. [1] argues that conventional metrics such as cost, schedule, quality,
and security should be used to support the system improvement and not as isolated parameters
that request an individual response. The authors explain that evaluating together cost and schedule
indicates whether the system is stable or corrective actions are needed, while the cost in conjunction
with safety parameters suggest if the work is planned and performed correctly [1].
Ogunlana [4] highlight as future research to focus on the integration of the organisations KPIs
with the following aspects: “operational (time, cost and quality), life cycle (maintenance capacity, energy
consumption and user satisfaction), strategic (inter-organisational co-operation, organisational learning)
and socio-economic (social and human development)”. Moreover, the authors assured that the criteria
for measuring the success of the projects should be based on strategy, sustainability, and safety.
Yang et al. [25] describe two categories of KPIs: Financial (e.g., increased sales and decreased material,
inventory, and transport costs) and operational (e.g., cycle time, utilisation rate, delivery time,
forecast accuracy). All the categorisation mentioned by the authors is summarised on Table 1.
These categories are an example of the categories that may be considered within an organisation.

Table 1. Key performance indicators (KPIs) categories list.

KPIs Categories Indicators Sources

Increases in sales [25]
Decreases in material [25]
Financial
Inventory [25]
Transportation expenses [25]
Cost [18]
Quality [18]
Flexibility [18]
Strategic
Stock [18]
Inter-organisational cooperation [4]
Organisational learning [4]
Tactic [18]
Cycle time [25]
Utilisation rate [25]
Lead time [25]
Forecast accuracy [25]
Time [4]
Cost [4]
Quality [4]
Customer satisfaction [4]
Overall satisfaction of stakeholders [4]
Project team’s ability to manage project risks [4]
Project (operational)
Ability to resolve project problems [4]
Efficiency of execution [4]
Managerial and organisational implications [4]
Personal growth [4]
Manufacturer’s ability [4]
Business performance [4]
Maintenance capacity [4]
Life cycle Energy consumption [4]
User satisfaction [4]
Safety [4]
Sustainability (socio-economic aspect) Social and human development in the area [4]

Cortes et al. [18] proposed a Lean Six-Sigma framework based on lean indicators for management
support during lean implementation intending to lead tactical and operational decisions for performance
Sustainability 2020, 12, 5977 7 of 18

improvement and maintenance. Lean tools such as the “five whys” and the root cause and effect
analysis can be used for establishing performance indicators and for identifying improvement metrics.
The work of Dombrowski et al. [8] proposed specific criteria of a measurement system for
performance indicators in product development context. The criteria to be taken into account consists
in: Relevance for the enterprise targets, quality data (based on the validity and timeliness of data),
compatibility with the hierarchy, variability (react quickly to changes), periodicity, visualisation,
and effort. Furthermore, Kerzner [13] describes six fundamental characteristics for project-oriented KPIs:
Predictive (future), measurable (quantitative), actionable (changes to correct), relevant (relationship to
project success/failure), automated (reports minimise human error), and few in number (those needed).
Table 2 shows the characteristics that the KPIs should accomplish according to the literature analysed.

Table 2. KPI characteristics.

Dombrowski et al. [8] Iuga et al. [2] Kerzner [13] Toor & Ogunlana [4]
Actionable X
Automated X
Compatibility (hierarchy) X X
Effort X
Few in number X
Helpful X
Measurable X X
Objective X
Periodicity X
Predictive X
Relevant X X X
Timeliness X
Valid X X
Variability (react quickly to changes) X
Visualisation X

3.4. Project Success

As has been said, project success as the heart of project management, and the factors that
affect it, is a commonly discussed topic in research in project management that has been studied by
practitioners and academics since 1960s (it started with the definition of success in terms of the iron
triangle, time, cost and quality [27]; and remains relevant to the present day); however, there is not a
unified definition [15]. Moreover, we can distinguish two different components: Project success factors:
Elements of a project that if influenced, increase the like hood of success (e.g., stakeholder, risk and
quality management, etc.) and project success criteria: Measures used to judge on the success or failure
of a project (e.g., stakeholder satisfaction, cost, scope, time, etc.) [28].
The first impression of project success is a project implemented among the constraints of time, cost,
and quality; however, project success is more than implementing the project within this iron triangle.
Constrains and exceeds from those boundaries do not mean that the project was or not successful.
Public opinion considers the Sydney Opera House as a successful project, even if it was 14 times over
budget and time [29]. Some researchers carried on upon that project and other similar ones led to
highlighting two main issues. First of all, the differences in the perception of project success among
different stakeholders, and secondly, the fact that project success is beyond controlling the iron triangle
constraints and that more dimensions should be considered.
Based on questions like “What factors lead to project management success?” or “What factor
lead to a successful project?”, Cooke-Davies [30] defined the success criteria (SC) as indicators for
measuring the success or failure of the projects, and identified 12 factors that are in one way or
another, critical to project success. As well, Lim and Mohamed [28] helped to define the success factors
as the set of principles or standards for judgement about the success of a project. Among the top
frequently cited on the literature we can identify support from senior management [31], clear and
realistic objectives [27,32], strong/detailed plan kept up to date [33], good communication/feedback [30],
Sustainability 2020, 12, 5977 8 of 18

among others. Top management support was introduced as SF by Müller and Turner [31], and by Pinto
and Slevin [27], project ownership [34]. Also, Chan et al. [35] stated that project team commitment,
contractor competencies, risk and liability assessment, client competencies, users’ needs, and constraints
imposed by users are project success factors (PSFs).
There are plenty of studies about the PSFs, each of which represented a wide range of success
factors; however, these factors are usually listed or in a very general way or, with such specificity
that can only be applied to a particular variety of projects. Nonetheless, in their research, Belassi and
Tukel [33] stressed grouping success factors and explain the interaction between them, putting aside
the focus from the identification or specificity of such individual factors.
Although varied PSFs are introduced through numerous studies, Cooke-Davies [30] stated that
finding the projects’ real success factors is important. However, some of this PSFs are extremely
important and must exist to ensure the project success; these are called critical success factors (CSFs).
In their work, Fortune [36] carried out a review of a series of publications that focused on CSFs,
allowing them to identify which were the most frequent success factors mentioned on theoretical or
empirical studies and the different stages of the project where these factors were evaluated, letting them
conclude that the evaluation of some success factors on different phases of a particular project can help
to determine if it is going to be successful or not. Pinto and Slevin [27,32] represented a list of CSFs,
including: Technical tasks, client acceptance, power and politics, communication, client consultation,
top management support, urgency, environmental events, and characteristics of the project manager,
troubleshooting, and personnel recruitment.
As illustrated in Table 3, various factors contribute positively or negatively to project success.
Nevertheless, analyses of all success factors are extremely hard. Therefore, some authors grouped the
CSFs under “success dimensions”, which make the evaluation and interpretation of project success
more understandable. Different authors offered different dimensions to be used as criteria to judge
project success.

Table 3. Project success dimension.

Project Success Dimension References

Mission, top management support, schedule, client consultation, personnel, technical,
[27,32]
client acceptance, communication, feedback, and trouble-shooting
Communication, time, mission, project management competences, the project delivering
[16]
the strategic benefits and top management.
Policy and strategy, mission [16,30,37]
Project-related, human-related factors, process-related factors, input-related factors,
[37]
output-related factors
The future potential of the projects in terms of innovations, generating a new product
[15,16,38–40]
line or new technological capability
Project management factors: planning, scheduling, monitoring and control, quality
[27,30,32,38]
management, and risk management
Meet stakeholders’ expectations, benefit to the stakeholder group, client/customer
[16,27,32,38,40,41]
specific, Client acceptance and consultant.
Project efficiency, impact on customers, business and direct success and strategic
potential (preparing for the future). Time and costs were considered as resources and [42]
quality as customers. Satisfaction in contrast to using them as separate entities
Goals and objectives, performance monitoring, decision-maker(s), transformations,
[36]
communication, environment, boundaries, resources, continuity

4. Defining a Lean-Based KPIs Identification Model

In this study, a model for the identification of KPIs in a project context was developed. For the
design of the model, the theoretical concepts previously analysed were used as a basis. Additionally,
Sustainability 2020, 12, 5977 9 of 18

concepts from success factors and lean models related to performance measurement identified in other
studies were considered in the design of the model.
Leading KPIs are established in a way that impacts the most relevant results of the organisation.
Although the main indicators vary from one organisation to another, a process based on lean thinking
can determine the KPIs of a company. The first step for defining the KPIs is to identify the organisation
strategic objectives and the different impacted levels [18]. Cortes et al. [18] proposed a KPI classification,
based on the work of Pakdil and Leonard [43], and Gopinath and Freiheit [44], in line with the main
lean fundamentals. Roberts and Latorre [45] in their research made a critical analysis of the KPI
measurement system. The model proposed is based on selecting the categories from SLR, so the needs
identified from the customers (VoC) and the organisational strategies and objectives (VoB) can be set in
an organised form. Once that process is completed, the identification of the parameters to be measured
as a driver to accomplish the future KPI. Next, the identification of the CTQs gives the information
required to raise the performance indicators. Following this, we propose a tie in a measurement to that
indicator (e.g., number of, percentage of, amount of, etc.), and finally, the organisation goal must be set.
The proposed model to defining and establishing KPIs for projects is resumed in Table 4.

Table 4. Lean-based model for identifying KPIs.

Categories VoC Drivers CTQ Measurements Target

The Data at a single
Customer’s Parameters to Organisation’s
Proposed KPIs performance point in time,
need be measured goal
categories detected indicators specific, measurable
in the SLR Organisation’s
strategies

5. Case study—CFAA (Advanced Manufacturing Centre for Aeronautics)

Looking for strategies to increase the innovative capacity of universities and thus boost the local
economy, the collaboration between companies and universities has been an institutional strategy
used to guide ideas, inventions, and innovations generated in universities and transmit them to the
industrial and social network [46]. The machine-tool and advanced aeronautical manufacturing sectors
have been some of the strategic areas for the economy of the Basque Country in Spain over the last few
decades, generating a turnover of up to 3.83 billion Euros [47]. In this context, the creation of a Research,
Development, and Innovation (R&D&I) Centre with focus on advanced aeronautical manufacturing
technologies that could integrate these two sectors with the University and allow the easy and fast
transmission to the industrial production ecosystem associated with the value chain was a fairly
straightforward decision [48]. The result was the Centre for Advanced Manufacturing in Aeronautics
(CFAA, in Spanish), an open and shared space for researchers, students, and professionals of the
sector, companies, and research centres at national and international level, where applied knowledge,
technologies and new methodologies for the previously mentioned sectors are being developed.
CFAA was created to operate within the structure of the University of the Basque Country /Euskal
Herriko Unibersitatea (UPV/EHU), and emerges from the agreement signed between the UPV/EHU
and a group of aeronautical and machine tool companies, and is supported in its origin, constitution,
and in the acquisition of facilities and machinery by the Regional Government of Bizkaia and by the
Society for Competitive Transformation (SPRI)
The R&D&I projects carried on at CFAA are located between Technology Readiness Level (TRL)
5–7 [49], which guarantee a quick knowledge transfer, and promote mutual benefits to companies and
universities from a micro up to a macro-economic point of view, so that universities obtain funding to
conduct their research and train staff to enable them responding positively to the demands of the labour
market [50]. In addition, it advocates for fostering the relations between companies from different
sectors, and in the university–business axis [51,52].
Sustainability 2020, 12, 5977 10 of 18

Current Situation
The role of the CFAA within the projects is, on the one hand, to filter out projects that are in line
with their strategy, and from those that result, test their economic, scientific, and technical feasibility
in an industrial environment designed and equipped to simulate a real factory. On the other hand,
CFAA on its own, and thanks to the interaction of different scientific groups, proposes, develops,
and tests advanced manufacturing technologies, techniques, and applied knowledge, born from
state-of-the-art research.
CFAA is a project-based organisation, where the success of their projects takes great relevance
and a determinant role in the present and future opportunities for the Centre, in the form of being
able to participate, or present itself to European calls, taking part in international and specialised
clusters, or attracting young and professional talent. However, some studies carried on CFAA had
demonstrated that there are still significant improvement opportunities to measure and improve the
rate of success in this Centre [53].
To date, almost 300 projects have been carried out with a success rate close to 70%, meaning that
the results obtained are being used by the company (or group of companies) that leads the project.
This result is thanks to close co-operation between researchers, technicians, University experts,
project managers and company staff, collaborating on the prevention of wrong pathways taken at early
stages of the project.
Since its inception, CFAA has been committed to boosting scientific activity and contributing to
various scientific publications e.g., journals with different impact rates, conferences, book chapters, etc.
From 2017, the impact of the CFAA and its manufacturing groups is reflected on more than 140 scientific
publications (March 2020) which have generated more than 840 citations. Also, 11 doctoral theses
have been developed. Currently, CFAA is working on the implementation of a project management
methodology developed ad hoc for the Centre, whose objective is to manage projects, programmes,
and portfolios, and push the organisation towards more agility and efficiency [54].
Despite the good state of the CFAA regarding its scientific production, use of resources,
collaboration with institutions, and participation in co-operation projects at European level, several
of the general objectives set for the Centre [55] and those described in the Centre’s Quality Manual,
are not being adequately studied, described, and measured.
To date, a few quantifications of consumed hours in projects, number of projects developed,
and hours spend using the resources of the CFAA, are being measured (Figure 2). Leaving room for a
new setSustainability
of KPIs oriented to measure the performance of organisational and production needs.
2020, 12, x FOR PEER REVIEW 11 of 19

Figure 2. Performance information Centre for Advanced Manufacturing in Aeronautics (CFAA).

Figure 2. Performance information Centre for Advanced Manufacturing in Aeronautics (CFAA).
6. Applying the Lean-Based KPIs Identification to the Case Study
Burimova et al. [56] carried on a study in order to obtain a clear picture of what “Success” means
for CFAA and their most important stakeholders. The study was constructed following one of the
most critical remarks behind project success, which is that the success of the project depends on which
area (or dimension) is inspected, therefore these areas need to be addressed individually, including
Sustainability 2020, 12, 5977 11 of 18

6. Applying the Lean-Based KPIs Identification to the Case Study

Burimova et al. [56] carried on a study in order to obtain a clear picture of what “Success” means
for CFAA and their most important stakeholders. The study was constructed following one of the
most critical remarks behind project success, which is that the success of the project depends on which
area (or dimension) is inspected, therefore these areas need to be addressed individually, including not
just the business side, but also people management. Consequently, it is crucial to define and evaluate
the success criteria at different dimensions. For that study, the success dimensions chosen were:

• Project Management Success Dimensions (PMSD): Focus on the aspects that are necessary for
the right management of the project, e.g., control of time, cost and scope, compliance of quality
standards, resources and stakeholder’s management, etc.
• Delivery Activities Success Dimensions (DASD): Focus on the processes that are used to create
the deliverables.
• Deliverable Success Dimensions (DSD): Describing the output of the project, including the success
criteria needed to verify that the final result of the project matches the scope.
• Operations Success Dimensions (OSD): Includes the success criteria needed to verify that the
operation ensures that the ongoing process of the project is carried on in an appropriate way.

Once the study was conducted, the resume of the results of the most important success factors can
be seen at Table 5.

Table 5. Most important success factors for CFAA and stakeholders.

No. Success Factors Success Dimension References

1 Workplace Safety OSD [57]
2 Project goal was achieved PMSD [27,32]
3 Customer satisfaction regarding the quality of delivery activities of the specific project PMSD [58]
4 Reputation of the organisation has increased PMSD [59]
Knowledge generation regarding project activities (e.g., tools, techniques,
5 DASD [60]
approaches, processes)
6 Customer satisfaction regarding the management of the specific project. PMSD [61]
7 Customer satisfaction regarding the deliverable. DSD [62]
8 Degree to which the deliverable meets its intended purpose. DSD [63]
9 Return on Investment of the project PMSD [57]
10 Workplace security OSD [57]
11 Completed within defined and agreed scope PMSD [31,64]
12 Completed within defined and agreed time PMSD [31,64]
13 Completed within defined and agreed costs PMSD [31,64]

The results have shown that the most important success factors are related to the safety of the
workplace, also if the project goal was achieved and how the customer feels about the quality of the
different deliverables. These results are quite valuable as an input of the strategic needs that CFAA
must necessarily focus on.
The analysis of the organisation strategies and needs were found seeking for CFAA documents
with relevant information that may guide the identification of performance indicators, e.g., data from a
survey about project success for stakeholders and personnel from the organisation, and a benchmarking
with a similar Centre like CFAA. Later, a stakeholder validation was carried out to find out their
perception of the performance indicators.

6.1. Ruhrvalley Innovation Cluster

As has been mentioned before, the performance parameters used by centres alike need to be
analysed in order to compare and understand the rationalisation of their measurement parameters from
Sustainability 2020, 12, 5977 12 of 18

a more mature point of view. For this research, “ruhrvalley” was chosen among different innovation
centres due to different aspects:

– Region: The companies that conform the ruhrvalley innovation cluster are SMEs focused on small
but very advanced niches that some of them, regardless of their size, stand as world markets
leaders in their domain.
– Cluster: ruhrvalley was formed to provide wider solutions in areas like eMobility, renewable
energy systems, and digital transformation.
– Impacts: The expected impact on solutions for urban mobility and energy systems, and the
innovation community in the region, develop a strong innovation profile for universities and
development of technology-driven start-ups and SMEs on the region.
– Collaboration: An effective interaction model between leading universities and research centres,
scientific and industrial associations, and important players from society and politics [65].

ruhrvalley seeks a combination of applied research, academic education with a strong application
focus (high TRL), industry co-operation, and a strong rivet on innovation and entrepreneurship [66],
as does CFAA. Sastoque et al. [51] mentioned that CFAA and ruhrvalley have had a strategic matching
to increase an impact of R&D results in the scientific community. They also have some contextual
factors in common, like their location, due to both of them being placed in industrial urban areas
under processes of digital transformation through technology- and innovation-driven approaches.
Regarding the TRL, ruhrvalley goes one step further than CFAA, with TRL 6–7 for CFAA and TRL 6–8
for ruhrvalley. Also, they both focus on Industry 4.0 technologies.

6.2. KPIs Construction

In order to assemble the KPIs for CFAA, the classification was made based on the study carried
by Wolff and Nuseibah [65] and divided on four different scopes: Innovation, region, research and
development, and university. Next, the strategic needs for each one of those scopes were inferred from
the information sources previously mentioned Table 6.

Table 6. Strategic needs—CFAA.

Scope Strategic needs

Create and marketing new services, product lines and technological capabilities
Innovation
Institutional support of SMEs with innovation impulses in the development of new business models
CFAA as a place of interest for local and international partners to develop projects.
Increase the attractiveness of the Basque Country for research, innovation, employment and start-ups
Region
Intensify knowledge transfer
Interact with society, strengthen actors and civil society
Gather and generate experience, new knowledge and understanding from activities and management of the
Research project (e.g., tools, techniques, approaches or processes)
Increase the impact of R&D in the scientific community
Professionalizing project management
Promote compliance with the goals and objectives for which CFAA was created.
University
Securing the freedom, financial and personnel basis for research and transfer activities in the long term
Strengthen collaboration with University in the implementation and development of projects

At the end of the analysis of the information gathered, besides these 12 strategic needs,
51 operational needs and 56 internal factors (drivers) were also identified. However, for this paper,
the result of the study of the strategic and operational needs for the scope of innovation will be shown.
The CTQ started by identifying the critical Xs for the first strategic need (Y): Creating and
marketing new services, product lines, and technological capabilities. Next, the specific measurements
required to fulfil the quality requirements were previously identified. A target value for each of the
measurements must be set according to the organisation expectations (Table 7; Table 8).
Sustainability 2020, 12, 5977 13 of 18

Table 7. Strategic need: Creating and marketing new services, product lines, and technological capabilities (Y).

Operational Need Category Drivers CTQ Measurements Targets

Number of complaints received vs. attended. 100% complaints received vs. attended.
Strengthen strategic Number of wellbeing activities programmed 100% wellbeing activities programmed vs. realised
human resources Watch over the rights and vs. realised At least one well-being activity programmed vs. realised
Sustainability
development as a interests of the workers. Number of Safety programme activities 100% of Safety programme activities programmed vs. realised
research priority programmed vs. realised 85% of participation from personnel at Safety programme
activities programmed vs. realised
Contribute to sustainability Number of good practices implemented in 100% compliance with good practices implemented in the use
Avoidance of waste
in the creation, design and the use of waste of waste
result of deliverables from Hazard materials Kg per type of hazard materials treated 100% of hazard materials treated
the projects Sustainability through treatments Notices received by the authority for Cero notices received by the authority for mismanagement
creation, design and mismanagement of hazards of hazards
Sustainability
result of deliverables Number of safety programme activities
100% of safety programme activities programmed vs. realised
from the projects Safety and security programmed vs. realised
initiatives to avoid Number of security programme activities
100% of security programme activities programmed vs. realised
accidents or incidents programmed vs. realised
Number of safety events At least 5 safety events programmed per year
Number of security events At least 5 security events programmed per year
Develop and implement Marketing of R&D and Marketing campaigns of
Marketing campaigns of R&D and At least one marketing campaigns of R&D and innovation
marketing of the R&D Technical innovation service R&D and innovation
innovation per innovation per innovation
service portfolio portfolio service portfolio

New methods vs. Number of new methods vs. solutions At least 3 publication by each new methods vs. solutions
solutions published in published in refereed journals. produced at CFAA.
scientific journal Number of new methods vs. solutions At least 3 attendances by each new methods vs. solutions
presented at refereed conferences produced at CFAA
Generate new usable Usable knowledge and Number of new patents registered 100% of new patents registered
knowledge and Operational engineering solutions Number of articles published in refereed
engineering solutions generated New patents At least 3 articles published in refereed journals.
journals regarding uses of patents.
Number of presentations at refereed
At least 3 presentations at refereed conferences.
conferences regarding use of patents.
Number of marketing campaigns per patent At least one marketing campaigns per patent.
R&D strategy workshop
Number of participations at R&D or
participations vs. 85% of participations at R&D or innovation workshops
Increase demand Transfer-oriented innovation workshops
Strategic Organized
orientation in transfer demand Innovative activities Number of R&D or innovation
At least one R&D or innovation workshop organised by CFAA
developed workshop organised
Number of projects developed characterised Success rate > 80% of projects developed characterised as
Introduce continuous Continuous innovation Innovation projects as “Innovative” from partners “Innovative” from partners
Technical management Number of projects developed characterised Success rate > 80% of projects developed characterised as
innovation management developed
introduction as “innovative” from CFAA “innovative” from CFAA
Sustainability 2020, 12, 5977 14 of 18

Table 8. Strategic need: Institutional support of SMEs with innovation impulses in the development of new business models (Y).

Operational Needs Category Driver CTQ Measurements Targets

Formation to encourage Number of courses related//needed to
Formation to develop innovation skills At least 2 courses per year
innovation skills encourage innovation skills
Enable and encourage Students at CFAA doing their TFM Number of new students doing TFM
Technical Students participation at CFAA At least 30 students per year
talent early on or TFG or TFG
Students at CFAA doing Students at On-the-job formation Number of new students doing
At least 10 students per year
On-the-job formation at CFAA On-the-job formation
70% of partners participating in at
Partner participation in projects % of Partners participation in projects
Expand existing least one project per year
Cooperation in strategic
cooperation into strategic Strategic Effective cooperation development plan Plan’s implementation 30% of partners involved in the plan
innovation partnerships
innovation partnerships Partners involved in the
Framework agreement with all partners 30% of partners involved in the plan
Framework agreement
Foundation, establishment and
Spin-offs from CFAA Number of new spin-offs from CFAA At least 1 initiative per year
accompaniment of spin-offs
Foundation, establishment and
Promote the foundation, Initiatives enabled and encouraged Initiatives enabled and encouraged At least 1 initiative per year
accompaniment of spin-offs
establishment and
Technical Number of recruitments of innovative
accompaniment of Start-ups projects activated At least 1 initiative per year
start-ups and joint ventures
spin-offs
Business start-ups provoked Coordination and cooperation with Number of new cooperation
start-up support entities (Universities, agreements with associations and At least 3 agreements signed per year
governments, private investors) business development agencies
Sustainability 2020, 12, 5977 15 of 18

7. Conclusions and Future Research

Lean thinking can lead to knowing an organisation in a deeper way, asking questions that were
overlooked, and reaching conclusions of highly strategic value for the company. In this research,
through an SLR, the literature on the process of KPI definition was analysed through a lean approach
in project-based organisations.
Through the concepts gathered in the SLR, a classification of KPIs was identified, and a lean-based
model was developed to define the KPIs. This model was applied to a case study conducted within a
R&D project-based organisation (CFAA). Following a series of steps based on lean tools and principles,
the model worked as a guide for the identification of KPIs. The DMAIC methodology phases were
used in order to keep an organised process flow for the model implementation.
The first step was to establish the current status of the company as well as explore the stakeholders
needs and requirements with the VoC. To complement the initial information and to determine
the strategic needs (VoB), the company quality manual was reviewed. The literature highlights the
importance of KPI formulation based on strategic business objectives as well as corporate culture.
Although this is the starting point towards the identification of performance indicators, the scope
of this research does not include the analysis of corporate strategy and culture. Furthermore, it was
considered useful to benchmark with a similar leading organisation (ruhrvalley) in order to get baseline
performance indicators. After obtaining the data to define the CTQs, we proceeded to measure and
analyse the internal factors, measurements, targets, and performance indicators that constitute the KPIs.
The result was a series of qualitative and quantitative KPIs that evaluates the strategic and
operational needs of a project-based organisation and helps to understand and improve their
performance criteria. However, those defined KPIs cannot be unmovable. The continuous changes
in the market or the research methodologies require KPIs to be constantly redefined and updated,
in order to ensure that KPIs are suitable for the current environment of the organisation [18].
The scope of this research was limited to proposing a model for the identification of KPIs using the
lean approach, therefore for future research, it is suggested to continue with the following actions of
validating, communicating, reporting, and controlling the adequacy of the KPIs. “Visual management”
is a suitable technique to manage the quality of the KPIs [2]. For managers and project managers,
the use of KPIs “dashboards” are recommended during to report, monitor, and control the KPIs [11].
As a next step, a project plan must be developed to ensure the right applicability of the KPIs at
CFAA and a system to measure their effectiveness in the organisation. Additionally, the integration
of the information coming from different areas of the organisation (IoTs, edge computing devices,
project reports, etc.) to ensure the correct functioning of the KPIs should be assured. It is a process as
important as the formulation of the KPIs itself, for the use of dashboards, scorecards, and reports that
show information almost in real time is highly recommended.
The result of this research is of high value for project-based organisations, especially those new
ones that are not apprised of how to correctly formulate a series of KPIs, or whose path is still not clear.

Author Contributions: C.C.V., L.S.P., and J.R.O.O. proposed the methodology; N.T.G. and N.L.d.L. gave ideas
and analysis of CFAA KPIs; C.C.V., L.S.P., and N.L.d.L. participated in writing the article. All authors have read
and agreed to the published version of the manuscript.
Funding: Authors are grateful to Basque government group IT IT1337-19 and the Spanish Ministry of Science
Mineco REF DPI2016-74845-R and project PID2019-109340RB-I00.
Acknowledgments: The authors wish to express their gratitude to all those who have contributed to the study
described in this article, with special thanks to the School of Industrial Engineers of Bilbao (University of the
Basque Country UPV/EHU), and at the Centre of Advanced Aeronautical Manufacturing “CFAA” of Bizkaia.
Special thanks to all people working hard against COVID19, medical people, nurses and all the people staying at
home making all of us safer.
Conflicts of Interest: The authors declare no conflict of interest.
Sustainability 2020, 12, 5977 16 of 18

References
1. España, F.; Tsao, C.C.; Hauser, M. Driving continuous improvement by developing and leveraging lean
key performance indicators. In Proceedings of the Annual Conference of the International Group for Lean
Construction, San Diego, CA, USA, 18–20 July 2012.
2. Iuga, M.V.; Kifor, C.V.; Rosca, L.-I. Lean information management: Criteria for selecting key performance
indicators at shop floor. ACTA Univ. Cibiniensis 2015, 66, 67–72. [CrossRef]
3. Zakaria, Z.; Yaacob, Z.; Noordin, N.; Sawal, M.Z.H.M.; Zakaria, Z. Key performance indicators (KPIs) in the
public sector: A study in Malaysia. Asian Soc. Sci. 2011, 7, 102. [CrossRef]
4. Ogunlana, S.O. Beyond the ‘iron triangle’: Stakeholder perception of key performance indicators (KPIs) for
large-scale public sector development projects. Int. J. Proj. Manag. 2010, 28, 228–236.
5. Thangarajoo, Y.; Smith, A. Lean thinking: An overview. Ind. Eng. Manag. 2015, 4, 1000159. [CrossRef]
6. Al, E.; Ali, Z.; Türkyılmaz, A.; Zaim, S. Lean principles in R&D projects. In Proceedings of the Global
Conference on Engineering and Technology Management 2014, Istanbul, Turkey, 23–26 June 2014.
7. Madsen, D.Ø.; Berg, T.; Stenheim, T.; Moum, J.V.; Bordewich, I.O.; Storsveen, M. The long-term sustainability
of lean as a management practice: Survey evidence on diffusion and use of the concept in Norway in the
period 2015–2017. Sustainability 2019, 11, 3120. [CrossRef]
8. Dombrowski, U.; Schmidtchen, K.; Ebentreich, D. Balanced key performance indicators in product
development. Int. J. Mater. Mech. Manuf. 2013, 1, 27–31. [CrossRef]
9. Majerus, N. Leveraging lean principles in R&D: The experience of The Goodyear Tire & Rubber Company
shows how applying to Lean principles to R&D can improve the efficiency and effectiveness of new product
development. Res. Technol. Manag. 2017, 60, 17–25.
10. Foruhi, T.; Behzad, M.; Amiri, Z.A.; Felekari, M.; Havangi, H. Employing lean concepts and tools in innovative
and R&D based organizations. Int. J. Appl. Optim. Stud. 2018, 1, 71–85.
11. Marti, F. Lean Six Sigma method in phase 1 clinical trials: A practical example. Qual. Assur. J. Qual. Assur. J.
Pharm. Health Environ. Prof. 2005, 9, 35–39. [CrossRef]
12. Mir, F.A.; Pinnington, A.H. Exploring the value of project management: Linking project management
performance and project success. Int. J. Proj. Manag. 2014, 32, 202–217. [CrossRef]
13. Kerzner, H. Project Management Metrics, KPIs, and Dashboards: A Guide to Measuring and Monitoring Project
Performance; John Wiley & Sons: Hoboken, NJ, USA, 2017; ISBN 1119427282.
14. Bizan, O. The determinants of success of R&D projects: Evidence from American-Israeli research alliances.
Res. Policy 2003, 32, 1619–1640.
15. Shenhar, A.J.; Tishler, A.; Dvir, D.; Lipovetsky, S.; Lechler, T. Refining the search for project success factors:
A multivariate, typological approach. R&D Manag. 2002, 32, 111–126.
16. Davis, K. Different stakeholder groups and their perceptions of project success. Int. J. Proj. Manag. 2014, 32,
189–201. [CrossRef]
17. Panagopoulos, I.; Atkin, C.J.; Sikora, I. Developing a performance indicators lean-sigma framework for
measuring aviation system’s safety performance. Transp. Res. Proc. 2017, 22, 35–44. [CrossRef]
18. Cortes, H.; Daaboul, J.; Le Duigou, J.; Eynard, B. Strategic Lean Management: Integration of operational
performance indicators for strategic lean management. IFAC-PapersOnLine 2016, 49, 65–70. [CrossRef]
19. Cherrafi, A.; Elfezazi, S.; Chiarini, A.; Mokhlis, A.; Benhida, K. The integration of lean manufacturing,
Six Sigma and sustainability: A literature review and future research directions for developing a specific
model. J. Clean. Prod. 2016, 139, 828–846. [CrossRef]
20. Panat, R.; Dimitrova, V.; Selvamuniandy, T.S.; Ishiko, K.; Sun, D. The application of Lean Six Sigma to the
configuration control in Intel’s manufacturing R&D environment. Int. J. Lean Six Sigma 2014, 5, 444–459.
21. Kim, Y.W.; Han, S.H. Implementing Lean Six Sigma: A case study in concrete panel production. In Proceedings
of the 20th Annual Conference of the International Group for Lean Construction, San Diego, CA, USA,
18–20 July 2012.
22. Gaskin, S.P.; Griffin, A.; Hauser, J.R.; Katz, G.M.; Klein, R.L. V oice of the c ustomer. In Wiley International
Encyclopedia of Marketing; John Wiley & Sons: Hoboken, NJ, USA, 2010.
23. Ferrús, R.M.; Somonte, M.D. Design in robotics based in the voice of the customer of household robots.
Robot. Auton. Syst. 2016, 79, 99–107. [CrossRef]
Sustainability 2020, 12, 5977 17 of 18

24. Six Sigma Institute. Six Sigma DMAIC Process—Define Phase—Capturing Voice of Customer (VOC).
Available online: https://www.sixsigma-institute.org/Six_Sigma_DMAIC_Process_Define_Phase_Capturing_
Voice_Of_Customer_VOC.php (accessed on 15 June 2020).
25. Yang, H.M.; Choi, B.S.; Park, H.J.; Suh, M.S.; Chae, B.K. Supply chain management six sigma: A management
innovation methodology at the Samsung Group. Supply Chain Manag. Int. J. 2007, 12, 88–95. [CrossRef]
26. Brînduşa Maria Popa. Challenges when developing performance indicators. J. Def. Res. Manag. 2015, 6,
111–114.
27. Pinto, J.K.; Slevin, D.P. Critical success factors in effective project implementation. Proj. Manag. Handb. 1988,
479, 167–190.
28. Lim, C.S.; Mohamed, M.Z. Criteria of project success: An exploratory re-examination. Int. J. Proj. Manag.
1999, 17, 243–248. [CrossRef]
29. Wikipedia. Sydney Opera House. Available online: https://en.wikipedia.org/wiki/Sydney_Opera_House
(accessed on 24 January 2020).
30. Cooke-Davies, T. The “real” success factors on projects. Int. J. Proj. Manag. 2002, 20, 185–190. [CrossRef]
31. Müller, R.; Turner, J.R. Matching the project manager’s leadership style to project type. Int. J. Proj. Manag.
2007, 25, 21–32. [CrossRef]
32. Pinto, J.K.; Slevin, D.P. Critical factors in successful project implementation. IEEE Trans. Eng. Manag. 1987,
EM-34, 22–27. [CrossRef]
33. Belassi, W.; Tukel, O.I. A new framework for determining critical success/failure factors in projects. Int. J.
Proj. Manag. 1996, 14, 141–151. [CrossRef]
34. Baccarini, D. The logical framework method for defining project success. Proj. Manag. J. 1999, 30, 25–32.
[CrossRef]
35. Chan, A.P.C.; Ho, D.C.K.; Tam, C.M. Design and build project success factors: Multivariate analysis. J. Constr.
Eng. Manag. 2001, 127, 93–100. [CrossRef]
36. Fortune, J.; White, D. Framing of project critical success factors by a systems model. Int. J. Proj. Manag. 2006,
24, 53–65. [CrossRef]
37. Angus, G.Y.; Flett, P.D.; Bowers, J.A. Developing a value-centred proposal for assessing project success. Int. J.
Proj. Manag. 2005, 23, 428–436.
38. Abdullah, W.M.W.; Ramly, A. Critical Factors in Project Success. In Proceedings of the Pacific Association of
Quantity Surveyors (PAQS) Congress, Edmonton, AL, CA, 17–18 August 2009.
39. Jugdev, K.; Müller, R. A retrospective look at our evolving understanding of project success. Proj. Manag. J.
2005, 36, 19–31. [CrossRef]
40. Cserháti, G.; Szabó, L. The relationship between success criteria and success factors in organisational event
projects. Int. J. Proj. Manag. 2014, 32, 613–624. [CrossRef]
41. Project Management Institute. A Guide to the Project Management Body of Knowledge Project Management
Institute, 6th ed.; Project Management Institute: Newtown Square, PA, USA, 2017; ISBN 9781628251845.
42. Shrnhur, A.J.; Levy, O.; Dvir, D. Mapping the dimensions of project success. Proj. Manag. J. 1997, 28, 5–13.
43. Pakdil, F.; Leonard, K.M. Criteria for a lean organisation: Development of a lean assessment tool. Int. J.
Prod. Res. 2014, 52, 4587–4607. [CrossRef]
44. Gopinath, S.; Freiheit, T.I. A waste relationship model and center point tracking metric for lean manufacturing
systems. IIE Trans. 2012, 44, 136–154. [CrossRef]
45. Roberts, M.; Latorre, V. KPIs in the UK’s construction industry: Using system dynamics to understand
underachievement. Rev. Constr. 2009, 8, 69–82.
46. De Falco, S. Monitoring the performance of university technology transfer offices: The bias control.
Arch. Bus. Res. 2015, 3. [CrossRef]
47. AFM. 40 Young Talents Specialised in Machine-Tool and Aeronautics Received Their Diplomas at the CFAA.
Available online: https://www.afm.es/en/news/40-young-talents-specialised-in-aeronautics-and-machine-
tool-received-their-diplomas-at-the-cfaa (accessed on 15 June 2020).
48. Sastoque Pinilla, L.; Llorente Rodríguez, R.; Toledo Gandarias, N.; López de Lacalle, L.N.; Ramezani
Farokhad, M. TRLs 5–7 Advanced manufacturing centres, practical model to boost technology transfer in
manufacturing. Sustainability 2019, 11, 4890. [CrossRef]
49. NASA. Technology Readiness Level. Available online: https://www.nasa.gov/directorates/heo/scan/
engineering/technology/txt_accordion1.html (accessed on 20 April 2020).
Sustainability 2020, 12, 5977 18 of 18

50. Poyago-Theotoky, J.; Beath, J.; Siegel, D.S. Universities and fundamental research: Reflections on the growth
of university–industry partnerships. Oxf. Rev. Econ. Policy 2002, 18, 10–21. [CrossRef]
51. Pinilla, L.S.; Bengfort, S.; Mikhridinova, N.; de Lacalle, N.L.; Wolff, C.; Gandarias, N.T. Patterns for
international cooperation between innovation clusters: Cases of CFAA and ruhrvalley. In Proceedings of the
2020 IEEE European Technology and Engineering Management Summit (E-TEMS), Dortmund, Germany,
5–7 March 2020; pp. 1–7.
52. Syreyshchikova, N.V.; Pimenov, D.Y.; Mikolajczyk, T.; Moldovan, L. Development of a risk management
technique in strategic planning of universities: Case study of a Polytechnical Institute. Proced. Manuf. 2020,
46, 256–262. [CrossRef]
53. Artelt, S. Mitigate the Risk of Project Schedule Overruns and Project Delay Chains in a TRL 5–7 Research
Centre: Improving the Quality of Project Estimating and Scheduling in a Multi-Project Environment. Master’s
Thesis, University of the Basque Country/Euskal Herriko Universitatea, Basque Country, Spain, 2020.
54. Farokhad, M.R.; Otegi-Olaso, J.R.; Pinilla, L.S.; Gandarias, N.T.; de Lacalle, L.N.L. Assessing the success of
R&D projects and innovation projects through project management life cycle. In Proceedings of the 2019
10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems:
Technology and Applications (IDAACS), Metz, France, 18–21 September 2019.
55. CFAA. General Objectives—Aeronautics Advanced Manufacturing Center–UPV/EHU. Available online:
https://www.ehu.eus/en/web/cfaa/helburu-nagusiak (accessed on 15 June 2020).
56. Burimova, A.; Artelt, S.; Sastoque Pinilla, L.; Toledo Gandarias, N.; Lopez de Lacalle, N. Project success
evaluation of a TRL 5–7 public-private research centre. Manuscript in preparation. 2020.
57. Project Management Institute. Delivering Value: Benefits Focus in Project Execution. Available online: https:
//www.pmi.org/learning/thought-leadership/pulse/focus-on-benefits-during-project-execution (accessed on
20 April 2020).
58. McLeod, L.; Doolin, B.; MacDonell, S.G. A perspective-based understanding of project success. Proj. Manag.
J. 2012, 43, 68–86. [CrossRef]
59. Khang, D.B.; Moe, T.L. Success criteria and factors for international development projects: A life-cycle-based
framework. Proj. Manag. J. 2008, 39, 72–84. [CrossRef]
60. Dvir, D.O.V.; Sadeh, A.; Malach-Pines, A. Projects and project managers: The relationship between project
managers’ personality, project types, and project success. Proj. Manag. J. 2006, 37, 36–48. [CrossRef]
61. Müller, R.; Jugdev, K. Critical success factors in projects: Pinto, Slevin, and Prescott-the elucidation of project
success. Int. J. Manag. Proj. Bus. 2012, 5, 757–775. [CrossRef]
62. Martens, M.L.; Carvalho, M.M. Sustainability and success variables in the project management context:
An expert panel. Proj. Manag. J. 2016, 47, 24–43. [CrossRef]
63. Agutter, C. ITIL® Foundation Essentials–ITIL 4 Edition; IT Governance Ltd: Ely, UK, 2019; ISBN 1787781186.
64. Project Management Institute. Success in Disruptive Times. Pulse of the Profession 2018. Available
online: https://www.pmi.org/learning/thought-leadership/pulse/pulse-of-the-profession-2018 (accessed
on 20 April 2020).
65. Wolff, C.; Nuseibah, A. A projectized path towards an effective industry-university-cluster: Ruhrvalley.
In Proceedings of the 12th International Scientific and Technical Conference on Computer Sciences and
Information Technologies (CSIT), Lviv, Ukraine, 5–8 September 2017.
66. Ruhrvalley. Mobility and Energy for Metropolitan Change. Available online: https://www.ruhrvalley.de/en
(accessed on 22 June 2020).

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).