J Intell Manuf (2019) 30:743–754

https://doi.org/10.1007/s10845-016-1276-0

Towards user empowerment in product design: a mixed reality

tool for interactive virtual prototyping
Pierre-Antoine Arrighi1 · Céline Mougenot1

Received: 18 August 2016 / Accepted: 26 October 2016 / Published online: 11 November 2016
© Springer Science+Business Media New York 2016

Abstract Designing new products according to user needs Context: designing products that meet customers’
and requirements is a key success factor for companies. How- needs
ever, the characterization of user requirements in the early
design stages is not an easy task due to the subjective nature Necessity to integrate customer needs at the beginning
of these requirements and because of the communication gap of the design process
between users and designers. Digital design tools have the
potential to enable users to actively participate in the design In today highly competitive markets, companies need to
process and to directly interact with representations of the set apart their products from their competitors not only
future product. Yet, they suffer a trade-off between their abil- with functions, price and quality, but also acknowledge
ity to accurately represent the user experience with the future needs and tastes of their customers (Petiot and Yannou
product and their capacity to offer simple interfaces for the 2004). As a consequence, the design processes need to
end user to manipulate. To overcome this issue, we introduce improve the attractiveness of products to satisfy customer’s
a new modular digital tool that allows users to become active psycho-physiological needs, by essence subjective. Subjec-
participants of the design process through a high level of both tive customer preferences are formed on the basis of a
immersion and control. The tool consists in a combination customer’s perceptions of the value of a product, from its
of a virtual reality environment for immersion and intuitive acquisition to discard, during its all cycle of use (Zeithaml
physical interfaces for direct control, resulting in a mixed 1988). Examples of such attributes include product func-
reality hardware/software system. The hardware is made of tionality, design, quality, brand image, user experience, and
modular tangible user interfaces (TUIs), custom-made by 3D ergonomics. Therefore, customer needs have to be accurately
printing and powered by a 3D game engine while the interac- and efficiently transformed into functional and psychological
tive content is displayed in virtual reality. The modularity of requirements (Füller and Matzler 2007).
the system allows several TUIs and 3D content behaviours While systematic design methods (Pahl and Beitz 1988)
configurations to bring user friendliness and intuitively for are successful for objective requirements, they suffer limited
each specific design project. input methods for subjective requirements. Design alter-
ations become more difficult and expensive towards the end
Keywords User/customer requirements · Digital design of the design process, thus it is necessary to collect customer
tool · Mass customization · Mixed reality · Tangible user feedback in the early design stages called “Fuzzy Front End”
interfaces (Khurana and Rosenthal 1997), even before physical proto-
types are tested with users.
Besides user’s requirements are difficult to collect by
designers, as users sometimes do not explicitly know the
B Pierre-Antoine Arrighi requirement themselves. In the case users describe their
[email protected]
preferences verbally, the collected information might be
1 School of Engineering, Tokyo Institute of Technology, 2-12-1 imprecise, as described by Lin and Zhang (2006).
i6-22 Ookayama, Meguro-ku, Tokyo 152-8550, Japan

123
744 J Intell Manuf (2019) 30:743–754

Collecting user requirements by integration of the users Computer-aided design tools can support integration of
in the design process user requirements

Products are sometimes designed to fit designer taste rather For an optimal compatibility with existing digital design
than user needs (Hsu 2010). To avoid this type of mismatch, process of most new products, we concentrated our research
directly collecting the users’ requirements is an interesting on Computer Aided Design (CAD) tools. CAD tools are
option. The involvement of users is crucial at each stage of the shown to be powerful tools for collecting data on the design
product development cycle, to leverage their contributions in process, as CAD representations are not prone to subjectiv-
a cumulative way and thus to maximize their effects (Shah ity and require a limited interpretation (Arrighi et al. 2015).
and Robinson 2007); it is also especially recommended to The tools are able to track the interactions between users,
involve users in the initial design stage, as it a critical phase they support the design process itself and produce represen-
of the design process (Sanders and Stappers 2008; Wang et al. tations of the concept, from early steps to final completion
2002). of the physical object. With CAD conceptual modeling and
The main benefits of early user involvement are an simulation, designers can successfully anticipate the interac-
increased access to and understanding of the user’s needs, tion of products with users in a real life environment (Van
experiences, and expectations and can offer an increase in der Vegte and Horváth 2002).
the functionality, usability, and quality of the future new However, CAD tools suffer from specific flaws. Because
products. Early user involvement in the design process has of the number of functions they support, CAD tools require
also the potential to deepen designers’ understanding of the a long training with steep learning curves. They require good
user needs, experiences, and expectations, to lead to better perceptive and imaginative skills along with years of techni-
design of devices and user interfaces and to a better integra- cal educational background (Hamade and Artail 2008). Their
tion of functionality, usability and quality of these devices. complexity notably lies in their human machine interfaces,
It can also facilitates the generation of requirements for the which impair collaboration and direct interaction with the
engineers and defining goals for the project (Fukuda and Mat- CAD objects (Sidharta et al. 2006). Keyboard and mouse
suura 1993). widely dominate work environments and offer a limited user
One interesting option for users’ co-design is to provide experience for non-expert users. These interfaces also require
them design tools that are compatible with the designers’. dexterity and precision. The compatibility of CAD tools with
Design tools support the design process and produce repre- early design stages, when the user should be involved, could
sentations of the concept which are key for collecting data also be improved.
about ergonomics, usability, user acceptance, body metrics
and user’s feeling (Nagamachi 2010; Schütte et al. 2004).
Design tools have the potential to facilitate communication Literature review: existing CAD tools and user
between designers and final users in a “co-design” approach participation in design
(Sanders and Stappers 2008). This extended collaboration
between designers and users’ leverages user-centred design We identified two types of CAD tools that collect and imple-
approaches to active co-design. Hence, design tools could ment user feedbacks in the design process: (1) CAD tools
enable users to participate in the fuzzy front end and thus that allow the users to manipulate rich representations of the
allow the collection and use of user data in the early design future product, with a high level of detail and similarity with
phase. the future product and (2) CAD tools with high level of inter-
Computational tools using fuzzy logic have been devel- activity which allow users to directly actively co-design some
oped to support the collection of customer requirements in specific parts of the product.
terms of the function, ergonomics and aesthetics and their
relationships in terms of design (Cai et al. 2016). The study Virtual prototyping evaluation in virtual reality
aimed at supporting the disambiguation of subjective infor-
mation, through a systematic categorization of imprecise Virtual product and virtual prototyping in virtual reality
information into three types: incompleteness, uncertainty and
ambiguity. In a co-design situation, information expressed by First, we describe CAD tools that provide realistic and
a user to describe their product preferences might be ambigu- detailed representations. Under specific conditions virtual
ous, e.g., “a luxurious car”, “modern furniture”, “feminine prototyping (VP) has proven to be an effective method to
glasses”, thus, by using other users’ input than verbal descrip- allow designers and engineers to determine various design
tion of their preferences, we expect to overcome the problem problems and errors early in the product development process
of inexact interpretation by a design of ambiguous descrip- (Bordegoni and Ferrise 2013; Bordegoni et al. 2011). The
tion given by a user. benefits of a VP approach include the improvement of the

123
J Intell Manuf (2019) 30:743–754 745

communication of ideas between designers and customers Customer kansei collection with “Kansei Engineering Type
(Carulli et al. 2013). VP is a powerful resource to meet IV”
the demands of co-design (Füller and Matzler 2007). VP
processes can help designers in testing the future product Ergonomics, usability, user acceptance, body metrics and
characteristics with end users from the very beginning of adequacy to the user’s feelings are key drivers of the final
the design process and possibly change the product design product properties and should be considered from the very
specifications. Once created, virtual prototypes are easy to early design stages. They are all related to what is called the
modify, share, represent in different ways and they offer flex- user kansei (Nagamachi 2010), a Japanese word for “sub-
ibility and limited cost compared to traditional prototyping jective impression”, “emotion” or “affect”. The emotional,
(Bordegoni and Ferrise 2013; Ferrise et al. 2015). psychological feelings of users when interacting with a prod-
Virtual reality (VR) refers to a computer-generated virtual uct are referred as the Japanese term “kansei ( )”. Kansei
world that is very close to reality, and simulated experiences Engineering is the methodology which aims to design and
that closely resemble the real world (Nagamachi 2008). Cre- develop product/systems based on the obtained kansei infor-
ating a computer image of a product is cheaper than creating mation of the user (Nagamachi 2010).
the product itself. By utilizing the simulated experience, the However, collecting and using the kansei requirements
human experience of a situation using computer technology in the early phase of a design project is a major challenge,
is like a walk-through of the actual situation. because of the simultaneity of the collection of both psycho-
VR environments can be characterized by two parameters logical and physiological user requirements. As described
identified by Slater et al. (1996) immersion and presence. by Lin and Zhang (2006), the user involvement in the design
Immersion is an objective description of what any particular process is required, as the designer needs to know the require-
system does provide and related to technology. It is linked ment from the beginning. The key point is how to explicitly
to the number of senses solicited (vision, hearing, touch) get the user’s requirement, as users are sometimes not aware
and the system mimics accurately the natural perception of of their preferences and requirements and are sometimes
the world. Other sensory modalities are usually not consid- not able to describe them in a precise, complete and non-
ered because of minor importance. Through haptic devices, ambiguous manner.
it is possible to simulate the touch and physical interaction, We choose to focus on VP as a tool to enable co-design
through sound devices realistic sounds can be reproduce, and between a designer and user. This way, verbal descriptions
with the technologies available for the visualization channel of subjective users requirements, which might be ambiguous,
we can return to the user a realistic and 3D representation are not necessary. VP plays an important role in collecting
of the product (Bordegoni and Ferrise 2013; Ferrise et al. kansei data, as it enables users to express their unspoken
2015). Therefore, a good immersion requires good graphics, feelings. Usually based on images of similar products and
low latency, effective tracking, high-resolution images and questionnaire, recent researches suggest VR can play an
3D sound. Presence is a state of consciousness, a cognitive interesting role and leverage the fidelity of the representa-
sense of being in the virtual environment, and corresponding tions presented to the users, and in consequence the sincerity
modes of behavior. Presence requires the user feel to be in and accuracy of their feedback for generating their kansei.
the virtual environment. Interactions and adequate reactions With a “Kansei Engineering Type IV” approach (Naga-
(pushing an object makes it move) increase the presence of machi 2010), a system of kitchen customization has been
the user in the virtual environment. High number of possible developed with Matsushita company (Imamura et al. 1994;
interactions are usually relating to higher sense of presence. Enomoto et al. 1995). A reference database of 10,000 real
VP in VR intends to “mimic” the behavior, response, kitchen designs has been built and an Artificial Intelligence
appearance and geometry of a target system (or object), with (AI) system based on users emotions, lifestyle evaluation
a degree of realism comparable to the actual system or object. and kitchen engineering rules has been implemented. When
Virtual prototyping techniques are being extensively used in a user want to change an existing design, the existing design
industry worldwide, as they provide cross-functional evalua- is modified by a technician and displayed on the screen after a
tion at a lower cost, enable engineers to consider downstream few seconds. This AI system is able to show a million differ-
issues earlier in the product design cycle (Cecil and Kan- ent kitchen designs. The user can explain his/her preferences
chanapiboon 2006). This technique increases the fidelity of to the technician who would in real time apply modification
the 3D model representation and makes it closer to what will to the virtual kitchen, so the user can see in almost real time
be the future product. It provides users with a perceptual feed- the result of the modifications asked for. Anything unsatisfac-
back as much as possible similar to that she can experience tory would be amended directly by the technician through the
while using a physical prototype of the same product, for a computer tool. The system is able to show a million different
much lower cost. kitchen designs (Fig. 1).

123
746 J Intell Manuf (2019) 30:743–754

Fig. 2 Collecting the VOC inside a highly immersive environment

(Carulli et al. 2013)
Fig. 1 Kansei Engineering Type IV: users can access a million different
kitchen design configurations (Nagamachi 2010)

However, this approach considers the user as a source of

design information and requirements but not as an active
participant and co-design partner. The user is not capable to
directly change the computer model and each modification
requires an expert designer to manipulate the representation
with an external system, with a specific set of functions. Iter-
ations can take time and might not be always completely
satisfy the user due to language and description issues.

“Voice of the Customer” collection with very high level of

immersion
Fig. 3 “Virtual Reality” design tools: high sensory immersion, lim-
The collection of the customers’ requirements during a ited control. For visibility the sensory feedback for the designer is not
illustrated
design process is sometimes qualified as Voice Of the Cus-
tomer collection (Carulli et al. 2013). This process can be
successfully achieved with the VP techniques which would
provide a high level of immersion, regarding the great sen- to a nearby designer. These parameters about force feedback
sory experience of the user during the interaction with the can then be reproduced by the designers in the real proto-
product (Fig. 2). type, by manipulating or mixing the components that affect
Even without any expertise, users are immersed in a the force feedback, like for example the material of collid-
VR environment where they can manipulate the VP. Latest ing components, spring constants, superficial roughness of
experiments use very detailed and realistic systems that can sliding parts (Bordegoni et al. 2011).
simultaneously stimulate sight, hearing and touch senses of However, in a similar fashion to kansei engineering type
users, with a high level of fidelity in the case of the visual IV, the optimized immersion of the user in the virtual envi-
and auditory interfaces (Bordegoni et al. 2011; Ferrise et al. ronment which contains the under design product, does not
2015). offer the potential and tools for direct modifications. The user
In some cases, haptic interfaces with force feedback allow has to formulate requests to the designer about his/her pre-
user to evaluate complex products behaviours, such as the ferred alteration of the design parameters (Fig. 3). His/her
resistance of a button and the feeling of a handle or door, with interactions with the product are therefore mediated and the
a high fidelity. When the user manipulates the digital mock- level of direct control on the actual design inexistent. The
up in the VR environment and interacts with the tangible user experiments the VR representation of the product as a
interfaces, he would give recommendations and comments spectator giving feedback rather than a co-designer.

123
J Intell Manuf (2019) 30:743–754 747

Fig. 4 Online configurators (from left to right: NIKEiD, MINI website, HomeByMe)

No immersion but good control: mass customization their valuation of the customizable attributes. In any case,
and configurators the mass customization system must be simple and allow
customers to customize a given product quickly. Mass cus-
Another part of CAD tools literature focus on the customiza- tomization relies on specifications from customers to design
tion options offers to the users to create their own specific and produce customized products (Helms et al. 2008; Ninan
and fitting product. Mass customization is acknowledged as and Siddique 2006).
a strategy to differentiate products and services according For leveraging mass customization applications, the litera-
to consumers’ preferences, thereby increasing the perceived ture cites the product configurators as capable tools. Indeed,
value of a product or service and supporting a business’ prof- product configurators are a class of applications designed
itability (Davis 1987; Huyett and Viguerie 2005; Lihra et al. to help companies offering product variety and customiza-
2008; Pine 1993; Thornton 2006; Toffler 1970; Wan and tion carry out the product configuration process (Trentin
Bullard 2008). et al. 2014). Due to trends toward product proliferation and
Mass customization (MC) provides customized goods or customization that have characterized many diverse indus-
services promptly upon request by a customer at prices at tries (Bayus and Putsis Jr 1999; Cox and Alm 1998; Pine
or near that of standard products by using methods com- 1993), product configurators have increasingly drawn the
bining manufacturing flexibility and mass production. MC attention of companies over the past few years. Config-
has been implemented successfully and profitably by manu- uration design necessitates the coordination of customer
facturers of consumer goods located in countries with high requirements, product characteristics, production processes,
manufacturing costs. A customized product will gain cus- and logistics networks, in order to achieve rapid response
tomer preference only when its perceived value exceeds that to customer orders. Under specific circumstances the use
of a mass produced, standard product (Du et al. 2006). MC is of a product configurator positively affects product qual-
considered an efficient way to match a firm’s offerings with ity (Trentin et al. 2012). However, Product configurators are
individual customer preferences (Fornell et al. 1996; Perdue mainly used as sales tools and fail to account for the require-
and Summers 1991). ments of the entire customer order fulfillment process (Helo
Because difficulties arise for manufacturers when required 2006)
to select and provide the most beneficial customization Examples of application of VP as a mass-customization
attributes to customers to differentiate products from compet- and product configurators tool are NIKEiD (customization
ing ones (Hart 1995; Lihra et al. 2012) an interesting strategy of shoes and accessories, Fig. 4), MINI website (customisa-
is to offer the users to directly customize themselves the tion of a MINI car) and HomeByME (decoration of home
products. Advantages of mass customization: the ability to interiors with furniture). These services provide customers
provide your customers with anything they want profitably, the controllability to alter parts or arrange positions of com-
any time they want it, anywhere they want it, any way they ponents in a preferable way. Product configurators have also
want it (Silveira et al. 2001). been essential components of successful mass customiza-
The time and effort that customers are willing to spend for tion strategies in many firms, such as Dell Computers
the customization of a product is limited (Jiao et al. 2003; (Magretta 1998), Cisco Systems (Tseng and Piller 2003),
Kahn 1998) so the tools are required to be intuitive and American Power Conversion (Hvam 2006) or Reebok
pleasant to use. The complexity of customization can lead (Piller 2007). These customised products make it possible
to mental fatigue and confusion of customers, preventing the to maximize individual customer satisfaction under specific
closing of an order (Hunt 2006; Rautenstrauch et al. 2012). circumstances (Hart 1995) (Fig. 5).
The time and effort considered acceptable for customization These product configurators provide extensive options and
can vary considerably depending on customers’ profiles and control level to the users and are engaging because they

123
748 J Intell Manuf (2019) 30:743–754

Fig. 5 “Configurator” design tools: high control, limited sensory

immersion

Table 1 Synthesis of the literature review for design tools for end users’
involvement
Category Configurators Virtual reality evalu-
ation
Fig. 6 A new tool that simultaneously offers high level of control and
Process User can change the User can experience immersion
configurations and the product and
generate an give design
original product instructions to a
To combine the benefits of both approaches, our objective
designer
is to create a new tool to support the integration of the user and
Main advantages The user designs Interaction,
himself/herself solicitation of offer simultaneously better immersion and control. We for-
senses mulate the following hypothesis: with a direct involvement
High level of High level of of the user in the design process, the outcome of the design
control immersion process will better match the user requirements. A new com-
Main limitations Only 2D Mediated puter aided co-design tool could support users participation
visualization of the interactions with in the design process with a high level of both immersion and
virtual prototype the virtual
prototype
control (Fig. 6).

Result: a new mixed reality tool for co-design

enable to create a product that matches individual needs. Con-
figurators offer a high level of control for product definition Principles and requirements of the design tool
by the user by providing freedom to actively and repeti-
tively change his/her design. However, the current state of The tool needs to be compatible with the current design
the art, part link to the general availability of tablets and 3D process, namely existing digital design tools and methods.
computers screen and the inherited complexity of modifying We use digital tools and the manipulation of CAD data for
3D models limits the realism of the available representa- a better integration. The system will focus on the initial
tions. Most popular configurators are available through web
browsers and give visual feedback to users through mono-
scopic representations. The interactions available are limited
to keyboard/mouse (desktop and laptops) or touch screen
interfaces (tablets or smartphones).

Research gap and objectives

From the aforementioned literature review, design tools fall

in two categories, “configurators” and “Virtual Reality eval-
uation”, which respectively give users a high level of control
on one hand and a high level of immersion on the other hand, Fig. 7 New “Mixed Reality” design tool: high control, high sensory
as described in the Table 1. immersion

123
J Intell Manuf (2019) 30:743–754 749

Fig. 8 The “virtuality

continuum” (Milgram and
Kishino 1994)

design steps for the definition of user requirements and dur- immersion. The flow chart of a MR system for interactive
ing the design iterations and evaluations of digital prototypes. VP is described in Fig. 9.
Physical limitations, such as degree of freedom and material
resistance are embedded in the CAD data. This can allow
Description of the system
more flexibility and the exploration of several design con-
cepts even if the user has a limited knowledge of the technical
We used a mix of VR and AR to create a MR design tool
constraints (Medyna et al. 2012). CAD data can be used for
(Figs. 10, 11). For immersion, we provide a full stereoscopic
generating and immersive environment and for stimulating
vision through head mounted display (HMD) for better visu-
touch, sight and hearing of users
alization of the product and an augmented tangible user
The tool needs to integrate the user in the design process
interfaces for better haptic feedbacks. The Oculus Rift
by giving him/her simple interfaces to collaborate actively
is a cheap head-mounted display that can display stereo-
with the designers. This can lead to a better user satisfaction
scopic images using a low persistence OLED display to
(Hideyoshi and Fukuda 2004) and therefore make the user
eliminate motion blur and judder, two of the biggest contrib-
willing to use the design tool. We also need the system to be
utors to simulator sickness. Low persistence makes the scene
capable of changing configurations for fitting different users.
appear visually stable, increasing the potential for presence.
One of the solutions is to create the interfaces on demand for
It is installed with an additional positional tracking and can
each use case and ensure their compatibility with the system
accurately map all of the user’s head movements. For the
with a modular architecture (Fig. 7).
stimulation of user hearing sense, we used spatialized sound
in the software and stereo headphones. The sound tracking
Introduction to MR is done by using the input from the Oculus tracking system.
At every moment, the position of the user is computed and
Mixed reality (MR) is defined as a particular subset of VR, the stereo sound modified accordingly.
related technologies that involve the merging of real and For the haptic direct interactions, we created and used 3D
virtual worlds somewhere along the ”virtuality continuum” printed TUIs. TUIs allow a natural and easy interaction with
(Milgram and Kishino 1994) that spans between completely CAD models. With TUIs the users are allow to see 3D models
real environments and completely virtual ones and illustrated in VR and to manipulate them naturally as if they were real,
in Fig. 8. physical objects, simply by grasping the physical interfaces.
One of the major advantages of this type of interfaces Additional benefits of TUIs include allowing multiple users
is the possibility of leveraging the real world with virtual to simultaneously view, feel and manipulate a physical shape
elements [as in augmented reality (AR)] or to map virtual instead of an abstract graphical representation (Leithinger
elements to real physical objects manipulated by the user (as et al. 2011). TUIs also bridges the gap between the worlds of
in AR). MR systems can facilitate the integration of users bits and atoms through graspable objects and ambient media
in the design process because they offer an enhanced per- in physical environment (Ishii and Ullmer 1997). They can
ception of the future product. The VR technology enables also serve as boundary objects (Carlile 2002) and close the
the user to experience the design model with high quality of gap between the interfaces and their functionalities for an
presence by using the interactive three-dimensional image increased affordance and discoverability.
without making an actual product from the early design These interfaces were combined with AR technology to
stages (Ogi et al. 2010). Shared representations and boundary allow accurate 3D tracking and precise positioning in the
objects can allow better communication between designers virtual world. The AR markers are simply printed on regular
and users. Boundary objects (Carlile 2002) work to estab- sheets of paper and glued on the top of the TUIs. We use a
lish a shared context between designers and can facilitate the Creative 720p webcam for the tracking of TUIs and a ded-
depiction of a future product form and function to a non- icated tracking algorithm developed by XZIMG company.
expert user. Visual representation of requirements can also With our system we are capable of tracking up to 20 markers
support a more effective collaboration between participants simultaneously. The markers are 2 cm wide and their position
of a design project (Richards et al. 2003). We want to mix the is detected in 3D at a distance up to 1 m. The behaviours of
advantages of MR, namely virtual and AR tools with Tan- the 3D objects (translation & rotation) in the virtual environ-
gible User Interfaces (TUIs) for simultaneous control and ment are synchronized with the movements of the TUIs in

123
750 J Intell Manuf (2019) 30:743–754

Fig. 9 Flow chart of a mixed

reality system for interactive
virtual prototyping

the physical environment by a script coded in the Unity soft- and complex objects. The 3D printed interfaces are made
ware. Hence, users are able to interact and change their design from simplified shapes of the early design subcomponents
by perceiving visual and haptic feedback from the system, of the product. The 3D printed TUIs are produced ad-hoc to
which guarantees high-immersion and great controllability be simultaneously easy to manipulate and track through AR
of this tool. The TUIs can be used either as 3D elements of markers printed on paper and glued on the TUIs. The TUIs
the virtual environment or as actuators for entirely virtual are produced with a Zortrax M200 3D printer, an affordable
interfaces inside the VR environment. desktop model and easy to operate. Their shape is different
The TUIs are 3D printed on demand for each project so the user can differentiate them only by using hi/her sense
so they can fit both the project theme and the user who of touch. Each TUI takes less than 20 min to 3D print and is
contributes to the design process. 3D printing is a fast an inex- ready to be used once the AR marker is glued on it (Fig. 12).
pensive manufacturing product that can create very detailed

123
J Intell Manuf (2019) 30:743–754 751

Fig. 10 System configuration of a mixed reality system for interactive virtual prototyping

Fig. 11 Schematic view of the

system

The Unity software implementation offers modularity play. With Unity software, it is possible to assign to each
and simplified user experience. Unity is a flexible develop- TUIs several behaviours depending on the specific design
ment platform for creating multiplatform 3D and 2D games task and the user (Fig. 13).
and interactive experiences. We use it to connect the TUIs to A first version of the tool was successfully implemented
the virtual objects of the 3D scene to the head-mounted dis- and tested with an interior design scenario, as illustrated

123
752 J Intell Manuf (2019) 30:743–754

Conclusion

By combining VR tools and TUIs, we created a MR

computer-aided design system that provides a high level of
immersion and control. Our system enables end-users of a
future product to visualize a virtual prototype of this product
in three dimensions, to manipulate it and to modify it in a
direct, natural and intuitive way. The stimulation of visuali-
sation, hearing and touch ensures a high level of immersion.
The Tangible User Interfaces allow direct and intuitive inter-
Fig. 12 3D printed TUIs equipped with AR markers glued on top of
actions within the virtual environment, therefore they offer a
them
high level of control over the virtual environment and virtual
in Fig. 14. A virtual room environment was created with models
UNITY software; it includes walls, windows, decorative ele- The key benefit of this tool is in that it allows to involve
ments (tapestry, paintings) and floor. Outside the room, we users in the early stages of the design process and to directly
designed a realistic scene with grass, sky and trees to pro- include their taste, feedback and requirements in the design
vide immersion and context to the virtual scene. For the first models. The virtual environment and Tangible User Inter-
prototype, the interactions are controlled with eight TUIs. faces can be easily edited to fit specific design cases. The
Each TUI is linked to a given piece of furniture (e.g. bed, system is now being tested in an experimental protocol aim-
chair, table, desk) and can be used to control the location ing at assessing the satisfaction and the perceived feelings of
and the behaviour (motion) of the virtual piece of furniture. “immersion” and “control” of users, when they are involved
When the user manipulates a TUI, the related piece of fur- in a design task.
niture is placed and moved in the virtual room accordingly. Our future work includes a full experimental evaluation
Thus, through the use of TUIs, the user is able to test several of the usability of the MR tool, as well as its impact on the
design configurations and make design decisions in a natural design process and the design outcomes. The development of
and intuitive way.

Fig. 13 Schematic illustrations

of user interactions with the
mixed reality tool. Interactions
with the left TUI produces a
movement of left object in the
virtual environment (flash)
while interactions with the right
TUI makes the right object
(camera) move

Fig. 14 User interactions with

the mixed reality design tool.
The tested environment is made
of an empty room and eight
pieces of furniture

123
J Intell Manuf (2019) 30:743–754 753

such a digital tool is expected to help designers to closely co- new products. Technovation, 27(6–7), 378–387. doi:10.1016/j.
design with end-users in a direct and efficient way and thus technovation.2006.09.005.
Hamade, R. F., & Artail, H. A. (2008). A study of the influence of
to create user-friendly products and satisfactory subsequent technical attributes of beginner CAD users on their performance.
user experiences. Computer-Aided Design, 40(2), 262–272. doi:10.1016/j.cad.2007.
11.001.
Hart, C. W. L. (1995). Mass customization: Conceptual underpinnings,
opportunities and limits. International Journal of Service Industry
Management, 6(2), 36–45. doi:10.1108/09564239510084932.
References Helms, M. M., Ahmadi, M., Jih, W. J. K., & Ettkin, L. P. (2008). Tech-
nologies in support of mass customization strategy: Exploring the
Arrighi, P.-A., Le Masson, P., & Weil, B. (2015). Managing radical inno- linkages between e-commerce and knowledge management. Com-
vation as an innovative design process: Generative constraints and puters in Industry, 59(4), 351–363. doi:10.1016/j.compind.2007.
cumulative sets of rules. Creativity and Innovation Management, 09.003.
24(3), 373–390. doi:10.1111/caim.12135. Helo, P. (2006). Product configuration analysis with design structure
Bayus, B., & Putsis, W. P. Jr. (1999). Product proliferation: An empir- matrix. Industrial Management & Data Systems, 106(7), 997–
ical analysis of product line determinants and market outcomes. 1011.
Marketing Science. Hideyoshi, Y., & Fukuda, S. (2004). Development of interactive indus-
Bordegoni, M., & Ferrise, F. (2013). Designing interaction with con- trial design support system considering customer’s evaluation.
sumer products in a multisensory virtual reality environment: This JSME International Journal Series C, 47(2), 762–769. doi:10.
paper shows how virtual reality technology can be used instead of. 1299/jsmec.47.762.
Virtual and Physical Prototyping, 8(1), 51–64. Hsu, J.-M. (2010). Design and evaluation of virtual home objects with
Bordegoni, M., Ferrise, F., & Lizaranzu, J. (2011). Use of interactive music interaction in smart homes. Journal of Intelligent Manufac-
virtual prototypes to define product design specifications: A pilot turing, 23(4), 1281–1291. doi:10.1007/s10845-010-0411-6.
study on consumer products. In 2011 IEEE international sym- Hunt, D. (2006). A consumer perspective on mass customization.
posium on VR innovation (pp. 11–18). doi:10.1109/ISVRI.2011. Ph.D Dissertation, Faculty of the Graduate School, University of
5759592. Missouri-Columbia.
Cai, M. Y., Lin, Y., Han, B., Liu, C. J., & Zhang, W. J. (2016). On a Huyett, W., & Viguerie, S. (2005). Extreme competition. McKinsey
simple and efficient approach to probability distribution function Quarterly, 1, 46–57.
aggregation. IEEE Transactions on System, Man and Cybernetics, Hvam, L. (2006). Mass customisation in the electronics industry: Based
Part A. doi:10.1109/TSMC.2016.2531647. on modular products and product configuration. International
Carlile, P. R. (2002). A pragmatic view of knowledge and boundaries: Journal of Mass Customisation, 1(4), 410–426.
Boundary objects in new product development. Organization Sci- Imamura, K., Nomura, J., & Nagamachi, M. (1994). Virtual space
ence, 13(4), 442–455. doi:10.1287/orsc.13.4.442.2953. decision support system using kansei engineering. In Japan–USA
Carulli, M., Bordegoni, M., & Cugini, U. (2013). An approach for symposium on flexible automation, II (pp. 549–555).
capturing the voice of the customer based on virtual prototyping. Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless inter-
Journal of Intelligent Manufacturing, 24(5), 887–903. faces between people, bits and atoms. In Proceedings of the
Cecil, J., & Kanchanapiboon, A. (2006). Virtual engineering approaches SIGCHI conference on human factors in computing systems CHI
in product and process design. The International Journal of 97 (Vol. 39, pp. 234–241). doi:10.1145/258549.258715
Advanced Manufacturing Technology, 31(9–10), 846–856. doi:10. Jiao, J., Ma, Q., & Tseng, M. (2003). Towards high value-added prod-
1007/s00170-005-0267-7. ucts and services: Mass customization and beyond. Technovation,
Cox, W., & Alm, R. (1998). The right stuff: America’s move to mass 23(10), 809–821.
customization. Economic Review-Federal Reserve Bank of Dallas, Kahn, B. (1998). Variety: From consumer’s perspective. In T. H. Ho & C.
3, 48. S. Tang (Eds.), Product variety management: Research advances
Da Silveira, G., Borenstein, D., & Fogliatto, F. (2001). Mass customiza- (pp. 19–37). Berlin: Springer.
tion: Literature review and research directions. International Khurana, A., & Rosenthal, S. (1997). Integrating the fuzzy front end of
Journal of Production Economics, 71(1), 1–13. new product development. MIT Sloan Management Review, 38(2),
Davis, S. (1987). Future perfect. Reading, MA: Addison-Wesley Pub- 103–120.
lishing. Leithinger, D., Lakatos, D., DeVincenzi, A., Blackshaw, M., & Ishii, H.
Du, X., Jiao, J., & Tseng, M. (2006). Understanding customer sat- (2011). Direct and gestural interaction with relief. In Proceedings
isfaction in product customization. The International Journal of of the 24th annual ACM symposium on user interface software
Advanced Manufacturing Technology, 31(3–4), 396–406. and technology—UIST ’11 (p. 541). New York, NY: ACM Press.
Enomoto, N., Nomura, J., Sawada, K., Imamura, K., & Nagamachi, M. doi:10.1145/2047196.2047268
(1995). Kitchen planning system using kansei VR. Advances in Lihra, T., Buehlmann, U., & Beauregard, R. (2008). Mass customisation
Human Factors/Ergonomics, 20, 185–190. of wood furniture as a competitive strategy. International Journal
Ferrise, F., Graziosi, S., & Bordegoni, M. (2015). Prototyping strate- of Mass Customisation, 2(3), 200–215.
gies for multisensory product experience engineering. Journal of Lihra, T., Buehlmann, U., & Graf, R. (2012). Customer preferences
Intelligent Manufacturing. doi:10.1007/s10845-015-1163-0. for customized household furniture. Journal of Forest Economics,
Fornell, C., Johnson, M., & Anderson, E. (1996). The American 18(2), 94–112. doi:10.1016/j.jfe.2011.11.001.
customer satisfaction index: Nature, purpose, and findings. The Lin, Y., & Zhang, W. J. (2006). Integrated design of function, usability,
Journal of Marketing, 60, 7–18. and aesthetics for automobile interiors: State-of-the-art, chal-
Fukuda, S., & Matsuura, Y. (1993). Prioritizing the customer’s require- lenges, and solutions. Proceedings of the IMechE Part I, Journal
ments by AHP for concurrentdesign. In ASME design engineering of Systems and Control Engineering, 220(18), 697–708.
division publications DE (Vol. 52, pp. 13–19). New York: ASME. Magretta, J. (1998). The power of virtual integration: An interview with
Füller, J., & Matzler, K. (2007). Virtual product experience and Dell Computers’ Michael Dell. Harvard Business Review, pp. 74–
customer participation—A chance for customer-centred, really 84.

123
754 J Intell Manuf (2019) 30:743–754

Medyna, G., Nonsiri, S., Coatanéa, E., & Bernardb, A. (2012). Mod- Schütte, S. T. W., Eklund, J., Axelsson, J. R. C., & Nagamachi, M.
elling, evaluation and simulation during the early design stages: (2004). Concepts, methods and tools in Kansei engineering. Theo-
Toward the development of an approach limiting the need for retical Issues in Ergonomics Science, 5(3), 214–231. doi:10.1080/
specific knowledge. Journal of Integrated Design and Process Sci- 1463922021000049980.
ence, 16(3), 111–131. doi:10.3233/jid-2012-0006. Shah, S. G. S., & Robinson, I. (2007). Benefits of and barriers to
Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual involving users in medical device technology development and
displays. IEICE Transactions on Information and Systems. The evaluation. International Journal of Technology Assessment in
Institute of Electronics, Information and Communication Engi- Health Care, 23(1), 131–137. doi:10.1017/S0266462307051677.
neers, 72(12), 1321–1329. Sidharta, R., Oliver, J., & Sannier, A. (2006). Augmented tangible
Nagamachi, M. (2008). Perspectives and the new trend of Kan- interfaces for product assembly planning. International Journal
sei/affective engineering. The TQM Journal, 20(4), 290–298. of Product Lifecycle Management, 1(3), 321–333.
doi:10.1108/17542730810881285. Slater, M., Linakis, V., Usoh, M., Kooper, R., & Street, G. (1996).
Nagamachi, M. (2010). Kansei/affective engineering. New York: CRC Immersion, presence, and performance in virtual environments:
Press. An experiment with tri-dimensional chess. In ACM virtual reality
Ninan, J. A., & Siddique, Z. (2006). Internet-based framework to software and technology (VRST) (pp. 163–172). Hong Kong: ACM
support integration of customer in the design of customizable Press.
products. Concurrent Engineering, 14(3), 245–256. doi:10.1177/ Thornton, L. G. (2006). Survey of US business leaders (12th ed., 16 pp).
1063293X06068391. Chicago, IL: Grant Thornton International.
Ogi, T., Tateyama, Y., & Haruyama, S. (2010). Education on human Toffler, A. (1970). Future shock. New York: Amereon Ltd.
centered design using virtual environment. In Volume 3: 30th Com- Trentin, A., Perin, E., & Forza, C. (2012). Product configurator impact
puters and information in engineering conference, parts A and B on product quality. International Journal of Production Eco-
(pp. 667–672). ASME. doi:10.1115/DETC2010-28624. nomics, 135(2), 850–859. doi:10.1016/j.ijpe.2011.10.023.
Pahl, G., & Beitz, W. (1988). Engineering design: A systematic Trentin, A., Perin, E., & Forza, C. (2014). Increasing the consumer-
approach. NASA STI/Recon Technical Report A, 89, 47350 perceived benefits of a mass-customization experience through
Perdue, B., & Summers, J. (1991). Purchasing agents’ use of negotiation sales-configurator capabilities. Computers in Industry, 65(4), 693–
strategies. Journal of Marketing Research, 28(2), 175–189. 705. doi:10.1016/j.compind.2014.02.004.
Petiot, J.-F., & Yannou, B. (2004). Measuring consumer perceptions for Tseng, M. M., & Piller, F. T. (2003). The customer centric enterprise:
a better comprehension, specification and assessment of product Advances in mass customization and personalization (536 pp).
semantics. International Journal of Industrial Ergonomics, 33(6), Berlin, Heidelberg: Springer.
507–525. doi:10.1016/j.ergon.2003.12.004. Van der Vegte, W. F., & Horváth, I. (2002). Consideration and modeling
Piller, F. (2007). Observations on the present and future of mass of use processes in computer-aided conceptual design: A state of
customization. International Journal of Flexible Manufacturing the art review. Journal of Integrated Design and Process Science,
Systems, 19(4), 630–636. 6(2), 25–59.
Pine, B. (1993). Mass customization: The new frontier in business com- Wan, Z., & Bullard, S. (2008). Firm size and competitive advantage
petition. Academy of Management Review, 19(3), 588–592. in the US upholstered, wood household furniture industry. Forest
Rautenstrauch, C., Seelmann-Eggebert, R., & Turowski, K. (2012). Products Journal, 58(1/2), 91–96.
Moving into mass customization: Information systems and man- Wang, L., Shen, W., Xie, H., Neelamkavil, J., & Pardasani, A.
agement principles. Berlin/New York: Springer. (2002). Collaborative conceptual design-state of the art and future
Richards, D., Boettger, K., Fure, A. B., & Oscar, A. (2003). Require- trends. Computer-Aided Design, 34(13), 981–996. doi:10.1016/
ments reconciliation using a computer supported collaborative S0010-4485(01)00157-9.
approach. Journal of Integrated Design and Process Science, 7(2), Zeithaml, V. A. (1988). Consumer perceptions of price, quality, and
113–129. value: A means-end model and synthesis of evidence on JSTOR.
Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the Journal of Marketing. doi:10.2307/1251446.
new landscapes of design. CoDesign, 4(1), 5–18. doi:10.1080/
15710880701875068.

123