TECHNOLOGIES AND TECHNIQUES FOR COLLABORATIVE

ROBOTICS IN ARCHITECTURE

- establishing a framework for human-robotic design exploration

MADS BRATH JENSEN1 and AVISHEK DAS2

1,2
Department of Architecture, Design and Media Technology
1,2
{mbje|adas}@create.aau.dk

Abstract. This study investigates the technological and methodological

challenges in establishing an indeterministic approach to robotic
fabrication that allows for a collaborative and creative design/fabrication
process. The research objective enquires into how robotic processes in
architecture can move from deterministic fabrication processes towards
explorative and indeterministic design processes. To address this
research objective, the study specifically explores how an architect and
a robot can engage in a process of co-creation and co-evolution, that
is enabled by a collaborative robotic arm equipped with an electric
gripper and a web camera. Through a case-based experiment, of
designing and constructing an adjustable façade system consisting
of parallel wood lamellas, designer and robotic system co-create by
means of interactive processes. The study will present and discuss
the technological implementations used to construct the interactive
robotic-based design process, with emphasis on the integration of visual
analysis features in Grasshopper and on the benefits of establishing a
state machine for interactive and creative robotic control in architecture.

Keywords. Design cognition; Digital fabrication; Construction;

Human-computer interaction.

1. Introduction
The background for this work is the capacity for Industrial robotic arms to
engage and change the way architects explore and fabricate novel structures
and material compositions. Also, the advancement of computational design
processes and CAD-related technologies have made possible the modelling,
analysis and simulation of complex performance-driven constructs. The recent
development of robotic arms, with their versatile and highly customizable setup,
has made the fabrication of these architectural constructs feasible. In the
most exceptional cases, the robotic arm is such a well-integrated aspect of the
design exploration that one cannot separate the resulting design from its means
of fabrication. The use of industrial robots has become a “transformational
technology in architecture” (Daas and Wit, 2018). Despite these advancements, in

RE: Anthropocene, Proceedings of the 25th International Conference of the Association for Computer-Aided
Architectural Design Research in Asia (CAADRIA) 2020, Volume 2, 293-302. © 2020 and published by the
Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong.
294 M.B. JENSEN AND A. DAS

the majority of robotic-driven architectural projects, the robot, however complex

tasks its performing, is following the deterministic set of commands given by the
architect through a file-to-factory procedure (Pigram, Maxwell and Mcgee, 2016).
This pre-planned procedure precludes the explorative element from the robotic
fabrication process, as the architect has no option of intervening by re-directing
or altering the fabrication process. This places the architect in the role of
disengaged spectator. To successfully integrate robotic-driven design exploration
the fabrication process needs to allow for human intervention and support an
indeterministic search of the problem-solution space. This issue is discussed by
Bryan Lawson in his seminal work on design thinking (Lawson, 2005) and later
by Mary-Lou Maher who suggested a cognitive model for co-evolutionary design
featuring two parallel search spaces; the problem space and the solution space
(Maher, 1994). The work of Kees Dorst and Nigel Cross has also supported
Maher’s co-evolution model and use it to explain the behaviour found in their
protocol studies of experienced designers regarding the nature of creativity in
design (Dorst and Cross, 2001).
Therefore, this study investigates the technological and methodological
challenges in establishing a framework for robot-based design exploration that
allows for a collaborative and creative fabrication-driven design process. By
taking advantage of existing methods and technological advancements in the field
of computational design and architectural robotics, the study aims to establish and
showcase suitable methods and procedures for connecting these fragments into a
framework for design exploration with collaborative robotics.
Previous work within the field of interactive robotic-driven processes in the
architectural domain already exists. Through their ‘Mixed Reality Modeling’
project, Johns et al. (Johns, Kilian and Foley, 2014) showcased an iterative
process of robotic heat-gun melting of wax. Seeking to include “human in the
loop modifications” their proposed design process allowed the designer to either
manually remove wax from the physical object or to spray paint the object and
thereby directly inform the robot where to perform the heat-gun treatment. The
relevance of feedback loops has also been discussed by Dubor et al. (Dubor et al.,
2016) who by exploring a series of case studies propose a framework for human
interaction and machine response. In their paper, they conclude that ‘collaboration
between robots and human can enhance creativity and innovation by supporting
designer and researcher while exploring complex material systems.’ (Dubor et
al., 2016). Similar work has been conducted by Moorman et al. (Moorman,
Sabin and Liu, 2016) through the construction of a framework for dynamic
robotic fabrication. Their proposed RoboSense framework seeks to promote a
feedback-oriented design process where the robot shifts from being an executor of
explicit commands to an “actor in a dynamic and reciprocal relationship with its
fabrication environment” (Moorman, Sabin and Liu, 2016).
The projects referenced above presents exciting advancements in the
construction of technological and methodological frameworks for robotic
interaction in a design content and how these fabrication processes can support
design exploration. At the same time, it is essential to recognize that all of
the above projects work with either purely interactive processes or re-active
 TECHNOLOGIES AND TECHNIQUES FOR COLLABORATIVE 295
ROBOTICS IN ARCHITECTURE

processes, as in the wine pouring case study in the RoboSense project. To obtain
a collaborative process the robot needs to perform in a way that exceeds what
can be anticipated by the designer - a robotic agent that contributes with actions
and intentions that assist the human designer in exploring unknown areas and
connections between a given problem-solution space.
In the field of architectural robotics, a growing range of technologies and
methods are emerging, of which some hold great potential for supporting a
framework for collaborative human-robot design exploration. As displayed in the
diagram in Figure 1, the construction of a framework for collaborative robotic
design processes requires certain key elements. While some are well described,
and standard in the field of computational architecture and robot-based design,
others need to be adopted from other research fields. Another important aspect for
a proposed framework is the option of parallel exploration of the physical design
object and its digital twin, which demands that the physical object (or system) can
be reproduced for further exploration in a CAD environment.

Figure 1. Framework for collaborative human-robot design exploration.

This study aims to utilize and explore the missing links needed to connect these
fragments into a combined framework for design exploration with collaborative
robotics. Therefore it is helpful to briefly look at the existing work for each of the
key elements displayed in Figure 1.
Most of the elements needed to construct a collaborative framework already
exist; however, a method for connecting and controlling all the sequential
processes needs to be defined. As the behaviour of each key element is likely
to pass through a series of clearly defined steps, triggered by input from either
the internal processes of the system or from external user input, the concept of
state machines are interesting. A state machine can be defined “by identifying
what states the system can be in, what inputs or events trigger state transitions,
and how the system will behave in each state” (Wright, 2005) and can be used to
control the behaviour of simple systems, as in the example in Figure 2, or very
complex UI systems. As GH is based on dataflow programming, suitable methods
will have to be investigated to ‘break’ this flow and construct a customizable state
machine.
296 M.B. JENSEN AND A. DAS

Figure 2. Example of a State Machine Diagram, with the key concepts ‘State’, ‘Event’,
‘Action’ and ‘Transition’.

The paper will present and discuss the implementations of existing

technologies and methods to construct a new framework that supports a
collaborative robotic-based design exploration. The paper will elaborate on the
integration of visual analysis features in GH and on the benefits and challenges of
establishing a state machine for controlling sub-routines. By presenting a design
example, the paper will showcase the collaborative design process made possible
by utilizing the proposed explorative framework.

2. Methods
To investigate the technologies and techniques needed to support a collaborative
robotic-based design process, the study applied a research-by-design strategy
(Hauberg, 2011). The strategy relies on physical and digital prototyping for
uncovering possible solutions and allows for a continuous and parallel process of
designing the framework and designing with the framework. The design process
thereby informs the development of the framework and vice versa.

2.1. THE PHYSICAL DESIGN SYSTEM

To facilitate continuous investigation and development of suitable methods
and techniques for robotic-based design exploration, a bespoke physical
design system is developed during the study. This material-based system is
developed on the criteria that both designer and robot should be capable of
manipulating and changing, in a reversible manner, the system’s inherent design
variables. These geometric configurations also have to permit both analogue
qualitative user-driven design evaluations and numeric-based performance-driven
computational simulations. To meet these criteria, a façade element consisting
of twenty-four identical wood lamellas within a steel frame was designed and
constructed (see Figure 3). Each lamella was constrained to a pre-made groove
in the bottom rail, allowing for 90-degree stepwise rotation, and fixed in a
rotatable acrylic disc within the top rail, allowing for 45-degree stepwise rotations.
Reconfiguration of the lamellas’ bottom parts was restricted to manual user
interaction. In contrast, the top parts could be rotated through a collaborative
process with a robot performing the rotational movement of the lamella and a
human user removing and inserting locking pins to fixate the lamella.
On the robotic side, the physical design system consisted of a UR10 robotic
arm from Universal Robots equipped with a RG6 electric gripper from OnRobot
onto which Logitech’s Brio 4K Ultra HD webcam was mounted. To enable user
 TECHNOLOGIES AND TECHNIQUES FOR COLLABORATIVE 297
ROBOTICS IN ARCHITECTURE

input, two simple push-buttons were connected to the digital inputs in the control
box of the UR10.

Figure 3. The physical design system consist of 24 wood lamellas mounted in a steel frame.
Variations caused by rotation of individual elements creates potential for directed views and
intentional blocking of sunlight.

2.2. THE COMPUTATIONAL DESIGN SYSTEM

The study utilized the Rhino-Grasshopper environment for developing a
computational design framework integrating the following five sub-systems. A
‘Computational Design System’ containing the generation and geometric-driven
manipulation of the virtual façade system. An ‘Environmental Simulation’ that
calculates the sun shading and view-blocking performance of the virtual façade
system. A ‘Robotic Simulation and Control System’ that based on availability,
ease of use and utility used Vicente Soler’s (UCL Bartlett) Robots for generating
target planes, simulating the robotic movements and streaming code to the robot.
A ‘Visual Analysis System’ containing custom components that utilize computer
vision libraries from OpenCV for tracking the position of the physical wood
lamellas. A ‘State Machine’ which, based on components from the MetaHopper
add-on by Andrew Heumann, handles the transitions between predetermined states
and triggers actions in the order they need to be executed.
298 M.B. JENSEN AND A. DAS

3. Results
3.1. COMPUTATIONAL FRAMEWORK FOR COLLABORATIVE DESIGN
EXPLORATION
The study resulted in a collaborative framework that integrates visual analysis
methods and a state machine to successfully allow for human-robot design
exploration of a material system. The computational design framework allows for
an interactive design exploration where a human agent, guided by design intentions
regarding obstruction of desired view lines and sun shading, can manually alter
the rotation of the wood lamellas. Subsequently, the robotic agent can be initiated
and via the mounted camera register the current rotation for each lamella. This
information allows the computational design model to perform environmental
simulation based on a series of alternative lamella configuration and suggest a
new and improved version by robotic rotation of the wood lamellas. As visualised
in the flow chart in Figure 4, the framework allows interactions by the human
designer (cyan coloured circles in the flow chart) to occur both during the physical
material-based design exploration and the robotic fabrication process.

Figure 4. Flow chart for the proposed human-robot framework for collaborative design
exploration. The cyan-coloured circles represent human actions during the design process
while the three types of colored flow-lines refer to processes within the ‘Human mode’ (pink),
‘Robot mode 1’ (orange), and ‘Robot mode 2’ (green).

The main result of the study is the design of the collaborative framework in its
entirety and the design process it supports. However, two aspects were crucial for
successfully achieving this objective; the integration of visual analysis features in
GH and the introduction of a state machine for controlling the interactive robotic
processes.
 TECHNOLOGIES AND TECHNIQUES FOR COLLABORATIVE 299
ROBOTICS IN ARCHITECTURE

3.2. VISUAL ANALYSIS IN GH

Integration of visual analysis was achieved by implementing functions from
the OpenCV library into custom GH-components, thereby enabling tracking of
individual lamella positions. As the standard python component in GH does not
allow the use of libraries like OpenCV and NumPy, the custom VA-components
were instead build using GH Python Remote by Digital Structures. The
VA-components allowed two types of tracking; registration of ArUco markers
and recognition of custom coloured markers. The ArUco marker, developed by
Rafael Muñoz and Sergio Garrido (Garrido-Jurado et al., 2014), is a synthetic
square marker identifiable by an inner binary matrix surrounded by a black border
(OpenCV - Open Source Computer Vision, 2019). By attaching ArUco markers
on each lamella the visual system, using the cv2.aruco module in the OpenCV
library, was able to distinguish between the 0- or 90-degree rotation allowed for
the bottom part of the lamella system. Custom red and cyan colour markers were
applied to the top of each lamella, and by colour recognition, the VA-system could
calculate the individual rotation vectors. The VA-process for tracking the custom
markers consisted of a set of discrete steps, starting with the capture of two images
containing the twelve wood lamella for the left and right side, respectively. As
can be seen from the diagram and the pictures in Figure 5, the captured images
are cropped to the boundary zone of the lamella ruling out any unwanted artefacts
and allowing for efficient computation of the lamella rotation angles. Next, two
image masks are created based on the predefined red and cyan colour in HSV
colour mode. These masks are used to subtract the background leaving only red
and cyan colour allowing a calculation of the contour for each coloured area and
a subsequent approximation of the outline of these regions. A boundary rectangle
for each coloured region serves to locate the individual centroids and from these
establish the two-point vectors (with direction from red-centroid to cyan-centroid)
that informs about the rotation of the wood lamella. Based on the bottom and top
vector the twisting of each lamella was determined and the digital version of the
design system could be updated so as to mirror its physical “twin”. Each of the
GH components developed for visual analysis were designed with an input field for
activation of the internal logic and an output field with a boolean value flagging its
successful completion - simple features that were crucial for integration of a state
machine.

3.3. STATE MACHINE IN GH

Construction of a custom state machine inside GH was achieved through custom
python components and use of components from the MetaHopper add-on, as
shown in Figure 6. The ‘SetObj’ component from MetaHopper allowed the
framework to use the Boolean value from selected components to control the value
of standard GH-buttons, in other words getting a ‘True/False’ output from one
component would ‘Push/Release’ another GH-button. The state machine allowed
control and activation of sequential stages in the established human-robot design
process.
300 M.B. JENSEN AND A. DAS

Figure 5. Top: Diagram of the visual analysis process from image data to rotational data.
Bottom: Example of the visual analysis routine performed on the top side of the wood lamellas
by using OpenCV in Python. The first picture shows the cropped image recorded by the
robot-mounted webcam. The second picture shows the masking out of all unwanted colours.
The third picture shows the result of applying the mask to the cropped image. .

Figure 6. Example of a simple State Machine in Grasshopper. The Grasshopper definition uses
custom python components and MetaHopper components to change and keep track of states.
The State Machine integrated in the proposed framework is an expansion of this setup.
 TECHNOLOGIES AND TECHNIQUES FOR COLLABORATIVE 301
ROBOTICS IN ARCHITECTURE

4. Discussion
Following a research-by-design approach, this study has investigated and
established a framework that allows a designer to engage directly with a physical
design object, while in succession obtaining new design suggestion from a robotic
system. The development process and the final result reveals essential aspects of
human-robot design exploration.
The camera used to capture the coloured markers and the ArUco markers has an
automatic focus feature which often affects the image capturing process and results
in blurry photos with a negative effect on the colour detection procedure. During
the prototyping process, this was resolved by inserting a time delay (approx. 2
seconds) between robot (and camera) arriving at capture position and the actual
capturing process. Another challenge, well-known in the field of visual analysis,
is the importance of lighting conditions. The colour detection algorithm used
in the framework takes in the absolute HSV values from the colour system and
detects accordingly within a given range the varying hue, saturation and value. In
the physical setup, due to the presence of a skylight in the indoor environment,
the ambient light varied by a visible spectrum to cause significant error in the
colour detection. This issue can be mitigated using complete artificial light or by
minimizing the effect of varying coloured light in the system.

Figure 7. Physical demonstrator placed in an outdoor environment. The façade system clearly
displays its environmental performance towards shading the sun and directing views.

When taking part in a creative and collaborative design process, the experience
of time and the maintenance of creative flow is essential. An important aspect
of successful collaborative work is knowing the intention of the co-workers - a
challenging aspect when working with robots. Not knowing what goes on “behind
the scenes” during ex the time-span of computational performance search, which
often took 30-90 seconds, leaves the designer in a state of passive waiting. Initial
experiments show a significant difference in running the collaborative process with
or without the opportunity to see GH-based visualisations of the computational
302 M.B. JENSEN AND A. DAS

performance search. Through parallel design and evaluation of the framework, it

was evident that the additional time used to ex resolve the issue of automatic focus,
as mentioned above, could be solved through technological changes/upgrades.
Optimising the algorithms used in the framework is another area for optimisation.
Many of the computational processes run in a series, and the initiation of a task is
often dependent on the completion of previous tasks. In some cases, it would be
more efficient to employ multiprocessing to ex. move the robot between the image
capture targets, while simultaneously running the image analysis on previously
captured images - the proposed framework currently waits for each image to be
captured and processed before moving the robot to the next target position.
The study has sought to carry out foundational work on which to base
applied research. The paper has focused on the technological and system-oriented
aspects of collaborative human-robot design exploration and shown that the
proposed framework can support such processes. For future work it will be
important to investigate to what extent this robotic-based approach will affect the
creative design process and if a fruitful “dialogue” can be established between
a holistic-driven human designer and a performance-driven solution-proposing
robot.

References
“OpenCV - Open Source Computer Vision” : 2019. Available from <https://docs.opencv.
org/trunk/d5/dae/tutorial�
e xtbackslash_aruco�e xtbackslash_detection.html> (accessed 10th
December 2019).
Daas, M. and Wit, A.J. 2018, Introduction, in M. Dass and A.J. Wit (eds.), Towards a Robotic
Architecture, Applied Research and Design Publishing, Novato, CA, 8-11.
Dorst, K. and Cross, N.: 2001, Creativity in the design process: Co-evolution of
problem-solution, Design Studies, 22(5), 425-437.
Dubor, A., Camprodom, G., Diaz, G.B., Reinhardt, D., Saunders, R., Dunn, K., Niemel”a, M.,
Horlyck, S., Alarcon-Licona, S., Wozniak-O’Connor, D. and Watt, R.: 2016, Sensors and
Workflow Evolutions: Developing a Framework for Instant Robotic Toolpath Revision,
Robotic Fabrication in Architecture, Art and Design 2016, Cham, 410-425.
Garrido-Jurado, S., Mu\˜noz-Salinas, R., Madrid-Cuevas, F. and Mar’in-Jim’enez, M.: 2014,
Automatic generation and detection of highly reliable fiducial markers under occlusion,
Pattern Recognition, 47(6), 2280-2292.
Hauberg, J.: 2011, Research by design: a research strategy, Architecture \& Education Journal,
1(2), 1-11.
Johns, R.L., Kilian, A. and Foley, N.: 2014, Design Approaches Through Augmented
Materiality and Embodied Computation, Robotic Fabrication in Architecture, Art and
Design 2014, Cham, 319-332.
Lawson, B.: 2005, How Designers Think - The Design Process Demystified, Taylor \& Francis.
Maher, M.L.: 1994, Creative design using a genetic algorithm, Computing in Civil Engineering,
Nwe York, 2014-2021.
Moorman, A., Sabin, J.E. and Liu, J.: 2016, RoboSense: Context-Dependent Robotic Design
Protocols and Tools, ACADIA 2016 Posthuman frontiers, 174-183.
Pigram, D., Maxwell, I. and Mcgee, W.: 2016, Towards Real-Time Adaptive Fabrication-Aware
Form Finding in Architecture, Robotic Fabrication in Architecture, Art and Design 2016, 1,
427-437.
Wright, D.R.: 2005, “Finite State Machines” . Available from <http://www4.ncsu.edu/̃drwrigh
3/docs/courses/csc216/fsm-notes.pdf>.