AUGMENTED ROBOTIC BRICKLAYING

An Experiment in Remote Programming Robotic Assembly Using Augmented Reality

for Brick-Based Structures

YANG SONG1and SOOMEEN HAHM2

1
Department of Architecture, University of Liverpool
2
the Southern California Institute of Architecture (SCI-Arc)
1
[email protected], 0000-0003-0340-8629
2
[email protected], 0000-0003-2868-9375

Abstract. After experiencing the Covid-19 pandemic, remote

communication became one of the key issues in almost every field and
discourse. Digital fabrication is no exception, and architects hope to
seek a user-friendly way for human-machine interactions. This paper
presents experimental research using Augmented Reality (AR) for
robotic remote programming. The research tries to develop a unique
pipeline and workflow which allows users from different locations to
program robots and communicate with machines through AR. A sample
workflow has been tested as a series of simple brick assemblies in an
online workshop with remote participants. The pipeline allows all users
to be able to remotely program and control a robot in AR. For this
workshop, we transform the robotic coding method from the traditional
computer science way to the plugin-oriented AR visual programming
way in Grasshopper. As for the physical outcomes, participants all
assembled brick-based structures successfully by programming and
operating the robotic arm in AR remotely at the end. Associating the
interaction in AR with the robotic arm and programming it with
interactive visual input methods will make it easier for architectural
practitioners to simulate and control industrial robots for complex
structure assembly.

Keywords. Augmented Reality (AR), Remote Programming, Robotic

Assembly, Brick-based Structure, Online Workshop.

1. Introduction
Due to the Covid-19 breakout and the social distancing requirements, students were
restricted from using on-campus facilities, which disrupted experiential learning that
relies heavily on face-to-face interaction (Tatiana et al., 2021). Under the influence of
this pandemic, education and practice have to move remotely online. Both instructors
and students are experiencing dramatic changes in their modes of teaching and
practising (Duaa et al., 2021). In architectural education fields, with few relevant

HUMAN-CENTRIC, Proceedings of the 28th International Conference of the Association for Computer-Aided
Architectural
– LEAVE Design Research
THIS WHITE BOXinON
Asia (CAADRIA)
PAGE01!!– If it has2023, Volume
moved, 1, 323-332.
you can cut and paste it © 2023
back and1, published
to page right click onby
thethe
Association for Computer-Aided Architectural Design Research in Asia (CAADRIA), Hong Kong.
boundary and choose 'More Layout Options...' and then under 'Vertical', choose 'Absolute position' - 24 cm (below Page).
324 Y. SONG AND S. HAHM

resources and experience, the above challenges have inspired many researchers to
explore and develop teaching and practising methods in remote communication modes
(Antonio and Lucas, 2021). Moreover, automated architectural fabrication practices,
which proceed by machines instead of humans, have become an efficient way to
produce physical outcomes from digital design to deal with the Covid-19 situation
(Regiane and Luiz, 2020). However, current research gaps exist because the remote
software and interfaces are not clear and straightforward enough to complete complex
teaching and practising tasks. According to the above changes brought about by the
pandemic, there is increasing interest in integrating easier and more intuitive ways for
remote education and fabrication in the architectural areas.
The mechanization and industrialization of architecture were a dream of the
modernists and gained momentum due to increased access to robotics (Gilles, 2016).
Moreover, the coupling of parametric computer-aided design (CAD) with robotic
fabrication equipment has enabled the materialization of designs with previously
unfathomable levels of complexity and variation (Ryan and Jeffrey, 2018). Current
research gaps also exist in that conventional robotic operation requires not only the
corresponding computer science knowledge but also demands code debugging and
simulation on-site. Even so, the entire operation process is tedious and full of
unpredictable errors, requiring people with solid background skills to solve or assist
during the process, which makes it challenging and unsafe to introduce to architectural
practitioners in a short period. The attempt to introduce robots into the construction
industry is not a new research topic (Ines and Merav, 2015). However, exploring a
remote and user-friendly robotic operation method convenient for architectural
participants is the research in line with the current pandemic trends in the architectural
digital fabrication field.
With the development of mixed reality (MR) technology, AR has gradually entered
the field of digital fabrication research. The AR system has the following
characteristics: combining real and virtual objects in a natural environment, running
interactively and in real-time, and registering real and virtual objects contextually in
3D (Bhaskar and Eliot, 2019). Moreover, based on the above unique features, AR is an
interfacing technology that has recently become popular in remote architectural
practice and industrial robotic fabrication sectors (Chu et al., 2020). Intertwining AR
technology with architectural practice opens up possibilities for a remote environment
where humans can intuitively communicate with digital content, objects, machines,
and spatial context (Song et al., 2021).
This paper presents experimental research using AR technology as an interface for
robotic remote programming in architectural practice. The research contributes to
developing a unique remote mode and workflow that allows users from different
locations to program and operate industrial robotic arms through AR for architectural
proposal fabrication. This sample workflow will be tested as a series of simple robotic
brick-based structure assemblies through the online workshop with remote
participants. Additionally, this online workshop is used as an example to reflect on this
new mode of online teaching and remote digital fabrication in architecture education
and fabrication reacting to the Covid-19 pandemic.
 AUGMENTED ROBOTIC BRICKLAYING 325

2. Methodology
The Augmented Robotic Bricklaying workshop proposes an experiment in remote
robotic programming consisting of two phases: A) robotic operation in AR, in which
interactive inputs will communicate with the robotic programming process in AR
through the screen-based user interface (UI) (e.g., smartphones or tablets) to achieve
the trajectory planning and end-effector commanding; B) remote robotic assembly, in
which user can edit and preview on the AR virtual robot anywhere for grabbing
commands and assembly sequences, and map the operations remotely to the robotic
arm in the lab for the physical assembly (Figure 1). This research aims to provide
architects with a convenient and easy operation method for robotic fabrication in line
with their corresponding knowledge reserves and develop a remote robotic control
method that conforms to the pandemic needs. This research adopts the method of
preliminary online workshop experimental verification from the participants to validate
the feasibility of the above aims and summarise the findings and limitations.

Figure 1. This is the flow chart of the Augmented Robotic Bricklaying workshop, including Phase A
(Robotic Operation in AR) and Phase B (Remote Robotic Assembly) (in blue). The related plugin
for each critical step (in red) and the outcomes of each phase (in green) are also illustrated.

For this workshop, we transform the robotic coding method from the traditional
computer science way to the plugin-oriented visual programming method in the
Grasshopper environment, which is familiar for architectural practitioners to
understand and manipulate. The employed programming method is driven by an
instant connection between the development environment (Grasshopper), AR
interactive immersion plugin (Fologram), and robotic operation plugin (Robots).
Fologram is an AR plugin developed by architects, which can translate interactive
inputs, such as hand gestures, screen taps, device location and QR codes, into digital
data in the Grasshopper. It bridges humans and machines between physical and virtual
through AR. Robots is a robotic plugin developed by architects, which enables the user
to program the robotic arm in Grasshopper, such as trajectory planning, gripper
commands, and other operational details. Implementing these plugins will provide
326 Y. SONG AND S. HAHM

architectural practitioners with an easy robotic programming method in AR as a

medium. The above plugins work with their integrated graphical algorithm editor,
Grasshopper, a standard tool in architectural fields, and can easily be integrated into
the established immersive robotic programming workflow.
The hardware in this workshop includes mobile devices (smartphones or tablets)
for AR-assisted robotic programming, and a Universal Robot 10e robotic arm with
RobotiQ 2F-140 gripper in the lab for the remote robotic assembly. Additionally,
workshop participants need their personal computers for the AR-assisted robotic
programming development in Grasshopper and the back-end programs running; as
well as the lab-based operator needing a laptop for activating and debugging the robotic
operations. Except for the robotic arm, we have to ensure that any software and
hardware used in this workshop should be ubiquitous, so that all participants can assess
and manipulate the corresponding operations easily and remotely.
This online workshop was taken by 5 bachelor students majoring in architectural
design. They all have 3D-modeling experience, but none had prior knowledge and skill
in robotic fabrication. Either Fologram or Robots plugin was new to them. Moreover,
due to the robotic lab safety restrictions, foam bricks are chosen instead of standard
bricks for the physical experiments.
The original contribution of our workshop is to integrate and combine the different
functions from various plugin applications into the remote robotic programming
method through AR for the brick-based structure assembly and verify the feasibility of
this remote method on the participant's experience and outcomes.

3. Experiments and Findings

The online workshop was limited to three days, and its schedule was split between
introduction lectures, plugin tutorials, digital workflow development, discussions,
remote robotic assembly experiments, and final reviews.
On the first day, participants received introductory lectures about AR and its
interactions in architectural and robotic fields, as well as information about the topic
and schedule of this workshop. After that, we brought the Fologram tutorials to the
participants on their mobile devices, as well as computers, to demonstrate the essential
interactive AR functions, including interactive input methods, holographic preview,
QR code recognition, and AR user interface (UI) developments. These example files
are based on the Rhino/Grasshopper environment.
On the second day, we delivered the Robots tutorial to the participants, so they
could understand and program robotic operations in Rhino/Grasshopper, such as
trajectory planning, grabbing commands, and other robotic operation settings. By
combining the AR skills from the first day, participants practiced programming the
robot in AR through UI and previewing the robotic operation simulation as 3D
holograms to check the details. Furthermore, time slots were set for discussing the
robotic operation in AR (Phase A experiment).
On the last day, the participants were asked to play with the prepared brick-based
structure AR design scripts developed by Song (Song et al., 2022) for parametric brick-
based structure design. After that, participants were asked to program and operate the
robotic arm for pick and place assembly remotely. The physical robotic operation of
 AUGMENTED ROBOTIC BRICKLAYING 327

brick-based structure assembly (using foam bricks for our experiments) was live-
streamed from the lab. The end of the day was dedicated to the final review and
extensive discussion about the findings and user experience of the remote robotic
program and operation (Phase B experiment).
In this section, the two phases of workshop outcomes and research findings, robotic
operation in AR and remote robotic assembly process, are illustrated and concluded.

3.1. PHASE A - ROBOTIC OPERATION IN AR

Phase A proposes a practical and user-friendly method for architects to control
industrial robots in AR, instead of the conventional programming process. Compared
with mastering cumbersome computer programming languages, users only need to be
familiar with the fundamental common sense of AR and the essential interactive
function of AR UI, which will be easy and controllable for beginners. The pre-set
background scripts are developed and run in Grasshopper. Participants only need to
understand the script's logic, but do not need to edit them unless they need other
customized robotic functions.
To start the robotic operation in AR, first, participants should scan the Fologram
QR code generated from Rhino with their mobile app to activate and connect their AR
environment with the laptops. Second, participants can select the serial robotic arm (UR
10e for this workshop) and preview the virtual machine as holograms through the AR
UI. Next, after understanding the safety information, such as the working radius of the
selected machine, the user can start programming the trajectory through screen-based
AR UI inputs. By tapping the mobile screen at the correct position and angle, the
waypoints with corresponding coordinate information will be added to the trajectory
for participants to preview and modify in AR. Moreover, from the AR UI, users can
edit more specific operation settings, including robotic speed, liner or joint movement
mode, end-effector commands, and node delay time. Last, participants can preview the
virtual robotic movements as 3D AR animations to simulate trajectory and end-effector
operation in AR (Figure 2).

Figure 2. This is the screenshot of the live stream demonstration for the robotic operation simulation
in AR during the workshop tutorials. The pre-set script works in the AR UI from Fologram App
(left), and Grasshopper (right).
328 Y. SONG AND S. HAHM

After the demonstration during the workshop, participants started to get familiar
with the AR UI operating environment and method and use the pre-set scripts to
program and simulate the robotic operation through AR, including trajectory planning,
end-effector commands, movement mode, operation speed, and operation node delay
time (Figure 3). The participants were asked to document their operation process as
AR videos, share and discuss the experiments at the end of the day. Furthermore, it was
assessed whether the interactive AR programming was well engaged, e.g., regarding
the operation of AR UI and the simulation of customized commands.

Figure 3. This is the AR UI (Fologram app) screenshot of participants programming robotic

trajectory and related detail settings in the AR environment.

The findings of phase A suggest that compared with the conventional coding-based
robotic programming method, this screen-based AR programming method can allow
the user to demonstrate robotic operation, set commands, and preview the simulation
efficiently. After the basic AR interactive function introduction, without any robotic
skills, participants can easily program and simulate the robot within ten minutes on
their mobile devices. The simulation results are all consistent with the participants'
settings. Moreover, the users' feedback was good during the AR programming process.
All participants, without a computer science background knowledge, indicated that
they could use AR UI easily and freely to program the robotic arm and preview the
simulation after the simple pieces of training. However, the commands for the end-
effector are currently limited to 'open' and 'close' due to the function of our robotic
gripper. But this pre-set script is based on the Grasshopper, which is believed to be
friendly to architectural practitioners and conducive to developing other subsequent
robotic functions depending on different end-effectors.

3.2. PHASE B - REMOTE ROBOTIC ASSEMBLY

Phase B proposes a remote robotic operation method through AR for brick-based
structures. Compared with the inflexible traditional on-site operation method,
participants can set, edit, and preview the robotic assembly sequence anywhere through
mobile devices (smartphones or tablets) in AR, and transmit commands to the lab-
based or on-site industrial robots to accomplish the physical assembly, which
 AUGMENTED ROBOTIC BRICKLAYING 329

accommodate the requirements of the pandemic situation. This pre-set script is

developed and runs in Grasshopper. The remote robotic operation is transmitted by
storing and reading command data on QR codes through AR UI. Participants should
understand the AR remote operation process logic, but do not need to edit the script
unless they need other customized operations. Besides that, participants also need to
master the method of storing and reading QR codes to realize the remote control.
To start the remote robotic assembly, the user should first design the brick-based
structures as the assembly targets using the pre-set or customized design scripts
provided by the workshop instructor. Second, participants need to scan the Fologram
QR code generated from Rhino to pair and activate their mobile app with Grasshopper
and import the brick-based structures into the AR environment. After selecting the
serial robotic arm (UR 10e) and gripper (RobotiQ 2F-140), participants can preview
the brick-based design and the virtual machine as holograms through the AR UI. Third,
participants can modify the assembly sequence by touching the holographic bricks one
by one from the designed structure on the screen through AR and preview the assembly
simulation. Moreover, the simulation can be edited in operation speed, liner or joint
movement mode, and node delay time through mobile devices. The determined robotic
assembly simulation will be shown as holograms once the AR programming
modifications are done. Last, the robotic commands with assembly sequences will be
generated as QR codes for remote operations (Figure 4).

Figure 4. The screenshot of the live stream demonstration for the remote robotic assembly simulation
in AR during the workshop tutorials. The pre-set script works in the AR UI from Fologram App
(left), and Grasshopper (right).

After the demonstration during the workshop, participants started to design the
brick-based structures, and use the pre-set scripts to program the remote robotic
assembly through AR UI, including the sequence, movement mode, operation speed,
and node delay time. After confirming that the simulations were correct, participants
generated the corresponding QR codes for the remote operations. After that, the lab-
based operator used the AR UI to scan and run the related commands from the QR
codes to help the remote robotic assembly process (Figure 5). Participants were asked
to document their remote assembly process as AR videos and the physical outcomes
330 Y. SONG AND S. HAHM

for the final review, share and discuss the experiments at the end of the workshop.
Furthermore, it was assessed whether the remote robotic operation was well engaged
in AR, e.g., regarding the remote operation of AR UI and the outcomes of physical
assembly.

Figure 5. This is the remote robotic assembly of the brick-based structures. The lab-based operator
scans the QR codes generated by the participants and runs the robotic assembly process in the
sequence set by the participants through AR UI.

Figure 6. The physical brick-based structure outcomes of participants' designs were assembled by the
remote robotic operation method during the Augmented Robotic Bricklaying workshop.

The findings of phase B show that the AR UI programming method fulfilled the
remote operation requirements reacting to the pandemic situation. After the basic AR
introduction and demonstration, participants can easily program and simulate the
robotic assembly operation anywhere and beam the commands remotely to the lab-
based or on-site robots in a second for the physical structures assembly. The brick-
based structure outcomes are all consistent with the participants' design (Figure 6).
Moreover, the users' feedback was good during the program and remote assembly
process. All participants indicated that they could use AR UI easily and intuitively to
simulate and operate the remote robotic assembly after the simple pieces of training.
 AUGMENTED ROBOTIC BRICKLAYING 331

However, the lab-based operator is still required to assist with on-site remote robotic
assembly. It is because the Fologram and Robots plugins are temporarily unable to
synchronize for remote operation due to network IP address issues. The operator needs
to assist in identifying the QR codes, ensuring the assembly operation of the robotic
arm on-site, as well as dealing with malfunctions or other special errors. With the
update of software and technology, this shortcoming will be solved in the future.

4. Conclusion
Associating the interaction in AR with the robotic arm and programming it with
intuitive input methods will make it easier for architectural practitioners to simulate
and control industrial robots for complex structure assembly. Using AR as a media
between physical and virtual, it simplifies the conventional robotic coding method and
replaces it with a user-friendly AR-assisted interface for the safe human-robot
collaborative process. This AR programming method also gives the possibility of
remote control for robotic assembly, which meets the pandemic requirements for
remote communications and makes up for the disadvantage that participants cannot
visit the lab in person due to Covid-19 restrictions.
Closely witnessing the participants' experiments, as well as analysing the
Augmented Robotic Bricklaying workshop outcomes, it can be concluded that
architectural practitioners can quickly employ this augmented programming workflow
after an essential AR skill tutorial. Moreover, learning the corresponding interactive
input method intuitively and visually through AR and smartphones is much simpler,
safer, and more convenient than mastering coding languages. The workshop outcome
shows that by programming and remote operating the robot in AR, participants can try
digital fabrication tools, such as industrial robots, without any computer science
background knowledge. The participants eliminated their fear of industrial machines
due to the lack of relevant skills and stimulated their interest in digital fabrication.
Through mobile device screen sharing and group discussions during the workshop,
participants can learn from each other and exchange their experiences to gain a deeper
understanding and reflection.
In conclusion, AR programming and remote robotic assembly is an innovative
approach, and this Augmented Robotic Bricklaying workshop has also implemented
remote digital fabrication methods as online teaching experiments in architectural
education and fabrication fields. Future investigations suggest developing more robotic
end-effector functions to fit different customized designs. The Grasshopper backstage
scripts for the AR programming and remote control should be more straightforward
and editable that can be quickly developed by architectural practitioners. In the future,
with the updates and developments of plugins and devices, the lab-based operator is
expected to be replaced to achieve complete remote robotic control and human-
machine interaction through the AR environment.

Acknowledgements
This paper presents the process and outcomes of the AR-assisted Robotic Assembly in
Architecture online workshop from SDPlatform in August 2022, instructed by Yang
Song (https://soomeenhahm.com/shop/workshop/sdp-2022-aug/). The above selected
332 Y. SONG AND S. HAHM

experiments in this paper were developed by the following workshop participants:

Soomeen Hahm, Park Jongwook, Yejun Yoon, Solah Yoo, and Sowoon Lee.

References
Antonio, R.M.O & Lucas, B.A. (2021). Design and Digital Manufacturing: changes and
challenges in product development in the context of remote learning. In the XXV
International Conference of the Ibero-American Society of Digital Graphics (SiGraDi)
2021, (pp.1345-1357), Brazil.
Bhaskar, B. & Eliot, H.W. (2019). Augmented Reality via Expert Demonstration Authoring
(AREDA). In the Computers in Industry, 105 (2019) 61-79.
https://doi.org/10.1016/j.compind.2018.04.021
Chu, C.H., Liao, C.J. & Lin, S.C. (2020). Comparing Augmented Reality-assisted Assembly
Functions: a case study on Dougong structure. In the Applied Sciences, 2020, 10, 3383.
https://doi.org/10.3390/app10103383
Duaa, A.M., Saba, A. & Amer, A. (2021). Transforming Learning for Archi-tecture: online
design studio as new norm for crises adaptation under Covid-19. In the 9th International
Conference of the Arab Society for Computer Aided Architectural Design (ASCAAD)
2021, (pp. 129-141), Cairo.
Gilles, R. (2016). Discrete Assembly and Digital Materials in Architecture. In the 34rd
International Conference on Education and Research in Computer Aided Architectural
Design in Europe (eCAADe) 2016, Volume 1, (pp. 143-151), Oulu.
Ines, A. & Merav, G. (2015). On-site Robotic Assembly of Double-curved Sefl-Supporting
Structures. In the 19th International Conference of the Ibero-American Society of Digital
Graphics (SiGraDi) 2015, (pp.746-753), Florianópolis
Regiane, T.P. & Luiz, S.R.G. (2020). The importance of collaborative design process and
fabrication during Covid-19 emergency - case in Brazil. In the XXVI International
Conference of the Ibero-American Society of Digital Graphics (SiGraDi) 2020, (pp.357-
362), Medellin.
Ryan, L.J. & Jeffrey, A. (2018). Interfaces for Adaptive Assembly. In the international
conference of the Association for Computer Aided Design in Architecture (ACADIA)
2018, (pp. 126-135), Mexico City.
Song, Y., Koeck, R. & Luo, S. (2021). Review and analysis of augmented reality (AR)
literature for digital fabrication in architecture. In the Automation in Construction,
Volume 128, 2021, 103762. https://doi.org/10.1016/j.autcon.2021.103762
Song, Y., Agkathidis, A. & Koeck, R. (2022). Augmented Masonry Design: a design method
using Augmented Reality (AR) for customized bricklaying design algorithms. In the 10th
International Conference of the Arab Society for Computer Aided Architectural Design
(ASCAAD) 2022, (pp. 703-712), Debbieh.
Tatiana, E., Vincent, H. & Lena, M. (2021). The Digital Design Build: modes of experiential
learning in the pandemic era. In the 26th International Confer-ence of the Association for
Computer-Aided Architectural Design Research in Asia (CAADRIA) 2021, Volume 2,
(pp. 41-50), Hong Kong.