robotic bricklaying, towards adaptive robotic programing
DESCRIPTION
After the recent move of the field of architecture towards fabrication, the use of industrial robots surfaced as a replacement of custom fabrication machines. Robots are typically controlled with strict offline programing to produce controlled results similar to how robots are used in manufacturing. However, in construction robots did not manage to replace humans in on site operations. In the bricklaying profession, the craftsman uses his tuition to adapt to the material he is working with and to the changing work environment. This raised the question of the ability to build a system that can respond to changes in order to replicate the craft using robots. The research focuses on two main aspects, of bricklaying: The craft of laying brick and mortar itself and the ability to keep track of the work on a larger scale. In order to replicate these aspects an adaptive program had to be built with reliance on sensors and feedback to control the robotic arms. The robotic arm managed to achievTRANSCRIPT
1 `
MSc Adapt ive Arch i tecture and computat ion Bart let t school of graduate Studies Univers i ty Col lege London September 2013
Robotic Bricklaying Towards adaptive robotic programing Khaled ElAshry
2 `
Word Count: 10891
This dissertation is submitted in partial fulfilment of the requirements for the degree
of Master of Science in Adaptive Architecture & Computation from Bartlett School of
Graduate Studies, University College London.
I, Khaled ElAsrhy confirm that the work presented in this thesis is my own. Where
information has been derived from other sources, I confirm that this has been
indicated in the thesis.
September 2013
Abstract Robotic Bricklaying
3
ABSTRACT
After the recent move of the field of architecture towards fabrication, the use
of industrial robots surfaced as a replacement of custom fabrication machines.
Robots are typically controlled with strict offline programing to produce controlled
results similar to how robots are used in manufacturing. However, in construction
robots did not manage to replace humans in on site operations. In the bricklaying
profession, the craftsman uses his tuition to adapt to the material he is working with
and to the changing work environment. This raised the question of the ability to build
a system that can respond to changes in order to replicate the craft using robots. The
research focuses on two main aspects, of bricklaying: The craft of laying brick and
mortar itself and the ability to keep track of the work on a larger scale. In order to
replicate these aspects an adaptive program had to be built with reliance on sensors
and feedback to control the robotic arms. The robotic arm managed to achieve the
tasks in question using two software: A new plugin to control the movements of the
robots and a server that is capable of dealing with feedback.
Key words: Robotic Arms, adaptive programing, bricklaying, construction,
architecture, fabrication, feedback, computer vision, unpredictable material, Mortar,
Brick.
Contents Robotic Bricklaying
4
CONTENTS
Abstract ............................................................................................................... 3
Contents .............................................................................................................. 4
Acknowledgment ................................................................................................ 6
1 Introduction ............................................................................................... 9
2 Background .............................................................................................. 12
2.1 The craft of bricklaying ......................................................................... 12
2.2 Robots in architecture .......................................................................... 15
2.2.1 Early adoptions ............................................................................. 15
2.2.2 The current state of robotic research in architecture. ................. 16
2.3 Observations during the Robotic foaming Workshop, SG2013 ........... 17
2.3.1 Robotic brick work ........................................................................ 19
3 Methodology ........................................................................................... 21
3.1 The robotic arm .................................................................................... 21
3.2 Onix Grasshopper plugin ...................................................................... 23
3.2.1 Kinematics of the UR robotic arm. ............................................... 24
3.2.2 Forward kinematics ...................................................................... 25
3.2.3 Inverse kinematics ........................................................................ 26
3.2.4 Solution toggles ............................................................................ 27
3.2.5 Angles calculation ......................................................................... 30
3.2.6 Execution path generation ........................................................... 31
3.2.7 Robot program and program uploader ........................................ 32
3.3 The end‐effectors ................................................................................. 33
3.4 Robot execution paths ......................................................................... 34
3.4.1 Pick and place of bricks. ................................................................ 35
Contents Robotic Bricklaying
5
3.4.2 Accompanying tasks. .................................................................... 35
3.4.3 Laying mortar ................................................................................ 36
3.5 Brick laying server ................................................................................ 38
3.5.1 Program operation sequence ....................................................... 39
3.5.2 Interface ........................................................................................ 40
3.5.3 Computer vision and sensing ........................................................ 41
3.5.4 TCP/IP Feedback. .......................................................................... 43
3.5.5 Bricklaying server to Onix communication ................................... 44
3.6 Wall building tests ................................................................................ 44
4 Discussion and results .............................................................................. 47
4.1 Testing Results ..................................................................................... 47
4.2 Critical assessment ............................................................................... 48
5 Conclusion ................................................................................................ 51
6 Appendix .................................................................................................. 53
7 References ............................................................................................... 59
Acknowledgment Robotic Bricklaying
6
ACKNOWLEDGMENT
I take this opportunity to express my profound gratitude and deep regards to my
supervisors Sean Hanna and Ruairi Glynn for their endless support and guidance,
Angelos Chronis for his valuable feedback, the Bartlett CADCAM team for their great
help, my gurus Mohamed Zaghloul and Ebtissam Farid for their inspiration, my
friends for always being there for me, my father Mohamed ElAmir ElAshry and sisters
Kholoud and Tayseer for their endless support and encouragement, and last but not
least my late mother Neveen ElMaghloub for her love and endless belief in me.
List of figures Robotic Bricklaying
7
LIST OF FIGURES
Figure 1 the tomb of Saminids Bukhara, 900 A.D (Campbell & Pryce 2003) ............. 12
Figure 2 Bricklaying on‐site ........................................................................................ 14
Figure 3 left: First commercial ABB robot IR6 1974 Right: KUKA first robotic arm,
Famulus 1973 ............................................................................................................. 15
Figure 4 Robot brick laying system ((G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle
1994) .......................................................................................................................... 16
Figure 5 Experiments during the Robotic foaming workshop SG2013 ...................... 18
Figure 6 R.O.B system introduced (Gramazio & Kohler 2008, p.57) .......................... 19
Figure 7 DimRob system (Helm et al. 2012, p.1) ....................................................... 20
Figure 8 : To the left Six degrees of freedom around a point .................................... 22
Figure 9: To the right a sample of the solutions of rotations around a point using a
UR10 robot and Onix .................................................................................................. 22
Figure 10 left : Spherical axis robotic arm Right: non sperical axis robotic arm ....... 22
Figure 11 Onix grasshopper plugin example file ........................................................ 24
Figure 12 Universal Robot UR10 servo axis diagram ................................................. 25
Figure 13 Forward kinematics component in Onix .................................................... 25
Figure 14 Inverse kinematics component in Onix ...................................................... 26
Figure 15 First IK solution toggle diagram ................................................................. 27
Figure 16 Second IK solution toggle diagram ............................................................. 28
Figure 17 Third IK solution toggle diagram ................................................................ 29
Figure 18 Angle calculation digram ............................................................................ 30
Figure 19 Move, I/O and program uploader components in Onix ........................... 32
Figure 20 Components of the gripper end‐effector .................................................. 33
Figure 21 right: End‐effector attached to the robotic arm. Left: The trowel attached
to the handle .............................................................................................................. 34
Figure 22 Pick and place sequence ............................................................................ 35
Figure 23 Gripper holding the trowel for mortar operation ...................................... 36
Figure 24 laying mortar execution path sequence .................................................... 37
Figure 25 robot executing mortar laying ................................................................... 37
Figure 26 robot removing excess material using the trowel ..................................... 38
List of figures Robotic Bricklaying
8
Figure 27 the bricklaying server in operation. ........................................................... 38
Figure 28 Operation sequence of the bricklaying program ....................................... 39
Figure 29 Detailed operation sequence of the bricklaying program ......................... 40
Figure 30 Bricklaying server interface ........................................................................ 40
Figure 31 Intel procedural computing camera .......................................................... 41
Figure 32 Intel camera point cloud used for surface approximation ........................ 42
Figure 33 Limit switch testing brick hieght ................................................................ 43
Figure 34 right: Sequence of communication between the bricklaying server and Onix
for a single operation left: program dispatcher in Grasshopper ...................... 44
Figure 35 first test to check mortar laying movements and server operation .......... 45
Figure 36 Second test to check error handling and feedback ................................... 46
Figure 37 third test to check program adaptation to different rotations ................. 46
Figure 38 operation sequence ................................................................................... 53
Figure 39 IK solver (Left Square) and uploading component (Right Square) location in
the Onix plugin example file ...................................................................................... 56
Figure 40 IK solver component parts cluster ............................................................. 56
Figure 41 Point A calculation ..................................................................................... 56
Figure 42 Point B calculation ...................................................................................... 56
Figure 43 Point C calculation ...................................................................................... 57
Figure 44 Angle sorting algorithm cluster .................................................................. 57
Figure 45 Robot uploading component cluster ......................................................... 57
Figure 46 the eight possible solutions for a single tip plane using Onix IK solver ..... 58
Introduction Robotic Bricklaying
9
1 INTRODUCTION
Robotics is a fairly new field in technology; The term robot dates to the 1920s (Spong
et al. 2005, p.1). Robots have come a long way since then, growing more complex
and capable. Robots are now a key part in manufacturing industries such as the
automotive industry. However, in construction, they did not play an important role.
Human labour remained the main asset for many reasons. e.g., the work
environment. Construction is a very complex work field; tasks that are very easy for
human to achieve (like avoiding an obstacle or moving from one work location to the
other) are big challenges for robots. In the 1980s (Bechthold 2010, p.119) numerous
trials to integrate autonomous machines in construction surfaced especially in japan,
these trials were faced by another major drawback; the economical visibility of the
integration of such technology. These machines were huge and expensive which
meant that they weren’t a viable solution economically to manual labour. Also,
construction now might take the advantage of machinery; they are most of the times
directly controlled by humans and very far from being autonomous. Recently, the
interest in robotic in construction has surfaced again as robots are getting cheaper
while labour in western societies is getting more expensive.
Robotic fabrication has been traditionally associated with high‐tech industrial environments, where fixed positioning and constant conditions determine the role that the robot may undertake in the fabrication process. Unlike in stationary facilities, a construction site is a spatially complex and heterogeneous environment in which a robotic system may be exposed to continuous change and unpredictable events. (Helm et al. 2012, p.1)
Bricklaying is a construction profession that depends mainly on manual labour.
The craft of bricklaying hasn’t changed dramatically through history. The tool used in
the modern construction site is as old as the profession itself(Campbell & Pryce 2003,
p.303). The craftsman throughout history played a huge role in the realization of
several great structures that are still standing till now. Architects had a huge
influence on bricklaying as they themselves were sometimes considered master
craftsmen (Hill 2005, p.14). Nowadays, the craft is becoming more of a profession of
filling the gaps between prefabricated parts of the buildings in the modern western
society. However, Architects are returning to their historic close relation to the
Introduction Robotic Bricklaying
10
realization of their designs in the real world, researching new techniques in
fabrication and construction, retaking their role as master builders (Kolarevic 2005,
p.65).
In Architecture, robotics has become a major research arena over the past eight
years. The introduction of robotics in architecture research has started with the move
of the field towards research in fabrication. As research grew more complex in the
virtual form finding field, Architects were getting more and more interested in the
art of making. Industries like ship construction(Kolarevic 2005, p.9) had a great
impact on research in fabrication. Industrial robotic arms which were a crucial part
of such industries for years, are getting much cheaper. Which made them a viable
replacement to getting huge specialized milling or routing machines. The versatility
of the robots meant there are several uses that go beyond just that and new research
by pioneers in the field started to explore their potential. Robotic arms are very
versatile machines, their use is fundamentally determined by the program that
operates them and the tool attached to it. They were able to work with composite
materials for example, and were as well able to do jobs like pick and place very
efficiently. Architects were also researching in the software side of robotics, linking
robotic operation to CAD software for easier generation of the controlling program
for instance.
In fabrication, Material behaviours plays a big role, the complexity of the
integration of material behaviour in the process is determent by the unpredictability
of the material itself. The exploration of the use of the robots with messy materials
and composite fabrication projects required more advanced programing for the
robots. This time using senses and feedback to produce an autonomous program that
can adapt and change to external stimuli, making them more efficient and
productive. This research into sensing techniques and adaptable programing is
getting us closer to achieve robotic integration in heterogeneous environments like
construction sites.
Bricklaying is a craft that require practice, tuition and adaptable technique with
the need to work with relatively unpredictable materials. Robots, also are powerful
and accurate machines, are generally very inadequate at the execution of such set of
Introduction Robotic Bricklaying
11
qualities. This raised the question of the ability of an autonomous robot to replicate
this craft using the same low tech tools used by a bricklayer. In order to achieve that
the research the primary focus of this research is to develop an adaptive operating
workflow that can address two specific qualities of the bricklaying workmanship
using robots:
The craft of working with messy materials like mortar in combination
with other components like bricks with the ability to shift between
them
The ability to keep track of the larger plan and execute error correction.
Following the introduction, the background chapter focuses on the
bricklaying craft, discussing the role of the bricklayer, followed by the recent related
research in the fields of robotics and fabrication in architecture. Next chapter is the
methodology, this chapter focuses on the creation of the software platforms and the
process that allowed the testing of the craft reproduction using robotic arms, and
also the tests conducted to build wall samples. Upon that the discussion chapter
discuss the results and outcomes of the tests. And the research then ends up with
conclusion.
Background Robotic Bricklaying
12
2 BACKGROUND
2.1 The craft of bricklaying
2.1.1 The evolution of bricklaying
The Hanging Gardens of Babylon, one of the seven wonders; the Great Wall of China, the largest manmade object in the planet; the hag Sofia, one of the most beautiful churches ever built …, they were built out of bricks. Brick is at once the simplest and most versatile of materials, the most ubiquitous and the least regarded, all too familiar yet strangely neglected (Campbell & Pryce 2003, p.13)
Brick is one of the oldest building materials. Moulded brick is dated back to the 5000
BC (Campbell & Pryce 2003, p.15). Brick building evolved significantly through history
with inventions like fired bricks and the use of concrete in Greece. Historic Brick
building varies according to the type of brick, type of bonding material used and the
technique used by the bricklayer. Bricklaying craftsmanship played a big role in the
construction of the masterpieces all around the world. Throughout the history the
relation between the Architect and the craftsman in this case the Bricklayer was
strong. Architects were considered master masons in the middle ages. However, In
the Italian Renaissance architects started distancing themselves from the craftsmen
focusing more on the building representation (Hill 2005). Like many labour
dependent workmanship, the bricklaying skill level required in ancient times is no
longer in demand no today and skills that are not in demand diminish by time
(Campbell & Pryce 2003).
Figure 1 the tomb of Saminids Bukhara, 900 A.D (Campbell & Pryce 2003)
Background Robotic Bricklaying
13
The Bricklaying craft was always accompanied through history by the
profession of brick–making .In the western world nowadays, brick‐making is almost
extinct. Bricks now are manufactured mechanically and transported to site. On the
other hand, bricklaying seems to be fairly untouched by modern technology
(Campbell & Pryce 2003, p.303). Machines that take bricklayer place are still not
seen. This is the very distinct difference between the evolution of manufacture and
construction in recent history. The brick can be mass produced cheaply in a factory
in a relatively controlled automated manner. While a bricklayer has to deal with the
changing work environment, adapting to it with each different job on site.
2.1.2 Observations of the bricklaying craft on-site
Bricklaying is a profession that requires years of experience. The work of a
master worker can be easily differentiated from a novice one especially when dealing
with facing bricks. When the research started there was a necessity to watch a
bricklayer craftsman and also try brick laying to get to know the profession. The
observation and personal tries of bricklaying on‐site resulted in the Focus on two
main aspects that defines the craft.
1. the act of brick and mortar laying itself
2. The focus on larger scheme of things.
A perfect technique will cut time and waste material, and also result in a much
better end product. Technique and material greatly affect the result. Personal tries
revealed how the craft is a combination of skill that can be taught and expertise that
is compiled over time. The bricklayer almost does his craft unconsciously. The tool is
a major part of the process. The trowel itself is as old as the profession of brick laying
and haven’t changed significantly along its history. Using such low tech tools, human
tuition plays a huge rule in deterring the movement according to the different
features of the material itself like weight and consistency.
Background Robotic Bricklaying
14
Figure 2 Bricklaying on‐site
A bricklayer might seems to only focus on laying a brick but he also keeps an
eye on the larger plan. This is vital in order for the brick rows and columns to end up
matching in the whole building. Bricklaying is accompanied by other work on‐site like
building openings and reinforced concrete works for example. A simple error in
calculation can easily propagate to produce huge problems in the end that may
require major error fixing or even the destruction of wall portions resulting in badly
finished product.
Bricklaying is still very far from mechanization but the quality of the craft is
declining in western high tech world. Prefabricated parts is being used when possible.
This lean towards prefabrication wasn’t only for machined parts that may require
special manufacturing, but even the parts of the brick building that required most of
the skill like the arches for example are shipped fully assembled and put in place in
construction site. This is due to the higher expenses required to hire a quality
craftsman to work on‐site. And at the same time ability to better control construction
time and increases efficiency as much as possible.
Brick laying is still a relatively untackled territory when it comes to automation.
This gives a large space for what can be achieved. The innovation in the field brick
design require new tools that can reproduce the technique and quality of ancient
brick laying and go beyond that. At the same time increase efficiency and decrease
time of construction, while also filling the need for such craft in unhuman
environments for instance.
Background Robotic Bricklaying
15
2.2 Robots in architecture
2.2.1 Early adoptions
Industrial robotic arms have been offered commercially since the 1970s
(international fedration of robotics 2012, p.3) Machinery has been a core topic in
manufacturing for many years pushing the limits of research. Productivity on such
fields have increased significantly in the last decade as opposed to the construction
industry (Balaguer & Abderrahim 2008, p.2). Robots however seems to be always
constrained to work in highly controlled facilities. Many robots barely have any
sensing systems of their environment, perhaps a few sensors to detect errors like
motors overloading but little else. When the robotic on‐site construction idea was
explored in the 1980s and 1990s (Bechthold 2010, p.119) the formula that succeeded
in the manufacturing industry didn’t work here. Robots were never developed to
adapt to changing environments or to deal with a messy type of data. In fact they
were developed to do very specific tasks with very specific programs that may take
years to develop and test and then run for the life time of the machine (Keating &
Oxman 2013, p.1).
Figure 3 left: First commercial ABB robot IR6 1974 Right: KUKA first robotic arm, Famulus 1973
In 1980s, during the Japanese booming era, up to 200 systems of automated
construction were developed (Bechthold 2010, p.3). During this era the Japanese
economy was very strong and the construction rate was very high. The profession of
a construction worker on the other hand was very alienating for its high risk and low
payment. The scarcity of construction workers in opposition of high demand required
searching for new possibilities. Robots seemed to be the solution, and numerous
Background Robotic Bricklaying
16
automated systems were developed. These were huge custom machines that
required large amount of money to build and operate. By the end of the 1990s, the
economy of Japan was hit and the research in automated construction stopped.
The Idea continued with a slower pace. In 1994 some papers were published
during the 11th ISARC conference. One of the papers was discussing the autonomous
machine to build brick walls (G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994).
Another system under the name of ROCCO was also proposed as a solution for
robotic masonry work on‐site (Andres et al. 1994). The system was referred to as a
“Fault Tolerant Assembly Tool”. As inferred from the title it tried to rely on sensors
and feedback for fault correction .It was designed as a purpose specific robotic
system. This time it was smaller and more approachable than the systems that were
developed in japan. It also proposed a link between CAD and construction directly.
The research in the platform continued with two more papers in 1995 and 1996, but
it wasn’t put in practical use.
Figure 4 Robot brick laying system ((G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994)
2.2.2 The current state of robotic research in architecture.
In recent years. The architecture practice has been changing and branching
with more research into material, construction and fabrication, forming new modes
of collaboration between disciplines. Architects this time w developing their own
design culture (Glynn & Sheil 2011, p.117). In this new turn architects are searching
into means to represent their digitally created work in the real world. This turn
towards fabrication made research in robotics surface again with huge momentum.
This time using the same robots that has been used by the industry for years instead
Background Robotic Bricklaying
17
of using huge custom robots (Braumann J. and S. Brell‐Cokcan. 2013, p.8). Robots like
these are getting more affordable thus becoming more and more accessible for both
architectural research and even practices. Pioneering offices like Snøhetta and
Foster+Parteners are getting their own robotic arms in order to experiment with
fabrication. For the major part of the research robots are usually used as either a
sophisticated 3D printing or milling machine executing specific preconfigured paths
in a controlled manner similar to the way it is used in the industry. However, this time
architects are finding better much faster ways of planning these fabrication programs
that is more approachable and simpler. This advanced the research of robotics in
architecture significantly in the field of fabrication. It allowed a faster learning curve
and ability to easily link to geometry directly from CAD software.
The real world is very complex which make modelling it in a virtual
environment is usually quite a challenge. Another side of research in robotic
fabrication explores a more sophisticated operational programing. This requires
more knowledge of the field robotics and links to other fields like computer vision,
mechanical engineering and mathematics. . Research in this field rely on using
feedback loops and more sophisticated sensing methods to read from the
surrounding environment to allow a more dynamic approach that can adapt with the
material in use.
2.2.3 Observations during the Robotic foaming Workshop, SG2013
During the participation in the “Robotic foaming workshop” (Colletti et al. 2013) in
the 2013 smart geometry conference, the problem with direct blind execution from
digital to physical was visible. The workshop was built on the idea of working with
spray foam (a very unpredictable material), using the precisely controlled robots. This
subject matched perfectly the conference theme “Constructing for Uncertainty” in
theory. Even though very interesting forms merged form the workshop, it was far
from perfect in the process front. During the workshop, foam was mixed manually
with other ingredients like hardener and water. The process produced a very elastic
fibrous mixture. It was then attached to plates on the three robotic arms. The arms
then took the part of stretching the mixture. During the testing the foam mix
Background Robotic Bricklaying
18
produced was very inconsistent, and even mixtures that were produced in a very
similar and timed process still behaved differently.
Figure 5 Experiments during the Robotic foaming workshop SG2013
The material was delicate and required the constant change of speeds while
executing the program. Due to that, running the preconfigured program directly
always resulted in a complete tear of the material and the failure of the test. On the
other hand, the tests run by pulling it manually by the participants were very
successful. In order to achieve the same result using the machines, the robots had to
be controlled almost in a total non‐autonomous manual manner using the control
joystick, this defied the purpose of the controlled offline designed program. The work
on this project put into perspective the importance of having an online feedback
loops and vision systems that can deal with such materials while keeping the
automation. The importance of using sensors and adaptable programing was
investigated in other similar projects recently like The magnetic architecture (Dubor
& Diaz 2012). The project was dealing with form finding using iron filling and
magnets. The same result was concluded from the research; Digital representation
only can’t achieve accurate results with such messy material. This deemed the linear
process of offline program generation not reliable.
Background Robotic Bricklaying
19
2.2.4 Robotic brick work
ETH Zurich is considered one of the major pioneer institutes that work on the
integration of robotics in architecture. One of their major areas of research is the
construction related manufacturing, especially with bricks. Gramazio and Kohler
succeeded in creating a practice that can combine research and architectural work
with many projects designed and constructed in the past years. Many of the brick
walls manufacture were usually produced in a lab environment and then shipped to
the construction site like the Gantenbein vineyard façade project for example
(Gramazio & Kohler 2008, p.95). Although these walls were very innovative visually,
the product was never structural and resembled a cladding system more. This was a
great potential worth exploring, building structural Brick walls that can also benefit
from the innovative visual effect.
Figure 6 R.O.B system introduced (Gramazio & Kohler 2008, p.57)
Other construction related robotic research at ETH Zurich considers the use of
robotic arms outside the laboratory setting. In 2007 R‐O‐B was introduced (Gramazio
& Kohler 2008, p.57), it is a mobile manufacturing facility that included an ABB
industrial robotic arm at its core (Figure 6). This allowed the exploration of the
production of brick structures on‐site. The unit was moved around the world in
locations in Europe and the USA and produced brick walls on‐site. Another recent
project had more emphasis on feedback. DimRob is mobile construction system
designed also in ETH Zurich. The system relied of a mobile robotic arm. The size of
Background Robotic Bricklaying
20
the proposed system allowed the movement of the robots trough standard openings
in site (Figure 7). It heavily relied on sensors to respond to the uniqueness of each
construction site(Helm et al. 2012, p.1) along with more collaboration with humans.
Figure 7 DimRob system (Helm et al. 2012, p.1)
The research in the field of on‐site construction and dealing with changing
environments and materials is still relatively young in the architecture field. The
Pioneers in robotics and fabrication are researching more in this field of adaptive
robot programing and the use of sensing capabilities. Such field have several
applicable potentials. Bricklaying is considered a great candidate for the integration
for such process.
Methodology Robotic Bricklaying
21
3 METHODOLOGY
In this research, the robotic arm was an important part along the software that runs
it. The understanding of such tool was crucial. However, this was not a straight
forward process the novelty of the robots used during the research and the lack of
any technical support meant it was generally a process of trial and error. This also
required building new platforms from scratch to explore the controlling possibilities
of these robots before the research can proceed with the testing of its hypothesis.
3.1 The robotic arm
Robots are fascinating electromechanical machines. It is defined as “a machine
capable of carrying out a complex series of actions automatically, especially one
programmable by a computer”(Oxford dictionaries n.d.). As the definition implies,
robots are usually defined by two main aspects: their physical mechanical aspect and
the software that runs them .In this essence the robotic revolution went hand in hand
with the finding and explorations in the field of computer science.
The most common multifunctional robots are robot Manipulators “Robot
Manipulators are composed of links connected by joints to form a kinematic chain.
Joints are typically rotary (revolute) or linear (prismatic).”(Spong et al. 2005, p.3). This
study has focused on the most common type which is the articulated rotary robotic
arms. Robotic arms are considered rigid body systems and usually are defined by its
degrees of freedom. They are the possibilities of movement of the end effector tip.
The most common number of degrees in industrial robotic arms is six degrees where
the end is able to translate in three dimensions and also rotate around three axes
(Figure 8); this allow all the possible motions around a point, which renders more
degrees in most cases useless. The degree number is also directly related to the
number of servos in the robot.
Methodology Robotic Bricklaying
22
Figure 8 : To the left Six degrees of freedom around a point
Figure 9: To the right a sample of the solutions of rotations around a point using a UR10 robot and Onix
The setting of the robotic arm’s servos affects directly how the robot is
controlled. The most common type of robotic arms manufactured by major
manufacturers is the spherical wrist robotic arms. In this type the three wrist axes of
rotations intersect at one point. The non‐spherical wrist type which is relatively
uncommon was introduced recently by the Danish company universal robots. The
latter type has advantages over the spherical wrist type but needs a more complex
inverse solution for its kinematic chain.
Figure 10 left : Spherical axis robotic arm Right: non sperical axis robotic arm
In 2012 The Bartlett School of architecture received two universal robots for
the sake of expanding its fabrication facilities. The UR10s are six degree of freedom
non‐spherical axis robots with 10 kg pay load. The problem the research faced along
other initial users of the robots is the lack of any controlling software or technical
Methodology Robotic Bricklaying
23
support for these robots as opposed to others manufactured by the other companies
like ABB and KUKA. The robots initially were only controlled via its control pendent
and could only be programed for simple pick and place programs. The uncommon
joint configuration design meant the available tools for other robots cannot be
reused to control the URs at the same time they lacked any sort of publicly available
control software either by the manufacturing company itself or third party
developers. In the search for tools to control the robot many suggestions were tested
like using the Blender modelling software to control the robots via bones and
animation (developed by Tom savilans). The initial blender model relied on the
animation functions in blender and required to send the instructions to the robot one
by one. The tool however was unreliable and a new tool was needed to control the
robots
3.2 Onix Grasshopper plugin
In the beginning of the research, it was very crucial to build a control software that
will enable the researcher to work with the robots in a meaningful way. It was
decided to build tool from scratch that will allow planning robot jobs easily with the
ability to directly upload to the program. This also allows having access and the ability
to fiddle with all its components according to the needed task. Grasshopper was
chosen as the base platform to develop the software.
Grasshopper is a .net plugin that runs on top of the 3D NURBS modelling
software rhino. It combines ease of use through its VPL interface and also the ability
to program in VB, C# or python languages directly inside it. The adoption of several
of practices and universities to using grasshopper made it a very good candidate to
build a controller software. Grasshopper also has several built‐in tools to create
plans. The planes contain all the required data (three translations and the rotation)
in order to calculate the kinematic chain of the robot. These tools become very useful
to build paths that the robot can execute along points, curves or even surfaces. Other
solutions for controlling robotic arms that were built on it for other manufactures
succeed in the architecture related research environment. In a comparison between
the use of KUKA|PRC (a grasshopper plugin for offline controlling of KUKA robots)
Methodology Robotic Bricklaying
24
against the company supplied path planning software was found to increase
productivity and reduce time to move from design to fabrication (Braumann & Brell‐
Cokcan 2011) .
Figure 11 Onix grasshopper plugin example file
The result definitions during the research lead to the creation of the Onix
grasshopper plugin. The plugin allowed the control of the robot directly through
streaming from the grasshopper interface. It included Inverse and forward Kinematic
solver for the universal robots in addition of upload, communication and end–
effector tools which will be detailed further in the upcoming part (Figure 11). The
plugin was packaged and released as an open source plugin and is used as a
controlling platform for other research projects.
3.2.1 Kinematics of the UR robotic arm.
The robotic arms are considered rigid bodies. When using robotic arms the user
usually is majorly concerned mainly by the position and orientation of the tip of the
end effector (The end‐effector refer to the tool used at the end of the robot that may
have extra transformations). In order to represent this tip point, six number are
needed which represent transformation location and rotation of the point, this set
of data will be referred to as planes during the research. The set of servo rotation
angles that achieves this tip plane position is called configuration(Spong et al. 2005,
p.4). The universal robots have six servos referred to by the manufacturer as (base,
shoulder, elbow, wrist 1, wrist 2, wrist 3) (Figure 12) The calculation of kinematic
Methodology Robotic Bricklaying
25
chain for any configuration require two major function to be computed: Forward and
inverse kinematics.
Figure 12 Universal Robot UR10 servo axis diagram
3.2.2 Forward kinematics
Figure 13 Forward kinematics component in Onix
Forward kinematics deal with the relationship between the joints positions the
position and orientation of the end‐effector(Spong et al. 2005, p.68) . The forward
kinematic fundamentally solves the location of the tip according to the joint position.
The related component in the Onix plugin relied on an input of the six rotation angles
(the configuration) of the robots servos (Figure 13). The plugin then applies a set of
transformation and rotations to the model of the robot until achieving the tip
position. It was also used also to show the representation of the robot along with the
solution of the tip plane.
Methodology Robotic Bricklaying
26
3.2.3 Inverse kinematics
For the Inverse kinematics solution, the function is presented with the tip plane as
the input and calculates the set of the joint positions needed to achieve this location.
Inverse kinematics is generally nonlinear meaning it results in multi solutions for a
single input. In this part the difference between the universal robots and their
counterpart are the most noticeable. The less common new servo setting of the
universal robots meant fewer singularities (Singularities in robotics refer to a tip
plane that don’t have one unique solution in the joint solution space. This either
refers to having infinite solutions or no solution at all) that may occur in the middle
of the configuration space. The UR robots are capable of eight different solutions for
most of their reach space (Appendix III) as opposed to four in the spherical wrist type.
In order to calculate those solutions, the inverse kinematic solver in Onix was built
from scratch directly in grasshopper using a geometric approach rather than using a
solely mathematical one. This allowed it to be easier to debug and edit in
grasshopper.
The input of the inverse kinematic solver in Onix consist is the plane of the tip
point. Starting with this input, the kinematic chain is computed then the angles are
calculated accordingly. The inverse kinematic chain is calculated by solving the
location of three key points along the chain. Each of these points have multiple
solutions for each tip plane input, thus three more toggles were added to the input
of the inverse kinematic solver in order to switch between the eight different
solutions.
Figure 14 Inverse kinematics component in Onix
Methodology Robotic Bricklaying
27
3.2.4 Solution toggles
Due to the novelty of the universal robots, there were no information about how to
solve their inverse kinematic chain and the initial trials with using the same methods
for spherical robots directly was unsuccessful. The new proposed solution however
came from the observation of the robots themselves. During their operation it was
noted that some configurations can have unlimited solutions in the joint space,
where the fifth axis can rotate continuously altering the rest for the chain while
maintain the tip poison. The solver was built initially to solve this configuration as it
allowed the second point in the chain to be disregarded (point B as referred to latter
on) as any solution for it is legitimate. After calculating the geometrical solutions for
the first and third points, finding the solution for the second point for the rest of the
configurations was much easier and the kinematic chain was finally constructed.
Figure 15 First IK solution toggle diagram
The solutions calculation starts with the inputs of the tip end plane. The first
point along the chains (point D) (Figure 15) is determent as the offset of the tip plan
in the direction of its normal; this determines the location of the fifth axis. Afterwards
the different solutions are calculated. First solution toggle results from the
calculation internal tangents between two circles.
1. One with the point D as its centre, and radius equal to the distance
between axes five and six. (These distances are constant and
comesfrom the robot manufacturing dimensions).
Methodology Robotic Bricklaying
28
2. The second circle with the base origin coordinates as its centre,
projected on the first circle corresponding plane, with radius is equal
the distance between the origin and the second axis.
These two circles represent the unlimited solution that the fifth and second
axis servo rotations. The intersection between the tangents and the resulting circles
represents two unique solutions that achieve the desired tip position. Point A on the
chain is determent according to the toggle choice.
Figure 16 Second IK solution toggle diagram
The Second solution toggle calculation depends on the solution of two
intersecting planes. One representing the tip plane and the other is the plane normal
to the perpendicular vector on the chosen tangent (from the first solution)(Figure
15). The two solution for this toggle result from intersection between the resulting
straight line and the circle representing all the possible solutions of the forth servo
rotation around the six. The point B Figure 16) represents the selected solution point.
This solution toggle is the major difference between the URs and the spherical wrist
robots.
Methodology Robotic Bricklaying
29
Figure 17 Third IK solution toggle diagram
The third solution toggle is determined by the calculation of the intersection
between two circles with centres A and B (Figure 17) (determined from the first two
solutions). A similar approach for the calculation of this solution was published by
Daniel piker for the calculation of the angles of the KUKA robots (Piker 2013).
The radius of circle with centre point A is the distance between the
second and third axis.
The radius of the circle with centre point B is the distance between the
third and fourth axis.
The solution results from intersection between the two circles. The
chosen solution is point C.
`
Methodology Robotic Bricklaying
30
3.2.5 Angles calculation
Figure 18 Angle calculation digram
The angles for the joint position are calculatedaccording to the kinematic chain
determined from the solution toggles.
1. The first servo angle (base) has is the angle between the X direction
vector of the origin plane and the vector represented by the direction
between the origin and the projection of point A (the base rotation
vector) on the origin plane.
2. The second servo angle (Shoulder) is the angle between the
perpendicular vector to the base rotation vector and vector in the
direction A‐C.
3. The third axis (elbow) angle is the angle between the vector C‐A and C‐
B.
4. The forth angle (wrist 1) is the angle between vector B‐C and the vector
B‐D.
5. The Fifth angle (wrist 2) is the angle between the normal direction
vector of the tip plane and the base rotation vector
6. The sixth angle (wrist 3) is the angle between vector B‐D and the x axis
of the tip plane.
Methodology Robotic Bricklaying
31
3.2.6 Execution path generation
The universal robots servos are capable of moving 720 degrees, from positive 360
degrees to negative 360 degrees for all its joints. This resulted in having 2 legitimate
joint solutions in any servo that can produce the same solution. For example if the
axis angle result from the inverse kinematic algorithm is 50° this means that ‐310° is
also a legitimate angle to active the same location. This increases the total solution
possible for any tip plane to 512 and the solutions that can achieve one of the eight
kinematic solutions to 64. This became a challenge when there is several inputs for
the inverse kinematic algorithm (when dealing with a continuous path for example),
the IK solver calculates the solution for the inputs individually only in the angle space
from 0° to positive 360°. In the initial tests, this resulted in some major errors in the
robot program when executed that may occur randomly when moving from one
configuration to the next. This might have catastrophic results duo collision for the
robot with the surround environment or the model it is working on. The solution for
this was building a sorting algorithm that allows the grouping of the closest solutions
when dealing with a continuous program.
It was crucial for the program when dealing with paths to put the timeline and
previous positions into consideration. To achieve that the IK algorithm calculate the
First configuration for the first tip position (referred to as home position usually)
which the user can override manually using angle toggles (Figure 14), and accordingly
he selects from two sets of automatically generated angles list for the whole path for
each servo. One list starts with the angle that is the closest to zero and one starts
with the angle that is the furthest. For example The IK for a single servo generated
numbers like (100, 10, 330 and 20) for the points along a hypothetical path. If this list
is executed without sorting, the servo will rotate 320° form the second state to the
third state and back 310° degrees. After the sorting of the solutions, the two
generated lists are (100, 10, ‐30 and 20) or (‐260,‐350, ‐30 and ‐340). These two lists
have both evade this problem resulting in a much smaller changes between states.
The inverse kinematic solver and path calculations are all grouped into one
component in the Onix plugin, allowing easy conversion from a set of planes to a set
Methodology Robotic Bricklaying
32
of corresponding configurations in one step. An error system was also added to the
component to stop program from execution if a problem with angle lists is still
present in one of the lists or if the Inver kinematics fails to produce a viable solution
for the configuration.
3.2.7 Robot program and program uploader
Figure 19 Move, I/O and program uploader components in Onix
After the calculation of the set of joint locations that represent the path. The
“Move” component is responsible of converting these values to the URscript
language. The URscript is a set of instruction that the universal robots can execute
(universal robots n.d.). For each of the joint positions, the robot required one line of
code starting with either “movej” or “movel” followed by the eight numbers , six of
them refer to the joint positions of the servos then either the speed and velocity, or
time and rounding. The difference between the two is related to the intermediate
movements between states. The movej allow the movement to happen in the joint
space while the “movel” required the movement to be calculated according to the
tip position to maintain a linear movement of the tip.
The set of moves are then combined in the uploading component. The upload
component is responsible for the conversion of the lines of “Move” commands along
with the I/O sates to a working program. The program is then uploaded via TCP/IP
connection to the robot and is executed instantly.
Methodology Robotic Bricklaying
33
3.3 The end-effectors
End‐effectors are an important part of any robot system. They define the rule
and use of the robot. In the beginning of the research, the end effectors were
designed to test different stages of the process which will requires either using multi
robots or changing the tools for each task manually. In the end a single system was
designed to transfer from one task to the other seamlessly. The three major tasks
needed to be covered by the end‐effector were: brick pick and place, laying mortar
and sensing.
Figure 20 Components of the gripper end‐effector
The end effector was designed around an industrial pneumatic gripper. The
chosen gripper had 3 cm stock and gripping force of 600 newton while keeping a
weight of 1 kg. Adjustable holding fingers was then machined and attached to the
gripper. The pneumatic air flow to the gripper was controlled by a 24 volt five way
solenoid valve. The valve allowed direct control over it from the robots control box
through the robot program itself. This was crucial to achieve synchronized robot
movements and end effector actions
Methodology Robotic Bricklaying
34
Figure 21 right: End‐effector attached to the robotic arm. Left: The trowel attached to the handle
Achieving the second task required using a traditional trowel. A special handle
was created in the same width as the bricks in use (100 mm) to allow the gripper to
pick the trowel and use it. The force of the pneumatic gripper in addition to using
rubber eliminated the problem of the trowel slipping while in use.
The sensing was covered by a depth camera a limit switch. Due to wiring, these
two needed to be attached directly to the end effector. The end‐effector design was
tested virtually to achieve the best configuration that would decrees the incidents of
collision and also keep the camera and the limit switch out of the mortar. The camera
was controlled by the server computer, while the limit switch was controlled directly
by the robot control box, connected in this case to a digital input.
3.4 Robot execution paths
Onix allowed an easy way to create, edit and test the execution paths of the
robots. Tasks needed to fulfil the brick laying process were divided in relatively small
job specific paths for each task. Each of these execution paths are constructed in a
parametric manner, changing and updating according to its inputs. Some of these
inputs were determined once in the first calibration phase, while other change each
time they are executed.
Methodology Robotic Bricklaying
35
3.4.1 Pick and place of bricks.
Figure 22 Pick and place sequence
A simple path was generated to pick one block from one location to the other.
The program has three inputs which are the brick stack location, brick number in the
stack and the placing location and rotation. The brick stack location is defined once
while the brick stack number and placing location change according to the order of
the program.
It was noted that during the placing the bricks on top of the mortar layer, the
forces required to place the brick in its location increase dramatically if there is excess
amount of mortar under. This is due the fact that the robot tries to push lots of
material out and is faced with the opposite force that and results in the emergency
stop of the robot. This led to the failure of the placement of the brick and the need
of re‐initialization of the robot. After some tests the movement was altered to have
rotational and side movements while placing the brick. This allowed the material to
move and level easily and decreased the perpendicular forces facing the movement
of the brick before.
3.4.2 Accompanying tasks.
A set of simple paths were designed to allow the shift between tools. First path
is the one which picks and leaves the trowel form a known location. The second path
was moving the camera to a perpendicular position to the mortar location. The
Methodology Robotic Bricklaying
36
location of the mortar working board is also fixed and given as a constant. The third
program is picking the mortar. The input for this program is the point determined by
the depth camera, indicating how much the trowel need to move to get the right
amount of mortar.
Figure 23 Gripper holding the trowel for mortar operation
3.4.3 Laying mortar
Laying mortar is the most interesting part of the process. After the gripper pick the
trowel, the end‐effector is adjusted in grasshopper and the working plane is shifted
from the middle of the gripper to the middle of the trowel surface. The degree of
success of the laying moment was determent by two factors; First, the ability to move
all the mortar from the trowel to the brick surface. Second is the creation of a
consistent layer of mortar on top of the brick.
A professional bricklayer usually uses one continues movement to lay the
mortar on the brick. In order to mimic that with the robots, the tool needed to move
very quickly to achieve a good result in one movement. After observing bricklayers
do their work, it was noted that there are numerous factors which the movements
of the trowel is adjusted accordingly. During the first tests the teaching option of the
universal robots was used. In this mode the robots can be moved very easily by hand
while streaming the joint positions through the TCP/IP connection. In grasshopper
the movement are then recorded in real time creating the path. This made it possible
to try to mimic the basic movement that the bricklayer does when using his own tool.
Methodology Robotic Bricklaying
37
Figure 24 laying mortar execution path sequence
Figure 25 robot executing mortar laying
In order to achieve a more universally successful results, the laying process was
broken into smaller movements that were tested individually with different mortar
consistencies. First was the movement to place the mortar on the surface of the
brick. The location of the placing of the mortar was very important to achieve a
consistent layer latter on. The next movements to be tested was the set of
movements that ensure that the all material left the trowel. The trowel paused and
moved up and down a number of times to ensure all the material left it and then the
trowel moves linearly to achieve a consistent layer of the mortar on the surface and
remove any excess material. The mortar program only requires one input which is
the location of the brick surface it is supposed to lay mortar on top.
Methodology Robotic Bricklaying
38
Figure 26 robot removing excess material using the trowel
After applying the mortar and laying the next set of Bricks. The robot picks the
trowel again and remove the excess material resulting from pushing the mortar out
by the bricks. The process is one continouse movement from one side of the brick
row to the other side. The input of this path is the start and end location of the brick
row and the row hight.
3.5 Brick laying server
Figure 27 the bricklaying server in operation.
In the beginning ONIX was used on its own to control the robot. The major
challenges were the limitation of grasshopper as real‐time controller. Also the real
time visualization of the robot was integrated in grasshopper and allowed some
features like path recording, problems with speed and inability to work with
computer vision came between using grasshopper alone as the brick server. The
Methodology Robotic Bricklaying
39
major problem with grasshopper by far was the way it is built in the first place.
Grasshopper is very linear in the way it executes its programs. Even with direct
programing it still faced lots of problems dealing with things like loops and retaining
variable data.
This required building another program that can be the server controlling the
bricklying process. The program was built using processing programing language
version 2.0 and was used to control the second major aspect of the research which
is keeping track of the major plan and fixing general errors. It was also in control of
the computer vision part of the process.
Figure 28 Operation sequence of the bricklaying program
3.5.1 Program operation sequence
The processing server works as the controller of the operation. It collects all the
data and control the execution of the robot paths. It allowed the shift between
different programs and tools. After the user selects the operations he needs to be
executed, the server initiates a logical sequence of operations to be processed.
Execution commands are then sent one by one to be executed in the Onix plugin. The
server always keeps track of the execution state of the program using feedback from
the robot, and sends the next program as soon as the job is completed. It shows visual
feedback in the interface about the progress of the work, and also allows the user to
overwrite the program in the case of errors.
Methodology Robotic Bricklaying
40
In order to achieve that, the program is built on a set of If statements each one
representing a task. The program won’t go in the execution of one until it ensures
the one before is executed. Each statement also keeps track of the locations of the
tasks allowing the server to go from one location to the next. The server also
automatically compensate for errors.
Figure 29 Detailed operation sequence of the bricklaying program
3.5.2 Interface
The simple user interface of the program is built using the ContolP5 (Schlegel
2012) plugin for processing. The user interface is used to input the basic information
like the width and height of the brick, the number of brick rows and columns of bricks,
the spacing between the rows and columns of the bricks and the brick stack amount
for both full size and half bricks. The interface also is used to choose the programs
tasks needed to be executed by the robot, like brick pick and place, laying mortar,
pausing the program and emergency stop. The interface helps monitor progress and
errors and accept edition during execution.
Figure 30 Bricklaying server interface
Methodology Robotic Bricklaying
41
3.5.3 Computer vision and sensing
Mortar in the right consistency is very close to liquid but still solid enough to
keep its shape. This makes mortar a very unpredictable material specially when
dealing with using low tech tools. When adding or subtracting from the material, it
will not level and would keep its shape and change slowly with time. The challenge
was getting the right amount of mortar on the trowel each time. This required having
a way to determine the arbitrary surface of the mortar each time before digging in
with the trowel.
Figure 31 Intel procedural computing camera
Depth cameras came as a solution for this problem. The focus was to have all
the tools needed on the robot itself without having a general vision system that
overlooks the whole process. This made the Kinect camera unsuitable for the task as
it is bulky in comparison to the size of the universal robots. Nevertheless, the new
Intel Perceptual computing camera prototype many advantage. It is much smaller
which made it possible to attach easily to the rest of the end‐effector and is unlike
the Kinect camera it was able to work on a range from 0 to 1.5 meters. This made it
possible for the robot to move and peek at the objects at close range getting a higher
density point cloud of the scanned object.
Methodology Robotic Bricklaying
42
Figure 32 Intel camera point cloud used for surface approximation
The point cloud you get from the camera suffers from the same noise problem
as the Kinect camera. But because it can work at a very close range meant that the
noise reduction would have less effect on the accuracy of the scanned surface. For
the noise reduction, the scanning algorithm would store the point cloud data five
times in a frame of a second and then the average of each of its point location is used
to resemble the surface.
The surface if the mortar is then determent and the volume is calculated. Each
point of the point cloud is considered a rectangular box with fixed width and depth
and variable height. The height data are calibrated according to 4 points that
represent the base working surface. These points are determent on the first run of
the program. The algorithm then moves on the surface of the mortar from right to
left row by row determining how far the towel should go until it have the amount of
mortar needed for the application.
Methodology Robotic Bricklaying
43
Figure 33 Limit switch testing brick hieght
In order to check the height of the placement of the bricks for errors, the first
plan was to use the depth camera to check the surface of the brick after laying it on
top of the mortar surface. Due to the noisy data of the camera this was deemed
unreliable. Instead a limit switch was added to the end effector. The limit switch was
connected to the digital input of the robot control box. During operation the robot
would move to a point which would make the limit switch tip 5mm away from the
supposed surface of the brick. Then it will move at very slow speed until the limit
switch is pressed and the location of the new point is recorded at the server program.
3.5.4 TCP/IP Feedback.
The universal robots are transmitting their state continually through their
TCP/IP connection. In initial tests it was difficult to decrypt the message that was sent
back form the robot due to the lack of support from the manufactures. Instead the
server relied on the python‐urx plugin (Roulet n.d.). The plugin was used to read the
message back from the robot and decrypt it. The message was then sent to both the
processing server and the onix plugin via the User Datagram Protocol (UDP)
connection.
For the processing server the message included the state of the program
running. This allowed the server to send the next command to the Onix plugin as
soon as the previous task was executed. The location of the tip was also sent to the
server in order to determine the height of the point acquired after the limit switch is
Methodology Robotic Bricklaying
44
pressed. This allowed to check the deviation of the bricks surface form the original
design. For the Onix plugin however the message included used the joint locations.
This allowed the plugin to show the robot virtual representation in real tam while
excursing the program while also record movements if needed.
3.5.5 Bricklaying server to Onix communication
The communication between the two programs was achieved using UDP
connection. Using the protocol the processing server sends an execution command
to grasshopper starting with a specific identifier. This ID allowed the program to be
distributed to its corresponding robot parametric path in onyx. The message also
included the information needed of the parametric grasshopper definition. For
example the message sent for a pick and place job would be “B%10,50,0%‐
300,0%3,True” In grasshopper the message will be broken intro
Program ID : B (Brick pick and place)
Location of the brick to be placed is ( 10 mm , 50 mm , 0 mm ) from
origin
Brick Stack location ( ‐300mm , 0 mm ) from origin
Number of the brick in stack is 3
The numbers are then dispatched to it corresponding parametric input and the
program is altered and sent to the robot to execute.
Figure 34 right: Sequence of communication between the bricklaying server and Onix for a single operation left: program dispatcher in Grasshopper
3.6 The building tests
The First bricklaying experiment was conducted to test the mortar laying
movements. The aim was to build two rows of bricks with the least excess material.
Methodology Robotic Bricklaying
45
It was repeated a number of times with alternations to the program until the desired
result was achieved. The flow of the program was also tested. The server program
shifted from the tasks of brick laying with the gripper directly to gripping the trowel
and changing the program to deal with mortar and back to laying bricks and removing
the excess amount of mortar on the side after.
Figure 35 first test to check mortar laying movements and server operation
The second test was conducted to build a higher wall. This allowed the testing
of the sensor data. The location of the brick wall needed to be shifted from the table
surface to the level of the ground to allow the robot to build higher and safer. One
of the first problems was the adaptation of the movements that is built in
grasshopper to the new position. The movements were built initially to work in the
space in front of the robot and relative to the fixed world coordinates. This created
a problem that was generally visible when dealing with trowel. The programs were
then rebuilt for this task to work according to the location relative to the robot. This
allowed the program to be executed in any location around the robot easily without
the need to alter the program. The test also was done using bricks and half bricks.
The bricks were cut manually and stacked in a secondary location. The server shifted
between the locations of the half bricks and the full ones according to the need. The
rows were built one by one and the height was tested for each row before laying the
new mortar layer.
Methodology Robotic Bricklaying
46
Figure 36 Second test to check error handling and feedback
The Third test was conducted to test the ability to use the program to build
different design formations. This time the rotation angles for each brick changed in
order to create the desired pattern both for the brick pick and place program and
mortar laying. Another challenge accrued due to the use of the gripper. The new
formation of the bricks meant that the gripper will collide with the previous brick
when exciting pick and place. This required altering the picking location of the bricks
according to the rotation.
Figure 37 third test to check program adaptation to different rotations
Discussion and results Robotic Bricklaying
47
4 DISCUSSION AND RESULTS
4.1 Testing Results
The experimentation for the program started with testing of the success of task
specific operations. Tasks like pick and place and camera movement were straight
forward and high success rate was achieved form initial tests.in the during the test
of laying mortar on the other hand, trying to replicate the bricklayer movements
directly were unsatisfying and had low success rate. This was due to the inconsistency
of the mortar material itself. When a bricklayer usually lays the mortar with the
trowel it is easy for him to determine the consistency of the mortar and adjusts the
movement accordingly, most of the time subconsciously. The robot on the other
hand at this point has simpler inputs to its program and is unable to determine the
consistency of the material it is holding. Several of other challenges occurred in the
creation of the mortar laying path. When the trowel is left for some time while the
robot is laying bricks for example the mortar layer on the trowel starts to harden as
it loses moisture. This makes the trowel lose its smooth surface which makes it harder
for the mortar to leave the trowel after picking it up.
The successful set of movements to lay mortar however was different from the
ones used by a brick layer. This time the process was much longer and included
several movements that might not be needed at all times, but is needed to achieve a
global success rate with different scenarios. After several tests the brick laying
movement had a high success rate to achieve a consist layer of mortar between
bricks. The Onix plugin managed to be a successful tool to produce and control these
paths, test them and stream them to the robot with the robots.
With regards to computer vision, the depth camera was successful to
determine the arbitrary surface of the mortar. The program managed to determine
depth needed for the trowel to go into the mortar to get the desired amount. During
the tests the process required a large amount of mortar to be placed on the working
board. When dealing with a small amount, the mortar usually moved along with the
movement of the trowel, resulting in an unreliable outcome.
Discussion and results Robotic Bricklaying
48
The next tests were conducted to test the ability of the bricklaying server to
control the process autonomously. The server relied on the data coming back from
the robots like the statues of the execution of the programme and the location of the
tip to alter and proceed with its sequence of operations. The tests showed that
building an adaptable program for the robots to perform a multi‐task craft as
bricklaying is achievable. The server program succeeded in the shift between
different tasks while keeping a record of execution progress.
The test also exposed how the error factor due to the complexity of the real
world can accumulate. During the test of the height of each of the layers of bricks
was shifted from the original virtual drawing. This was mostly due to the mortar
settling under pressure from the layers on top. The brick itself was a part of this issue,
even though the brick used were mechanically machined engineering bricks. There
were small differences in the dimension of the brick in comparison to its advertised
size. This resulted in up to three mm difference by the end the fourth row of bricks.
The server was able to keep track of the error and try to compensate it in the mortar
horizontal spacing.
The last test revealed the possibility to use the same process to produce
different brick designs. The parametric inputs of the execution paths allowed easy
manipulation of the bricks rotation angles and locations for each row while keeping
the automation of the program. The pick and place and the mortar laying movements
both adapted to the changes in design.
4.2 Critical assessment
For years the digital field was the major breakthrough arena in research and practice,
the move from digital drawing to the physical creation was left for other disciplines
without the control of the designer. However, there is more focus recently on
physical realisation of ideas. During the robotic brick laying research the emphasis
was on building a robotic program that can adapt and change. This nevertheless was
not a case of direct execution digital to physical. The reliance on an adapting program
meant that a dialogue can be maintained to some extent between the real
environment and the virtual one.
Discussion and results Robotic Bricklaying
49
To achieve the goals of this study, three main aspects needed to be addressed:
the tool, the software that runs it and the task. In this case it was the robotic arm,
the controlling software and the craft of bricklaying. Because of the novelty of the
tool, a part of the research has been conducted to understand how it works, how to
communicate to it and how to control it. This nevertheless resulted in a new platform
that can be used by other users to easily control the robots in future research. The
two software platforms used in the process are still in their early stages and are still
in a constant development. The Onix grasshopper plugin can get more and more
research on board to work with the robots with its simple interface and fast learning
curve. It contained a set of easy tools that can be used directly to run the robots, or
can be altered for more advanced users. Although this platform is capable to work
on its own in the field of fabrication for example, it is still not able to work with
feedback in a way that is more significant than the visualization of real time robot
movement. This made the data flow in one direction from digital to physical and back
but without a way to readapt the program automatically. These drawbacks are
related to the grasshopper platform itself and how the programs is executed inside
it and also other problems with speed for example as grasshopper wasn’t built for
real‐time execution in the first place.
The brick laying server program is the other software developed for the
research .on the contrary was able to work with the feedback with great success. The
use of programing for this tool allowed the program to easily adapt, keeping record
of errors and building up data during the execution. In the same time signalling Onix
to execute its paths with ease. Working with coding allowed the designing of a task
specific interface that can keep the execution and data gathered easily visible for the
user. This puts focus on the importance of programing as a language for controlling
design. The server software is very far from being used with unspecialized architects
in a way that is more significant that dealing with the simple interface. Combining
the two platforms in a one working workflow benefited from both, allowing an easy
familiar way to build robotic paths while maintaining the more specialized adaptable
program.
Discussion and results Robotic Bricklaying
50
Bricklaying as task was a major challenge, Even though several researches were
inducted dealing with bricks. The more interesting challenges emerged from using
mortar rather than the simple task of precise pick and place. Pick and place is a task
taken for granted in robotics for years. On the other side, working with mortar put
emphasis on the problem of dealing with a relatively unpredictable material. Adding
this to the equation of pick and place rendered the precision of it on its own less
efficient to predict the precision of the final result. Although the test was conducted
in a small scale, it showed how the errors can build but quickly which may lead to a
very unstable structure in the end if it wasn’t addressed during the execution.
Mortar also exposed the importance of intuition in such craft as brick laying.
While a brick layer might do one movement to lay mortar with high success. The
initial tests that were conducted to directly copy these movements to the robots
weren’t very satisfying. Achieving a good rate of success with the same tool required
tests and numerous adjustments. Quite a few of the moves in the executed path
might not be required all the time but is there to produce a more general success
rate. The performance of the program was measured by the ability of the robot to
achieve a mortar layer that cover all the surface without having a large amount of
excess material that can come in between laying the next brick. The success rate in
doing that was high in the last test but still was far from producing a very immaculate
end result as produced by a skilled worker. Another drawback was the inability of the
program to fix the errors directly, for example pushing a brick that is higher than the
rest of the row to level it up. This was due to the limitation of the payload of the
universal robots. The robots usually went into emergency stop very easily when
bushing against mortar. This kept the error correction only on a larger scale dealing
with the general execution of the program rather than on the scale of a single brick.
Conclusion Robotic Bricklaying
51
5 CONCLUSION
The research intended to shed the light on the potential of adaptive programing as a
mean to control robotics in the field of architecture. Using such method robots can
become more responsive to the task they are programed to execute. This potentially
can lead to better results working with messy materials and heterogeneous
environments. It can also enhance the design process with more integration of
material behaviour and help overcome the difficulties facing the introduction of
robotics in construction.
The choice of bricklaying as the field of research was successful. It had all the
potentials required to test the hypotheses. It is an important part of construction
industry, with a large space for innovation, at the same time it is a craft that heavily
dependent on human labour. Working on such craft allowed the testing of the ability
of the robots to work with multi material using different tools and the ability to shift
between them seamlessly. Mortar was the most interesting material during the
process. The movement executed by the bricklayer to produce a consistent layer of
mortar is highly based on skill, technique and tuition. Achieving high success rates
with robots needed several tests. The resulting process is still not as immaculate as
the work of a skilled bricklayer. However, the process achieved high success rate in
the end.
On the software side of the process, using such novel tool as the universal robot
resulted in the creation of the Onix plugin, a new plugin specially designed to control
universal robots directly from grasshopper. The potential of the plugin goes beyond
this research and hopefully will continue as potential controlling software for other
researches as well. The plugin allowed easy creation and edit of execution paths for
the robots. This was very useful during the testing and editing of such paths for the
tasks needed to fulfil the process, like brick pick and place and mortar laying. The
process needed another piece of software that is capable to achieve the adaptive
features of the process. The Brick laying server worked as the controller for the whole
process, keeping track of the progress of the program and also dealing with the
Conclusion Robotic Bricklaying
52
feedback from the robot, form the sensors and vision and also controlling the onix
plugin itself.
The small scale tests of the program succeeded in achieving its goals, however
the process is still very far from being used in a construction environment. Humans
still have vastly more sophisticated senses and capabilities and are still also the
economical choice. Nevertheless, it is a step closer to achieving such integration. The
field nevertheless still have great potential of further exploration for example in the
design side of the process, in the mobility of the robots or the integration of machine
learning.
The programing aspect of the research still has great space for further
exploration. The server program was only dealing with a part of the data available
for it. The end‐effector and the robot itself offer large amount of data that is still to
be utilized. Like live video feed from the Intel camera and torque data from the robot.
The increased amount of information available can result in a more sophisticated
program. And with more data, Finding patterns and making sense of this data can
sometimes be more than a human can handle. Research in machine learning can be
a logical next step for robotic adaptive programing. Producing a program that is not
only adaptable but can grow more complex on its own learning from its mistakes and
taking program beyond the designer’s scope.
Appendix Robotic Bricklaying
53
6 APPENDIX
6.1 Appendix I: bricklaying server program operation sequence
persuade code
Figure 38 operation sequence
User selects desired operations and bricks. Pick and place = True Mortar = True Check and adjust Height = True Read robot execution statues While (X < Number of operations)
{ If (communication established && Pick and place && Robot is Idle)
{ Send message “B + Brick location + stack number + True” to Onix X+=1 Brick +=1 If (Brick == column number)
{ Pick and place = False Brick = 0 }
} If (communication established && Mortar && Row > 1 && ! Pick and place && Robot is Idle)
{ Check Height sequence While (Brick < column)
{ Send “H+ Brick location” To Onix Brick + 1 Read Tip location Calculate difference Shift locations of bricks If (Brick == column number)
{ Brick = 0 Height checked = True Mortar = Ture }
} }
If (communication established && Mortar && ! Pick and place && Robot is Idle)
Appendix Robotic Bricklaying
54
{ Pick trowel sequence Send message “T + True” to Onix Trowel picked = true Mortar = false }
If (communication established && Trowel picked && Row > 1 && ! Pick and place && Robot is Idle)
{ Remove excess mortar sequence Send “r+ row height” X+=1 Trowel picked = false Surface checked = true Excess removed = True }
While ( Brick < column) {
If (communication established && excess removed || Row=1 && && ! Pick and place && Robot is Idle)
{ Run Check motor surface sequence Send “C + True” to Onix Run depth calculation algorithm Trowel picked = false Surface checked = true }
If (communication established && surface checked && ! Pick and place && execution not in progress)
{ Pick mortar from result location Send “P+ result location from depth” to Onix Surface checked=false Mortar picked = true }
If (communication established && mortar picked && ! Pick and place && Robot is Idle)
{ Lay mortar from result location Send “m+ Brick location” Brick +=1 X+=1 Mortar picked = false If (Brick == column number)
{ Brick = 0 Mortar laid = True }
} If (communication established && mortar picked && ! Pick and place && Robot is Idle)
{
Appendix Robotic Bricklaying
55
Lay mortar from result location Send “M+ Brick location” to Onix Brick +=1 X+=1 Mortar picked = false If (Brick == column number)
{ Brick = 0 Mortar laid = True }
}
} If (communication established && Mortar laid = && ! Pick and place && Robot is Idle)
{ Leave Trowel sequence Send “L” to Onix Mortar =false Pick and place = True Trowel left = false }
} If X = Number of operations
{ STOP PROGRAME }
}
Appendix Robotic Bricklaying
56
6.2 Appendix II: Inverse Kinematic and uploading components
Figure 39 IK solver (Left Square) and uploading component (Right Square) location in the Onix plugin example file
Figure 40 IK solver component parts cluster
Figure 41 Point A calculation
Figure 42 Point B calculation
Appendix Robotic Bricklaying
57
Figure 43 Point C calculation
Figure 44 Angle sorting algorithm cluster
Figure 45 Robot uploading component cluster
Appendix Robotic Bricklaying
58
6.3 Appendix III : universal robots possible solutions via Onix IK solver
Figure 46 the eight possible solutions for a single tip plane using Onix IK solver
References Robotic Bricklaying
59
7 REFERENCES
Andres, J. et al., 1994. First Results of the Development of the Masonry Robot System ROCCO: a Fault Tolerant Assembly Tool. Automation and Robotics in Construction XI, 11, pp.87–93.
Balaguer, C. & Abderrahim, M., 2008. Trends in Robotics and Automation in Construction. , pp.1–22.
Bechthold, M., 2010. The Return of the Future: A Second Go at Robotic Construction. Architectural Design, 80(4), pp.116–121.
Braumann J. and S. Brell‐Cokcan., 2013. Rob|Arch 2012: Robotic Fabrication in Architecture, Art and Design, Vienna: Springer Vienna Architecture.
Braumann, J. & Brell‐Cokcan, S., 2011. Parametric Robot Control: Integrated CAD/CAM for Architectural Design. In ACADIA.
Campbell, J.W.P. & Pryce, W., 2003. Brick: A World History, London, UK: Thames & Hudson.
Colletti, M. et al., 2013. Robotic foaming cluster SG2013. Available at: http://smartgeometry.org/index.php?option=com_content&view=article&id=219:robotic‐foaming&catid=48:sg2012‐london [Accessed April 20, 2013].
Dubor, A. & Diaz, G.‐B., 2012. Magnetic Architecure. In Rob\Arch. p. 7.
G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle, J.H., 1994. Application specific realisation of a mobile robot for on‐site construction of masonry. ISARC Proceedings, 11, pp.95 –102.
Glynn, R. & Sheil, B., 2011. Fabricate: Making Digital Architecture, Riverside Architectural Press.
Gramazio, F. & Kohler, M., 2008. Digital Materiality in Architecture: Gramazio & Kohler, Boden: Lars Muller Publishers.
Helm, V. et al., 2012. Mobile robotic fabrication on construction sites: DimRob. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp. 4335–4341.
Hill, J., 2005. Building the Drawing. Architectural Design, 75(4), pp.13–21.
international fedration of robotics, 2012. History of Industrial Robots, From the first installation until today, Frankfurt. Available at: http://www.ifr.org/uploads/media/History_of_Industrial_Robots_online_brochure_by_IFR_2012.pdf.
Keating, S. & Oxman, N., 2013. Compound fabrication: A multi‐functional robotic platform for digital design and fabrication. Robotics and Computer‐Integrated Manufacturing, 29(6), pp.439–448.
References Robotic Bricklaying
60
Kolarevic, B., 2005. Architecture in the Digital Age: Design and Manufacturing, Taylor & Francis.
Oxford dictionaries, robot: definition of robot in Oxford dictionary (British & World English). Available at: http://oxforddictionaries.com/definition/english/robot [Accessed August 28, 2013].
Piker, D., 2013. Lobster grasshopper group. Available at: http://www.grasshopper3d.com/group/lobster [Accessed April 1, 2013].
Roulet, O., python‐urx. Available at: https://github.com/oroulet/python‐urx [Accessed August 1, 2013].
Schlegel, A., 2012. ControlP5 plugin. Available at: http://www.sojamo.de/libraries/controlP5/ [Accessed April 1, 2013].
Spong, M.W., Hutchinson, S. & Vidyasagar, M., 2005. Robot Modeling and Control, John Wiley & Sons.
universal robots, The URScript Programming Language. , p.30. Available at: http://www.wmv‐robotics.de/home_htm_files/scriptmanual_en_1.5.pdf [Accessed June 1, 2013].