robotic bricklaying, towards adaptive robotic programing

1 `

MSc Adapt ive Arch i tecture and computat ion Bart let t school of graduate Studies Univers i ty Col lege London September 2013

Robotic Bricklaying Towards adaptive robotic programing Khaled ElAshry

2 `

Word Count: 10891

This dissertation is submitted in partial fulfilment of the requirements for the degree

of Master of Science in Adaptive Architecture & Computation from Bartlett School of

Graduate Studies, University College London.

I, Khaled ElAsrhy confirm that the work presented in this thesis is my own. Where

information has been derived from other sources, I confirm that this has been

indicated in the thesis.

September 2013

Abstract Robotic Bricklaying

3

ABSTRACT

After the recent move of the field of architecture towards fabrication, the use

of industrial robots surfaced as a replacement of custom fabrication machines.

Robots are typically controlled with strict offline programing to produce controlled

results similar to how robots are used in manufacturing. However, in construction

robots did not manage to replace humans in on site operations. In the bricklaying

profession, the craftsman uses his tuition to adapt to the material he is working with

and to the changing work environment. This raised the question of the ability to build

a system that can respond to changes in order to replicate the craft using robots. The

research focuses on two main aspects, of bricklaying: The craft of laying brick and

mortar itself and the ability to keep track of the work on a larger scale. In order to

replicate these aspects an adaptive program had to be built with reliance on sensors

and feedback to control the robotic arms. The robotic arm managed to achieve the

tasks in question using two software: A new plugin to control the movements of the

robots and a server that is capable of dealing with feedback.

Key words: Robotic Arms, adaptive programing, bricklaying, construction,

architecture, fabrication, feedback, computer vision, unpredictable material, Mortar,

Brick.

Contents Robotic Bricklaying

4

CONTENTS

Abstract ............................................................................................................... 3

Contents .............................................................................................................. 4

Acknowledgment ................................................................................................ 6

1 Introduction ............................................................................................... 9

2 Background .............................................................................................. 12

2.1 The craft of bricklaying ......................................................................... 12

2.2 Robots in architecture .......................................................................... 15

2.2.1 Early adoptions ............................................................................. 15

2.2.2 The current state of robotic research in architecture. ................. 16

2.3 Observations during the Robotic foaming Workshop, SG2013 ........... 17

2.3.1 Robotic brick work ........................................................................ 19

3 Methodology ........................................................................................... 21

3.1 The robotic arm .................................................................................... 21

3.2 Onix Grasshopper plugin ...................................................................... 23

3.2.1 Kinematics of the UR robotic arm. ............................................... 24

3.2.2 Forward kinematics ...................................................................... 25

3.2.3 Inverse kinematics ........................................................................ 26

3.2.4 Solution toggles ............................................................................ 27

3.2.5 Angles calculation ......................................................................... 30

3.2.6 Execution path generation ........................................................... 31

3.2.7 Robot program and program uploader ........................................ 32

3.3 The end‐effectors ................................................................................. 33

3.4 Robot execution paths ......................................................................... 34

3.4.1 Pick and place of bricks. ................................................................ 35

Contents Robotic Bricklaying

5

3.4.2 Accompanying tasks. .................................................................... 35

3.4.3 Laying mortar ................................................................................ 36

3.5 Brick laying server ................................................................................ 38

3.5.1 Program operation sequence ....................................................... 39

3.5.2 Interface ........................................................................................ 40

3.5.3 Computer vision and sensing ........................................................ 41

3.5.4 TCP/IP Feedback. .......................................................................... 43

3.5.5 Bricklaying server to Onix communication ................................... 44

3.6 Wall building tests ................................................................................ 44

4 Discussion and results .............................................................................. 47

4.1 Testing Results ..................................................................................... 47

4.2 Critical assessment ............................................................................... 48

5 Conclusion ................................................................................................ 51

6 Appendix .................................................................................................. 53

7 References ............................................................................................... 59

Acknowledgment Robotic Bricklaying

6

ACKNOWLEDGMENT

I take this opportunity to express my profound gratitude and deep regards to my

supervisors Sean Hanna and Ruairi Glynn for their endless support and guidance,

Angelos Chronis for his valuable feedback, the Bartlett CADCAM team for their great

help, my gurus Mohamed Zaghloul and Ebtissam Farid for their inspiration, my

friends for always being there for me, my father Mohamed ElAmir ElAshry and sisters

Kholoud and Tayseer for their endless support and encouragement, and last but not

least my late mother Neveen ElMaghloub for her love and endless belief in me.

List of figures Robotic Bricklaying

7

LIST OF FIGURES

Figure 1 the tomb of Saminids Bukhara, 900 A.D (Campbell & Pryce 2003) ............. 12

Figure 2 Bricklaying on‐site ........................................................................................ 14

Figure 3 left: First commercial ABB robot IR6 1974 Right: KUKA first robotic arm,

Famulus 1973 ............................................................................................................. 15

Figure 4 Robot brick laying system ((G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle

1994) .......................................................................................................................... 16

Figure 5 Experiments during the Robotic foaming workshop SG2013 ...................... 18

Figure 6 R.O.B system introduced (Gramazio & Kohler 2008, p.57) .......................... 19

Figure 7 DimRob system (Helm et al. 2012, p.1) ....................................................... 20

Figure 8 : To the left Six degrees of freedom around a point .................................... 22

Figure 9: To the right a sample of the solutions of rotations around a point using a

UR10 robot and Onix .................................................................................................. 22

Figure 10 left : Spherical axis robotic arm Right: non sperical axis robotic arm ....... 22

Figure 11 Onix grasshopper plugin example file ........................................................ 24

Figure 12 Universal Robot UR10 servo axis diagram ................................................. 25

Figure 13 Forward kinematics component in Onix .................................................... 25

Figure 14 Inverse kinematics component in Onix ...................................................... 26

Figure 15 First IK solution toggle diagram ................................................................. 27

Figure 16 Second IK solution toggle diagram ............................................................. 28

Figure 17 Third IK solution toggle diagram ................................................................ 29

Figure 18 Angle calculation digram ............................................................................ 30

Figure 19 Move, I/O and program uploader components in Onix ........................... 32

Figure 20 Components of the gripper end‐effector .................................................. 33

Figure 21 right: End‐effector attached to the robotic arm. Left: The trowel attached

to the handle .............................................................................................................. 34

Figure 22 Pick and place sequence ............................................................................ 35

Figure 23 Gripper holding the trowel for mortar operation ...................................... 36

Figure 24 laying mortar execution path sequence .................................................... 37

Figure 25 robot executing mortar laying ................................................................... 37

Figure 26 robot removing excess material using the trowel ..................................... 38

List of figures Robotic Bricklaying

8

Figure 27 the bricklaying server in operation. ........................................................... 38

Figure 28 Operation sequence of the bricklaying program ....................................... 39

Figure 29 Detailed operation sequence of the bricklaying program ......................... 40

Figure 30 Bricklaying server interface ........................................................................ 40

Figure 31 Intel procedural computing camera .......................................................... 41

Figure 32 Intel camera point cloud used for surface approximation ........................ 42

Figure 33 Limit switch testing brick hieght ................................................................ 43

Figure 34 right: Sequence of communication between the bricklaying server and Onix

for a single operation left: program dispatcher in Grasshopper ...................... 44

Figure 35 first test to check mortar laying movements and server operation .......... 45

Figure 36 Second test to check error handling and feedback ................................... 46

Figure 37 third test to check program adaptation to different rotations ................. 46

Figure 38 operation sequence ................................................................................... 53

Figure 39 IK solver (Left Square) and uploading component (Right Square) location in

the Onix plugin example file ...................................................................................... 56

Figure 40 IK solver component parts cluster ............................................................. 56

Figure 41 Point A calculation ..................................................................................... 56

Figure 42 Point B calculation ...................................................................................... 56

Figure 43 Point C calculation ...................................................................................... 57

Figure 44 Angle sorting algorithm cluster .................................................................. 57

Figure 45 Robot uploading component cluster ......................................................... 57

Figure 46 the eight possible solutions for a single tip plane using Onix IK solver ..... 58

Introduction Robotic Bricklaying

9

1 INTRODUCTION

Robotics is a fairly new field in technology; The term robot dates to the 1920s (Spong

et al. 2005, p.1). Robots have come a long way since then, growing more complex

and capable. Robots are now a key part in manufacturing industries such as the

automotive industry. However, in construction, they did not play an important role.

Human labour remained the main asset for many reasons. e.g., the work

environment. Construction is a very complex work field; tasks that are very easy for

human to achieve (like avoiding an obstacle or moving from one work location to the

other) are big challenges for robots. In the 1980s (Bechthold 2010, p.119) numerous

trials to integrate autonomous machines in construction surfaced especially in japan,

these trials were faced by another major drawback; the economical visibility of the

integration of such technology. These machines were huge and expensive which

meant that they weren’t a viable solution economically to manual labour. Also,

construction now might take the advantage of machinery; they are most of the times

directly controlled by humans and very far from being autonomous. Recently, the

interest in robotic in construction has surfaced again as robots are getting cheaper

while labour in western societies is getting more expensive.

Robotic fabrication has been traditionally associated with high‐tech industrial environments, where fixed positioning and constant conditions determine the role that the robot may undertake in the fabrication process. Unlike in stationary facilities, a construction site is a spatially complex and heterogeneous environment in which a robotic system may be exposed to continuous change and unpredictable events. (Helm et al. 2012, p.1)

Bricklaying is a construction profession that depends mainly on manual labour.

The craft of bricklaying hasn’t changed dramatically through history. The tool used in

the modern construction site is as old as the profession itself(Campbell & Pryce 2003,

p.303). The craftsman throughout history played a huge role in the realization of

several great structures that are still standing till now. Architects had a huge

influence on bricklaying as they themselves were sometimes considered master

craftsmen (Hill 2005, p.14). Nowadays, the craft is becoming more of a profession of

filling the gaps between prefabricated parts of the buildings in the modern western

society. However, Architects are returning to their historic close relation to the


10

realization of their designs in the real world, researching new techniques in

fabrication and construction, retaking their role as master builders (Kolarevic 2005,

p.65).

In Architecture, robotics has become a major research arena over the past eight

years. The introduction of robotics in architecture research has started with the move

of the field towards research in fabrication. As research grew more complex in the

virtual form finding field, Architects were getting more and more interested in the

art of making. Industries like ship construction(Kolarevic 2005, p.9) had a great

impact on research in fabrication. Industrial robotic arms which were a crucial part

of such industries for years, are getting much cheaper. Which made them a viable

replacement to getting huge specialized milling or routing machines. The versatility

of the robots meant there are several uses that go beyond just that and new research

by pioneers in the field started to explore their potential. Robotic arms are very

versatile machines, their use is fundamentally determined by the program that

operates them and the tool attached to it. They were able to work with composite

materials for example, and were as well able to do jobs like pick and place very

efficiently. Architects were also researching in the software side of robotics, linking

robotic operation to CAD software for easier generation of the controlling program

for instance.

In fabrication, Material behaviours plays a big role, the complexity of the

integration of material behaviour in the process is determent by the unpredictability

of the material itself. The exploration of the use of the robots with messy materials

and composite fabrication projects required more advanced programing for the

robots. This time using senses and feedback to produce an autonomous program that

can adapt and change to external stimuli, making them more efficient and

productive. This research into sensing techniques and adaptable programing is

getting us closer to achieve robotic integration in heterogeneous environments like

construction sites.

Bricklaying is a craft that require practice, tuition and adaptable technique with

the need to work with relatively unpredictable materials. Robots, also are powerful

and accurate machines, are generally very inadequate at the execution of such set of


11

qualities. This raised the question of the ability of an autonomous robot to replicate

this craft using the same low tech tools used by a bricklayer. In order to achieve that

the research the primary focus of this research is to develop an adaptive operating

workflow that can address two specific qualities of the bricklaying workmanship

using robots:

The craft of working with messy materials like mortar in combination

with other components like bricks with the ability to shift between

them

The ability to keep track of the larger plan and execute error correction.

Following the introduction, the background chapter focuses on the

bricklaying craft, discussing the role of the bricklayer, followed by the recent related

research in the fields of robotics and fabrication in architecture. Next chapter is the

methodology, this chapter focuses on the creation of the software platforms and the

process that allowed the testing of the craft reproduction using robotic arms, and

also the tests conducted to build wall samples. Upon that the discussion chapter

discuss the results and outcomes of the tests. And the research then ends up with

conclusion.

Background Robotic Bricklaying

12

2 BACKGROUND

2.1 The craft of bricklaying

2.1.1 The evolution of bricklaying

The Hanging Gardens of Babylon, one of the seven wonders; the Great Wall of China, the largest manmade object in the planet; the hag Sofia, one of the most beautiful churches ever built …, they were built out of bricks. Brick is at once the simplest and most versatile of materials, the most ubiquitous and the least regarded, all too familiar yet strangely neglected (Campbell & Pryce 2003, p.13)

Brick is one of the oldest building materials. Moulded brick is dated back to the 5000

BC (Campbell & Pryce 2003, p.15). Brick building evolved significantly through history

with inventions like fired bricks and the use of concrete in Greece. Historic Brick

building varies according to the type of brick, type of bonding material used and the

technique used by the bricklayer. Bricklaying craftsmanship played a big role in the

construction of the masterpieces all around the world. Throughout the history the

relation between the Architect and the craftsman in this case the Bricklayer was

strong. Architects were considered master masons in the middle ages. However, In

the Italian Renaissance architects started distancing themselves from the craftsmen

focusing more on the building representation (Hill 2005). Like many labour

dependent workmanship, the bricklaying skill level required in ancient times is no

longer in demand no today and skills that are not in demand diminish by time

(Campbell & Pryce 2003).

Figure 1 the tomb of Saminids Bukhara, 900 A.D (Campbell & Pryce 2003)


13

The Bricklaying craft was always accompanied through history by the

profession of brick–making .In the western world nowadays, brick‐making is almost

extinct. Bricks now are manufactured mechanically and transported to site. On the

other hand, bricklaying seems to be fairly untouched by modern technology

(Campbell & Pryce 2003, p.303). Machines that take bricklayer place are still not

seen. This is the very distinct difference between the evolution of manufacture and

construction in recent history. The brick can be mass produced cheaply in a factory

in a relatively controlled automated manner. While a bricklayer has to deal with the

changing work environment, adapting to it with each different job on site.

2.1.2 Observations of the bricklaying craft on-site

Bricklaying is a profession that requires years of experience. The work of a

master worker can be easily differentiated from a novice one especially when dealing

with facing bricks. When the research started there was a necessity to watch a

bricklayer craftsman and also try brick laying to get to know the profession. The

observation and personal tries of bricklaying on‐site resulted in the Focus on two

main aspects that defines the craft.

1. the act of brick and mortar laying itself

2. The focus on larger scheme of things.

A perfect technique will cut time and waste material, and also result in a much

better end product. Technique and material greatly affect the result. Personal tries

revealed how the craft is a combination of skill that can be taught and expertise that

is compiled over time. The bricklayer almost does his craft unconsciously. The tool is

a major part of the process. The trowel itself is as old as the profession of brick laying

and haven’t changed significantly along its history. Using such low tech tools, human

tuition plays a huge rule in deterring the movement according to the different

features of the material itself like weight and consistency.


14

Figure 2 Bricklaying on‐site

A bricklayer might seems to only focus on laying a brick but he also keeps an

eye on the larger plan. This is vital in order for the brick rows and columns to end up

matching in the whole building. Bricklaying is accompanied by other work on‐site like

building openings and reinforced concrete works for example. A simple error in

calculation can easily propagate to produce huge problems in the end that may

require major error fixing or even the destruction of wall portions resulting in badly

finished product.

Bricklaying is still very far from mechanization but the quality of the craft is

declining in western high tech world. Prefabricated parts is being used when possible.

This lean towards prefabrication wasn’t only for machined parts that may require

special manufacturing, but even the parts of the brick building that required most of

the skill like the arches for example are shipped fully assembled and put in place in

construction site. This is due to the higher expenses required to hire a quality

craftsman to work on‐site. And at the same time ability to better control construction

time and increases efficiency as much as possible.

Brick laying is still a relatively untackled territory when it comes to automation.

This gives a large space for what can be achieved. The innovation in the field brick

design require new tools that can reproduce the technique and quality of ancient

brick laying and go beyond that. At the same time increase efficiency and decrease

time of construction, while also filling the need for such craft in unhuman

environments for instance.


15

2.2 Robots in architecture

2.2.1 Early adoptions

Industrial robotic arms have been offered commercially since the 1970s

(international fedration of robotics 2012, p.3) Machinery has been a core topic in

manufacturing for many years pushing the limits of research. Productivity on such

fields have increased significantly in the last decade as opposed to the construction

industry (Balaguer & Abderrahim 2008, p.2). Robots however seems to be always

constrained to work in highly controlled facilities. Many robots barely have any

sensing systems of their environment, perhaps a few sensors to detect errors like

motors overloading but little else. When the robotic on‐site construction idea was

explored in the 1980s and 1990s (Bechthold 2010, p.119) the formula that succeeded

in the manufacturing industry didn’t work here. Robots were never developed to

adapt to changing environments or to deal with a messy type of data. In fact they

were developed to do very specific tasks with very specific programs that may take

years to develop and test and then run for the life time of the machine (Keating &

Oxman 2013, p.1).

Figure 3 left: First commercial ABB robot IR6 1974 Right: KUKA first robotic arm, Famulus 1973

In 1980s, during the Japanese booming era, up to 200 systems of automated

construction were developed (Bechthold 2010, p.3). During this era the Japanese

economy was very strong and the construction rate was very high. The profession of

a construction worker on the other hand was very alienating for its high risk and low

payment. The scarcity of construction workers in opposition of high demand required

searching for new possibilities. Robots seemed to be the solution, and numerous


16

automated systems were developed. These were huge custom machines that

required large amount of money to build and operate. By the end of the 1990s, the

economy of Japan was hit and the research in automated construction stopped.

The Idea continued with a slower pace. In 1994 some papers were published

during the 11th ISARC conference. One of the papers was discussing the autonomous

machine to build brick walls (G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994).

Another system under the name of ROCCO was also proposed as a solution for

robotic masonry work on‐site (Andres et al. 1994). The system was referred to as a

“Fault Tolerant Assembly Tool”. As inferred from the title it tried to rely on sensors

and feedback for fault correction .It was designed as a purpose specific robotic

system. This time it was smaller and more approachable than the systems that were

developed in japan. It also proposed a link between CAD and construction directly.

The research in the platform continued with two more papers in 1995 and 1996, but

it wasn’t put in practical use.

Figure 4 Robot brick laying system ((G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle 1994)

2.2.2 The current state of robotic research in architecture.

In recent years. The architecture practice has been changing and branching

with more research into material, construction and fabrication, forming new modes

of collaboration between disciplines. Architects this time w developing their own

design culture (Glynn & Sheil 2011, p.117). In this new turn architects are searching

into means to represent their digitally created work in the real world. This turn

towards fabrication made research in robotics surface again with huge momentum.

This time using the same robots that has been used by the industry for years instead


17

of using huge custom robots (Braumann J. and S. Brell‐Cokcan. 2013, p.8). Robots like

these are getting more affordable thus becoming more and more accessible for both

architectural research and even practices. Pioneering offices like Snøhetta and

Foster+Parteners are getting their own robotic arms in order to experiment with

fabrication. For the major part of the research robots are usually used as either a

sophisticated 3D printing or milling machine executing specific preconfigured paths

in a controlled manner similar to the way it is used in the industry. However, this time

architects are finding better much faster ways of planning these fabrication programs

that is more approachable and simpler. This advanced the research of robotics in

architecture significantly in the field of fabrication. It allowed a faster learning curve

and ability to easily link to geometry directly from CAD software.

The real world is very complex which make modelling it in a virtual

environment is usually quite a challenge. Another side of research in robotic

fabrication explores a more sophisticated operational programing. This requires

more knowledge of the field robotics and links to other fields like computer vision,

mechanical engineering and mathematics. . Research in this field rely on using

feedback loops and more sophisticated sensing methods to read from the

surrounding environment to allow a more dynamic approach that can adapt with the

material in use.

2.2.3 Observations during the Robotic foaming Workshop, SG2013

During the participation in the “Robotic foaming workshop” (Colletti et al. 2013) in

the 2013 smart geometry conference, the problem with direct blind execution from

digital to physical was visible. The workshop was built on the idea of working with

spray foam (a very unpredictable material), using the precisely controlled robots. This

subject matched perfectly the conference theme “Constructing for Uncertainty” in

theory. Even though very interesting forms merged form the workshop, it was far

from perfect in the process front. During the workshop, foam was mixed manually

with other ingredients like hardener and water. The process produced a very elastic

fibrous mixture. It was then attached to plates on the three robotic arms. The arms

then took the part of stretching the mixture. During the testing the foam mix


18

produced was very inconsistent, and even mixtures that were produced in a very

similar and timed process still behaved differently.

Figure 5 Experiments during the Robotic foaming workshop SG2013

The material was delicate and required the constant change of speeds while

executing the program. Due to that, running the preconfigured program directly

always resulted in a complete tear of the material and the failure of the test. On the

other hand, the tests run by pulling it manually by the participants were very

successful. In order to achieve the same result using the machines, the robots had to

be controlled almost in a total non‐autonomous manual manner using the control

joystick, this defied the purpose of the controlled offline designed program. The work

on this project put into perspective the importance of having an online feedback

loops and vision systems that can deal with such materials while keeping the

automation. The importance of using sensors and adaptable programing was

investigated in other similar projects recently like The magnetic architecture (Dubor

& Diaz 2012). The project was dealing with form finding using iron filling and

magnets. The same result was concluded from the research; Digital representation

only can’t achieve accurate results with such messy material. This deemed the linear

process of offline program generation not reliable.


19

2.2.4 Robotic brick work

ETH Zurich is considered one of the major pioneer institutes that work on the

integration of robotics in architecture. One of their major areas of research is the

construction related manufacturing, especially with bricks. Gramazio and Kohler

succeeded in creating a practice that can combine research and architectural work

with many projects designed and constructed in the past years. Many of the brick

walls manufacture were usually produced in a lab environment and then shipped to

the construction site like the Gantenbein vineyard façade project for example

(Gramazio & Kohler 2008, p.95). Although these walls were very innovative visually,

the product was never structural and resembled a cladding system more. This was a

great potential worth exploring, building structural Brick walls that can also benefit

from the innovative visual effect.

Figure 6 R.O.B system introduced (Gramazio & Kohler 2008, p.57)

Other construction related robotic research at ETH Zurich considers the use of

robotic arms outside the laboratory setting. In 2007 R‐O‐B was introduced (Gramazio

& Kohler 2008, p.57), it is a mobile manufacturing facility that included an ABB

industrial robotic arm at its core (Figure 6). This allowed the exploration of the

production of brick structures on‐site. The unit was moved around the world in

locations in Europe and the USA and produced brick walls on‐site. Another recent

project had more emphasis on feedback. DimRob is mobile construction system

designed also in ETH Zurich. The system relied of a mobile robotic arm. The size of


20

the proposed system allowed the movement of the robots trough standard openings

in site (Figure 7). It heavily relied on sensors to respond to the uniqueness of each

construction site(Helm et al. 2012, p.1) along with more collaboration with humans.

Figure 7 DimRob system (Helm et al. 2012, p.1)

The research in the field of on‐site construction and dealing with changing

environments and materials is still relatively young in the architecture field. The

Pioneers in robotics and fabrication are researching more in this field of adaptive

robot programing and the use of sensing capabilities. Such field have several

applicable potentials. Bricklaying is considered a great candidate for the integration

for such process.

Methodology Robotic Bricklaying

21

3 METHODOLOGY

In this research, the robotic arm was an important part along the software that runs

it. The understanding of such tool was crucial. However, this was not a straight

forward process the novelty of the robots used during the research and the lack of

any technical support meant it was generally a process of trial and error. This also

required building new platforms from scratch to explore the controlling possibilities

of these robots before the research can proceed with the testing of its hypothesis.

3.1 The robotic arm

Robots are fascinating electromechanical machines. It is defined as “a machine

capable of carrying out a complex series of actions automatically, especially one

programmable by a computer”(Oxford dictionaries n.d.). As the definition implies,

robots are usually defined by two main aspects: their physical mechanical aspect and

the software that runs them .In this essence the robotic revolution went hand in hand

with the finding and explorations in the field of computer science.

The most common multifunctional robots are robot Manipulators “Robot

Manipulators are composed of links connected by joints to form a kinematic chain.

Joints are typically rotary (revolute) or linear (prismatic).”(Spong et al. 2005, p.3). This

study has focused on the most common type which is the articulated rotary robotic

arms. Robotic arms are considered rigid body systems and usually are defined by its

degrees of freedom. They are the possibilities of movement of the end effector tip.

The most common number of degrees in industrial robotic arms is six degrees where

the end is able to translate in three dimensions and also rotate around three axes

(Figure 8); this allow all the possible motions around a point, which renders more

degrees in most cases useless. The degree number is also directly related to the

number of servos in the robot.


22

Figure 8 : To the left Six degrees of freedom around a point

Figure 9: To the right a sample of the solutions of rotations around a point using a UR10 robot and Onix

The setting of the robotic arm’s servos affects directly how the robot is

controlled. The most common type of robotic arms manufactured by major

manufacturers is the spherical wrist robotic arms. In this type the three wrist axes of

rotations intersect at one point. The non‐spherical wrist type which is relatively

uncommon was introduced recently by the Danish company universal robots. The

latter type has advantages over the spherical wrist type but needs a more complex

inverse solution for its kinematic chain.

Figure 10 left : Spherical axis robotic arm Right: non sperical axis robotic arm

In 2012 The Bartlett School of architecture received two universal robots for

the sake of expanding its fabrication facilities. The UR10s are six degree of freedom

non‐spherical axis robots with 10 kg pay load. The problem the research faced along

other initial users of the robots is the lack of any controlling software or technical


23

support for these robots as opposed to others manufactured by the other companies

like ABB and KUKA. The robots initially were only controlled via its control pendent

and could only be programed for simple pick and place programs. The uncommon

joint configuration design meant the available tools for other robots cannot be

reused to control the URs at the same time they lacked any sort of publicly available

control software either by the manufacturing company itself or third party

developers. In the search for tools to control the robot many suggestions were tested

like using the Blender modelling software to control the robots via bones and

animation (developed by Tom savilans). The initial blender model relied on the

animation functions in blender and required to send the instructions to the robot one

by one. The tool however was unreliable and a new tool was needed to control the

robots

3.2 Onix Grasshopper plugin

In the beginning of the research, it was very crucial to build a control software that

will enable the researcher to work with the robots in a meaningful way. It was

decided to build tool from scratch that will allow planning robot jobs easily with the

ability to directly upload to the program. This also allows having access and the ability

to fiddle with all its components according to the needed task. Grasshopper was

chosen as the base platform to develop the software.

Grasshopper is a .net plugin that runs on top of the 3D NURBS modelling

software rhino. It combines ease of use through its VPL interface and also the ability

to program in VB, C# or python languages directly inside it. The adoption of several

of practices and universities to using grasshopper made it a very good candidate to

build a controller software. Grasshopper also has several built‐in tools to create

plans. The planes contain all the required data (three translations and the rotation)

in order to calculate the kinematic chain of the robot. These tools become very useful

to build paths that the robot can execute along points, curves or even surfaces. Other

solutions for controlling robotic arms that were built on it for other manufactures

succeed in the architecture related research environment. In a comparison between

the use of KUKA|PRC (a grasshopper plugin for offline controlling of KUKA robots)


24

against the company supplied path planning software was found to increase

productivity and reduce time to move from design to fabrication (Braumann & Brell‐

Cokcan 2011) .

Figure 11 Onix grasshopper plugin example file

The result definitions during the research lead to the creation of the Onix

grasshopper plugin. The plugin allowed the control of the robot directly through

streaming from the grasshopper interface. It included Inverse and forward Kinematic

solver for the universal robots in addition of upload, communication and end–

effector tools which will be detailed further in the upcoming part (Figure 11). The

plugin was packaged and released as an open source plugin and is used as a

controlling platform for other research projects.

3.2.1 Kinematics of the UR robotic arm.

The robotic arms are considered rigid bodies. When using robotic arms the user

usually is majorly concerned mainly by the position and orientation of the tip of the

end effector (The end‐effector refer to the tool used at the end of the robot that may

have extra transformations). In order to represent this tip point, six number are

needed which represent transformation location and rotation of the point, this set

of data will be referred to as planes during the research. The set of servo rotation

angles that achieves this tip plane position is called configuration(Spong et al. 2005,

p.4). The universal robots have six servos referred to by the manufacturer as (base,

shoulder, elbow, wrist 1, wrist 2, wrist 3) (Figure 12) The calculation of kinematic


25

chain for any configuration require two major function to be computed: Forward and

inverse kinematics.

Figure 12 Universal Robot UR10 servo axis diagram

3.2.2 Forward kinematics

Figure 13 Forward kinematics component in Onix

Forward kinematics deal with the relationship between the joints positions the

position and orientation of the end‐effector(Spong et al. 2005, p.68) . The forward

kinematic fundamentally solves the location of the tip according to the joint position.

The related component in the Onix plugin relied on an input of the six rotation angles

(the configuration) of the robots servos (Figure 13). The plugin then applies a set of

transformation and rotations to the model of the robot until achieving the tip

position. It was also used also to show the representation of the robot along with the

solution of the tip plane.


26

3.2.3 Inverse kinematics

For the Inverse kinematics solution, the function is presented with the tip plane as

the input and calculates the set of the joint positions needed to achieve this location.

Inverse kinematics is generally nonlinear meaning it results in multi solutions for a

single input. In this part the difference between the universal robots and their

counterpart are the most noticeable. The less common new servo setting of the

universal robots meant fewer singularities (Singularities in robotics refer to a tip

plane that don’t have one unique solution in the joint solution space. This either

refers to having infinite solutions or no solution at all) that may occur in the middle

of the configuration space. The UR robots are capable of eight different solutions for

most of their reach space (Appendix III) as opposed to four in the spherical wrist type.

In order to calculate those solutions, the inverse kinematic solver in Onix was built

from scratch directly in grasshopper using a geometric approach rather than using a

solely mathematical one. This allowed it to be easier to debug and edit in

grasshopper.

The input of the inverse kinematic solver in Onix consist is the plane of the tip

point. Starting with this input, the kinematic chain is computed then the angles are

calculated accordingly. The inverse kinematic chain is calculated by solving the

location of three key points along the chain. Each of these points have multiple

solutions for each tip plane input, thus three more toggles were added to the input

of the inverse kinematic solver in order to switch between the eight different

solutions.

Figure 14 Inverse kinematics component in Onix


27

3.2.4 Solution toggles

Due to the novelty of the universal robots, there were no information about how to

solve their inverse kinematic chain and the initial trials with using the same methods

for spherical robots directly was unsuccessful. The new proposed solution however

came from the observation of the robots themselves. During their operation it was

noted that some configurations can have unlimited solutions in the joint space,

where the fifth axis can rotate continuously altering the rest for the chain while

maintain the tip poison. The solver was built initially to solve this configuration as it

allowed the second point in the chain to be disregarded (point B as referred to latter

on) as any solution for it is legitimate. After calculating the geometrical solutions for

the first and third points, finding the solution for the second point for the rest of the

configurations was much easier and the kinematic chain was finally constructed.

Figure 15 First IK solution toggle diagram

The solutions calculation starts with the inputs of the tip end plane. The first

point along the chains (point D) (Figure 15) is determent as the offset of the tip plan

in the direction of its normal; this determines the location of the fifth axis. Afterwards

the different solutions are calculated. First solution toggle results from the

calculation internal tangents between two circles.

1. One with the point D as its centre, and radius equal to the distance

between axes five and six. (These distances are constant and

comesfrom the robot manufacturing dimensions).


28

2. The second circle with the base origin coordinates as its centre,

projected on the first circle corresponding plane, with radius is equal

the distance between the origin and the second axis.

These two circles represent the unlimited solution that the fifth and second

axis servo rotations. The intersection between the tangents and the resulting circles

represents two unique solutions that achieve the desired tip position. Point A on the

chain is determent according to the toggle choice.

Figure 16 Second IK solution toggle diagram

The Second solution toggle calculation depends on the solution of two

intersecting planes. One representing the tip plane and the other is the plane normal

to the perpendicular vector on the chosen tangent (from the first solution)(Figure

15). The two solution for this toggle result from intersection between the resulting

straight line and the circle representing all the possible solutions of the forth servo

rotation around the six. The point B Figure 16) represents the selected solution point.

This solution toggle is the major difference between the URs and the spherical wrist

robots.


29

Figure 17 Third IK solution toggle diagram

The third solution toggle is determined by the calculation of the intersection

between two circles with centres A and B (Figure 17) (determined from the first two

solutions). A similar approach for the calculation of this solution was published by

Daniel piker for the calculation of the angles of the KUKA robots (Piker 2013).

The radius of circle with centre point A is the distance between the

second and third axis.

The radius of the circle with centre point B is the distance between the

third and fourth axis.

The solution results from intersection between the two circles. The

chosen solution is point C.

`


30

3.2.5 Angles calculation

Figure 18 Angle calculation digram

The angles for the joint position are calculatedaccording to the kinematic chain

determined from the solution toggles.

1. The first servo angle (base) has is the angle between the X direction

vector of the origin plane and the vector represented by the direction

between the origin and the projection of point A (the base rotation

vector) on the origin plane.

2. The second servo angle (Shoulder) is the angle between the

perpendicular vector to the base rotation vector and vector in the

direction A‐C.

3. The third axis (elbow) angle is the angle between the vector C‐A and C‐

B.

4. The forth angle (wrist 1) is the angle between vector B‐C and the vector

B‐D.

5. The Fifth angle (wrist 2) is the angle between the normal direction

vector of the tip plane and the base rotation vector

6. The sixth angle (wrist 3) is the angle between vector B‐D and the x axis

of the tip plane.


31

3.2.6 Execution path generation

The universal robots servos are capable of moving 720 degrees, from positive 360

degrees to negative 360 degrees for all its joints. This resulted in having 2 legitimate

joint solutions in any servo that can produce the same solution. For example if the

axis angle result from the inverse kinematic algorithm is 50° this means that ‐310° is

also a legitimate angle to active the same location. This increases the total solution

possible for any tip plane to 512 and the solutions that can achieve one of the eight

kinematic solutions to 64. This became a challenge when there is several inputs for

the inverse kinematic algorithm (when dealing with a continuous path for example),

the IK solver calculates the solution for the inputs individually only in the angle space

from 0° to positive 360°. In the initial tests, this resulted in some major errors in the

robot program when executed that may occur randomly when moving from one

configuration to the next. This might have catastrophic results duo collision for the

robot with the surround environment or the model it is working on. The solution for

this was building a sorting algorithm that allows the grouping of the closest solutions

when dealing with a continuous program.

It was crucial for the program when dealing with paths to put the timeline and

previous positions into consideration. To achieve that the IK algorithm calculate the

First configuration for the first tip position (referred to as home position usually)

which the user can override manually using angle toggles (Figure 14), and accordingly

he selects from two sets of automatically generated angles list for the whole path for

each servo. One list starts with the angle that is the closest to zero and one starts

with the angle that is the furthest. For example The IK for a single servo generated

numbers like (100, 10, 330 and 20) for the points along a hypothetical path. If this list

is executed without sorting, the servo will rotate 320° form the second state to the

third state and back 310° degrees. After the sorting of the solutions, the two

generated lists are (100, 10, ‐30 and 20) or (‐260,‐350, ‐30 and ‐340). These two lists

have both evade this problem resulting in a much smaller changes between states.

The inverse kinematic solver and path calculations are all grouped into one

component in the Onix plugin, allowing easy conversion from a set of planes to a set


32

of corresponding configurations in one step. An error system was also added to the

component to stop program from execution if a problem with angle lists is still

present in one of the lists or if the Inver kinematics fails to produce a viable solution

for the configuration.

3.2.7 Robot program and program uploader

Figure 19 Move, I/O and program uploader components in Onix

After the calculation of the set of joint locations that represent the path. The

“Move” component is responsible of converting these values to the URscript

language. The URscript is a set of instruction that the universal robots can execute

(universal robots n.d.). For each of the joint positions, the robot required one line of

code starting with either “movej” or “movel” followed by the eight numbers , six of

them refer to the joint positions of the servos then either the speed and velocity, or

time and rounding. The difference between the two is related to the intermediate

movements between states. The movej allow the movement to happen in the joint

space while the “movel” required the movement to be calculated according to the

tip position to maintain a linear movement of the tip.

The set of moves are then combined in the uploading component. The upload

component is responsible for the conversion of the lines of “Move” commands along

with the I/O sates to a working program. The program is then uploaded via TCP/IP

connection to the robot and is executed instantly.


33

3.3 The end-effectors

End‐effectors are an important part of any robot system. They define the rule

and use of the robot. In the beginning of the research, the end effectors were

designed to test different stages of the process which will requires either using multi

robots or changing the tools for each task manually. In the end a single system was

designed to transfer from one task to the other seamlessly. The three major tasks

needed to be covered by the end‐effector were: brick pick and place, laying mortar

and sensing.

Figure 20 Components of the gripper end‐effector

The end effector was designed around an industrial pneumatic gripper. The

chosen gripper had 3 cm stock and gripping force of 600 newton while keeping a

weight of 1 kg. Adjustable holding fingers was then machined and attached to the

gripper. The pneumatic air flow to the gripper was controlled by a 24 volt five way

solenoid valve. The valve allowed direct control over it from the robots control box

through the robot program itself. This was crucial to achieve synchronized robot

movements and end effector actions


34

Figure 21 right: End‐effector attached to the robotic arm. Left: The trowel attached to the handle

Achieving the second task required using a traditional trowel. A special handle

was created in the same width as the bricks in use (100 mm) to allow the gripper to

pick the trowel and use it. The force of the pneumatic gripper in addition to using

rubber eliminated the problem of the trowel slipping while in use.

The sensing was covered by a depth camera a limit switch. Due to wiring, these

two needed to be attached directly to the end effector. The end‐effector design was

tested virtually to achieve the best configuration that would decrees the incidents of

collision and also keep the camera and the limit switch out of the mortar. The camera

was controlled by the server computer, while the limit switch was controlled directly

by the robot control box, connected in this case to a digital input.

3.4 Robot execution paths

Onix allowed an easy way to create, edit and test the execution paths of the

robots. Tasks needed to fulfil the brick laying process were divided in relatively small

job specific paths for each task. Each of these execution paths are constructed in a

parametric manner, changing and updating according to its inputs. Some of these

inputs were determined once in the first calibration phase, while other change each

time they are executed.


35

3.4.1 Pick and place of bricks.

Figure 22 Pick and place sequence

A simple path was generated to pick one block from one location to the other.

The program has three inputs which are the brick stack location, brick number in the

stack and the placing location and rotation. The brick stack location is defined once

while the brick stack number and placing location change according to the order of

the program.

It was noted that during the placing the bricks on top of the mortar layer, the

forces required to place the brick in its location increase dramatically if there is excess

amount of mortar under. This is due the fact that the robot tries to push lots of

material out and is faced with the opposite force that and results in the emergency

stop of the robot. This led to the failure of the placement of the brick and the need

of re‐initialization of the robot. After some tests the movement was altered to have

rotational and side movements while placing the brick. This allowed the material to

move and level easily and decreased the perpendicular forces facing the movement

of the brick before.

3.4.2 Accompanying tasks.

A set of simple paths were designed to allow the shift between tools. First path

is the one which picks and leaves the trowel form a known location. The second path

was moving the camera to a perpendicular position to the mortar location. The


36

location of the mortar working board is also fixed and given as a constant. The third

program is picking the mortar. The input for this program is the point determined by

the depth camera, indicating how much the trowel need to move to get the right

amount of mortar.

Figure 23 Gripper holding the trowel for mortar operation

3.4.3 Laying mortar

Laying mortar is the most interesting part of the process. After the gripper pick the

trowel, the end‐effector is adjusted in grasshopper and the working plane is shifted

from the middle of the gripper to the middle of the trowel surface. The degree of

success of the laying moment was determent by two factors; First, the ability to move

all the mortar from the trowel to the brick surface. Second is the creation of a

consistent layer of mortar on top of the brick.

A professional bricklayer usually uses one continues movement to lay the

mortar on the brick. In order to mimic that with the robots, the tool needed to move

very quickly to achieve a good result in one movement. After observing bricklayers

do their work, it was noted that there are numerous factors which the movements

of the trowel is adjusted accordingly. During the first tests the teaching option of the

universal robots was used. In this mode the robots can be moved very easily by hand

while streaming the joint positions through the TCP/IP connection. In grasshopper

the movement are then recorded in real time creating the path. This made it possible

to try to mimic the basic movement that the bricklayer does when using his own tool.


37

Figure 24 laying mortar execution path sequence

Figure 25 robot executing mortar laying

In order to achieve a more universally successful results, the laying process was

broken into smaller movements that were tested individually with different mortar

consistencies. First was the movement to place the mortar on the surface of the

brick. The location of the placing of the mortar was very important to achieve a

consistent layer latter on. The next movements to be tested was the set of

movements that ensure that the all material left the trowel. The trowel paused and

moved up and down a number of times to ensure all the material left it and then the

trowel moves linearly to achieve a consistent layer of the mortar on the surface and

remove any excess material. The mortar program only requires one input which is

the location of the brick surface it is supposed to lay mortar on top.


38

Figure 26 robot removing excess material using the trowel

After applying the mortar and laying the next set of Bricks. The robot picks the

trowel again and remove the excess material resulting from pushing the mortar out

by the bricks. The process is one continouse movement from one side of the brick

row to the other side. The input of this path is the start and end location of the brick

row and the row hight.

3.5 Brick laying server

Figure 27 the bricklaying server in operation.

In the beginning ONIX was used on its own to control the robot. The major

challenges were the limitation of grasshopper as real‐time controller. Also the real

time visualization of the robot was integrated in grasshopper and allowed some

features like path recording, problems with speed and inability to work with

computer vision came between using grasshopper alone as the brick server. The


39

major problem with grasshopper by far was the way it is built in the first place.

Grasshopper is very linear in the way it executes its programs. Even with direct

programing it still faced lots of problems dealing with things like loops and retaining

variable data.

This required building another program that can be the server controlling the

bricklying process. The program was built using processing programing language

version 2.0 and was used to control the second major aspect of the research which

is keeping track of the major plan and fixing general errors. It was also in control of

the computer vision part of the process.

Figure 28 Operation sequence of the bricklaying program

3.5.1 Program operation sequence

The processing server works as the controller of the operation. It collects all the

data and control the execution of the robot paths. It allowed the shift between

different programs and tools. After the user selects the operations he needs to be

executed, the server initiates a logical sequence of operations to be processed.

Execution commands are then sent one by one to be executed in the Onix plugin. The

server always keeps track of the execution state of the program using feedback from

the robot, and sends the next program as soon as the job is completed. It shows visual

feedback in the interface about the progress of the work, and also allows the user to

overwrite the program in the case of errors.


40

In order to achieve that, the program is built on a set of If statements each one

representing a task. The program won’t go in the execution of one until it ensures

the one before is executed. Each statement also keeps track of the locations of the

tasks allowing the server to go from one location to the next. The server also

automatically compensate for errors.

Figure 29 Detailed operation sequence of the bricklaying program

3.5.2 Interface

The simple user interface of the program is built using the ContolP5 (Schlegel

2012) plugin for processing. The user interface is used to input the basic information

like the width and height of the brick, the number of brick rows and columns of bricks,

the spacing between the rows and columns of the bricks and the brick stack amount

for both full size and half bricks. The interface also is used to choose the programs

tasks needed to be executed by the robot, like brick pick and place, laying mortar,

pausing the program and emergency stop. The interface helps monitor progress and

errors and accept edition during execution.

Figure 30 Bricklaying server interface


41

3.5.3 Computer vision and sensing

Mortar in the right consistency is very close to liquid but still solid enough to

keep its shape. This makes mortar a very unpredictable material specially when

dealing with using low tech tools. When adding or subtracting from the material, it

will not level and would keep its shape and change slowly with time. The challenge

was getting the right amount of mortar on the trowel each time. This required having

a way to determine the arbitrary surface of the mortar each time before digging in

with the trowel.

Figure 31 Intel procedural computing camera

Depth cameras came as a solution for this problem. The focus was to have all

the tools needed on the robot itself without having a general vision system that

overlooks the whole process. This made the Kinect camera unsuitable for the task as

it is bulky in comparison to the size of the universal robots. Nevertheless, the new

Intel Perceptual computing camera prototype many advantage. It is much smaller

which made it possible to attach easily to the rest of the end‐effector and is unlike

the Kinect camera it was able to work on a range from 0 to 1.5 meters. This made it

possible for the robot to move and peek at the objects at close range getting a higher

density point cloud of the scanned object.


42

Figure 32 Intel camera point cloud used for surface approximation

The point cloud you get from the camera suffers from the same noise problem

as the Kinect camera. But because it can work at a very close range meant that the

noise reduction would have less effect on the accuracy of the scanned surface. For

the noise reduction, the scanning algorithm would store the point cloud data five

times in a frame of a second and then the average of each of its point location is used

to resemble the surface.

The surface if the mortar is then determent and the volume is calculated. Each

point of the point cloud is considered a rectangular box with fixed width and depth

and variable height. The height data are calibrated according to 4 points that

represent the base working surface. These points are determent on the first run of

the program. The algorithm then moves on the surface of the mortar from right to

left row by row determining how far the towel should go until it have the amount of

mortar needed for the application.


43

Figure 33 Limit switch testing brick hieght

In order to check the height of the placement of the bricks for errors, the first

plan was to use the depth camera to check the surface of the brick after laying it on

top of the mortar surface. Due to the noisy data of the camera this was deemed

unreliable. Instead a limit switch was added to the end effector. The limit switch was

connected to the digital input of the robot control box. During operation the robot

would move to a point which would make the limit switch tip 5mm away from the

supposed surface of the brick. Then it will move at very slow speed until the limit

switch is pressed and the location of the new point is recorded at the server program.

3.5.4 TCP/IP Feedback.

The universal robots are transmitting their state continually through their

TCP/IP connection. In initial tests it was difficult to decrypt the message that was sent

back form the robot due to the lack of support from the manufactures. Instead the

server relied on the python‐urx plugin (Roulet n.d.). The plugin was used to read the

message back from the robot and decrypt it. The message was then sent to both the

processing server and the onix plugin via the User Datagram Protocol (UDP)

connection.

For the processing server the message included the state of the program

running. This allowed the server to send the next command to the Onix plugin as

soon as the previous task was executed. The location of the tip was also sent to the

server in order to determine the height of the point acquired after the limit switch is


44

pressed. This allowed to check the deviation of the bricks surface form the original

design. For the Onix plugin however the message included used the joint locations.

This allowed the plugin to show the robot virtual representation in real tam while

excursing the program while also record movements if needed.

3.5.5 Bricklaying server to Onix communication

The communication between the two programs was achieved using UDP

connection. Using the protocol the processing server sends an execution command

to grasshopper starting with a specific identifier. This ID allowed the program to be

distributed to its corresponding robot parametric path in onyx. The message also

included the information needed of the parametric grasshopper definition. For

example the message sent for a pick and place job would be “B%10,50,0%‐

300,0%3,True” In grasshopper the message will be broken intro

Program ID : B (Brick pick and place)

Location of the brick to be placed is ( 10 mm , 50 mm , 0 mm ) from

origin

Brick Stack location ( ‐300mm , 0 mm ) from origin

Number of the brick in stack is 3

The numbers are then dispatched to it corresponding parametric input and the

program is altered and sent to the robot to execute.

Figure 34 right: Sequence of communication between the bricklaying server and Onix for a single operation left: program dispatcher in Grasshopper

3.6 The building tests

The First bricklaying experiment was conducted to test the mortar laying

movements. The aim was to build two rows of bricks with the least excess material.


45

It was repeated a number of times with alternations to the program until the desired

result was achieved. The flow of the program was also tested. The server program

shifted from the tasks of brick laying with the gripper directly to gripping the trowel

and changing the program to deal with mortar and back to laying bricks and removing

the excess amount of mortar on the side after.

Figure 35 first test to check mortar laying movements and server operation

The second test was conducted to build a higher wall. This allowed the testing

of the sensor data. The location of the brick wall needed to be shifted from the table

surface to the level of the ground to allow the robot to build higher and safer. One

of the first problems was the adaptation of the movements that is built in

grasshopper to the new position. The movements were built initially to work in the

space in front of the robot and relative to the fixed world coordinates. This created

a problem that was generally visible when dealing with trowel. The programs were

then rebuilt for this task to work according to the location relative to the robot. This

allowed the program to be executed in any location around the robot easily without

the need to alter the program. The test also was done using bricks and half bricks.

The bricks were cut manually and stacked in a secondary location. The server shifted

between the locations of the half bricks and the full ones according to the need. The

rows were built one by one and the height was tested for each row before laying the

new mortar layer.


46

Figure 36 Second test to check error handling and feedback

The Third test was conducted to test the ability to use the program to build

different design formations. This time the rotation angles for each brick changed in

order to create the desired pattern both for the brick pick and place program and

mortar laying. Another challenge accrued due to the use of the gripper. The new

formation of the bricks meant that the gripper will collide with the previous brick

when exciting pick and place. This required altering the picking location of the bricks

according to the rotation.

Figure 37 third test to check program adaptation to different rotations

Discussion and results Robotic Bricklaying

47

4 DISCUSSION AND RESULTS

4.1 Testing Results

The experimentation for the program started with testing of the success of task

specific operations. Tasks like pick and place and camera movement were straight

forward and high success rate was achieved form initial tests.in the during the test

of laying mortar on the other hand, trying to replicate the bricklayer movements

directly were unsatisfying and had low success rate. This was due to the inconsistency

of the mortar material itself. When a bricklayer usually lays the mortar with the

trowel it is easy for him to determine the consistency of the mortar and adjusts the

movement accordingly, most of the time subconsciously. The robot on the other

hand at this point has simpler inputs to its program and is unable to determine the

consistency of the material it is holding. Several of other challenges occurred in the

creation of the mortar laying path. When the trowel is left for some time while the

robot is laying bricks for example the mortar layer on the trowel starts to harden as

it loses moisture. This makes the trowel lose its smooth surface which makes it harder

for the mortar to leave the trowel after picking it up.

The successful set of movements to lay mortar however was different from the

ones used by a brick layer. This time the process was much longer and included

several movements that might not be needed at all times, but is needed to achieve a

global success rate with different scenarios. After several tests the brick laying

movement had a high success rate to achieve a consist layer of mortar between

bricks. The Onix plugin managed to be a successful tool to produce and control these

paths, test them and stream them to the robot with the robots.

With regards to computer vision, the depth camera was successful to

determine the arbitrary surface of the mortar. The program managed to determine

depth needed for the trowel to go into the mortar to get the desired amount. During

the tests the process required a large amount of mortar to be placed on the working

board. When dealing with a small amount, the mortar usually moved along with the

movement of the trowel, resulting in an unreliable outcome.


48

The next tests were conducted to test the ability of the bricklaying server to

control the process autonomously. The server relied on the data coming back from

the robots like the statues of the execution of the programme and the location of the

tip to alter and proceed with its sequence of operations. The tests showed that

building an adaptable program for the robots to perform a multi‐task craft as

bricklaying is achievable. The server program succeeded in the shift between

different tasks while keeping a record of execution progress.

The test also exposed how the error factor due to the complexity of the real

world can accumulate. During the test of the height of each of the layers of bricks

was shifted from the original virtual drawing. This was mostly due to the mortar

settling under pressure from the layers on top. The brick itself was a part of this issue,

even though the brick used were mechanically machined engineering bricks. There

were small differences in the dimension of the brick in comparison to its advertised

size. This resulted in up to three mm difference by the end the fourth row of bricks.

The server was able to keep track of the error and try to compensate it in the mortar

horizontal spacing.

The last test revealed the possibility to use the same process to produce

different brick designs. The parametric inputs of the execution paths allowed easy

manipulation of the bricks rotation angles and locations for each row while keeping

the automation of the program. The pick and place and the mortar laying movements

both adapted to the changes in design.

4.2 Critical assessment

For years the digital field was the major breakthrough arena in research and practice,

the move from digital drawing to the physical creation was left for other disciplines

without the control of the designer. However, there is more focus recently on

physical realisation of ideas. During the robotic brick laying research the emphasis

was on building a robotic program that can adapt and change. This nevertheless was

not a case of direct execution digital to physical. The reliance on an adapting program

meant that a dialogue can be maintained to some extent between the real

environment and the virtual one.


49

To achieve the goals of this study, three main aspects needed to be addressed:

the tool, the software that runs it and the task. In this case it was the robotic arm,

the controlling software and the craft of bricklaying. Because of the novelty of the

tool, a part of the research has been conducted to understand how it works, how to

communicate to it and how to control it. This nevertheless resulted in a new platform

that can be used by other users to easily control the robots in future research. The

two software platforms used in the process are still in their early stages and are still

in a constant development. The Onix grasshopper plugin can get more and more

research on board to work with the robots with its simple interface and fast learning

curve. It contained a set of easy tools that can be used directly to run the robots, or

can be altered for more advanced users. Although this platform is capable to work

on its own in the field of fabrication for example, it is still not able to work with

feedback in a way that is more significant than the visualization of real time robot

movement. This made the data flow in one direction from digital to physical and back

but without a way to readapt the program automatically. These drawbacks are

related to the grasshopper platform itself and how the programs is executed inside

it and also other problems with speed for example as grasshopper wasn’t built for

real‐time execution in the first place.

The brick laying server program is the other software developed for the

research .on the contrary was able to work with the feedback with great success. The

use of programing for this tool allowed the program to easily adapt, keeping record

of errors and building up data during the execution. In the same time signalling Onix

to execute its paths with ease. Working with coding allowed the designing of a task

specific interface that can keep the execution and data gathered easily visible for the

user. This puts focus on the importance of programing as a language for controlling

design. The server software is very far from being used with unspecialized architects

in a way that is more significant that dealing with the simple interface. Combining

the two platforms in a one working workflow benefited from both, allowing an easy

familiar way to build robotic paths while maintaining the more specialized adaptable

program.


50

Bricklaying as task was a major challenge, Even though several researches were

inducted dealing with bricks. The more interesting challenges emerged from using

mortar rather than the simple task of precise pick and place. Pick and place is a task

taken for granted in robotics for years. On the other side, working with mortar put

emphasis on the problem of dealing with a relatively unpredictable material. Adding

this to the equation of pick and place rendered the precision of it on its own less

efficient to predict the precision of the final result. Although the test was conducted

in a small scale, it showed how the errors can build but quickly which may lead to a

very unstable structure in the end if it wasn’t addressed during the execution.

Mortar also exposed the importance of intuition in such craft as brick laying.

While a brick layer might do one movement to lay mortar with high success. The

initial tests that were conducted to directly copy these movements to the robots

weren’t very satisfying. Achieving a good rate of success with the same tool required

tests and numerous adjustments. Quite a few of the moves in the executed path

might not be required all the time but is there to produce a more general success

rate. The performance of the program was measured by the ability of the robot to

achieve a mortar layer that cover all the surface without having a large amount of

excess material that can come in between laying the next brick. The success rate in

doing that was high in the last test but still was far from producing a very immaculate

end result as produced by a skilled worker. Another drawback was the inability of the

program to fix the errors directly, for example pushing a brick that is higher than the

rest of the row to level it up. This was due to the limitation of the payload of the

universal robots. The robots usually went into emergency stop very easily when

bushing against mortar. This kept the error correction only on a larger scale dealing

with the general execution of the program rather than on the scale of a single brick.

Conclusion Robotic Bricklaying

51

5 CONCLUSION

The research intended to shed the light on the potential of adaptive programing as a

mean to control robotics in the field of architecture. Using such method robots can

become more responsive to the task they are programed to execute. This potentially

can lead to better results working with messy materials and heterogeneous

environments. It can also enhance the design process with more integration of

material behaviour and help overcome the difficulties facing the introduction of

robotics in construction.

The choice of bricklaying as the field of research was successful. It had all the

potentials required to test the hypotheses. It is an important part of construction

industry, with a large space for innovation, at the same time it is a craft that heavily

dependent on human labour. Working on such craft allowed the testing of the ability

of the robots to work with multi material using different tools and the ability to shift

between them seamlessly. Mortar was the most interesting material during the

process. The movement executed by the bricklayer to produce a consistent layer of

mortar is highly based on skill, technique and tuition. Achieving high success rates

with robots needed several tests. The resulting process is still not as immaculate as

the work of a skilled bricklayer. However, the process achieved high success rate in

the end.

On the software side of the process, using such novel tool as the universal robot

resulted in the creation of the Onix plugin, a new plugin specially designed to control

universal robots directly from grasshopper. The potential of the plugin goes beyond

this research and hopefully will continue as potential controlling software for other

researches as well. The plugin allowed easy creation and edit of execution paths for

the robots. This was very useful during the testing and editing of such paths for the

tasks needed to fulfil the process, like brick pick and place and mortar laying. The

process needed another piece of software that is capable to achieve the adaptive

features of the process. The Brick laying server worked as the controller for the whole

process, keeping track of the progress of the program and also dealing with the

Conclusion Robotic Bricklaying

52

feedback from the robot, form the sensors and vision and also controlling the onix

plugin itself.

The small scale tests of the program succeeded in achieving its goals, however

the process is still very far from being used in a construction environment. Humans

still have vastly more sophisticated senses and capabilities and are still also the

economical choice. Nevertheless, it is a step closer to achieving such integration. The

field nevertheless still have great potential of further exploration for example in the

design side of the process, in the mobility of the robots or the integration of machine

learning.

The programing aspect of the research still has great space for further

exploration. The server program was only dealing with a part of the data available

for it. The end‐effector and the robot itself offer large amount of data that is still to

be utilized. Like live video feed from the Intel camera and torque data from the robot.

The increased amount of information available can result in a more sophisticated

program. And with more data, Finding patterns and making sense of this data can

sometimes be more than a human can handle. Research in machine learning can be

a logical next step for robotic adaptive programing. Producing a program that is not

only adaptable but can grow more complex on its own learning from its mistakes and

taking program beyond the designer’s scope.

Appendix Robotic Bricklaying

53

6 APPENDIX

6.1 Appendix I: bricklaying server program operation sequence

persuade code

Figure 38 operation sequence

User selects desired operations and bricks. Pick and place = True Mortar = True Check and adjust Height = True Read robot execution statues While (X < Number of operations)

{ If (communication established && Pick and place && Robot is Idle)

{ Send message “B + Brick location + stack number + True” to Onix X+=1 Brick +=1 If (Brick == column number)

{ Pick and place = False Brick = 0 }

} If (communication established && Mortar && Row > 1 && ! Pick and place && Robot is Idle)

{ Check Height sequence While (Brick < column)

{ Send “H+ Brick location” To Onix Brick + 1 Read Tip location Calculate difference Shift locations of bricks If (Brick == column number)

{ Brick = 0 Height checked = True Mortar = Ture }

} }

If (communication established && Mortar && ! Pick and place && Robot is Idle)


54

{ Pick trowel sequence Send message “T + True” to Onix Trowel picked = true Mortar = false }

If (communication established && Trowel picked && Row > 1 && ! Pick and place && Robot is Idle)

{ Remove excess mortar sequence Send “r+ row height” X+=1 Trowel picked = false Surface checked = true Excess removed = True }

While ( Brick < column) {

If (communication established && excess removed || Row=1 && && ! Pick and place && Robot is Idle)

{ Run Check motor surface sequence Send “C + True” to Onix Run depth calculation algorithm Trowel picked = false Surface checked = true }

If (communication established && surface checked && ! Pick and place && execution not in progress)

{ Pick mortar from result location Send “P+ result location from depth” to Onix Surface checked=false Mortar picked = true }

If (communication established && mortar picked && ! Pick and place && Robot is Idle)

{ Lay mortar from result location Send “m+ Brick location” Brick +=1 X+=1 Mortar picked = false If (Brick == column number)

{ Brick = 0 Mortar laid = True }

} If (communication established && mortar picked && ! Pick and place && Robot is Idle)

{


55

Lay mortar from result location Send “M+ Brick location” to Onix Brick +=1 X+=1 Mortar picked = false If (Brick == column number)

{ Brick = 0 Mortar laid = True }

}

} If (communication established && Mortar laid = && ! Pick and place && Robot is Idle)

{ Leave Trowel sequence Send “L” to Onix Mortar =false Pick and place = True Trowel left = false }

} If X = Number of operations

{ STOP PROGRAME }

}


56

6.2 Appendix II: Inverse Kinematic and uploading components

Figure 39 IK solver (Left Square) and uploading component (Right Square) location in the Onix plugin example file

Figure 40 IK solver component parts cluster

Figure 41 Point A calculation

Figure 42 Point B calculation


57

Figure 43 Point C calculation

Figure 44 Angle sorting algorithm cluster

Figure 45 Robot uploading component cluster


58

6.3 Appendix III : universal robots possible solutions via Onix IK solver

Figure 46 the eight possible solutions for a single tip plane using Onix IK solver

References Robotic Bricklaying

59

7 REFERENCES

Andres, J. et al., 1994. First Results of the Development of the Masonry Robot System ROCCO: a Fault Tolerant Assembly Tool. Automation and Robotics in Construction XI, 11, pp.87–93.

Balaguer, C. & Abderrahim, M., 2008. Trends in Robotics and Automation in Construction. , pp.1–22.

Bechthold, M., 2010. The Return of the Future: A Second Go at Robotic Construction. Architectural Design, 80(4), pp.116–121.

Braumann J. and S. Brell‐Cokcan., 2013. Rob|Arch 2012: Robotic Fabrication in Architecture, Art and Design, Vienna: Springer Vienna Architecture.

Braumann, J. & Brell‐Cokcan, S., 2011. Parametric Robot Control: Integrated CAD/CAM for Architectural Design. In ACADIA.

Campbell, J.W.P. & Pryce, W., 2003. Brick: A World History, London, UK: Thames & Hudson.

Colletti, M. et al., 2013. Robotic foaming cluster SG2013. Available at: http://smartgeometry.org/index.php?option=com_content&view=article&id=219:robotic‐foaming&catid=48:sg2012‐london [Accessed April 20, 2013].

Dubor, A. & Diaz, G.‐B., 2012. Magnetic Architecure. In Rob\Arch. p. 7.

G. Pritschow, M. Dalacker, J.Kurz, M.Gaenssle, J.H., 1994. Application specific realisation of a mobile robot for on‐site construction of masonry. ISARC Proceedings, 11, pp.95 –102.

Glynn, R. & Sheil, B., 2011. Fabricate: Making Digital Architecture, Riverside Architectural Press.

Gramazio, F. & Kohler, M., 2008. Digital Materiality in Architecture: Gramazio & Kohler, Boden: Lars Muller Publishers.

Helm, V. et al., 2012. Mobile robotic fabrication on construction sites: DimRob. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp. 4335–4341.

Hill, J., 2005. Building the Drawing. Architectural Design, 75(4), pp.13–21.

international fedration of robotics, 2012. History of Industrial Robots, From the first installation until today, Frankfurt. Available at: http://www.ifr.org/uploads/media/History_of_Industrial_Robots_online_brochure_by_IFR_2012.pdf.

Keating, S. & Oxman, N., 2013. Compound fabrication: A multi‐functional robotic platform for digital design and fabrication. Robotics and Computer‐Integrated Manufacturing, 29(6), pp.439–448.

References Robotic Bricklaying

60

Kolarevic, B., 2005. Architecture in the Digital Age: Design and Manufacturing, Taylor & Francis.

Oxford dictionaries, robot: definition of robot in Oxford dictionary (British & World English). Available at: http://oxforddictionaries.com/definition/english/robot [Accessed August 28, 2013].

Piker, D., 2013. Lobster grasshopper group. Available at: http://www.grasshopper3d.com/group/lobster [Accessed April 1, 2013].

Roulet, O., python‐urx. Available at: https://github.com/oroulet/python‐urx [Accessed August 1, 2013].

Schlegel, A., 2012. ControlP5 plugin. Available at: http://www.sojamo.de/libraries/controlP5/ [Accessed April 1, 2013].

Spong, M.W., Hutchinson, S. & Vidyasagar, M., 2005. Robot Modeling and Control, John Wiley & Sons.

universal robots, The URScript Programming Language. , p.30. Available at: http://www.wmv‐robotics.de/home_htm_files/scriptmanual_en_1.5.pdf [Accessed June 1, 2013].

robotic bricklaying, towards adaptive robotic programing

Documents

robotic bricklaying

robotic arms

adaptive programing

ive arch i tecture

computat ion bart

col lege london

industrial robots