Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

Research Article

An autonomous mobile manipulator to build outdoor structures consisting of heterogeneous brick patterns

Meysam Basiri1  · João Gonçalves1 · José Rosa1 · Alberto Vale2 · Pedro Lima1

Received: 23 September 2020 / Accepted: 12 March 2021 / Published online: 14 April 2021 © The Author(s) 2021 OPEN

AbstractThis paper describes the implementation of an autonomous mobile manipulator to build outdoor structures consisting of heterogeneous brick patterns, finding applications in different industrial automation, manufacturing and civil construc-tion scenarios. This system was developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020, to demonstrate the novel real-world application of constructing structures using teams of robots, showcasing once again the role of major scientific competitions in advancing the state of the art towards exploring solutions to open problems. The paper presents in detail the hardware and software architectures of the developed mobile manipulator, integrat-ing different research results and developments into a functional complex robot system, while proposing methods to detect, approach and manipulate differently sized/colored bricks to build a wall of predefined pattern given to the robot just before the building task starts.

* Meysam Basiri, meysam.basiri@tecnico.ulisboa.pt; João Gonçalves, joao.goncalves@tecnico.ulisboa.pt; José Rosa, jose.rosa@tecnico.ulisboa.pt; Alberto Vale, avale@ipfn.tecnico.ulisboa.pt; Pedro Lima, pedro.lima@tecnico.ulisboa.pt | 1Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais 1, 1049-001 Lisboa, Portugal. 2Instituto de Plasmas e Fusão Nuclear, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais 1, 1049-001 Lisboa, Portugal.

Article Highlights

– Full system description of an autonomous mobile manipulator for construction tasks tested in a realistic setting– Algorithms for detection, localization, picking and placement of heterogeneous building blocks to form large structures– Versatile service robot capable of smooth adaptation to other functions developed through scientific robot competitions

Keywords Autonomous mobile manipulator · Unmanned ground vehicle · Brick-shaped object detection · MBZIRC competition · Outdoor object perception and manipulation · Industrial automation and construction

1 Introduction

Scientific robot competitions provide a common framework for the rigorous comparison of intelligent and autonomous systems in real world conditions [1]. Such competitions play the role of scientific experiments while being appealing to both researchers and to the general public. Currently several major grand challenges exist in robotics around the world

e.g., the Mohamed Bin Zayed International Robotics Chal-lenge (MBZIRC), DARPA Grand Challenge, CYBATHLON or EuRoC challenges, where the aim is to push the state of the art toward the solution of open problems [2]. Robot compe-titions such as RoboCup or the European Robotics League encourage different sub-disciplines of AI and robotics to be integrated into complete robot systems to solve complex tasks in realistic environments, benchmarking the solutions

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

against a common problem [1, 3]. This paper describes our efforts towards the development of an autonomous mobile manipulator for the second challenge of the MBZIRC 2020 robot competition.1

A mobile manipulator is composed of a robotic manipu-lator mounted on a mobile base, offering both mobility and dexterity. Many different professional and consumer service applications have been envisioned for mobile manipulators, mainly to perform manufacturing assistance in industrial environments [4, 5], to execute domestic ser-vice tasks [3, 6] and human assistance [7, 8], to transport goods inside warehouses and stores [9, 10] and to manipu-late and transport objects in hazardous environments such as in search and rescue missions [11, 12] or in areas with radioactive or toxic debris [13]. There has been a growing interest in the research community in developing autono-mous mobile manipulators, ranging from wheeled-mobile robots [14, 15] to humanoids [16], legged robots [17] and even aerial robots [18]. To encourage research and devel-opments on mobile manipulators for challenging real-world problems, multiple grand robotic challenges such as DARPA [19] and MBZIRC [20] competitions have defined complex manipulation scenarios requiring robots to per-form a series of challenging tasks combining perception, localization, navigation, manipulation and even multi-robot cooperation. In [21] a custom mobile manipulator for operating a valve stem was developed under the MBZIRC 2017 challenge. The same competition encouraged devel-opment of innovative airborne robot solutions to locate, pick and place small-sized objects from the ground [22, 23]. In [24] a mobile manipulation robot was developed for the DARPA grand challenge to execute several challenging manipulation tasks in a disaster environment.

Employment of robots for civil construction can lead to significant cost and time savings and to satisfy the ris-ing demand for workforce and safety regulations. Robot fabrication is commonly referred to as stationary industrial robots that operate under fixed conditions with a limited

working area [25]. However, such approaches result in development of large stationary robots with specific and pre-determined roles while constraining the work-piece size that they can act upon. The use of versatile multi-purpose mobile manipulators can allow cooperative operation directly on construction sites to build complex structures with small building components. This has led to innovative efforts to develop multi-purpose ground mobile manipulators [26–28] and aerial robots [29, 30] for autonomous construction of structures. Aiming to push the boundaries of research on automated construction and robot-based 3D printing of civil structures, the MBZIRC 2020 competition hosted an innovative challenge where a team of ground and aerial robots were expected to construct a structure composed of heterogeneous bricks (Figs. 1, 2).

This system paper describes the development of a ver-satile mobile manipulator for the MBZIRC 2020 competi-tion with the purpose of operating in a semi-organized outdoor environment to locate, pick and place a set of

Fig. 1 Snapshot from the MBZIRC 2020 competition trial with the goal of constructing a wall composed of heterogeneous bricks, illustrating the bricks and the proposed mobile manipulator

1.8m

4m

0.4m

90o1.2m

0.3m

0.6m

4m

Fig. 2 Top view diagram illustrating the shape and dimensions of the wall to be built by the UGV in the second challenge of the MBZIRC 2020 competition along with the four different brick types available for the construction1 http:// mbzirc. com/ chall enge/ 2020.

Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7 Research Article

differently sized brick-shaped objects to construct a wall of predefined pattern. It describes integration of differ-ent research results and developments into a functional complex robot system for execution of a novel real-world application beyond what has been seen before. Our con-tribution includes a complete software pipeline for the full brick picking and placement operation including naviga-tion, brick detection, brick alignment, brick picking and brick placing. The integrated system and the function-alities developed for this scenario can directly benefit other purposes and applications such as logistic tasks in an automated warehouse. We point out that the versatil-ity of the system allowed the robot to be easily adapted and employed for another challenge of the MBZIRC 2020 competition, to locate and extinguish fires inside a build-ing, and was awarded as the winner of this challenge. Fur-thermore, the robot solution is currently being used in a project for autonomous inspection of solar panels under the DURABLE project.2

The outline of this paper is as follows: Sect. 2 provides the detailed description of the second challenge of the MBZIRC 2020 competition. Section 3 presents our pro-posed approach to the challenge by describing the hard-ware and software components and the main functionali-ties of the developed mobile manipulator. Functionalities such as robot localization and navigation, brick detection and alignment, vision-based brick picking and placing are described in this section. Section 4 describes the evalu-ation of the entire system as a whole in performing an autonomous brick picking and placing operation. Finally, Sect. 5 concludes the paper by providing the conclusions and future work.

2 Challenge description

In the second challenge of the MBZIRC 2020 competi-tion, a team of unmanned aerial vehicles (UAVs) and an unmanned ground vehicle (UGV) are expected to work together in an outdoor environment to autonomously locate, pick, transport and assemble a set of brick-shaped objects to construct pre-defined structures composed of multiple walls. Two separate walls are expected to be built by the robots, an L shaped wall that must be assembled completely by the UGV and a wall with U shaped channels to be completed by the UAVs. This paper focuses on the assignment defined for the UGV and describes our devel-oped solution for this purpose.

Four types of bricks with equal cross-section of (0.2m x 0.2m) but different lengths and weights are available in the arena:

– Red 0.3m (1.0kg)– Green 0.6m (1.0kg)– Blue 1.2m (1.5kg)– Orange 1.8m (2.0kg).

A small ( ∼ 0.3m length) ferromagnetic plate is attached on top center of all bricks allowing robots to grab the bricks using a magnetic gripper. This plate is made of a steel plate with 0.6mm thickness. The color of the ferro-magnetic plates is white and different from the base color of the brick surfaces.

The arena size for the challenge is approximately 50m × 60m. Initially bricks are stacked according to their colors in different piles. The robots are required to locate, approach and pick the bricks from the piles and to deliver and place them on top of a wall base to construct a multi-layer wall. Each layer of the wall is composed of a randomly gener-ated pattern of bricks that is communicated to the robots at the beginning of a trial. The location of the wall base is unknown, and the robots need to locate it during the trial. The maximum duration of a trial is 30 minutes. More details about the rules and procedure for the challenge can be obtained from the MBZIRC 2020 website.

3 Proposed method

To address the challenge described in Sect. 2, a custom mobile manipulator was developed through refurbishing an old four-wheeled differential drive mobile base (iRobot ATRV-Jr) and integrating it with a six degrees of freedom robotic arm (Universal robotics UR5e), a magnetic gripper and a set of sensors (detailed in Sect. 3.1). The developed mobile manipulator is used alongside a team of three UAVs to work towards completing the overall task.

The proposed method for solving this challenge is described as follows. Once the input wall pattern is com-municated to the team by the operator, the mission starts, and a UAV takes off to perform a quick search to find the approximate location of the brick piles and the wall base from the air. This is achieved simply by performing a sweep of the arena by the UAV and detecting the piles and the base through color thresholding, similarly to the method used by the mobile manipulator, described in Sect. 3.5. Upon identifying this information, the UAVs will then com-municate a waypoint near every brick pile and the wall base. The manipulator then moves to the waypoint of the desired brick pile based on the input wall pattern to pick up the first brick. This is achieved through a GPS-based 2 https:// www. durab lepro ject. eu.

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

localization system and a waypoint navigation algorithm (described in Sect. 3.3) that includes obstacle detection and avoidance based on a laser scanner sensor.

Once the robot is in the vicinity of a desired pile, a laser-based brick detection and alignment method is then employed (described in Sect. 3.4) allowing the robot to move precisely towards the center of the desired brick that is to be picked. A vision based algorithm (described in Sect. 3.5) that uses a color-depth camera attached near the end-effector of the robotic manipulator is then employed to confirm the desired brick color and to estimate the brick pose (position and orientation) as well as the center point of the brick. The robotic manipulator is then moved to have the magnetic gripper approach the center point of the brick with a force feedback to identify when the brick is reached. The motion of the robot base is independent of the manipulator and only after the base is aligned with the brick, the manipulator is then employed to pick up the brick.

Once the brick is attached to the gripper it is picked up by the manipulator and the waypoint navigation algo-rithm is employed once again to drive the mobile robot towards the wall base where the brick is to be placed. When the robot is in the vicinity of the wall, the laser rangefinder and the vision-based detection are employed (described in Sect. 3.6) to precisely adjust the position of the mobile manipulator and the end-effector to position the brick to its desired location. The brick is then released by deactivating the magnetic gripper and the robot moves towards the next brick in the pile.

3.1 Hardware description

The developed UGV is a mobile manipulator composed of two main parts: a mobile base and a UR5e robotic arm with six degrees of freedom (as shown in Fig. 3).

The mobile base was developed through refurbishment of an old iRobot ATRV-Jr robot with four wheels connected to 2 high torque, 24V DC servo motors. This robot was used in the past in the Portuguese national RESCUE project, [31], to foster research on multi-agent robotic systems for search and rescue operations, and later in the INFRANET project [32] to test a robotic solution for power line inspec-tion. The ATRV-Jr mobile base has 55 cm of height, 78 cm of length, weights about 50 Kg, and is capable of trans-porting an additional payload of 25 Kilograms at a maxi-mum speed of 1.4 m/sec.

The ATRV-Jr robot was fully refurbished and upgraded to be used in the challenge. Besides replacing many of the old power circuits and control boards, the UR5e con-trol boards were accommodated inside the mobile robot along with an Intel NUC7i7BNH to act as the new on-board computer. A NovAtel OEM2-G2L GPS antenna and receiver, a Crossbow DMU-6X-003 IMU positioned at the center of the robot and an MPU-6050 IMU positioned at the top of the robot were installed for the purpose of robot localiza-tion and navigation. A Hokuyo UTM-30LX laser scanner was also installed at the front of the robot, which provides two-dimensional laser scans from 0.1 to 30m within a field of view of 180◦.

The UR5e arm was assembled at the top front of the robot with all the control circuits fitted inside the mobile base. For this purpose, the base structure of the robot was reinforced to accommodate the fixation of the arm. Addi-tional sensors and actuators such as a RealSense D435, a

Fig. 3 The mobile manipulator developed from a refurbished ATRV-Jr mobile base and a UR5e robotic arm with a six degrees of freedom used for solving the second challenge of the MBZIRC 2020 competition

Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7 Research Article

Terabee thermal sensor and a HEM1080 magnetic grip-per were also added to the robot. The Intel RealSense D435 camera that comprises of a full-HD RGB and depth sensor is installed on the manipulator and near the end-effector allowing the environment to be actively perceived through the movement of the manipulator. The thermal camera placed at the front of the robot is only used to perform heat and fire detection for the third challenge of the MBZIRC competition and is not used in this challenge. The whole robot system can be powered either using two LifePO4 24V 20Ah batteries or using two Pb 12V 35Ah batteries providing an autonomy of more than 5 hours of continuous operation.

3.2 Software architecture and task events

The Robot Operating System (ROS) was used as a com-mon framework for integrating different software com-ponents and interacting with the robot hardware. The UGV operates by autonomously switching between a set of predefined states that are triggered autonomously by sensor inputs and the current status of the mission. A state-machine was built for this purpose using the SMACH library [33] allowing the definition of states and state tran-sitions and executing the mission plan. Each state uses simple actions that perform a basic operation with the aim of generating the complex global behaviour. Each independent action was implemented as a Python func-tion that requests operations from the running rosnodes through rosservice calls and rostopic messages. The states of the robot can be divided into four main states:

– Waypoint navigation;– Laser-based alignment;– Brick picking;– Brick placement.

Figure 4 describes a simplified diagram of the pick and place procedure and the order of the state transitions for the main objective of building a wall from a set of dif-ferent shaped bricks stacked in piles. The algorithm and sequence of expected events when executing the com-plete challenge is described in Algorithm 1. Details about each state and the corresponding actions are provided in the following sections.

3.3 Robot localization and waypoint navigation

A main requirement for a robot competing in the chal-lenge is to be able to navigate reliably in the environment. For example, to reach a waypoint in the proximity of a brick pile communicated by the UAVs or to navigate to a predefined waypoint provided by the user. To obtain an

accurate robot navigation functionality, two main system components were implemented:

– A robot localization system which estimates the posi-tion and orientation of the mobile base with respect to the arena frame.

– A waypoint path planner, which is in charge of comput-ing feasible paths for the robot to reach specific points in the environment.

The localization system is described in Fig. 5. Due to the outdoor and open-field nature of the problem, the localization system is developed to use the Odometry information from the wheel encoders along with the GPS and IMU measurements to estimate the pose of the robot. This is achieved using an Extended Kalman Filter (EKF) [34] that fuses the three sources of information together and outputs a reliable estimate of the robot’s pose. Note that the laser scanner is not employed for robot localization due to availability of GPS signals and since laser meas-urements only correspond to dynamic structures, i.e. the bricks. However, the laser scans are used for robot naviga-tion and obstacle avoidance as described later in this sec-tion. The use of advanced algorithms for highly dynamic environments [35, 36] could potentially be used to also integrate the laser scans in the localization system and further enhance the reliability of the localization system

The measurements are used as standard ROS mes-sages with associated covariances, enabling the filter to weight the information from the sensors according to its expected accuracy and to obtain a more precise pose estimation. The localization system allows the robot to estimate its pose at every time instance with respect to a

Fig. 4 Simplified diagram of the pick and place approach and the corresponding robot states for the scenario of building a wall from a pile of bricks

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

fixed reference frame and to compute its relative pose to other fixed frames in the arena (for example the starting pose, the wall and the piles of bricks). Figure 6 shows the localization results from an experiment performed in the Instituto Superior Técnico Lisbon campus, where the robot was driven on a path of several hundred meters consisting of a set of black lines used as the ground truth.

For the waypoint navigation, a global planner and a local planner are defined that work together to calculate the best path from the robot’s current location to the desired waypoint in the arena. The ROS navigation stack [37] is used for this purpose which benefits from cost-maps to represent obstacles and to build an occupancy grid representation of the arena. For the Global Planner, which is based on the Dijkstra’s Algorithm [38], a modified version of the ROS Navfn planner [39] is used to compute

the global path. However, the planner is allowed to create plans that traverse unknown spaces in the global costmap. This is because the map of the environment is not avail-able in advance and it is not possible to produce a map before the mission as the layout of the arena is expected to be frequently changing. The known space is considered as the area that can be perceived by the laser scanner.

Obstacles are represented in the costmap as occu-pied cells. In their vicinity, the cells are assigned with a value representing the probability of that cell also being occupied (as shown in Fig. 7). This probability decreases exponentially with the distance to the obstacle allowing the planner to compute a smooth path while avoiding collisions between the occupied cells and the cells inside the robot’s footprint. i.e. a cost of zero is assigned to free spaces, a cost of one is assigned to occupied cells and cells leading to collision, and a user specified exponential decay function is used to connect the free space and occupied cells

A Dynamic Window Approach (DWA) Local Planner [40] is used along with a local costmap of 10m × 10m to compute circular arc motions (expressed by their linear and angular velocity combinations) towards the local goal pose, taking into consideration the kinematic characteris-tics of the robot and the obstacles in the scene. By scoring each of the options, the best motion for the robot at each step is then selected. This is regularly repeated throughout the global path provided by the Global Planner until the final goal is reached.

When a path is computed, only the visible obstacles are taken into account. However, in case a new obstacle is suddenly detected by the laser scanner, the costmap is

Algorithm 1: Challenge 2 routine

Result:

Algorithm 1: Challenge 2 routine

construction of a wall with a predefined brick patterninput: wall patternbegin

translate the input wall pattern into a sequence of bricks S;wait to receive the wall and brick-pile waypoints from the UAVs;forall bricks s ∈ S do

navigate to the waypoint of the pile s;align the robot with the pile using laser sensor;detect brick s using vision;pickup brick s;navigate to the waypoint of the wall;align the robot with the wall;if s is the first brick of the wall then

detect wall base using vision;place the brick relative to the wall base;

elsevisually detect the reference brick on the wall for which the new brick isgoing to be adjacent to it;

place the brick relative to the reference brick;end

endend

Fig. 5 Diagram illustrating the EKF-based sensor fusion technique. The filter is composed of two nodes. The EKF node uses Odometry and IMU type ROS messages and fuses them together in a custom-ized way (for example, based on the weight defined for each meas-urement). The NAV-SAT-TRANSF node transforms the GPS data mes-sage into an Odometry type message that is consistent with the robot’s world frame

Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7 Research Article

instantly updated and the planner react accordingly. Fig-ure 8 shows two instances from the actual competition trials where an unexpected obstacle and a falling brick was identified and safely avoided by the robot.

3.4 Laser‑based alignment

The robot localization and navigation algorithm described before allows the robot to move autonomously and safely in the arena and to reach regions of interest such as the

Fig. 6 Results from an outdoor localization experiment. Left: Google Maps view with raw GPS coordinates. Right: EKF-based localization estimates using the Odometry + IMU + GPS information (red line) along with the covariance of the estimates (pink)

Fig. 7 Left: An example of the path planning solution in Gazebo simulation; Right: The correspondent costmap containing both the known (darker) and the unknown spaces (lighter) of the environ-

ment and the occupation probability of the cells are used by the planner to compute the path of the robot

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

wall or the pile of bricks. However, for approaching objects to interact with them, a more precise navigation method is required. To address this situation, a relative localization and navigation system based only on the measurements of the laser scanner is employed to position the robot pre-cisely close to the pile of bricks or the wall, hence allowing the robotic arm to reach, grasp and manipulate bricks.

Figure  9 illustrate the flowchart of the laser-based alignment method. This algorithm starts by segmenting the lines in the vicinity using the 2D point cloud obtained from the laser scanner. The method presented in [41] was employed to extract a list of all lines visible in the point cloud. Once a line that corresponds to a desired brick (or the wall in the case of placing a brick) is detected, the rela-tive angle and distance of the brick is computed and is used to guide the robot to reach within a given distance (30 cm in our examples) of the center of the brick. This

computation is updated with a frequency of 10Hz when-ever a new laser scan is available. Once the robot reaches the desired brick, it rotates to face the brick. Figure 10 illus-trates an example of the brick aligning procedure where the robot approaches and positions itself within manipula-tion range from the brick.

3.5 Brick picking

The previously described methods provide a reliable solution to navigate in the environment and to move the mobile base within manipulation range from the bricks. For picking and placing the bricks a vision-based algorithm was developed that allows the robot to pre-cisely locate the top surface of the bricks and to guide the robotic manipulator to pick the brick from the center. Furthermore, the vision-based brick detection allows the robot to also become aware of bricks on top of other bricks that are undetected by the laser scanner, for example the bricks in the brick pile or in the wall.

Figure 11 describes the brick detection and picking rou-tine. A RealSense D345 depth camera is attached near the end-effector of the robotic manipulator that is used for the vision-based object detection and grasping. Once the mobile base is in place and ready to pick a brick, the arm is initially extended out to a predefined position and orien-tation to have the camera above the brick and parallel to the ground (as shown by the camera image illustrated in Fig. 12). The top-view image of the camera is then filtered using a depth segmentation to remove the ground planes or points below the surface of the brick. Afterwards, white color segmentation and rectangle fitting is performed to detect the ferromagnetic area of the brick.

Once the metallic plate of the brick is identified, the center and the tilt angle of the fitted rectangle is then computed. In addition, the average depth of a set of pixels around the center point of the rectangle is used

Fig. 8 Snapshot from the actual MBZIRC 2020 competition trials illustrating the real-time path planning and the robust navigation even in the presence of unexpected obstacles

New line segment Segment within2m limits ?

Calculate relativedistance and angle to

segment

No

matches desiredbrick length?

No

Yes

Rotate towards thedesired trajectory

Forward robotmovement

Segment furtherthan 30cm?Yes

Rotate robot to alignwith segmentNo

Yes

Fig. 9 Flowchart describing the laser-based brick alignment rou-tine

Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7 Research Article

to compute the relative distance between the camera and brick. This information is used along with the cam-era intrinsic parameters to obtain the 3D coordinates of the center point and the orientation of the brick with respect to the robot. This information is forwarded to the manipulation planner that computes the inverse kinematic solution as well as the trajectory to have the gripper posi-tioned about 5cm above the brick’s center point. The gripper is then gradually lowered until it grasps the brick which is detected using the force feedback sensors of the

manipulator. The motion planner previously reported in [42] is used for computing an efficient inverse kinematics solution and to plan a collision free trajectory for the arm.

3.6 Brick placement

The brick placement algorithm follows a similar principle as the brick picking solution. However, the algorithm also takes into account a priori information on the current wall status and selects a brick on the wall as the reference brick

Fig. 10 Left: robot starting position; Center: robot final position (aligned with the brick surface); Right: the path calculated to perform the alignment using laser information as feedback

Fig. 11 Flowchart describing the brick detection and picking algorithm

Move manipulator topredefined top view

position

Pointcloud empty?

Magnetize gripperGround plane

removal by depthsegmentation

Brick detection bycolor segmentation

Acquire new image

3D center pointcalculation

No

Yes

Center point poseestimation

Brick attached?

Inverse kinematics computation

Manipulatormovement to target

position

Descend manipulator0.2cm

Manipulatormovement to

carrying position

Yes

No

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

to place the new brick with respect to this reference, i.e. to place it on the right, left or above the reference brick depending on the input wall pattern.

The flowchart in Fig.  13 describes the brick place-ment algorithm. After picking a brick the robot initially navigates to the wall location, using the waypoint naviga-tion method described in Sect. 3.3, and then approaches the desired region of the wall containing the reference brick and positions itself using the method described in Sect. 3.4. Once in position, the robot extends the arm out to have the camera above the reference brick similar to the picking algorithm described in Sect. 3.4. Note that the position and viewing angle of the camera is selected in

such a way that the picked brick does not interrupt the camera’s field of view.

Once a top-view image is acquired, the reference brick is then identified in the image using color segmentation and furthermore a rectangle is fitted to the top surface of the brick, as shown in Fig. 14. The rectangle parameters are then used to compute the edge coordinates of the refer-ence brick, allowing the picked brick to be placed relative to this reference edge. For this purpose, a set of points near the edge of this brick is selected and averaged to compute a reliable depth measurement for the edge. This is to avoid the noisy depth measurements that often appear near the edges of objects. The edge coordinate in the camera frame is then transformed to the robot’s frame and is used to

Fig. 12 Illustration of the vision based algorithm used for picking a brick. Left: A top-view image of the brick is used to perform white-color segmentation and to fit a rectangle to the brick’s metallic sur-

face. Right: The center pose with axis aligned to the brick is com-puted and communicated to the manipulator planner

Fig. 13 Flowchart describing the brick placement algorithm Move manipulator to

predefined top viewposition

Pointcloud empty?

Planar segmentationusing depth image

Reference brickdetection by color

segmentationNew image

3D edge pointcomputation

No

Yes

Edge point poseestimation

Force greater than 80% of the

initial force?

Inverse kinematics computation

Manipulatormovement to target

position

Descend manipulator0.2cm

Yes

NoDemagnetize gripper

Manipulatormovement to default

configuration

Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7 Research Article

compute the placement coordinate for the picked brick. Finally, the manipulator moves the brick above the desired location and gradually lowers the arm until the force feed-back from the manipulator indicates that the brick is in place. Figure 14 illustrates an experiment where the target position is computed based on the top-view image and the pose provided to the manipulator planner.

4 System evaluation

Extensive experiments performed in both simulation and on the real robot demonstrated the success of the over-all system and the employed algorithms in executing the challenge. Although it is not trivial or even possible to provide a fair comparison with other existing approaches, either because they do not exist or result from a different composition of subsystems based on different subsystems approaches, the results of our experiments have shown the functionality of the integrated system as a whole and the capability of successfully completing the task.

A Gazebo world was initially developed to simulate the robot and the scenario ( Fig. 15). Models of the four types of bricks were added to the world along with a L-shaped frame where the wall should be built on. Since Gazebo uses a physics engine it provides realistic rendering of the robot and the environment including gravity, inertial properties, collisions, lights, shadows and textures with a good approximation. Although simulations cannot com-pletely capture all the details of the real world, it played out to be very important in developing and testing our algorithms, especially regarding the design and testing of the state machines and execution of the mission routines. Figure 16 shows an example run from the simulator where

the robot builds a 2 layer wall composed of 6 bricks. The time to complete this scenario was measured over 5 runs and averaged 9 minutes and 12 seconds.

For the real experiments, multiple bricks with identical dimensions and weight as to the ones used in the actual challenge were developed. The robot was then instructed to autonomously complete a full pick and place cycle with-out any human interruption. This cycle included: naviga-tion to an area containing a brick, detection of the brick to be picked, alignment with the brick, grasping and deliver-ing the brick to the wall location and placing the brick with respect to a reference brick on the wall . Figure 17 illustrate a trial where the robot continuously performs the full pick and place operation. Experiments have shown the success of the system in reliably navigating to desired waypoints

Fig. 14 Illustration of the vision based algorithm used for placing a picked brick. Left: Color segmentation with rectangle fitting and then averaging samples at the edge is used to compute a reliable

position for the edge of the reference brick. Right: A placement pose is computed relative to the reference edge that is communi-cated to the manipulator planner

Fig. 15 Challenge 2 simulated Gazebo world with realistic meas-urements. The image represents an arena of 15x12m

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

with an accuracy of < 1 meter, aligning to the center of a target brick with an average error of ≈ 15cm and picking bricks from their center with a distant error of ≈ 5cm . Once the robot is in position, the time required for detection and localization of the brick, after an image is acquired, is about 6 milliseconds, and the time to pick or place a brick with the manipulator was configured to last around 20 seconds.

Finally, the robot was transferred to Abu Dhabi and evaluated during the actual MBZIRC competition and its challenging arena. Several problems were encoun-tered all due to the introduction of a new, unfamiliar and

challenging environment, such as the uneven terrain with large wooden ramp in the center of the arena, poor GPS signal reception and highly varying light conditions. However, these problems were all eventually eliminated only by tuning the algorithm parameters to successfully perform the full pick and place cycle autonomously. For example, to eliminate the problem of poor GPS reception at the arena which resulted in localization failures, the EKF parameters were tuned to heavily rely on the odometer and laser readings. For the problem of the uneven terrain and the curved wooden ramp which resulted in detection of imaginary obstacles, the costmap parameters and the

Fig. 16 Snapshots from a simulation where the mobile manipulator is building a 2 layer wall consisting of 6 bricks

Fig. 17 Snapshots from an outdoor experiment where the full pick and place cycle is performed autonomously. a The robot searching for a brick with laser scanning. b Robot detects the brick and aligns itself to the brick. c The brick is visually detected and grasped by the manipulator from the center. d The robot carries the brick to

the wall location. e Robot aligns itself with the wall using feedback from the laser scanner and uses the camera at the manipulator to localize the red reference brick on the wall. f The robot places the brick on the third layer while using the edge of the reference brick to calculate the placement position

Vol.:(0123456789)

SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7 Research Article

lidar thresholds were tuned to eliminate the appearance of such false obstacles in the costmap.

Highlights from some of the experiments and the com-petition runs are compiled in a video.3

5 Conclusions and future work

This paper describes the hardware and software architec-ture of an autonomous mobile manipulator, developed for the second challenge of the MBZIRC 2020 competition, with the aim of manipulating a set of differently shaped bricks to build a wall of predefined pattern. A set of tools and methods for autonomous brick picking and place-ment are presented, using a mobile manipulator while relying on vision-based measurements and 2D laser scan-ning. A complete software pipeline for the full brick pick-ing and placement operation was presented including navigation, brick detection, brick alignment, brick picking and brick placing. Solutions were successfully integrated and tested using a mobile manipulator, both in simulation and in reality, and at different test sites, demonstrating the success of the integrated system, the employed algorithms and the software pipeline for the intended brick pick and placing application. The presented work highlights the importance of major scientific competitions in contribut-ing to the development of real-world robotic solutions.

Acknowledgements This work was supported by a grant from the Mohamed Bin Zayed International Robotics Challenge (MBZIRC 2020) and partially funded by the INTERREG ATLANTIC EAPA_986/2018 pro-ject and LARSyS - FCT Plurianual funding 2020-2023. The authors would like to thank Rui Bettencourt and all other members of the Iberian Robotics team including the GRVC group of University of Seville and FADA-CATEC for their help and support.

Declarations

Conflict of interest On behalf of all authors, the corresponding au-thor states that there is no conflict of interest.

Open Access This article is licensed under a Creative Commons Attri-bution 4.0 International License, which permits use, sharing, adap-tation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.

References

1. Basiri M, Piazza E, Matteucci M, Lima P (2019) Benchmarking functionalities of domestic service robots through scientific competitions. KI-Künstliche Intell 33(4):357–367

2. Dias J, Althoefer K, Lima PU (2016) Robot competitions: what did we learn? IEEE Robot Autom Mag 23(1):16–18

3. Ventura R, Basiri M, Mateus A, Garcia J, Miraldo P, Santos P, Lima P (2016) A domestic assistive robot developed through robot competitions. In: IJCAI Workshop on Autonomous Mobile Ser-vice Robots, New York, USA

4. Meng J, Wang S, Li G, Jiang L, Zhang X, Liu C, Xie Y (2021) Iter-ative-learning error compensation for autonomous parking of mobile manipulator in harsh industrial environment. Robot Comput-Integ Manuf 68:102077. https:// doi. org/ 10. 1016/j. rcim. 2020. 102077

5. Sprunk C, Lau B, Pfaff P, Burgard W (2017) An accurate and effi-cient navigation system for omnidirectional robots in industrial environments. Autonom Robot 41(2):473–493

6. Stückler J, Holz D, Behnke S (2012) Demonstrating everyday manipulation skills in robocup@ home. Robot Autom Magaz 19(2):34–42

7. Smarr CA, Mitzner TL, Beer JM, Prakash A, Chen TL, Kemp CC, Rogers WA (2014) Domestic robots for older adults: attitudes, preferences, and potential. Int J Soc Robot 6(2):229–247. https:// doi. org/ 10. 1007/ s12369- 013- 0220-0

8. Caselli S, Fantini E, Monica F, Occhi P, Reggiani M (2003) Toward a mobile manipulator service robot for human assistance. In: 1st Robocare Workshop

9. Dömel A, Kriegel S, Kaßecker M, Brucker M, Bodenmüller T, Suppa M (2017) Toward fully autonomous mobile manipula-tion for industrial environments. Int J Adv Robot Syst. https:// doi. org/ 10. 1177/ 17298 81417 718588

10. Cosma C, Confente M, Governo M, Fiorini R (2004) An autono-mous robot for indoor light logistics. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol 3, pp 3003–3008 vol.3, https:// doi. org/ 10. 1109/ IROS. 2004. 13898 66

11. Rouček T, Pecka M, Čížek P, Petříček T, Bayer J, Šalanskỳ V, Heřt D, Petrlík M, Báča T, Spurnỳ V, et al. (2019) Darpa subterranean chal-lenge: Multi-robotic exploration of underground environments. In: International Conference on Modelling and Simulation for Autonomous Systesm, Springer, pp 274–290

12. Petrlík M, Báča T, Heřt D, Vrba M, Krajník T, Saska M (2020) A robust uav system for operations in a constrained environment. IEEE Robot Autom Lett 5(2):2169–2176. https:// doi. org/ 10. 1109/ LRA. 2020. 29709 80

13. West C, Arvin F, Cheah W, West A, Watson S, Giuliani M, Len-nox B (2019) A debris clearance robot for extreme environ-ments. In: Althoefer K, Konstantinova J, Zhang K (eds) Towards Auton Robot Syst. Springer International Publishing, Cham, pp 148–159

14. Korayem M, Azimirad V, Nikoobin A, Boroujeni Z (2010) Maxi-mum load-carrying capacity of autonomous mobile manipula-tor in an environment with obstacle considering tip over stabil-ity. Int J Adv Manuf Technol 46(5–8):811–829

15. Korayem A, Nekoo SR, Korayem M (2019) Sliding mode con-trol design based on the state-dependent riccati equation: theoretical and experimental implementation. Int J Contr 92(9):2136–2149

16. Bouyarmane K, Chappellet K, Vaillant J, Kheddar A (2018) Quad-ratic programming for multirobot and task-space force control. IEEE Trans Robot 35(1):64–77

17. Rehman BU, Focchi M, Lee J, Dallali H, Caldwell DG, Semini C (2016) Towards a multi-legged mobile manipulator. In: IEEE

3 https:// www. youtu be. com/ watch?v= 7Vq- pK2xv Ew.

Vol:.(1234567890)

Research Article SN Applied Sciences (2021) 3:558 | https://doi.org/10.1007/s42452-021-04506-7

International Conference on Robotics and Automation (ICRA), pp 3618–3624, https:// doi. org/ 10. 1109/ ICRA. 2016. 74875 45

18. Jimenez-Cano AE, Martin J, Heredia G, Ollero A, Cano R (2013) Control of an aerial robot with multi-link arm for assembly tasks. In: IEEE International Conference on Robotics and Automation, pp 4916–4921, https:// doi. org/ 10. 1109/ ICRA. 2013. 66312 79

19. Krotkov E, Hackett D, Jackel L, Perschbacher M, Pippine J, Strauss J, Pratt G, Orlowski C (2017) The darpa robotics challenge finals: results and perspectives. J Field Robot 34(2):229–240

20. Dias J, Lima PU, Seneviratne L, Khatib O, Tadokoro S, Dario P (2019) Journal of field robotics special issue on mbzirc 2017 challenges in autonomous field robotics. J Field Robot 36(1):3–5

21. Carius J, Wermelinger M, Rajasekaran B, Holtmann K, Hutter M (2018) Deployment of an autonomous mobile manipulator at mbzirc. J Field Robot 35(8):1342–1357. https:// doi. org/ 10. 1002/ rob. 21825

22. Spurný V, Báča T, Saska M, Pěnička R, Krajník T, Thomas J, Thakur D, Loianno G, Kumar V (2019) Cooperative autonomous search, grasping, and delivering in a treasure hunt scenario by a team of unmanned aerial vehicles. J Field Robot 36(1):125–148. https:// doi. org/ 10. 1002/ rob. 21816

23. Loianno G, Spurny V, Thomas J, Baca T, Thakur D, Hert D, Penicka R, Krajnik T, Zhou A, Cho A, Saska M, Kumar V (2018) Localization, grasping, and transportation of magnetic objects by a team of mavs in challenging desert-like environments. IEEE Robot Autom Lett 3(3):1576–1583. https:// doi. org/ 10. 1109/ LRA. 2018. 28001 21

24. Schwarz M, Beul M, Droeschel D, Klamt T, Lenz C, Pavlichenko D, Rodehutskors T, Schreiber M, Araslanov N, Ivanov I et al (2018) Drc team nimbro rescue: Perception and control for centaur-like mobile manipulation robot momaro. The DARPA robotics chal-lenge finals: humanoid robots to the rescue. Springer, Berlin, pp 145–190

25. Helm V, Willmann J, Gramazio F, Kohler M (2014) In-situ robotic fabrication: advanced digital manufacturing beyond the labo-ratory. In:Röhrbein F, Veiga G, Natale C (eds) Gearing up and accelerating cross-fertilization between academic and indus-trial robotics research in Europe. Springer Tracts in Advanced Robotics, vol 94. Springer, Cham. https:// doi. org/ 10. 1007/ 978-3- 319- 02934-4_4

26. Sandy T, Giftthaler M, Dörfler K, Kohler M, Buchli J (2016) Auton-omous repositioning and localization of an in situ fabricator. In: IEEE International Conference on Robotics and Automation (ICRA), pp 2852–2858, https:// doi. org/ 10. 1109/ ICRA. 2016. 74874 49

27. Lussi M, Sandy T, Dörfler K, Hack N, Gramazio F, Kohler M, Buchli J (2018) Accurate and adaptive in situ fabrication of an undu-lated wall using an on-board visual sensing system. In: IEEE International Conference on Robotics and Automation (ICRA), pp 3532–3539, https:// doi. org/ 10. 1109/ ICRA. 2018. 84604 80

28. Gawel A, Blum H, Pankert J, Krämer K, Bartolomei L, Ercan S, Farshidian F, Chli M, Gramazio F, Siegwart R, et al. (2019) A fully-integrated sensing and control system for high-accuracy mobile robotic building construction. arXiv preprint arXiv: 19120 1870

29. Willmann J, Augugliaro F, Cadalbert T, D’Andrea R, Gramazio F, Kohler M (2012) Aerial robotic construction towards a new field of architectural research. Int J Arch Comput 10(3):439–459. https:// doi. org/ 10. 1260/ 1478- 0771. 10.3. 439

30. Mirjan A, Augugliaro F, D’Andrea R, Gramazio F, Kohler M (2016) Building a bridge with flying robots. In: Reinhardt D, Saunders R, Burry J (eds) Robotic fabrication in architecture, art and design 2016. Springer, Cham. https:// doi. org/ 10. 1007/ 978-3- 319- 26378-6_3

31. Lima P, Custódio L, Ribeiro I, SantosVictor J (2003) The RESCUE project: Cooperative navigation for rescue robots. Proceedings do 1st International Workshop on Advances in Service Robotics. Bardolino, Italia, pp 94–101

32. Vale A, Gomes-Mota J (2007) Lidar data segmentation for track clearance anomaly detection on over-head power lines. In: Proc. IFAC Workshop, 5-9

33. Bohren J, Cousins S (2010) The smach high-level executive [ros news]. IEEE Robot Autom Magaz 17(4):18–20

34. Moore T, Stouch D (2014) A generalized extended kalman filter implementation for the robot operating system. In: Proceedings of the 13th International Conference on Intelligent Autonomous Systems (IAS-13), Springer

35. Li G, Huang Y, Zhang X, Liu C, Shao W, Jiang L, Meng J (2020) Hybrid maps enhanced localization system for mobile manipu-lator in harsh manufacturing workshop. IEEE Access 8:10782–10795. https:// doi. org/ 10. 1109/ ACCESS. 2020. 29653 00

36. Tipaldi GD, Meyer-Delius D, Burgard W (2013) Lifelong localiza-tion in changing environments. Int J Robot Res 32(14):1662–1678. https:// doi. org/ 10. 1177/ 02783 64913 502830

37. Marder-Eppstein E, Berger E, Foote T, Gerkey B, Konolige K (2010) The office marathon: Robust navigation in an indoor office environment. In: International Conference on Robotics and Automation

38. Dijkstra EW et al (1959) A note on two problems in connexion with graphs. Numerische mathematik 1(1):269–271

39. ROS (2014) “navfn” planner. http:// www. ros. org/ wiki/ navfn, accessed: 2019-10-30

40. Fox D, Burgard W, Thrun S (1997) The dynamic window approach to collision avoidance. IEEE Robot Autom Magaz 4(1):23–33. https:// doi. org/ 10. 1109/ 100. 580977

41. Pfister ST, Roumeliotis SI, Burdick JW (2003) Weighted line fit-ting algorithms for mobile robot map building and efficient data representation. In: IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422), vol 1, pp 1304–1311

42. Gonçalves J, Lima P (2019) Grasp planning with incomplete knowledge about the object to be grasped. In: 2019 IEEE Inter-national Conference on Autonomous Robot Systems and Com-petitions (ICARSC), pp 1–6

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.