Modelling of Human Welder for

Intelligent Welding and Welder Training*

YuMing ZhangUniversity of Kentucky

Lee KvidahlIngalls Shipbuilding

NSRP Welding Panel MeetingBethesda, Maryland

May 4-5, 2016

For Unlimited Distribution. Research funded by the NSF under grant “NRI-Small: Virtualized Welding: A NewParadigm for Intelligent Welding Robots in Unstructured Environment,” IIS-1208420, Sept. 2012-August 31, 2016and grant “Machine-Human Cooperative Control of Welding Process” CMMI-0927707, October 2009-Spet. 2013

Contents

1. Sensing of 3D Arc Weld Pool Surface: motivation, method, real time

2. Characterization: from numerous points to three characteristic parameters

3. Control of 3D Weld Pool Surface: control theory method

4. Human Welder Response: modeling and analysis, control using welder model

5. Welder Motion Response: human-robot system, speed adjustment, 3D adjustment

6. Future Directions

1. Sensing of 3D Arc Weld Pool Surface

(Human Response Input)

Weld pool: where complex phenomena originate; but only the surface is

visible; the major feedback information available to human welders

Measurement of weld pool surface temperature distribution: needs the

emissivity to determine the temperature the infrared radiation but the

emissivity is slope dependent

Weld Penetration:

(1) Surface Specular: use laser reflection; (2) Arc Radiation: use laser

reflection and intercept at a distance low power continuous laser for

continuous measurement, no need for a special camera

Laser: 20 mW, 685 nm

Y.M. Zhang, H.S. Song, and G. Saeed. Observation of a dynamic specular weld pool surface. Measurement Science & Technology, 17(6), 2006.

Reflection law, surface constraint, error evaluation

-40 -30 -20 -10 0 10 20 30 400

20

40

60

80

100

120

140

160

180

X/mm

Y/m

m

Reflected dots from image processing and reconstructed surface

Image processed dots

Dots reflected by reconstructed surface

Hongsheng Song. Machine Vision Recognition of Three-Dimensional Specular Surface for

Gas Tungsten Arc Weld Pool. ECE Department, University of Kentucky, 2007.

XiaoJi Ma. Measurement of Dynamic Weld Pool Surface in Gas Metal Arc Welding Process.

Department of Electrical and Computer Engineering, University of Kentucky, Feb. 2012.

Analytical Solution

Key for Real Time Measurement and Control

W.J. Zhang, X.W. Wang, Y.M. Zhang, 2013. “Analytical Real-time

Measurement of Three-dimensional Weld Pool Surface,” Measurement

Science and Technology, 24(11), article Number 115011 (18pp),

doi:10.1088/0957-0233/24/11/115011

2. Characterization of 3D Weld Pool Surface

Characteristic parameters should be used rather than a large set of 3D

coordinates.

Should keep the fundamental information in the weld pool surface about

the weld joint penetration.

W.J. Zhang, Y.K. Liu, X. W. Wang, Y.M. Zhang. Characterization of three-dimensional weld

pool surface in gas tungsten arc welding. Welding Journal, vol. 91, 2012.

Left: Measured 3D weld pool surface parameters from 36 experiments; Right: Least squares model fitting with 3-parameter model using the width, length, and convexity.

1.7906 0.5657 10.8057 0.9868bw W L C

3. Control of 3D Weld Pool Surface

Modeling: how the characteristic parameters respond to the change in

current and travel speed – extract the model from experimental data

Control: Model predictive control algorithm

Yukang Liu, YuMing Zhang. Control of 3D Weld Pool Surface. Control Engineering

Practice, 21(11), 2013.

Welding Experiments

Speed Disturbance

0 20 40 60 80 100 1200

2

4

6

Distance (mm)

Wb (

mm

)

10 20 30 40 50 60 70 80 90 100 110 1200

1

2

3

4

5

6

7

8

Time (s)

Input

Para

mete

rs

Current/10 (A)

Voltage/3 (V)

Speed (mm/s)

10 20 30 40 50 60 70 80 90 100 110 1200

1

2

3

4

5

6

7

Time (s)

Weld

Pool P

ara

mete

rs

Width (mm)

Length (mm)

10*Convexity (mm)

4. Modeling and Analysis of Human Welder Response to 3D Weld Pool Surface

(mechanized welding, human adjusts the current)

Welding Parameters

Current/A Welding speed/mm/s Arc length/mmArgon flow

rate/L/min

57~81 1~2 3.5-4.5 11.8

Monitoring Parameters

Project

angle/°Laser to weld pool

distance/mmImaging plane to weld pool distance/mm

35.5 24.7 101

Camera Parameters

Shutter speed

/msFrame rate/ fps Camera to imaging plane distance/mm

4 30 57.8

Manual control system of GTAW process

Skilled human welder holds the currentregulator while observing the geometry ofweld pool;

Adjusts the welding current to controlthe process to full penetration.

Experiment Parameters

Y.K. Liu, Y.M. Zhang, L. Kvidahl. Skilled Human Welder Intelligence Modeling and Control.

Welding Journal, 93, 2014.

Linear modeling result.

In general, the human intelligent model can be written as:

0 200 400 600 800 1000 1200 1400 1600-4

-2

0

2

4

Sample Number

dC

urr

ent

Measured dCurrent

Linear Estimated dCurrent

( )= ( ( 3), ( 3), ( 3), ( 1))f f fI k f W k L k C k I k

Following linear model can be identified using standard least squares method:

( )= 0.16 ( 3) 0.082 ( 3)+1.81 ( 3)+0.26 ( 1)f f fI k W k L k C k I k

Model comparison between linear and ANFIS model.

Model Comparison between Neuro-Fuzzy Model and linear model

50 100 150

-2

0

2

4

Sample Number

dC

urr

ent

Measured dCurrent

Linear

ANFIS Estimated dCurrent

500 520 540 560 580 600

-2

-1

0

1

2

3

4

5

Sample Number

dC

urr

ent

Measured dCurrent

Linear

ANFIS Estimated dCurrent

1360 1380 1400 1420 1440 1460

-2

-1

0

1

2

3

Sample Number

dC

urr

ent

Measured dCurrent

Linear

ANFIS Estimated dCurrent

Average Model

Error /ARMSE /A

Maximum Model

Error /A

Linear Model 0.52 0.79 3.15

ANFIS Model 0.50 0.76 3.03

Y.K. Liu, W.J. Zhang, Y.M. Zhang. Dynamic neuro-fuzzy-based human intelligence

modeling and control in GTAW. IEEE Transactions on Automation Science and

Engineering, 12, 2015.

Model ComparisonLinear Models

( )= 0.049 ( 3) 0.0049 ( 3)+1.73 ( 3)+0.72 ( 1)I k W k L k C k I k

( )= 0.16 ( 3) 0.082 ( 3)+1.81 ( 3)+0.26 ( 1)f f fI k W k L k C k I k

Novice welder

Skilled welder

Nonlinear Model Comparison

In normal cases the skilled welder'sadjustments are minimal which can preventlarge oscillation and overshoot novice weldermodel suffers;

In other cases where the convexity iseither considerably small or large, theadjustment made by the skilled welder islarger than that of the novice welder, whichcan provide shorter settling time than novicewelder does.

The skilled welder model does providebetter adjustment than the novice welder.

Nonlinear model surface of the neuro-fuzzy human welder model (left: novice welder, right: skilled welder) for convexity = (a) 0.10 mm (b) 0.18mm (c) 0.26mm. Previous response is zero for all cases.

3

4

5

6

3

4

5

6

7

-2

0

2

W (mm)

Novice Welder

L (mm)

I

(A

)

-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

3

4

5

6

3

4

5

6

7

-2

0

2

W (mm)

Skilled Welder

L (mm)

I

(A

)

-3

-2

-1

0

1

2

3

3

4

5

6

3

4

5

6

7

-2

0

2

W (mm)

Novice Welder

L (mm)

I

(A

)

-0.2

0

0.2

0.4

0.6

0.8

1

3

4

5

6

3

4

5

6

7

-2

0

2

W (mm)

Skilled Welder

L (mm)

I

(A

)

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

3

4

5

6

3

4

5

6

7

-2

0

2

W (mm)

Novice Welder

L (mm)

I

(A

)

0

0.2

0.4

0.6

0.8

1

3

4

5

6

3

4

5

6

7

-2

0

2

W (mm)

Skilled Welder

L (mm)

I

(A

)

-0.5

0

0.5

1

1.5

2

2.5

3

(a)

(b)

(c)

Control Experiments: Varying Initial Current

Control experiment result with different initial current (a) 52A; (b) 54A.

A B

A B

0 10 20 30 40 50 60 70 80 90 100 1100

1

2

3

4

5

6

7

8

Time (s)

Input

Para

mete

rs

Current/10 (A)

Voltage/3 (V)

Speed (mm/s)

BA

(a) (b)

A B

BA

10 20 30 40 50 60 70 800

1

2

3

4

5

6

7

8

Time (s)

Input

Para

mete

rs

Current/10 (A)

Voltage/3 (V)

Speed (mm/s)

BA

5. Welder Motion Response to Weld Pool Surface

(welding speed adjustment, 3D operation adjustment)

1. Equipment for experimental data: human-robot system

2. Extract good response from not-perfect performance of human welder

3. Quality evaluation model

4. Supervised learning using good data

Y.K. Liu, Y.M. Zhang, L. Kvidahl, 2014. “Skilled Human Welder Intelligence Modeling and

Control: Part I-Modeling,” Welding Journal, 93: 46s-52s.

Y.K. Liu, Y.M. Zhang, L. Kvidahl, 2014. “Skilled Human Welder Intelligence Modeling and

Control: Part II-Analysis and Control Applications,” Welding Journal, 93(5): 162s-170s.

Equipment - Virtualized Welding System

Illustration of virtualized welding operation.

Developed virtualized welding system.

In virtual station a humanwelder can view the mockup and moves the virtualwelding torch accordingly asif he/she is right in front ofthe work-piece;In welding station a robotarm (Universal Robot UR-5with six Degree of Freedom)equipped with the weldingtorch receives commands viaEthernet and performsGTAW.

Virtual Station and Welding Station

Virtual welding torch.

Equipment - Virtualized Welding System

Detailed view of the virtual station.

3D weld pool sensing system.

A low power laser (19 by 19 structure lightpattern) is projected to the weld pool surface;

Its reflection from the specular weld poolsurface is intercepted and imaged by Camera 1;

Weld pool image captured by Camera 2 (with orwithout virtual reality (VR) enhancement)

Major components in virtual station include aLeap motion tracking sensor, a mock up pipe, acomputer screen, and a projector.

3D scanning system

Visualization

system Motion sensor

Camera

Projector

Mockup

Screen

Detailed view of the visualization result.

Learning Experiments for Response Data

Teleoperation training experiments The welder observe the weld pool under

random welding current, and move the virtualwelding torch accordingly.

0 200 400 600 800 1000 1200 1400 1600 1800 20000

2

4

6

8

10

12

Sample Number

Input

and O

utp

ut

Speed*3 (mm/s) Current/5 (A) Speedf*3 (mm/s) Width (mm) Length (mm) Convexity*10 (mm)

Measured welding current, weld pool characteristic parameters and human arm

movement speed in thirteen training experiments.

Rating of Performance and Response Data

Welder Rating System To better distill the correct response of the human welder, the human

welder evaluates the measured data and corresponding back-side

weld penetration and assigns a rating (from 0 to 10) in each 5 s interval

(offline, thus less skill demanding).

0 200 400 600 800 1000 1200 1400 1600 1800 20000

5

10

Sample Number

Rating (

0-1

0)

Rating assigned by human welder in thirteen dynamic experiments.

Y.K. Liu, Y.M. Zhang, “Iterative local ANFIS based human welder intelligence modeling and

control in pipe GTAW process: A data-driven approach,” IEEE/ASME Transactions on

Mechatronics, DOI: 10.1109/TMECH.2014.2363050.

ANFIS Based Automated Rating of Data Quality

Welder Rating System

Both linear and ANFIS models are proposed to automate the

rating

Linear and ANFIS modeling of the rating (Welder Rating System)

0 200 400 600 800 1000 1200 1400 1600 1800 20000

2

4

6

8

10

12

Sample Number

Hum

an W

eld

er R

ating

Human Welder Rating

Linear Estimated Rating

ANFIS Estimated Rating

( ) ( ), ( ), ( ), ( )R k f W k L k C k S k

Supervised Modeling

Good Response data (with ratings greater than 8) are used to model how

human welder adjusts the speed per weld pool surface.

0 100 200 300 400 500 600 7000.6

0.8

1

1.2

Sample Number

Speedf

(mm

/s)

Measured Speedf Linear Estimated Speedf ANFIS Estimated Speedf

Linear and supervised ANFIS modeling of the welder response.

3D Operation Learning and Robot Implementation of Human Response

Experiment 1: Different Welding Current

Experiment 2: Speed Disturbance

40 50 60 70 80 90 100 1100

1

2

3

4

5

6

7

8

9

Time (s)

Weld

ing C

urr

ent

and P

ool P

ara

mete

rs

Current/10 (A)

Width (mm)

Length (mm)

Convexity*10 (mm)

40 50 60 70 80 90 100 110-0.5

0

0.5

1

Time (s)

Contr

ol In

puts

Speed (mm/s)

Z Adjustment (mm)

RX Adjustment/10 (deg)

RY Adjustment/10 (deg)

50 60 70 80 90 100 1100

2

4

6

8

10

Time (s)

Weld

ing C

urr

ent

and P

ool P

ara

mete

rs

Current/10 (A)

Width (mm)

Length (mm)

Convexity*10 (mm)

50 60 70 80 90 100 110-0.5

0

0.5

1

Time (s)

Contr

ol In

puts

Speed (mm/s)

Z Adjustment (mm)

RX Adjustment/10 (deg)

RY Adjustment/10 (deg)

6. Future Directions

o Varying Gap feedforward + feedback

o Operation inconsistence: automatic rating of operation quality

o Better Free Demonstration of Human Skills IMU sensor on torch, co-view

helmet

o What is an Co-View Helmet?

o Welder Operation Documentation: Heat Input/Welding Speed/Torch

Orientation/O’clock Position, Weld Pool Size and Shape

o Welder Operation Modeling and Diagnosis

o Improvement from Comparison with Skilled Welder Model

o Program Welding Robot for Intelligent Control

o Automatic Welding Parameters Adjustment Based on Speed and Torch

Orientation/Angle o

Key: Mobile Sensor

- Projective Torch : grid

pattern and dot matrix pattern

attached, as well as one IMU

- Shield glass (simulated by a

piece of glass covered by a paper)

- Sensory Helmet (camera

tripod to simulate the head

movement in 6 DOF)

- Weld pool (Convex spherical

mirror with known geometry)

W.J. Zhang, J. Xiao, Y.M. Zhang, 2016. “A mobile sensing system for three-dimensional weld

pool measurement in manual GTAW process," Measurement Science and Technology, 27 (2016)

045102 (24pp), doi:10.1088/0957-0233/27/4/045102.

Experimental system configuration

Simmer Wireless Inertia Measurement Unit includes:

1. tri-axial accelerometer (Freescale MMA7260Q)

2. tri-axial gyro sensor (InvenSense 500 series)

3. a magnetometer

4. a microprocessor (MSP430F1611)

5. a Bluetooth unit.

Key Component - Inertial Measurement Unit (IMU)

W.J. Zhang, J. Xiao, Y.M. Zhang, 2014. “Navigation of welding

torch for arc welding process,” Preprints of the 19th World

Congress of The International Federation of Automatic

Control, pp. 7158-7163, Cape Town, South Africa, August 24-

29, 2014.

Position experiment

The torch is smoothly moved

along the 3-D since curve. The results of torch trajectory position estimation

Measurement errors in Position Experiment 2

Tracking Accuracy Verification

W.J. Zhang, J. Xiao, Y.M. Zhang, 2014. “Navigation of welding torch for arc welding process,”

Preprints of the 19th World Congress of The International Federation of Automatic Control, pp.

7158-7163, Cape Town, South Africa, August 24-29, 2014.