MULTISENSOR INTEGRATION AND

FUSION

Presented by:

Prince Garg

CONTENTCONTENT INTRODUCTIONINTRODUCTION TYPES OF SENSORSTYPES OF SENSORS WHAT DOES MULTIPLE SENSORS MEAN?WHAT DOES MULTIPLE SENSORS MEAN? MULTI SENSOR INTEGRATION VS. FUSIONMULTI SENSOR INTEGRATION VS. FUSION ROLE OF MULTI SENSOR INTEGRATION AND FUSION IN A ROLE OF MULTI SENSOR INTEGRATION AND FUSION IN A

SYSTEMSYSTEM ADVANTAGESADVANTAGES APPLICATION APPLICATION LASER RADAR AND FORWARD LOOKING INFRARED LASER RADAR AND FORWARD LOOKING INFRARED

SENSORSSENSORS ADVANCEMENTADVANCEMENT REFERENCESREFERENCES

INTRODUCTIONINTRODUCTION Sensor is a device that Sensor is a device that

detects or senses the detects or senses the value or changes of value or changes of value of the variable value of the variable being measured. being measured.

The term sensor some The term sensor some times is used instead times is used instead of the term detector, of the term detector, primary element or primary element or transducer.transducer.

TYPES OF SENSORSTYPES OF SENSORS CONTACT SENSORS:CONTACT SENSORS: AA contact sensorcontact sensor

measure themeasure the response of a target to some form of response of a target to some form of physical contact .this group of sensors responds to physical contact .this group of sensors responds to touch, force torque,pressure,temperature or electrical touch, force torque,pressure,temperature or electrical quantities.quantities.

NONCONTACT SENSOR:NONCONTACT SENSOR: A A non contactnon contact sensor sensor measures the response broughtmeasures the response brought by by some form of some form of electromagnetic radiation. This group of sensors electromagnetic radiation. This group of sensors responds to light, x-ray, acoustic, electric or magnetic responds to light, x-ray, acoustic, electric or magnetic radiation.radiation.

WHAT DOES MULTIPLE WHAT DOES MULTIPLE SENSORS MEAN?SENSORS MEAN?

For these system to use effectively the multiple sensors ,we require some For these system to use effectively the multiple sensors ,we require some method for integrating the information provided by these sensors into the method for integrating the information provided by these sensors into the operation of the system.operation of the system.

In many multi sensor system, the information from each sensor serve as the In many multi sensor system, the information from each sensor serve as the separate input to the system.separate input to the system.

The actual combination or fusion of information prior to its use in the The actual combination or fusion of information prior to its use in the system has been a particularly active area of research.system has been a particularly active area of research.

Typical of the application that can benefit from the use of multiple sensor Typical of the application that can benefit from the use of multiple sensor are industrial task like assembly,military command and for battlefield are industrial task like assembly,military command and for battlefield management. management.

MULTISENSOR MULTISENSOR INTEGRATION VS INTEGRATION VS

FUSIONFUSION Multi sensor integration refers to the synergistic use of the information Multi sensor integration refers to the synergistic use of the information provided by the multiple sensory devices to assist in the accomplishment of a provided by the multiple sensory devices to assist in the accomplishment of a task by a system.task by a system.

Multi sensor fusion refers to any stage in the integration process where there Multi sensor fusion refers to any stage in the integration process where there is an actual combination of different sources of the information into one is an actual combination of different sources of the information into one representational format.representational format.

Multisensor integrationMultisensor integration is discussed in terms of basic integration functions is discussed in terms of basic integration functions whereas whereas multisensor fusionmultisensor fusion in terms of the different levels at which fusion in terms of the different levels at which fusion can take place. can take place.

This difference leads to the conclusion that in many multi sensor integrated This difference leads to the conclusion that in many multi sensor integrated system, The output from one sensor may be used to guide the operation of system, The output from one sensor may be used to guide the operation of other sensor in the system without ever actually fusing the sensor other sensor in the system without ever actually fusing the sensor information.information.

ROLE OF MULTISENSOR ROLE OF MULTISENSOR INTEGRATION AND INTEGRATION AND

FUSION IN A SYSTEMFUSION IN A SYSTEM The role of multi sensor integration and fusion can be The role of multi sensor integration and fusion can be

described by knowing that which type of information described by knowing that which type of information multiple sensor provide to the system.multiple sensor provide to the system.

In figure 1, n sensors are integrated to provide In figure 1, n sensors are integrated to provide information to the system.information to the system.

Fusion of the information takes place at the node as Fusion of the information takes place at the node as indicated in the fig. indicated in the fig.

The output x1 and x2 from the first two sensors are The output x1 and x2 from the first two sensors are fused into new one indicated as x1,2.fused into new one indicated as x1,2.

The output x3 from the third sensor could then be fused The output x3 from the third sensor could then be fused with x1,2 at the next node resulting in x1,2,3 ,which with x1,2 at the next node resulting in x1,2,3 ,which might then be fused with the higher node in the structure.might then be fused with the higher node in the structure.

GENERAL PATTERN OF GENERAL PATTERN OF MSIFMSIF

The dashed lines from the system to the node represent any of the signal sent from the integrating functions.

3 FUNCTION IN THE INTEGRATING SYTEM

SENSOR SELECTION: It selects the most appropriate group of sensors in response to changing conditions.

WORLD MODEL: It represents all the sensory information.

DATA TRANSFORMATION: From this all the information taken from the sensors need to be transformed before representing in the world model.

ADVANTAGES OF USING ADVANTAGES OF USING MULTIPLE SENSORMULTIPLE SENSOR

REDUNDANCY

COMPLEMENTARY INFORMATION

TIMELINESS

LESS COST OF INFORMATION

APPLICATION OF MULTI APPLICATION OF MULTI SENSOR INTEGRATION AND SENSOR INTEGRATION AND

FUSIONFUSION ROBOTICS:ROBOTICS:

Robots with multi sensor fusion and integration enhance Robots with multi sensor fusion and integration enhance theirtheir flexibilityflexibility and and productivity productivity in industrial application in industrial application such as material handling, part fabrication, inspection and such as material handling, part fabrication, inspection and assembly.assembly.

Mobile robot present one of the most importantMobile robot present one of the most important applicationapplication areas for multi sensor fusion and integration. areas for multi sensor fusion and integration. When operating in an uncertain or unknown environment, When operating in an uncertain or unknown environment, integrating and tuning data from multiple sensors enable integrating and tuning data from multiple sensors enable mobile robots to achieve quick perception mobile robots to achieve quick perception for navigation for navigation and obstacleand obstacle avoidance.avoidance.

perception, position location, obstacle avoidance vehicle control, path planning, and learning are necessary functions for an autonomous mobile robot.

Multi sensor fusion and integration of vision ,tactile,thermal,range,laser radar, and forward looking infrared sensors play a very important role for robotic system.

VISION:

Visual information is the most powerful single source of sensory information available to a system.

Many different types of non visual sensor are used in combination with vision sensors to compensate for some of the difficulties encountered in the machine processing of visual information.

Recent research efforts aimed at combining intensity and range features derived from visual images to determine the structure of 3-d objects.

TACTILE:

Combined vision and tactile sensor have been used with a great extent by industrial robot to perform both assembly and inspection task.

STEP 1: Initially 2-d vision is applied to determine bounded regions. The centroid of each region is calculated by using the matched 3-d so that it can be used as the starting point for tactile exploration of the region.

STEP 2: The tactile sensing system explores each region to determine if it is a hole,cavity or surface.

The tactile sensor used in this system is an octagonal cylinder covered with conducting surfaces mounted perpendicular to the mounting plate of the robot.

STEP 3: For the surface region ,the tactile sensing are integrated to create 3-d surface patch that can be

matched with model data base.Starting from the location of the contact with the surface ,the tactile sensors traces the surface.

The points along each trace are combined into cubic least square curves.

These curves are than used to fill in areas of the surface.

STEP 4 The surface patches and closed curves are matched

against the model data base to find an object that is consistent with the sensory information.

If more than one object is found, a probabilistic curve is used to order the objects for verifications.

LASER RADAR AND LASER RADAR AND FORWARD LOOKING FORWARD LOOKING INFRARED SENSORSINFRARED SENSORS

Recent research efforts have developed the method of fusing complementary information provided by laser radar and infrared sensors to segment and enhance features of man made objects in a cluttered back ground.

Once separated from the back ground, the laser radar image provides range information for areas with in afield of view and the infrared image can show special feature of the area.

The gradients of both images are used for both object enhancement and segmentation.

HONDA HUMANOID ROBOT

HONDA HUMANOID ROBOT

Military application

Remote sensing

ADVANCEMENT IN MSIFADVANCEMENT IN MSIF The 2010 IEEE International Conference on Multi sensor Fusion The 2010 IEEE International Conference on Multi sensor Fusion

and Integration for Intelligent Systems will take place at and Integration for Intelligent Systems will take place at historical Fort Douglas, University of Utah, Salt Lake City, historical Fort Douglas, University of Utah, Salt Lake City, Utah,USA, 5-7 September 2010.Utah,USA, 5-7 September 2010.

The theme of MFI 2010 is “Cognitive Sensor Fusion.” A major The theme of MFI 2010 is “Cognitive Sensor Fusion.” A major goal of multi sensor fusion systems is to achieve human-like goal of multi sensor fusion systems is to achieve human-like performance in terms of perception, knowledge extraction and performance in terms of perception, knowledge extraction and situation assessment, exploiting symbolic and/or dynamical situation assessment, exploiting symbolic and/or dynamical systems approaches. systems approaches.

Innovative results on multi sensor fusion topics are invited such Innovative results on multi sensor fusion topics are invited such as distributed methods, sensor/actuator networks, bio-based as distributed methods, sensor/actuator networks, bio-based systems, evolutionary methods, biomedical applications and many systems, evolutionary methods, biomedical applications and many more.more.

REFERENCESREFERENCES

I.P.Howard, Human Visual Orientation I.P.Howard, Human Visual Orientation chichester, U.K: wiley,1982chichester, U.K: wiley,1982

B.K.P.Horn, Robot vision B.K.P.Horn, Robot vision cambridge, MA:MIT Press cambridge, MA:MIT Press 19861986

M.J.Magee and J.K. Aggarwal,''using M.J.Magee and J.K. Aggarwal,''using multi sensor images to derive the multi sensor images to derive the structure of 3-d objects -A review''.structure of 3-d objects -A review''.

M.J.Magee, B.A Boyter ,C,Chien and M.J.Magee, B.A Boyter ,C,Chien and J.K Aggarwal,IEEE Trans Pattern anal. J.K Aggarwal,IEEE Trans Pattern anal. Machine Intell. Vol. PAM I-7,no.6,1985Machine Intell. Vol. PAM I-7,no.6,1985