Active vision system for embodied intelligence based on retina sampling model and hierarchical representation

Janusz A. Starzyk, Xinming YuOhio University, Athens, OH

INTRODUCTION

Retina structure is fundamental to human vision system, which is much more efficient than any of the current robotic vision systems.● Photoreceptors (CONE, ROD) are concentrated around fovea, for the highest resolution on target ● Retina processes the sampled scene using ganglion cells, and sends activation through optical nerve and LGN to the primary visual cortex (V1)● The neurons in V1 fire in groups responding to different visual features from the retina

The retina sampling model uses prespecifiedsampling density. Fig. 2 shows the distribution density curves of the cones inside retina.

● Correlation based sparse connections are used to mimic the neuron connections in V1● Neurons which are locally correlated connect to the same group of neurons in the higher layer● Winners and theirs neighbors fire and have weight adjusted together, for smooth processing and increased robustness.

Fig. 1: Retina structure

Fig. 2: Cone densitiesin human retina [1]

Retina sampling: Model of data Collection

The retina, unlike a camera, does not simply send a picture to the brain. The retina spatially encodes (compresses) the image to fit the limited capacity of the optic nerve.

● The photoreceptors are not evenly distributed inside retina. Most of them are concentrated on or around fovea●1D probability distribution curve is shown in Fig. 3.

● Cortex receives distorted images, which are sharper in the fovea area. ● Fovea is the reference point of gaze shifting, and focuses on the most interesting part of the scene.

Fig. 4 shows the sampling points for the retina model, with higher density in the center than on the periphery

Fig. 3: PDF of the photoreceptors (cones and rods) [2]

Fig. 4: Sampling points for retina model

● Using the retina sampling, the vision system receives more useful information.● Table 1 shows comparison of retina sampling model (human vision) and uniform sampling (computer vision). ● The resolution (density of the sampling points) in the center part of the retina sampling is much higher than that of uniform sampling.

100%100%Whole range

50%78%Part inside green

25%63%Part inside red

14%52%Part inside black

Uniform Sampling(Computer Vision)

Retina Sampling(Human Vision)

Percentage of sampling points

Table 1: Comparison (Human V.S. Computer vision)

Part inside blue 31% 4%

When this artificial retina sampling is applied to a visual scene, the vision system will receive much more data from object in focus, and still have a peripheral vision

Fig. 5: An example of retina samplingOriginal resolution: 900x900, resolution after sampling 60x60

● In primary visual cortex (V1), neurons are activated by the stimuli from similar groups of inputs. ● The connections built based on the correlation of the input reflect observedrelations in the real world. Fig. 6 shows the correlations based on real images.

Correlation-based Connection

Fig. 6: Correlation of the input data

● Linsker obtained useful features in visual field with a fixed connectivity model and noise input for self-organizing training. [3] The disadvantage of his model is that the fixed connectivity model ◘ May not deliver connections to remote but correlated areas of the visual field. Fig. 7 shows the existence of the remote but correlated area ◘ May not result in useful features on higher levels ◘ Local connectivity region is set arbitrarily

Fig.7: Correlation based connections with remote but correlated area

[1] Curcio, C.A., Sloan, K.R. Jr, Packer, O., Hendrickson, A.E. & Kalina, R.E. (1987). Distribution of cones in human and monkey retina: individual variability and radial asymmetry. Science 236, pp. 579-582. [2] Riedel G., Physiology of Human Cells, Available: http://www.aberdeen.ac.uk/sms/ugradteaching/course.php?ID=10[3] Linsker R., “From Basic Network Principles to Neural Architecture: Emergence of Spatial-Opponent Cells”, Proc. National Academy of Sciences, Vol. 83. pp. 7508-7512, 1986. [4] Oja E., “Simplified neuron model as a principal component analyzer”. Journal of Mathematical Biology 15 (3): pp. 267-273, 1982.

An active vision system for embodied intelligence based on retina sampling model and hierarchical representation is developed. The retina sampling model mimics efficiency of human vision system.A hierarchical representation is built up with sparse connections, which are locally generated from the neurons’ activity correlation.Using the goal creation system learning scheme, the active vision system can learn complex knowledge.Goals evolve from the simple ones through interaction with environment. Such organization of the learning process is conductive to creation of a general intelligence, with self-organizing structure and dynamic goals.

BIBLIOGRAPHY

CONCLUSIONS

● Local winners are used to

adjust the connection weights

◘ the local winners are

activated (e.g. the green

one in layer N)

◘ The weights of connections

to neighbors of local winner

are adjusted

◘ The local winners help the

neighbors to fire together (horizontal red arrows are excitatory)

◘ All groups of winner sets in layer N (local winners and their neighbors)

used to activate layer N+1

◘ Use Oja’s learning rule [4] to adjust the weights of connections to the winners

or

Procedure of the weight adjustment:

Activate layer N-1 find the strongest winners in layer N excite the neighbors

as co-winners adjust weights for all activated activate layer N+1

In active vision system, we apply a connection mechanism based on correlation between input neurons’ activation and the activation of local winners.

● Correlation between input neurons’ activation ◘ Use real image data instead of noise ◘ Use images organized in time sequences to obtain feedback connections for invariance building ◘ Process the input data from layer N-1 and calculate the correlations ◘ For each neuron, find out the best correlated set of neurons, and create connections to those neurons

Fig. 8: The excitation of local winner and its neighbors

An active servo system shown in Fig. 9 is being built with real-time video input, to demonstrate the active vision system for embodied intelligence.Both the retina sampling model and the correlation based connections are used to work with the servo system.◘ The webcam is used to capture the visual data,◘ The raw data is uniformly distributed (320x240 pixels), it will be processed first by retina model, compressed to 40x30 with little data loss in the center.◘ With the compressed data and the correlation based sparse connection, the active vision system processes the real-time input, finds the interesting object and generates the object coordinates.◘ The servo system receives the real-time coordinates and follows the object with laser pointer.

Fig. 9: Servo system

Fig. 10: Servo system is working with activevision system to follow the object in view

Building up memories from environment

Fig. 11: The pathways through which the system is built up from interactions with the unspecified environment

In learning, it is not easy to obtain examples of desired behavior that are both correct and representative of all the situations in which the agent has to act.

Reinforcement learning (RL) is a good choice for learning in unspecified external environment.

As shown in Fig. 12, the agent (A) receives data, which includes input (I) and reward (R) from the environment (E), and takes proper action (M) back to the environment. With the aid of the reward, the agent learns how to take correct action to have the maximum reward.

Goal Creation system provides a mechanism that organizes learning of intentional representations and associations between sensory and motor pathways. When an agent realizes that a specific action resulted in a desirable effect related to the current goal, it stores a representation of the perceived object involved in such action and learns associations between the sensory and motor pathways.

Hierarchical representations learning is based on external reinforcement for primitive goals and internal goal creation system for abstract goals and internal rewards

Fig. 12: The reinforcement learning model

A

E

I

RM

S

INPUT OUTPUT

Simulation or

Real-World System

TaskEnvironment

EI Architecture

Goal Creation

ActPerceive Competing goals

Planning

Pain

INPUT OUTPUT

Simulation or

Real-World System

TaskEnvironment

EI Architecture

Goal Creation

ActPerceive Competing goals

Planning

Pain