Automatic Analysis of Fish Behaviors and Abnormality

Detection

Myo Thida, How-Lung Eng and Boon Fong ChewInstitute for Infocomm Research

1 Fusionopolis Way, Singapore 138666mthida@i2r.a-star.edu.sg

Abstract

Current state-of-the-art sh monitoring systems arelack of intelligent in interpreting shes behaviors au-tomatically. To tackle these problems, we propose avision-based method that automatically analyze behav-iors of a group of shes in an aquarium and detect ab-normality precisely. Here we consider the problem intwo steps. First, we propose a new incremental spec-tral clustering method to extract frequently occurred keyswimming patterns of shes. Then, we present videosequences of shes into a trajectory through the spaceof these key patterns. Studying these trajectories pro-vides a new tool to analyze shes behaviors. Compar-isons of shes behaviors in clean water and water in thepresence of chemicals provides a new tool to detect anyabnormality. Experimental results illustrate that theprecision value of our proposed method is above 90%.

1 Introduction

Visual analysis of animals and insects behavior isa growing research area with a number of interestingapplications. For instance,

Biological Study on genetics of certain diseasesand

Biological early warning system: monitoring be-haviors of aquatic organisms such as shes to de-tect various chemicals in water.

The traditional way is to rst videotape the animalfor a period of time and then, observe animal behav-iors manually. But, this is time-consuming process andalso observation results is subjective to knowledge ofhuman observers. Therefore, automated system thatcould analyze the behaviors of animals and detect anyabnormal behaviors is a crucial task in above discussedapplications. In this paper, we present a framework forecient animals behavior analysis.

In particular, we emphasized on behaviors analysisof shes. Over the past decades, computer vision-basedsh behavior monitoring systems have been establishedas a way to detect toxicity in drinking water ([1, 2, 3][4] and the references therein). Swimming patterns andbehavior of shes are monitored by using computer vi-sion techniques to observe changes that could be in-dicative of developing toxicity conditions. As observed

in above literatures, more research eorts have beenput on analyzing behaviors of individual shes. Hence,most of the state-of-the-art sh behavior monitoringsystems are limited with restricted environments (forexample, individual sh is required to stay in a verynarrow chamber with electrodes). This makes the sys-tem costly, and also put sh in stressed environment.In this work, we focus on studying group behaviors ofshes in a standard aquarium rather than individualsh in a small chamber. In our knowledge, there is onesimilar work which uses a standard tank in literature[5]. In their work, parameters for 15 individual shesin a standard tank are tracked and analyzed using ar-ticial intelligence concept. But, in reality, tracking ofmultiple shes simultaneously is noisy due to occlusionand abrupt motion of shes.

In this work, based on group activities, we propose anovel approach for automatic clustering of shes swim-ming patterns and abnormal detection using spectralclustering method. The main steps and contributionsare as follows:

Key Patterns Extraction: For an input a longvideo sequence, we propose a new incrementalspectral clustering based method to extract fre-quently occurred key swimming patterns of shes.This summarizes long video sequence into a spaceof possible patterns.

Trajectory Presentation: We describe video se-quences of shes into a trajectory through thespace of extracted key patterns. Studying thesetrajectories provide a new tool to analyze shesbehaviors.

Abnormal Detection: Comparing shes behaviorsin clean water and water in the presence of chemi-cal, we can generate early warning signals of waterquality deterioration precisely.

Experiments on long video sequences, includingboth normal and abnormal conditions are performed.Experiments with dierent chemicals such as aldicarband chloramine are evaluated. The evaluation reportsthat our proposed method can detect any chemical inless than 10 minutes and precision value (correctly de-tected/number of images) is above 90%.

MVA2009 IAPR Conference on Machine Vision Applications, May 20-22, 2009, Yokohama, JAPAN8-18

278

2 Discovering Key Swimming Patterns

In this section, we review spectral clustering methodrst and explain our proposed method in details. Inaddition to developing an application of spectral clus-tering method to real-life videos, we contribute in 1)developing an algorithm to compute distance measurebetween two images, and 2) proposing a new methodto perform spectral clustering incrementally.

2.1 Basic Concepts on Spectral Clustering

This section gives a basic concepts of spectral clus-tering. For details, please refer to [7, 8, 9].

Given a set of data points, the rst step of spectralclustering is to compute a pairwise similarity matrix Wwhere wij represents similarity measure between datapoints i and j. There are a number of methods tocompute similarity matrix. One of the most popularmethods is to use Gaussian similarity function. Math-ematically, it is dened as:

wij = exp{ d(i, j)

2

22

}, (1)

where dij is distance measure between two data pointsand is a scaling parameter.

Then, a graph laplacian matrix is computed usingthe similarity matrix W . In literature on spectral clus-tering, a number of dierent laplacian matrix is beingused and comparisons of these matrixes are discussedin [8]. Here, we shows only normalized graph laplacianmatrix which applied in our proposed method.

L = I D1/2WD1/2, (2)

where I is identity matrix, W is similarity matrix(5)while D is diagonal matrix with values dii which isdened as:

dii =

j

wij . (3)

Once the laplacian matrix is dened, eigen-decomposition of this matrix is performed using singu-lar value decomposition. Finally, a grouping algorithmclusters data based on the eigenvalues and correspond-ing eigenvectors obtained.

2.2 Feature Extraction And Representa-tion

In this paper, we propose to use shape feature andrepresent shapes of shes using a signed-distance func-tion. After extracting contours(shape) of shes usingfast level set method [6], we represent contours as fol-lows:

(x, y) =

d((x, y), C), (x,y) lies outside C;0, (x,y) lies on the contour, C;d((x, y), C), (x,y) lies inside C.

(4)where d((x, y), C) is Euclidean distance between pixel(x, y) and contour C.

Figure 1: Swimming Pattern and its RepresentationUnder Abnormal Condition.

Figure 1 shows contours of shes and its represen-tation using signed distance function. Darker color tolighter color represents values of increasing from neg-ative to positive. This representation has been one ofpopular choices in shape representation due to its ex-ibility in topological changes. This is particulary use-ful in our application as it provides parameters suchas location of shes (darker regions), swimming direc-tion (movement of darker regions) and area occupiedby shes (clustering of shes) and many more.

2.3 Distance Measure

The success of spectral clustering depends heavily onthe choice of similarity or anity matrix. In our pro-posed method, similarity matrix, W is dened based onlocal scaling instead of xed scaling. Mathematically,each entry of the matrix, W , is dened as

wij = exp

{ d

2ij

2ij

}, (5)

where i = dik and k is the neighbor of data point i.In this work, we propose a new distance measure, dijbased on our shape representations:

d2ij =12

(i j)2(h(i) + h(j))dxdy, (6)

where h(i) is dened as:

h(i) =H(i)H(i)

, (7)

and

H(i) ={

1, if( < 5);0, otherwise. (8)

Please note that we give higher weight to the objectsinside. Table 1 compares two distance measures, Eu-clidean and our proposed Pseudo-distance for imagesshown in Figure 2. As shown, test image 1 and testimage 2 are signicantly dierence apparently. But,Euclidean distance measure does not show signicantdiscrepancy while our proposed measure does.

2.4 Clustering Similar Shapes with our In-cremental Approach

Knowing the pairwise distances between all frames,we can segment the sequence into clusters of similar

279

Table 1: Comparisons between two distance measures.

Proposed Dist Euclidean DistRef. vs Test Img 1 0.2 0.8Ref. vs Test Img 2 1.25 0.95

Figure 2: Two test images and reference image areshown (left to right).

shapes. But, when dealing with large amount of data,1) constructing large similarity matrix and 2) comput-ing eigenvalues of large matrix become challenges. Toovercome these limitations, a few methods [10, 11, 12]have been proposed in literature. One of the possiblesolutions is to use Nystorm extension [10] which is anextrapolating method to compute eigenvalues of largematrix. This extension provides a way to extrapolateeigenvectors of large matrix based on the eigenvaluescomputed on a portion of entire similarity matrix, W .However, Nystorm extension will not be accurate if thisportion does not include samples from small clusters.As a result, this method is not suitable if all entries inthe matrix W are not available at the same time. An-other possible solution is to perform spectral clusteringincrementally [12]. Our proposed method belongs tothe second approach. In this paper, we propose a newincremental method to cluster shapes from a large dataset. We perform our algorithm in three steps.

Initial Step The rst step performs traditional spec-tral clustering on a subset S1 of large data set Swhere S1 S.

Compute distance measure between all im-ages within S1 using the proposed distancemeasure (6).

Obtain Laplacian matrix by equation (2). Compute eigenvalues of Laplacian matrix,

and extract the rst K smallest eigenvaluesand corresponding eigenvectors.

Perform Kmeans clustering algorithm onthe extracted eigenvectors to group theshapes into K clusters.

Extract representative swimming patternwhich has minimum distance from any otherimages within the same cluster.

Incremental Step This step brings most of our con-tribution. We rst compute distance between newimage from data set to representative images ob-tained in the rst step. We require that its mini-mum distance from the current clusters must not

far away from the current cluster representatives.If it is, we must perform a new clustering. Hence,for every new image from complement set of S1,

Compute distance measures (6) between newimage and cluster representatives and selectthe minimum distance.

When the new images are far away from thecurrent representatives, we put the imageinto a set of cluster-to-be.

In contrast to other incremental spectralclustering approaches [12], we perform spec-tral clustering (step one) only when theset of cluster-to-be has sucient amount ofdata. This reduces computational cost sig-nicantly. Then, new cluster representativesare augmented into the previous cluster lists.

Rene Step This nal step renes the results ob-tained. The tightness of each cluster is computedby calculating the maximum distance of any twoimages within the cluster. If the maximum dis-tance is greater than a threshold, we re-performspectral clustering on that particular cluster.

2.4.1 Choice of Parameter K

In spectral clustering, one of the most important pa-rameter is the number of clusters, K. In our proposedmethod, we dene K based on matrix perturbation the-ory [13]. This theory states that the number of clustersin a data set depends on the stability of eigenvalueswhich is determined by the gap e between two consec-utive eigenvalues. Based on this theory, we dene thenumber K by nding for the maximum gap e over aset of eigenvalues:

K = argmaxi

| i i1 |, (9)

where i and i1 are two consecutive eigenvalues.

3 Interpreting Fishes Behaviors

Having key swimming patterns, we can describe asequence of video frames as a trajectory through thesepatterns. Figure (3) shows extracted key swimmingpatterns in 2D space. These swimming patterns areobtained from video of shes swimming in clean wa-ter (no chemical involved) using our proposed methodexplained in the above section.

Given a sequence of images, we can generate a tra-jectory through extracted key swimming patterns. Bystudying these trajectories, we can analyze behaviorsof shes. Figure (4) shows the trajectory of video ofshes in clean water (ve hour long video sequence,total images 110730). The dierent swimming behav-iors is clearly visible in its trajectory. The rst oneand half-hour (shown in green color) illustrates re-peated bottom-swimming behaviors while the secondpart (shown in black color) demonstrates more com-plex sh behaviors.

280

Figure 3: Projection of key swimming patterns in 2Dspace by using Principal component Analysis(PCA).

Figure 4: Video trajectory of shes swimming in cleanwater(no chemical involved).

4 Abnormal Behaviors Detection

Figure 5: Video trajectory of shes in clean wa-ter(green) and toxicity water (red).

Based on comparisons of shes behaviors in cleanwater and water in the presence of chemical, we cangenerate early warning signals of water quality deteri-oration. In order to detect abnormality, we rst extractkey patterns from two video sequences: one is video ofshes in clear water and another is video of shes in wa-ter with chemical. Projection of these key patterns into2D space are shown with circle in Figure (5). Videotrajectories of shes in clean water (green) and shesin the water with chemical (red) clearly illustrates dif-ferent behaviors of shes in clean water and toxicitywater.

For any test video, we rst segment video into Noverlapping segments where nth video segment con-tains M number of images. Then, we compute proba-bility of observing any abnormality as follows:

1. First, we assign every image of video segment, vn,into one of key patterns extracted from two videosequences. Hence, video trajectory for segmentbecomes vn = {k1, k2, ..., kM} where M is numberof frames and k is given as:

k = argmaxk

[exp( d2xk

22k)], (10)

where dxk is the distance measure (6) between newimage and representative image of cluster k and kis variance of the cluster.

2. Second, we compute normality score of shes be-haviors as follows:

Pnor =1M

Mi=1

[1 1

1 + exp(vin K)], (11)

where K is parameter estimated using the set ofvideo trajectories of shes swimming in clear wa-ter.

3. Finally, we assign an image as abnormal if

Pnor < kth, (12)

where kth is threshold dened empirically.

Table 2: Abnormal Detection Summary Results.

Video 1 Video 2 Video 3(clean water) (Chlora) (Aldicarb)

Num of Images 110730 77324 92567Correctly Detected 101995 73689 87250Falsely Detected 8735 3635 5317Precision (%) 92.11 95.30 94.26

5 Conclusion

In this paper, we have proposed a vision-basedanimals behaviors analysis system. Though we em-phasized on studying shes behaviors, our proposedmethod can be successfully applied to other animalssuch as ants, bees and mice due to its robust feature ex-traction and representation. Our proposed incrementalmethod provides a tool to project long video sequenceinto a space of possible patterns. Experimental resultsillustrates that our proposed method provides abnor-mal detection signal in less than 5 minutes while pre-cision value is maintained above 90%.

281

References

[1] S. Anitei, Fishes used against terrorist attacks, Septem-ber 2006, http://news.softpedia.com/news/Fishes-Used-Against-Terrorist-Attacks-36818.shtml.

[2] Small sh detect big problems: Environmental scien-tists use sh behavior to monitor water quality, Sci-ence Video, March 2007,http://www.sciencedaily.com/videos/2007/0303-small-sh-detect-big-problems.htm.

[3] S. Nogita, K. Baba, H. Yahagi, S. Watanabe, and S. Mori,Acute toxicant warning system based on a sh move-ment analysis by use of AI concept. in Proceedings ofthe International Workshop: Articial Intelligence forIndustrial Applications, Hitachi, Japan, 1988, pp. 273276.

[4] W. H. van der Schalie, T. R. Shedd, P. L. Knecht-ges, and M. W. Widder, Using higher organisms inbiological early warning systems for real-time toxicitydetection, Biosensors and Bioelectronics, vol. 16, no.7-8, pp. 457465, Spetember 2001.

[5] J.-y. Xu, X.-w. Miao, Y. C. Liu, and Shao-rong, Be-havioral response of tilapia (oreochromis niloticus) toacute ammonia stress monitored by computer vision,Journal of Zhejiang University Science, vol. 6B, no. 8,pp. 812816, May 2005.

[6] Y. Shi, , and W. C. Karl, Real-time tracking usinglevel sets, Computer Vision and Pattern Recognition,vol. 2, pp. 3441, June 2005.

[7] J. Shi and J. Malik, Normalized cuts and image seg-mentation, IEEE Transactions on Pattern Analysisand Machine Intelligence, vol. 22, no. 8, pp. 888905,2000.

[8] U. von Luxburg, A tutorial on spectral clustering,Max Planck Institute for Biological Cybernetics, Spe-mannstr. 38, 72076 Tbingen, Germany, Tech. Rep. TR-149, August 2006.

[9] A. Y. Ng, M. Jordan, and Y. Weiss, On spectral clus-tering: Analysis and an algorithm, in Advances inNeural Information Processing Systems, vol. 14, BritishColumbia, Canada, December 2002, pp. 849856.

[10] C. Fowlkes, S. Belongie, F. Chung, and J. Malik, Spec-tral grouping using the nystorm method, IEEE Trans-actions on Pattern Analysis and Machine Intelligence,vol. 26, no. 2, pp. 214225, February 2004.

[11] M. M. Naeini, G. Dutton, K. Rothley, and G. Mori,Action recognition of insects using spectral cluster-ing, in IAPR conference on Machine Vision Applica-tions, Tokyo, Japan, 2007, pp. 14.

[12] L. A. Valgren Christoer and, Duckett Tom and, In-cremental spectral clustering and its application to topo-logical mapping, in IEEE International Conference onRobotics and Automation, Roma, Italy, April 2007, pp.42834288.

[13] X. Zheng and X. Lin, Automatic determination ofintrinsic cluster number family in spectral clusteringusing random walk on graph, in International Confer-ence on Image Processing, ICIP, vol. 5, no. 3471-3174,2004.

282