Behav ResDOI 10.3758/s13428-014-0520-9

A method for resolving occlusions when multitrackingindividuals in a shoal

Ruth Dolado · Elisabet Gimeno · Francesc S. Beltran ·Vicenc Quera · Jose F. Pertusa

© Psychonomic Society, Inc. 2014

Abstract Studying the collective behavior of fishes oftenrequires tracking a great number of individuals. Whenmany fishes move together, it is common for individualsto move so close to each other that some fishes superim-pose themselves on others during one or several units oftime, which impacts on tracking accuracy (i.e., loss of fishtrajectories, interchange of fish identities). Type 1 occlu-sions arise when two fishes swim so near each other thatthey look like one long fish, whereas type 2 occlusionsoccur when the fishes’ trajectories cross to create a T- or X-shaped individual. We propose an image processing methodfor resolving these types of occlusions when multitrackingshoals in two dimensions. We assessed processing effective-ness after videorecording shoals of 20 and 40 individualsof two species that exhibit different shoal styles: zebrafish(Danio rerio) and black neon tetras (Hyphessobrycon her-bertaxelrodi). Results show that, although the number ofocclusions depended on both the number of individuals andthe species, the method is able to effectively resolve a greatdeal of occlusions, irrespective of the species and the num-ber of individuals. It also produces images that can be usedin a multitracking system to detect individual fish trajec-tories. Compared to other methods, our approach makes itpossible to study shoals with water depths similar to thoseseen in the natural conditions of the two species studied.

R. Dolado · E. Gimeno · F. S. Beltran (�) · V. QueraInstitute for Brain, Cognition and Behavior (IR3C), AdaptiveBehavior and Interaction Research Group (GCAI), Departmentof Behavioral Science Methods, University of Barcelona, CampusMundet, Passeig Vall d’Hebron, 171, 08035 Barcelona, Spaine-mail: francesc.salvador@ub.edu

J. F. PertusaDepartment of Functional Biology and Physical Anthropology,University of Valencia, Burjassot, Spain

Keywords Multitracking · Resolving occlusions ·Collective behavior · Danio rerio · Hyphessobryconherbertaxelrodi

Introduction

Fishes are currently very commonly used in experimen-tation in a wide range of research fields, such as genet-ics, embryology and toxicology (e.g., An et al., 2007;Guo, 2004; Kissling et al., 2006; Williams, Haash, andDasmahapatra, 2006), neurobiology and behavior (Guo,2004; Sison, Cawker, Buske, & Gerlain, 2006). Somespecies used in experimentation exhibit group behavior byforming shoals and moving around in a coordinated way.Depending on the species, shoals exhibit different patternswith varying global shapes, polarization (i.e., synchronizedswimming direction; Miller & Gerlai, 2012) and averageinter-individual distances (i.e., the average of all dyadicdistances). Although group behavior depends on interac-tions between individuals, determining whether these col-lective behaviors are emergent self-organized behaviors ordepend on factors such as leadership still generates intensiveresearch and discussion (e.g., Couzin et al., 2011; Parrish,Viscido, & Grunbaum, 2002; Reebs, 2000; Sumpter, 2010).

The biological consequences of shoaling behavior havebeen pointed out in several studies that show the effectsof aggregation on foraging, mating and avoiding predators(e.g., Pitcher & Parrish, 1993; Radakov, 1973). Moreover,changes in collective behavior can be used to evaluateexperimental treatments in the laboratory, especially in thezebrafish (Danio rerio), which has become a very com-mon subject of experimentation (e.g., Gerlai, Ahmad, &Prajapati, 2008; Gerlai, Lahav, Guo, & Rosenthal, 2000;Stewart, Nguyen, Wong, Poudei, & Kalueff, 2014). It is

Behav Res

usually necessary to measure behavioral parameters suchas nearest-neighbor distances, inter-individual distances,speed and polarization to assess the effects of experi-mental treatments (Delcourt & Poncin, 2012; Miller &Gerlai, 2012). However, when measuring the trajectories ofindividuals, researchers are faced with the technical diffi-culty of simultaneously tracking a great many individualsin the same area. A common solution involves video-recording shoal behavior for a predetermined period oftime while using computer software to analyze the images,and then automatically extracting the trajectories of theindividuals.

However, simultaneously tracking a great many individ-uals at each time unit is not an easy task. Because there aremany individuals moving together, some individuals may beswimming so close to each other either at the same heightor one below the other that occlusions can occur in a singletime unit or in several time units in succession. An occlusionis “the phenomenon of two or more tracked target imagesbecoming one during a time period” (Delcourt, Denoel,Ylieff, & Poncin, 2013, p. 193). Figure 1 shows twomain types of occlusion that can be observed when ashoal is tracked: (a) Type 1 occlusion, when two fishesare swimming so close that can be falsely perceived asa single long fish; and (b) type 2 occlusion, when twotrajectories cross and the two fishes are perceived as asingle T- or X-shaped individual. When a type 1 occlu-sion occurs, automatic tracking systems usually detect onlyone individual (the aggregate of the two individuals), andone of the trajectories is therefore lost. When a type 2occlusion occurs and the individuals then separate, auto-matic tracking systems usually detect two individuals and

resume recording two distinct trajectories, but the iden-tities of the individuals may be interchanged; i.e., theindividual labeled A may now be labeled B, and the pre-vious trajectory of A may now be assigned to B, andvice versa. The difficulty determining their identities aftercrossing depends on the angle at which the trajectoriesof the two fish cross, the fishes’ speed, and whetheror not they keep their headings after crossing (Delcourt,Becco, Vandewalle, & Poncin, 2009). Other occlusions canbe seen as intermediate steps between these two types(e.g., Delcourt et al., 2009, Fig. 7).

Detection of occlusions and identification of individ-uals during and after occlusion are problems that havebeen addressed using a variety of approaches, includingtracking in three dimensions with two video cameras, andtracking both the individuals and their shadows or theirreflections on a mirror (for a review, see Delcourt et al.,2013). Since the publication of a work by Kato et al. (2004),which presented a computer image processing system toseparate occluded images of two fishes, research on thistopic has attracted increasing interest. Attempts have beenmade to resolve the undesirable effects produced by occlu-sions by refining the procedure to track the individuals(Miller & Gerlai, 2012), by developing trackers that arespecifically designed to overcome the effects of the occlu-sions on the trajectories (Attanasi et al., 2013; Butail &Paley, 2012; Delcourt et al., 2006; Delcourt et al., 2009),and by developing quantitative analysis methods (Becco,Vandewalle, Delcourt, & Poncin, 2006; Leem, Jeon, Yun, &Lee, 2012). However, although there has been an increasein the power and effectiveness of such techniques in recentyears, results are still far from optimal (Delcourt et al.,

Fig. 1 Examples of occlusions: a Type 1 occlusion, when two fishes are swimming so close that their image looks like a very long fish; b type2 occlusion, when two fishes cross each other and their image looks like a single T- or X-shaped individual. Both images are snapshots of fortyzebrafish (Danio rerio)

Behav Res

2013). A new multitracking method has recently been devel-oped, based on extracting a signature that identifies themovement of each individual from occlusion-free portionsof the video and then uses the signatures to resolve occlu-sions and identity switches (Perez-Escudero, Vicente-Paige,Hinz, Arganda, & de Polavieja, 2014). However, the methodoptimally requires sufficient contrast of the animals againstthe background, the processing of lengthy videos (typically30 min) to obtain the signatures and a resolution of at least150 pixels per animal, which may require a camera reso-lution of about 1,000 × 1,000 pixels, according to theseauthors.

In this article, we propose an image processing methodfor resolving occlusions when multitracking shoals in twodimensions. The method consists of successively applyinga set of image processing functions to a series of videoframes, detecting both types of occlusions, and separatingthe individuals involved. It is a powerful pre-processing toolthat can be used as a complement to multitracking systems,including commercial ones (e.g., Image-Pro Plus� MediaCybernetics, Inc., 2009) and ad-hoc ones (e.g., Butail,Bartolini, & Porfiri, 2013; Delcourt et al., 2009), which,in most cases, perform preliminary image processing thatis not effective enough to obtain clear individual trajecto-ries. Our method provides processed images that facilitatethe task of obtaining the coordinates and trajectories ofeach individual in a shoal. It can also work on small-sizefishes, with low light intensity and does not require lengthyvideos.

The main factors causing occlusions and producingerrors in trajectories are population density, the height of thewater column and species behavior. The higher the popula-tion density, the more likely it will be for some individuals tobe occluded by others (Delcourt et al., 2013); and the higher

the water column, the more likely it will be for some indi-viduals to swim at different levels, which tends to increasethe number of type 2 occlusions. The species is also adetermining factor, as different species usually have dif-ferent shoal styles. Some species tend to move in morecompact shoals, while other species move in a more polar-ized way (Delcourt et al., 2013). In our laboratory we haveobserved that zebrafish (Danio rerio) shoals tend to be lesspolarized than those of black neon tetras (Hyphessobryconherbertaxelrodi). Type 2 occlusions are therefore likely tobe more frequent in zebrafish shoals than in black neon tetrashoals (see Fig. 2).

We tested the effectiveness of our method to detectand eliminate the effects of type 1 and type 2 occlusions.We videorecorded shoals of zebrafish and of black neontetras with varying numbers of individuals in a controlledaquarium setting and then processed the images using ourmethod.

Method

Image processing

We propose a two-step image processing method thatapplies successive image analysis procedures to clean upand simplify original 8-bit gray images. Based on previ-ous research (e.g., Kato et al., 2004; Delcourt et al., 2009;Miller & Gerlai, 2012), the goal of the first step is to selecta stack of 8-bit gray images and return a stack of binaryimages (Fig. 3). The following processes are carried out inthis stage: (a) The background is subtracted from all framesin order to remove noise in the images; (b) a median filteris applied to preserve the edges and reduce the noise caused

Fig. 2 Examples of two different shoal styles produced by different species observed in our laboratory: a A shoal of forty black neon tetras(Hyphessobrycon herbertaxelrodi); b a shoal of forty zebrafish (Danio rerio). Note that group a swims in a more polarized way than group b

Behav Res

by the fishes’ caudal fins; and (c) the images are segmentedand converted into binary form. The background image canbe obtained by either averaging multiple images from thevideo in which the shoal occupies different positions in thetank (Miller & Gerlai, 2012) or by simply taking a sin-gle image of the tank before introducing the fish. Medianfiltering consists of replacing the gray level of each pixelwith the median of the gray levels of its neighboring pix-els, a process that results in a more homogeneous and lessnoisy image where edges are more sharply demarcated, andthe image is thus easier to segment. Image segmentation isthe process of separating objects from their background byconverting a gray level image into a binary one, i.e., blackobjects on a white background. Several segmentation meth-ods exist, the commonest being thresholding based on thehistogram of gray levels (e.g. Boyle & Thomas, 1990). Theresulting binary images contain small black areas or clumps(Miller & Gerlai, 2012) that represent fish. The imagesmay still contain some scattered noise (i.e., small dirt par-ticles submerged in the water tank that move during therecording); a filter is therefore applied to remove all isolatedobjects whose area is considerably smaller than that of afish.

In the second step, the binary images are filtered andprocessed by applying two successive functions (water-shed and ellipse) in order to obtain occlusions-free images(see Fig. 4). In order to detect occlusions, visual inspec-tion of the binary images is first necessary to establishthe size of the occlusions and define a threshold. Becausethe size of an occlusion depends on the size of the fish,thresholds must be based on these criteria: (a) minimumarea occupied by occlusions at different heights in thewater column; and (b) maximum area occupied by a sin-gle fish swimming close to the water surface. Definedthresholds should guarantee that the maximum number ofocclusions is detected without including large single fish asocclusions.

As shown in Fig. 4: (a) the binary images are filteredin order to select clumps whose area is greater than thedefined threshold; (b) the watershed function (see below)is applied to the selected clumps, and the resulting splitclumps are resolved type 1 occlusions; (c) the imagesare filtered again in order to select clumps that are stillgreater than the threshold; (d) the ellipse function (seebelow) is applied to the selected clumps, and the result-ing split clumps are resolved type 2 occlusions; (e) finally,images resulting from the previous steps are merged withthe original images containing clumps that were smallerthan the threshold. It should be noted that the final imagesmay still contain unresolved occlusions, i.e., they may beeither clumps that, despite being occlusions, are smallerthan the threshold or clumps that were unsuccessfully split.The final stack of images can be compiled as an AVI

file and subsequently used in a multitracking softwareprogram.

It should be noted that the method does not consistof matching the big clumps to predefined templates thatcorrespond to types of occlusion. Instead, it uses thewatershed and ellipse functions blindly (i.e., it does notmatch templates) to split these clumps. We selected thesefunctions as appropriate for successfully splitting type 1and type 2 occlusions based on a preliminary study asto determine the most suitable kind of image process-ing to split the kinds of occlusion we observed in ourlaboratory.

The watershed function separates connected and adjacentobjects by eroding their frontier pixel line. This opera-tion consists of a series of consecutive erosions until thelast erodible pixel at the center of each object is reached.The single pixels are then dilated so the resulting objectsdo not touch (Fig. 5a) (Pertusa, 2010; see also Katoet al., 2004, Fig.2). The watershed function successfullyseparates adjacent objects when the length of their fron-tier pixel line is smaller than the object width; if thatis not the case, the watershed function does not succeedin separating them, and an alternative function must beused, such as the ellipse function. It consists of fittingan ellipse to a clump and splitting it along its shortestaxis, by subtracting the axis, conveniently widened, fromthe clump (Fig. 5b). Although the resulting objects nolonger have their characteristic spindle shape, they canbe detected as distinct objects; if the image still containsscattered noise (i.e., pixels that remain after applying theellipse function), isolated objects are deleted whose areais considerably smaller than that of a fish. Fig. 5b andc show two kinds of type 2 occlusion: T- and X-shapedocclusions.

Subjects

Forty adult zebrafish (Danio rerio, mean body length: 3.2cm) and forty adult black neon tetras (Hyphessobrycon her-bertaxelrodi, mean body length: 2.85 cm) were used in theexperiment. They were obtained from a local vendor (Coral-lus, Gava, Barcelona) six months before the beginning ofthe experiment. All the fishes were kept in several aquar-iums of identical size (40 × 43 × 30 cm), which weremaintained on a regular light/dark cycle. Water tempera-ture was maintained at 25±2◦C, pH at 7.8–8, total hardnesswas at 10–14◦ dH and carbonated hardness at 15◦ dH.The ammonium concentration was nil, while nitrate andnitrite levels were below 100 mg/L and 1 mg/L, respec-tively. Water quality was kept at optimal values by staticrenewal. Fishes were fed to satiation on Ocean NutritionTM

Community Formula Flakes (Ocean Nutrition Europe,Belgium).

Behav Res

Fig. 3 a An 8-bit gray image of twenty black neon tetras. b Its binary transformation, after background subtraction, median filtering andsegmentation

Fig. 4 Schematic diagram of the image processing applied to a video frame to resolve occlusions. The binary image is filtered successively andthe watershed and ellipse functions (highlighted) are applied to resolve occlusions. The process is repeated for each individual frame

Behav Res

Procedure

Fishes (n = 20 or n = 40, according to the experimentalcondition) were transferred into the experimental tank (100× 100 × 40 cm, with a water column height of 15 cm)and we recorded their activity for 10 min after 5 min ofhabituation to the new environment. The experimental tankwas indirectly lit by five lamps surrounding the apparatus,which provided a low light intensity. A fixed CCD camera(uEye UI-164xLE�) was mounted 2.2 m above the centerof the tank (see Fig. 6). The signal from the camera wasfed into a computer (HP Z420, Intel� Xeon� E5-1620)and was recorded using the uEye Cockpit� software pro-gram (IDS GmbH, 2010). We recorded our videos at 20 fps,8-bit gray, and each frame with a resolution of 1024 × 1002pixels, which corresponds to the tank area in our experimentsetting. Before introducing the fishes in the tank, the tankwas recorded to obtain a background image, as describedabove. We fragmented each video frame by frame using thefree VirtualDub 1.19.11 software program (Lee, 2013) toobtain four stacks of 12,000 frames each, one stack for each

experimental condition. Then, we randomly selected eightpacks of 500 frames per experimental condition to test theeffectiveness of our image processing at resolving occlu-sions. We analyzed 4,000 frames per experimental condition(16,000 frames in total).

The image processing techniques previously describedwere implemented as ImageJ macros (Rasband, 2014).They are available at www.ub.edu/gcai/soft/occlusions.zip.ImageJ is an open-source image-analysis software pro-gram that accepts third-party macros and plugins to per-form specific tasks. In order to determine the size of theocclusions and define thresholds for detecting them, wevisually inspected 1,000 frames (500 for each species)randomly chosen from stacks obtained after applyingthe first step. An occlusion threshold area of 135 pix-els was obtained for black neon tetras and one of 230pixels was obtained for zebrafish. Prospective users canchange the threshold size in the ImageJ macro to fit theirneeds.

After applying our method, the Image-Pro Plus� version7.0.1 software was used on the preprocessed videos to track

Fig. 5 a A type 1 occlusion split using the watershed function; fromleft to right, the initial binary image, the ultimate eroded points (onepixel each), a stage of the dilation process, and its final stage. b A T-shaped type 2 occlusion split using the ellipse function. c An X-shapedtype 2 occlusion split using the ellipse function. In b and c, from left toright, the initial binary image, the best fitting ellipse, the clumps split

along the ellipse’s short axis, and the resolved occlusion after filter-ing out noise. Fish centroids according to human inspection are shownas dots; the centroids obtained automatically before and after applyingthe split functions are shown as crosses. Note that the latter are closeto the former, and that the position error is less than one body length

Behav Res

a b

CCD camera

ComputerExperimental tank

Walls

Fig. 6 Schematic representation of the experiment setup. a Top viewof the tank and position of the lights. The black-filled bulbs represent23 W lights; the gray-filled bulbs correspond to 11 W lights. All lightsface away from the experimental tank. Because light is reflected on

the right wall, lower wattage bulbs were used on that side to obtainas homogeneous light intensity as possible throughout the tank; b aCCD camera above the experimental tank sends a digital signal to acomputer, which captures the images of the fishes’ positions

the fishes and obtain their trajectories. Image-Pro Plus� isa general purpose commercial software program for imageanalysis that includes multitracking among other functions.The program is able to track an unlimited number of indi-viduals, and the user does not have to specify the number ofindividuals in advance. However, when given either a rawvideo or a stack of background-subtracted binary imagesof a shoal, it produces many wrong trajectories and iden-tity switches because of occlusions. Many of these errorscan be prevented if the images are preprocessed using ourmethod.

The auto tracking option in Image-Pro Plus� was used,with the parameters shown in Table 1. Tracking predic-tion depth is a crucial parameter that indicates how manyprevious frames must be taken into account by the pro-gram in order to predict the next position of each fish. Thegreater the prediction depth, the higher the probability thatthe program will correctly identify individuals in occlusionsthat were previously resolved by our method. However, thegreater the depth, the slower the multitracking analysis. Dueto memory limitations in the 32-bit version of Image-ProPlus�, the maximum depth allowed by the program was 7frames for videos with 20 fishes and 6 frames for videoswith 40 fishes.

Results

Results concerning the effectiveness of the method forresolving occlusions are shown in Table 2, which summa-rizes the data shown in the Appendix. Occlusions weretallied as the number of big clumps occurring in the images.An occlusion could last longer than one frame, in whichcase the number of frames it occurred was tallied; a framecould also contain more than one big clump. Therefore,

the total number of occlusions listed in the table for a spe-cific segment can be greater than its number of frames, i.e.,500. The number of undetected occlusions was obtained byvisual inspection of the images resulting from the imageprocessing described. Extra code was included in the ImageJmacros in order to tally the number of type 1 and type 2occlusions detected and the number of errors in the occlu-sion splitting process. Given the occlusions detected andassuming that an occlusion always involves two individu-als, if the splitting process is error free, we would expectthe number of clumps obtained after splitting to be twicethe number of occlusions detected. For example, in the firstpack of images for black neon tetras, 189 type 1 occlusionsand 295 type 2 occlusions were detected; therefore, assum-ing that each occlusion involved two fishes, we expected968 clumps after splitting (i.e., 2 × (189 + 295)). Theerrors in the splitting process were either excess or defaulterrors. An excess error occurs when more than two clumpsare obtained after splitting an occlusion; a default erroroccurs when a type 2 occlusion is not split correctly usingthe ellipse function (e.g., because the ellipse’s minor axisis too short), resulting in a single clump. The effective-ness of image processing in resolving occlusions is calcu-lated as (1−number of errors/number of expected clumps)×100.

As we expected, as the number of individuals increased,the number of occlusions also increased (U = 8.5, p <

.001, r = .8). Globally, type 2 occlusions were more fre-quent than type 1 ones (Wilcoxon test, z = −3.41, p =.001, r = .6) (see Table 2). Type 2 occlusions weremore frequent in zebrafish than in black neon tetras, butonly when the number of individuals was 20 (U = 1.5,p < .001, r = .8).

Although the rate of occlusions per individual dependedon sample size and species, once occlusions were detected,

Behav Res

Table 1 Parameters used in the multitracking function of Image-ProPlus�

Parameters Values

Track parameters

Velocity limit (search radius) 40 pixels/frame

Acceleration limit 40 pixels/frame2

Minimum total track length 10 pixels

Predominant motion type Directional

Objects in tracks

Allow partial tracks, minimum length 20 frames

Tracking prediction depth 6–7 frames

they were resolved with a high degree of effectiveness (for20 black neon tetras, M = 93.51, SD = 6.65; 40 blackneon tetras, M = 96.45, SD = 0.95; 20 zebrafish, M =92.28, SD = 10.20; 40 zebrafish, M = 96.03, SD = 1.42;see Appendix). Effectiveness was not significantly differentbased on sample size (U = 124.5, p = .897) or species(U = 119.5, p = .752)

Table 3 shows the mean number of tracks and theirmean lengths (measured in frames) both when Image-ProPlus� was used on the videos obtained from the binaryimages and when it was used after applying the method, asa function of sample size and species. The anticipated num-ber of tracks equals the sample size (20 and 40) and theanticipated mean track length equals the number of framesanalyzed (500). Therefore, the closer the obtained values areto the expected ones, the more effective the method is. Forboth sample sizes and both species, applying our method

before multitracking caused the mean number of tracksto decrease and the mean track lengths to increase, thusapproaching their anticipated values. For the four samplesize and species conditions, the Wilcoxon signed rank testwas performed to compare the number of tracks obtainedwith and without our method and the results ranged from−2.10 to −2.58, p-values ranged from .036 to .011, andrs ranged from .7 to .9; the Wilcoxon signed rank test wasalso performed to compare track lengths and the resultsranged from −2.24 to −2.52, p-values ranged from .025 to.012, and rs ranged from .8 to .9. When the method wasapplied, the mean percentages of track lengths obtained toanticipated increased in all cases, especially in the case ofzebrafish.

Figure 7 provides an example of the effectiveness of ourmethod for resolving occlusions and facilitating multitrack-ing. It shows the results of tracking 20 zebrafish using thebinary images, i.e., only with previous background subtrac-tion and without resolving occlusions (Fig. 7a), and afterapplying the method (Fig. 7b). As shown, in this specificcase all the false trajectories caused by occlusions weresuccessfully removed when the method was applied beforemultitracking was performed.

Discussion

We proposed an image processing method to resolve occlu-sions when many individuals in a shoal are being trackedsimultaneously. It makes it possible to identify each indi-vidual by splitting occlusions. We videorecorded shoals in acontrolled aquarium setting and processed the videos using

Table 2 Total number of occlusions as a function of sample size and species, and average occlusion rate per individual. (Total number ofocclusions / n). These values summarize the data in the Appendix

n = 20 n = 40

Detected Detected

Total Undet. Total Type 1 Type 2 Total Undet. Total Type 1 Type 2

Black neon tetras

Frequency 330 65 265 93 172 4027 440 3587 1369 2218

Rate 16.5 3.3 13.3 4.7 8.6 100.7 11.0 89.7 34.2 55.5

Zebrafish

Frequency 1659 357 1302 656 646 4857 765 4092 1581 2511

Rate 83.0 17.9 65.1 32.8 32.3 121.4 19.1 102.3 39.5 62.8

Behav Res

our method to assess its effectiveness. Because the speciesand number of individuals are factors that increase the num-ber of occlusions, the method was applied to different sizegroups of individuals (20 versus 40) and different species(black neon tetras versus zebrafish).

As expected, the results showed that the number of occlu-sions detected increased when the number of individualsincreased. We also found differences between species in thenumber of occlusions detected; the zebrafish shoals showedmore occlusions than the black neon tetra shoals. How-ever, the effectiveness of the method was high regardless ofthe species and the number of individuals. As a result, theimages obtained after applying the method resolved enoughocclusions to ensure that a multitracking system could theneasily detect individual trajectories.

As shown in Table 2 and the Appendix, not all the occlu-sions were detected by the proposed method. As occurswith the occlusions detected, the more individuals, the moreocclusions that go undetected. Also, the more undetectedocclusions, the more excess errors, which are even higherin zebrafish, regardless of the number of individuals. Thenumber of undetected occlusions and excess errors is aconsequence of the threshold value used to detect occlu-sions, so that the choice of an appropriate value determinesprocess success. As mentioned, the appropriate thresholdvalue should depend on fish size. It should be greaterthan the minimum area occupied by occlusions at differ-ent heights in the water column, and less than the areaoccupied by a single fish swimming at the water surface.Therefore, if the chosen value is too small, all occlusionswill be detected, but excess errors will increase becauselarge individuals swimming at the surface will be consid-ered occlusions. Conversely, if the value is too large, many

occlusions will go undetected and excess errors willdecrease. Because varying the threshold value can have dra-matic effects on the process, it would be very advisable toexplore these effects systematically, a task to be addressedin future research.

The effectiveness of the proposed method is also evidentfrom the fact that the results of a multitracking softwareprogram, such as Image-Pro Plus�, are improved when thevideos are pre-processed using the method. When the mul-titracking software is used on a video with binarized imagesonly, the number of tracks obtained is usually greater thanthe number of individuals, and the track lengths are usuallyshorter than the video; this is the case because the soft-ware tends to lose track of the individuals involved whenocclusions occur, and assigns new identities to them. How-ever, when occlusions are resolved using our method priorto using a multitracking software, the number of tracksobtained by the software is closer to the actual number ofindividuals, and their lengths are closer to the length of thevideo.

The proposed method assumes that all detected occlu-sions involve only two individuals. Therefore, the imageprocessing method splits detected occlusions into twoclumps even if more than two individuals are involved. Iffishes tend to swim at similar depths, doubling the numberof individuals would not increase the probability of somefishes swimming on top of and crossing over others; there-fore, the number of type 2 occlusions would not necessarilyincrease if the number of individuals is doubled. That is pre-cisely what we see in the Appendix: the ratio of detectedtype 2 occlusions to the total number of detected occlu-sions is substantially identical in neon tetras (.69 for 20 neontetras vs .62 for 40 neon tetras), and the difference is small

Table 3 Mean number of tracks and mean track length (measured in frames) detected by Image-Pro Plus� before using the pre-processingmethod described in the article (i.e., using binarized videos only) and after using the method

Mean number of tracks Mean track length Mean % expected length

Binary Pre-processed Binary Pre-processed Binary Pre-processed

n = 20

Black neon tetras 22.8 20.8 439.1 483.0 87.8 96.6

Zebrafish 28.5 24.1 351.2 416.4 58.8 80.0

n = 40

Black neon tetras 66.0 50.0 293.8 399.8 70.2 83.3

Zebrafish 73.3 52.1 267.9 382.7 53.6 76.5

Behav Res

in zebrafish (.48 for 20 zebrafish vs .56 in 40 zebrafish).Therefore, as the fishes we observed tended to swim at

a

bFig. 7 Multitracking of twenty zebrafish (Danio rerio) obtained withImage-Pro Plus� on one of the image stacks analyzed. For a clearerpicture, only 200 of the 500 frames are shown here. a Trajectoriesfound by Image-Pro Plus� on a raw image stack with previous back-ground subtraction only. Dots indicate the positions of individualsdetected at the first frame; trajectories lacking a dot start at laterframes. Note that more than twenty trajectories are detected, eitherstarting at the first frame or later. Arrows indicate some of the falsetrajectories. b Trajectories found on the same image stack as in a afterbackground subtraction, binary conversion, and resolving type 1 andtype 2 occlusions. Note that the twenty trajectories anticipated weredetected

similar depths, occlusions involving more than two fisheswere infrequent and short-lived. However, a future update ofthe method should address occlusions involving more thantwo individuals.

The ellipse function splits both T- and X-shaped type 2occlusions. When two fishes cross, a T-shaped occlusionprecedes a sequence of X-shaped ones, which is followedby another T-shaped occlusion. There may be a moment inwhich the X-shape is completely symmetrical, in which casethe ellipse function may split it incorrectly, thus resultingin two clumps that correspond to two different body sec-tions of the same individual (head and tail). However, unlessthe temporal resolution is very high, complete symmetryis infrequent. Even so, if a completely symmetrical X-shaped occlusion is formed, it will probably last for a singleframe. Because the ellipse function can successfully resolvenon-symmetrical X-shaped and T-shaped occlusions, whenfishes cross, a sequence of occlusions will be resolved cor-rectly except for a possible completely symmetrical X-shapein the middle of the sequence. When a completely symmet-rical X-shape is split incorrectly using the ellipse function,the distance between the centroids of the two clumps andthe centroids of the fish is less than one body length. Anal-ogously to the example of a non-symmetrical X-shapedocclusion shown in Fig. 5c, the position error is not sub-stantial in this case. Therefore, when the ellipse functionsplits an X-shaped occlusion incorrectly, no individual islost, and only a small position error is obtained. When amultitracking program is subsequently applied, that error isnot a problem because the program should be able to pre-dict individual trajectories from past positions (as is the casewith Image-Pro Plus�).

Our method makes it possible to effectively resolveocclusions detected in two-dimensional videorecordings ofgroups of small-size fishes in low light intensity and ren-ders the use of mirrors and extra video cameras unnecessary(Delcourt et al., 2013). Unlike other methods that requirefishes to swim in tanks with a water depth of less than 5cm in order to prevent occlusions (Delcourt et al., 2006;Miller & Gerlai, 2012), ours can be applied with a depth ofup to 15 cm. Zebrafish and some neon tetra species inhabitshallow wetlands (FishBase, 2014). Our method can pro-vide descriptions of the collective behavior of shoals in suchspecies in experimental settings that are more similar to thenatural conditions of their habitats.

Author Notes This project was supported by grants from the Direc-torate General for Research of Catalonia (2009SGR-1492) and fromthe Ministry of Science and Innovation of Spain (PSI2009-09075,PSI2012-32007).

Behav Res

Appendix

Table 4 Results of occlusion detection in shoals of black neon tetras and zebrafish (n = 20 and n = 40 for each species). Each row corresponds toa 500 frame video

Occlusions

Species Detected Clumps Errors Effectiveness

and video Total Undetected Total Type 1 Type 2 expected Excess Default %

Black neon tetras

20 28 2 26 3 23 52 0 0 100.0

20 31 11 20 9 11 40 0 0 100.0

20 61 0 61 20 41 122 0 2 98.4

20 94 27 67 37 30 134 0 6 95.5

20 37 3 34 12 22 68 0 8 88.2

20 42 11 31 6 25 62 1 1 96.8

20 19 6 13 3 10 26 0 4 84.6

20 18 5 13 3 10 26 0 4 84.6

40 499 15 484 189 295 968 2 24 97.3

40 525 23 502 223 279 1004 2 21 97.7

40 573 94 479 186 293 958 3 36 95.9

40 336 51 285 95 190 570 0 23 96.0

40 784 47 737 258 479 1474 2 63 95.6

40 581 67 514 194 320 1028 1 31 97.0

40 393 60 333 112 221 666 3 16 97.1

40 336 83 253 112 141 506 0 25 95.0

Zebrafish

20 192 32 160 74 86 320 2 5 97.8

20 420 54 366 157 209 732 13 9 97.0

20 128 15 113 49 64 226 2 7 96.0

20 199 44 155 64 91 310 8 17 91.9

20 90 10 80 36 44 160 5 9 91.2

20 248 70 178 106 72 356 61 53 68.0

20 177 44 133 103 30 266 4 2 97.7

20 205 88 117 67 50 234 2 1 98.7

40 908 110 798 200 598 1596 16 46 94.5

40 924 97 827 275 552 1654 25 51 95.0

40 810 143 667 214 453 1334 10 64 96.1

40 641 141 500 253 247 1000 14 29 95.7

40 461 77 384 188 196 768 8 10 97.7

40 192 32 160 110 50 320 0 4 98.7

40 618 112 506 219 287 1012 17 26 95.7

40 303 53 250 122 128 500 10 14 95.2

Behav Res

References

An, L., Hu, J., Zhu, X., Deng, B., Zhang, Z., Yang, M. (2007). Cruciancarp (Carassius carassius) VTG monoclonal antibody: Develop-ment and application. Ecotoxicology and Environmental Safety,66(2), 148–153.

Attanasi, A., Cavagna, A., Del Castello, L., Giardina, I., Jelic, A.,Melillo, S. (2013). Tracking in three dimensions via multi-pathbranching. Retrieved from. arXiv:1305.1495

Becco, C., Vandewalle, N., Delcourt, J., Poncin, P. (2006). Experi-mental evidences of a structural and dynamical transition in fishschool. Physica A: Statistical Mechanics and its Applications, 367,487–493.

Boyle, R. D., & Thomas, R. C. (1990). Computer vision. A first course,2nd Ed. Oxford, UK: Blackwell.

Butail, S., Bartolini, T., Porfiri, M. (2013). Collective response ofzebrafish shoals to a free-swimming robotic fish. PloS One, 8(10),e76123.

Butail, S., & Paley, D. A. (2012). Three-dimensional reconstructionof the fast-start swimming kinematics of densely schooling fish.Journal of the Royal Society Interface, 9(66), 77–88.

Couzin, I. D., Ioannou, C. C., Demirel, G., Gross, T., Torney, C. J.,Hartnett, A. (2011). Uninformed individuals promote democraticconsensus in animal groups. Science, 334, 1578–1580.

Delcourt, J., Becco, C., Vandewalle, N., Poncin, P. (2009). A videomultitracking system for quantification of individual behavior ina large fish shoal: Advantages and limits. Behavior ResearchMethods, 41(1), 228–235.

Delcourt, J., Becco, C., Ylieff, M. Y., Caps, H., Vandewalle, N.,Poncin, P. (2006). Comparing the EthoVision 2.3 system and anew computerized multitracking prototype system to measure theswimming behavior in fry fish. Behavior Research Methods, 38,704–710.

Delcourt, J., Denoel, M., Ylieff, M., Poncin, P. (2013). Video mul-titracking of fish behaviour: A synthesis and future perspectives.Fish and Fisheries, 14(2), 184–204.

Delcourt, J., & Poncin, P. (2012). Shoals and schools: Back to theheuristic definitions and quantitative references. Reviews in FishBiology and Fisheries, 22(3), 595–619.

FishBase (2014). Danio rerio and Hyphessobrycon herbertaxelrodi.Retrieved from http://www.fishbase.org

Gerlai, R., Ahmad, F., Prajapati, S. (2008). Differences inacute alcohol-induced behavioral responses among zebrafishpopulations. Alcoholism, Clinical and Experimental Research, 32,1–11.

Gerlai, R., Lahav, M., Guo, S., Rosenthal, A. (2000). Drinks like a fish:Zebra fish (Danio rerio) as a behavior genetic model to study alco-hol effects. Pharmacology, Biochemistry and Behavior, 67, 773–782.

Guo, S. (2004). Linking genes to brain, behavior and neurologicaldiseases: What can we learn from zebrafish? Genes, Brain andBehavior, 3, 63–74.

IDS, GmbH. (2010). uEye Cockpit (version 4.20). Obersulm, Germany.http://www.mediacy.com/

Kato, S., Nakagawa, T., Ohkawa, M., Muramoto, K., Oyama, O.,Watanabe, A. (2004). A computer image processing system forquantification of zebrafish behavior. Journal of NeuroscienceMethods, 134(1), 1–7.

Kissling, G., Bernheim, N., Hawkins, W., Wolfe, M., Jokinen, M.,Smith, C. (2006). The utility of the guppy (Poecilia reticulata)and medaka (Oryzias latipes) in evaluation of chemicals forcarcinogenicity. Toxicological Sciences, 92, 143–146.

Lee, A. (2013). VirtualDub (version 1.19.11). http://www.virtualdub.org/

Leem, J., Jeon, W., Yun, C., Lee, S. (2012). Quantitative analysisof fish schooling behavior with different numbers of medaka(Oryzias latipes) and goldfish (Carassius auratus). Ocean ScienceJournal, 47(4), 445–451.

Media Cybernetics, Inc. (2009). Image-Pro Plus (Version 7.0.1). MD,USA: Bethesda.

Miller, N., & Gerlai, R. (2012). Automated tracking of zebra fishshoals and the analysis of shoaling behavior. In A. V. Kalueff,& A. M. Stewart (Eds.), Zebrafish protocols for neurobehav-ioral research (Vol. 66, pp. 217–230). New York: HumanaPress.

Parrish, J. K., Viscido, S. V., Grunbaum, D. (2002). Self-organizedfish schools: An examination of emergent properties. BiologicalBulletin, 202, 296–305.

Perez-Escudero, A., Vicente-Paige, J., Hinz, R. C., Arganda, S., DePolavieja, G. G. (2014). idTracker: Tracking individuals in a groupby automatic identification of unmarked animals. Nature Methods,11, 743–748.

Pertusa, J. F. (2010). Tecnicas de analisis de imagen. Aplicacionesen biologıa [Image analysis techniques. Applications in biol-ogy], 2nd Ed. Valencia: Spain: Publicacions de la Universitat deValencia.

Pitcher, T. J., & Parrish, J. K. (1993). Functions of Shoaling Behaviorin Teleosts. In T. J. Pitcher (Ed.), Behaviour of teleost fishes, 2ndEd, (pp. 363–440). London: Chapman & Hall.

Radakov, D. V. (1973). Schooling in the ecology of fish. New York:Halsted Press.

Rasband, W. (2014). ImageJ (version 1.45s). http://imagej.nih.gov/ij/Reebs, S. G. (2000). Can a minority of informed leaders determine the

foraging movements of a fish shoal. Animal Behaviour, 59, 403–409.

Sison, M., Cawker, J., Buske, C., Gerlain, R. (2006). Fishing for genesinluencing vertebrate behavior: Zebrafish making headway. LabAnimal, 35, 33–39.

Stewart, A., Nguyen, M., Wong, K., Poudei, M., Kalueff, A. V.(2014). Developing zebrafish models of autism spectrum disorder(ASD). Progress in Neuro-Psychopharmacology and BiologicalPsychiatry, 50, 27–36.

Sumpter, D. (2010). Collective animal behavior. Princeton: PrincetonUniversity Press.

Wang, X., Williams, E., Haash, M., Dasmahapatra, A. (2006).Japanese medaka (Oryzias latipes): Developmental model for thestudy of alcohol teratology. Birth Defects Research (Part B):Developmental and Reproductive Toxicology, 77(1), 29–39.