nowcasting with neural network using reflectivity images of meteorological radar

INSTITUTO DE PESQUISAS METEOROLÓGICASINSTITUTO DE PESQUISAS METEOROLÓGICAS

NOWCASTING WITH NEURAL NOWCASTING WITH NEURAL NETWORKNETWORK

USING REFLECTIVITY IMAGES OFUSING REFLECTIVITY IMAGES OF

METEOROLOGICAL RADARMETEOROLOGICAL RADAR

R. Machado (1,2) C. A. Thompson (2)

R. V. Calheiros (1)

(1) Meteorological Research Institute (IPMet)/UNESP, Bauru, SP, 17033-360, Brazil

(2) Polytechnic Institute (IPRJ)/UERJ, Nova Friburgo, RJ, 28601-970, Brazil

Lençóis Paulista/SP, may,25-04, 20UTC – F2

Palmital/SP, may,25-04, 17UTC – F3

Indaiatuba/SP, may,24-05, 20:30UTC – F3


OBJECTIVES & STATUSOBJECTIVES & STATUS

GENERAL :GENERAL : SUPPORT TO OPERATIONAL SUPPORT TO OPERATIONAL NOWCASTING IN CENTRAL SÃO PAULONOWCASTING IN CENTRAL SÃO PAULO

SPECIFIC :SPECIFIC : IMPLEMENT A NEURAL IMPLEMENT A NEURAL NETWORK APPROACH TO IPMET’S NETWORK APPROACH TO IPMET’S OPERATIONAL FORECASTING PRACTICESOPERATIONAL FORECASTING PRACTICES

AS OF NOW:AS OF NOW: PRELIMINARY TESTS OF PRELIMINARY TESTS OF NETWORK NETWORK PERFORMANCE RUN FOR PERFORMANCE RUN FOR DISTINCT COMPUTATION OF AVERAGES ON DISTINCT COMPUTATION OF AVERAGES ON STATISTICAL TEXTURE DESCRIPTORSSTATISTICAL TEXTURE DESCRIPTORS


DATA & AREADATA & AREA PRODUCT:PRODUCT: REFLECTIVITY CAPPIS REFLECTIVITY CAPPIS

AT 3,5 KM HEIGHT AGL, TO A 240 AT 3,5 KM HEIGHT AGL, TO A 240 KM RANGE FROM THE BAURU KM RANGE FROM THE BAURU RADAR (BRU)RADAR (BRU)

PERIOD:PERIOD: SUMMERS OF 2002/2003 & SUMMERS OF 2002/2003 & 2003/20042003/2004

DATA SET:DATA SET: 300 IMAGES GATHERED 300 IMAGES GATHERED IN INTERVALS OF 2 HOURS EACH, IN INTERVALS OF 2 HOURS EACH, COMPOSING TWO SUB SETS: COMPOSING TWO SUB SETS: CHARACTERIZED BY (1) RAIN AND CHARACTERIZED BY (1) RAIN AND (2) NORAIN SITUATION AT THE END (2) NORAIN SITUATION AT THE END OF THE TIME INTERNALOF THE TIME INTERNAL

DATA SAMPLE:DATA SAMPLE:

TARGET AREA: TARGET AREA: 15 KM RADIUS 15 KM RADIUS CIRCLE AROUND THE RADARCIRCLE AROUND THE RADAR


PROCESSINGPROCESSINGA)A)

•STATISTICAL TEXTURE DESCRIPTORS (ATTRIBUTES), STATISTICAL TEXTURE DESCRIPTORS (ATTRIBUTES), I.E. MEAN, STD DEVIATION, SKEWNESS AND KURTOSIS I.E. MEAN, STD DEVIATION, SKEWNESS AND KURTOSIS WERE COMPUTED FOR EACH IMAGEWERE COMPUTED FOR EACH IMAGE

•AVERAGES OF THE ATTRIBUTES WERE CALCULATED AVERAGES OF THE ATTRIBUTES WERE CALCULATED FOR ALL IMAGES WITHIN EACH 2 H INTERVALFOR ALL IMAGES WITHIN EACH 2 H INTERVAL

•RESULTED TWO SETS OF ATRIBUTE VECTORS: ONE RESULTED TWO SETS OF ATRIBUTE VECTORS: ONE CORRESPONDING TO THE PROCESSING OF EACH CORRESPONDING TO THE PROCESSING OF EACH IMAGE, AND THE OTHER FOR THE AVERAGE VALUES IMAGE, AND THE OTHER FOR THE AVERAGE VALUES OF EACH ATTRIBUTE (SEE TABLE 1)OF EACH ATTRIBUTE (SEE TABLE 1)


TABLE 1 : ATTRIBUTES FOR A SAMPLE TABLE 1 : ATTRIBUTES FOR A SAMPLE EVENT 30-JAN-2004 FROM THE DATA EVENT 30-JAN-2004 FROM THE DATA

BANK ( ALL TIMES LT = UTC – 3)BANK ( ALL TIMES LT = UTC – 3)

OBS.: STATISTICS WERE COMPUTED ON THE IMAGE PIXELS IN mm.hOBS.: STATISTICS WERE COMPUTED ON THE IMAGE PIXELS IN mm.h-1 -1

DERIVED WITH Z = 300RDERIVED WITH Z = 300R1,41,4

TIME EVENTTIME EVENT

ATTRIBUTEATTRIBUTE12:5312:53 13:2313:23 13:5313:53 14:2314:23 AVERAGESAVERAGES

MEANMEAN 0.15290.1529 0.20590.2059 0.26290.2629 0.26750.2675 0.22230.2223

STD STD DEVIATIONDEVIATION

-1.5224-1.5224 1.81061.8106 1.94871.9487 2.05322.0532 1.83371.8337

SKEWNESSSKEWNESS -0.1604-0.1604 -1.3625-1.3625 -1.9867-1.9867 -1.8567-1.8567 -1.3415-1.3415

KURTOSISKURTOSIS 9.11189.1118 8.17918.1791 6.22896.2289 5.25135.2513 7.19277.1927


B)B)

• ATRIBUTE VECTORS WERE USED AS INPUTS TO A NEURAL NETWORK ATRIBUTE VECTORS WERE USED AS INPUTS TO A NEURAL NETWORK CONSTITUTED OF 2 LAYERS, WITH 4 NEURONS IN THE HIDDEN LAYER AND CONSTITUTED OF 2 LAYERS, WITH 4 NEURONS IN THE HIDDEN LAYER AND 1 NEURON IN THE OUTPUT LAYER1 NEURON IN THE OUTPUT LAYER

LINE COMMAND WAS: LINE COMMAND WAS: net2=newff(minmax(p),[4,1],{‘logsig’,’logsig’},’traingda’);net2=newff(minmax(p),[4,1],{‘logsig’,’logsig’},’traingda’);

Newff = NETWORK WITH BACK-PROPAGATIONNewff = NETWORK WITH BACK-PROPAGATION

[4,1] =[4,1] = TWO LAYERS, 4 NEURONS IN THE HIDDEN LAYER AND 1 NEURON IN THE TWO LAYERS, 4 NEURONS IN THE HIDDEN LAYER AND 1 NEURON IN THE OUTPUT LAYER; ANDOUTPUT LAYER; AND

logsig = logsig = TRANSFER FUNCTION OF EACH NEURON, DIFFERENTIABLE, WITH TRANSFER FUNCTION OF EACH NEURON, DIFFERENTIABLE, WITH OUTPUT BETWEEN 0 AND 1.OUTPUT BETWEEN 0 AND 1.

• RESULTS OF RUNS WITH DIFFERENT NEURAL NETWORKS ARE RESULTS OF RUNS WITH DIFFERENT NEURAL NETWORKS ARE DEMONSTRATED FOR TWO OF THEMDEMONSTRATED FOR TWO OF THEM

• TRAINNING WAS EFFECTED FOR OUTPUT VALUES BETWEEN O AND 1, AS:IF TRAINNING WAS EFFECTED FOR OUTPUT VALUES BETWEEN O AND 1, AS:IF OUTPUT OUTPUT ≥ 0.5 → RAIN, OUTPUT< 0.5 → NO-RAIN≥ 0.5 → RAIN, OUTPUT< 0.5 → NO-RAIN

(OBS. AN AREA PROBABILITY OF RAIN, a, IF δi IS AN INDICATOR VARIABLE (OBS. AN AREA PROBABILITY OF RAIN, a, IF δi IS AN INDICATOR VARIABLE EQUAL TO 1 WHEN RAINS OCCURS AT A POINT 1, AND ZERO, IS EQUAL TO 1 WHEN RAINS OCCURS AT A POINT 1, AND ZERO, IS

FOR BRU, ECHO STATISTICS INDICATES a ~ 0.55 FOR SUMMER).FOR BRU, ECHO STATISTICS INDICATES a ~ 0.55 FOR SUMMER).

0

i

m

1i

Pa


• FIRST NETWORKFIRST NETWORK

1.1. TRAINING: PERFORMANCE RESULTTRAINING: PERFORMANCE RESULT


2.2.SIMULATION: SIMULATION: ATTRIBUTES FOR EACH ATTRIBUTES FOR EACH IMAGE WERE USED. THE FIRST 15 IMAGES IMAGE WERE USED. THE FIRST 15 IMAGES

WERE KNOWN TO RESULT IN RAIN, AND THE WERE KNOWN TO RESULT IN RAIN, AND THE LAST 15 IMAGES IN NO-RAIN,LAST 15 IMAGES IN NO-RAIN,

FIRST 15 IMAGES (VALUES SHOULD BE ≥ 0.5)FIRST 15 IMAGES (VALUES SHOULD BE ≥ 0.5)

0.83050.8305 0.79780.7978 0.7890.789 0.78820.78820.23810.2381

0.23160.2316 0.39010.3901 0.4210.421 0.8970.8970.74150.7415

0.57780.5778 0.60960.6096 0.8950.895 0.86030.86030.84770.8477

LAST 15 IMAGES (VALUES SHOULD BE < 0.5)LAST 15 IMAGES (VALUES SHOULD BE < 0.5)

0.15950.1595 0.11320.1132 0.240.24 0.3220.3220.27450.2745

0.22050.2205 0.20510.2051 0.1930.193 0.27260.27260.32010.3201

0.3030.303 0.25590.2559 0.30320.3032 0.31140.31140.26160.2616

TABLE 2 – RESULTS AT THE OUTPUT OF THE FIRST NETWORKTABLE 2 – RESULTS AT THE OUTPUT OF THE FIRST NETWORK

ERRORS (VALUE IN ERRORS (VALUE IN REDRED) = 4/30 ) = 4/30 13% ≅ 13% ≅ 87% OF SUCCESS 87% OF SUCCESS


•SECOND NETWORK (TRAINING NOT SHOWN)SECOND NETWORK (TRAINING NOT SHOWN)

ERRORS (VALUES IN ERRORS (VALUES IN REDRED) = 1/30 ) = 1/30 3% ≅ 3% ≅ 97% OF SUCCESS 97% OF SUCCESS

FIRST 15 IMAGES (VALUES SHOULD BE ≥ 0.5FIRST 15 IMAGES (VALUES SHOULD BE ≥ 0.5

0.73750.7375 0.66750.6675 0.80930.80930.85640.8564 0.42180.4218

0.67520.6752 0.84110.8411 0.90480.90480.5660.566 0.81660.8166

0.9720.972 0.87330.8733 0.95030.9503 0.95610.95610.94850.9485

LAST 15 IMAGES (VALUES SHOULD BE < 0.5LAST 15 IMAGES (VALUES SHOULD BE < 0.5

0.37490.3749 0.3480.348 0.03040.0304 0.1660.1660.02230.0223

0.05190.0519 0.20770.2077 0.14440.14440.0540.054 0.02610.0261

0.22310.2231 0.1080.108 0.0810.081 0.28160.28160.2630.263

SAME ATTRIBUTES AND CRITERIA FOR RAIN (SAME ATTRIBUTES AND CRITERIA FOR RAIN (≥ 0.5) AND ≥ 0.5) AND NO-RAIN (< 0.5 ) AS THE FIRST NETWORK, BUT USING NO-RAIN (< 0.5 ) AS THE FIRST NETWORK, BUT USING AVERAGES OF THE ATTRIBUTES TAKEN OVER EACH 2H AVERAGES OF THE ATTRIBUTES TAKEN OVER EACH 2H INTERVAL.INTERVAL.

TABLE 3 – RESULTS AT THE OUTPUT OF THE SECOND NETWORKTABLE 3 – RESULTS AT THE OUTPUT OF THE SECOND NETWORK


CHI – SQUARE TESTCHI – SQUARE TEST

TWO SAMPLES (2 H INTERVALS) TAKEN. FOR THE FIRST xTWO SAMPLES (2 H INTERVALS) TAKEN. FOR THE FIRST x11=22, n=22, nII =30 AND =30 AND

FOR THE SECOND: xFOR THE SECOND: x22=79, n=79, n22=90, WHERE x=90, WHERE xi=1,2i=1,2 = RAIN, n = RAIN, n i=1,2i=1,2 = SAMPLE SIZE. = SAMPLE SIZE.

NULL HYPOTHESIS IS FORMULATED, I. E. RAIN/ NO - RAIN RELATIONS ARE NULL HYPOTHESIS IS FORMULATED, I. E. RAIN/ NO - RAIN RELATIONS ARE TRUE. x TRUE. x i = 1,2 i = 1,2 IS A RANDOM VARIABLE. MODELING ITS BINOMIAL IS A RANDOM VARIABLE. MODELING ITS BINOMIAL

DISTRIBUTION BY A NORMAL DISTRIBUTION, THE PROPORTION OF THE DISTRIBUTION BY A NORMAL DISTRIBUTION, THE PROPORTION OF THE

SAMPLE TAKEN BY xSAMPLE TAKEN BY x ((θθ = = XX / / nn)) IF UNKNOWN, IS IF UNKNOWN, IS

FOR THE MODELED NORMAL DISTRIBUTION ,FOR THE MODELED NORMAL DISTRIBUTION ,

IF THE RANDOM INDEPENDENT VARIABLES zIF THE RANDOM INDEPENDENT VARIABLES z11 AND z AND z2 2 HAVE STANDARD HAVE STANDARD

NORMAL DISTRIBUTION, THEN (y= zNORMAL DISTRIBUTION, THEN (y= z22 + z + z22) HAS A CHI-SQUARE ) HAS A CHI-SQUARE DISTRIBUTION WITH m DEGREES OF FREEDOM.DISTRIBUTION WITH m DEGREES OF FREEDOM.

•COMPUTING Y WITH THE ABOVE NUMBERS : COMPUTING Y WITH THE ABOVE NUMBERS : θθ = 0.84 y = 3.495 = 0.84 y = 3.495

•FOR FOR α α = 0.05 AND = 0.05 AND υυ((mm) = 2 DEGREES OF FREEDOM ) = 2 DEGREES OF FREEDOM = 5.991(>3.495) = 5.991(>3.495)

•NULL HYPOTHESIS IS SATISFIED TO 95% OF CONFIDENCE, I. E., NULL HYPOTHESIS IS SATISFIED TO 95% OF CONFIDENCE, I. E., FORECASTS ARE NOT BIASED.FORECASTS ARE NOT BIASED.

225.0 ;

^̂

212,1

1

i

iii

n

nx ^̂

^̂ ^̂


NEXT STEPSNEXT STEPSA)A) IMPROVEMENT OF PERFORMANCEIMPROVEMENT OF PERFORMANCE

A . 1) ADD NEW TECHNIQUES, E. G.A . 1) ADD NEW TECHNIQUES, E. G.

• GABOR FILTERING AS A FIRST LAYER IN THE NETWORK SYSTEM TO GABOR FILTERING AS A FIRST LAYER IN THE NETWORK SYSTEM TO EXTRACT TEXTURAL FEATURES, WHICH WILL FEED THE INPUT LAYER EXTRACT TEXTURAL FEATURES, WHICH WILL FEED THE INPUT LAYER OF THE FORECASTING NETWORKOF THE FORECASTING NETWORK

(GABOR FILTERING HAS SHOWN TO IMPROVE THE PERFORMANCE OF (GABOR FILTERING HAS SHOWN TO IMPROVE THE PERFORMANCE OF NEURAL NETWORKS)NEURAL NETWORKS)

• FUZZY LOGIC, DUE TO THE FACT THAT THERE IS NO CLEAR FUZZY LOGIC, DUE TO THE FACT THAT THERE IS NO CLEAR SEPARATION BETWEEN SEASONS, DAILY INTERVALS, AND OTHER SEPARATION BETWEEN SEASONS, DAILY INTERVALS, AND OTHER STRAFICATION FACTORS.STRAFICATION FACTORS.

• GENETIC ALGORITHMS, WHICH USE TECHNIQUES OF BIOLOGICAL GENETIC ALGORITHMS, WHICH USE TECHNIQUES OF BIOLOGICAL DERIVATION THAT COULD BE APPLIED TO RAINFALL DERIVATION THAT COULD BE APPLIED TO RAINFALL CONFIGURATIONS SUCH AS : HERITAGE ( RAIN AT TCONFIGURATIONS SUCH AS : HERITAGE ( RAIN AT T0 0 IS RELATED TO TIS RELATED TO T00

– 1), MUTATION ( RAIN PATTERNS CHANGE STRUCTURE IN TIME), – 1), MUTATION ( RAIN PATTERNS CHANGE STRUCTURE IN TIME), NATURAL SELECTION (PREFERENTIAL DEVELOPMENT CONDITIONS NATURAL SELECTION (PREFERENTIAL DEVELOPMENT CONDITIONS EXIST), AND RECOMBINATIONS ( RAIN CELLS SPLIT AND MERGE IN EXIST), AND RECOMBINATIONS ( RAIN CELLS SPLIT AND MERGE IN TIME)TIME)


A.2) ADD NEW (METEOROLOGICAL) A.2) ADD NEW (METEOROLOGICAL) ATTRIBUTES, E. G.ATTRIBUTES, E. G.

• DOPPLER RADAR WINDSDOPPLER RADAR WINDS

• SATELLITE IMAGES (VIS, IR, WV & MW) SATELLITE IMAGES (VIS, IR, WV & MW) INDIVIDUALLY OR IN COMBINATIONS TO INDIVIDUALLY OR IN COMBINATIONS TO INFER, E. G. RAIN/NO - RAIN THRESHOLD.INFER, E. G. RAIN/NO - RAIN THRESHOLD.

• VARIABLES, LIKE TEMPERATURE, VARIABLES, LIKE TEMPERATURE, PRESSURE, HUMIDITY.PRESSURE, HUMIDITY.


B) VERIFICATION/VALIDATIONB) VERIFICATION/VALIDATION

COMPARISONS WITH OTHER NOWCASTING TECHNIQUES EITHER COMPARISONS WITH OTHER NOWCASTING TECHNIQUES EITHER IN TESTS OR OPERATIONAL, OR IN CONSIDERATION FOR IN TESTS OR OPERATIONAL, OR IN CONSIDERATION FOR OPERATIONAL USE, AT IPMET FORECASTING SECTOR.OPERATIONAL USE, AT IPMET FORECASTING SECTOR.

B.1) TITAN (THUNDERSTORM IDENTIFICATION, TRACKING, B.1) TITAN (THUNDERSTORM IDENTIFICATION, TRACKING, ANALYSIS AND NOWCASTING) PREDICTING ECHO CENTROID ANALYSIS AND NOWCASTING) PREDICTING ECHO CENTROID POSITION EVOLUTION. STATUS: UNDER OPERATIONAL POSITION EVOLUTION. STATUS: UNDER OPERATIONAL EVALUATIONEVALUATION

B.2) KAVVAS (ADAPTIVE EXPONENTIAL METHOD) PREDICTING B.2) KAVVAS (ADAPTIVE EXPONENTIAL METHOD) PREDICTING SHORT-TERM EVOLUTION (15 MIN. TO 2 H) OF CENTROID, BASED SHORT-TERM EVOLUTION (15 MIN. TO 2 H) OF CENTROID, BASED ON REFLECTIVITY AND VELOCITY (DOPPLER). STATUS: UNDER ON REFLECTIVITY AND VELOCITY (DOPPLER). STATUS: UNDER STUDYSTUDY

B.3) VIL (VERTICALLY INTEGRATED LIQUID WATER CONTENT) B.3) VIL (VERTICALLY INTEGRATED LIQUID WATER CONTENT) PREDICTOR IS WATER COLUMN FROM GROUND TO 12 KM AGL PREDICTOR IS WATER COLUMN FROM GROUND TO 12 KM AGL COMBINED WITH PRESENCE OF 45 dBZ ABOVE 3 KM. STATUS : COMBINED WITH PRESENCE OF 45 dBZ ABOVE 3 KM. STATUS : OPERATIONALOPERATIONAL


CONCLUSIONSCONCLUSIONS

•NEURAL NETWORK APPROCH TO RADAR BASED NEURAL NETWORK APPROCH TO RADAR BASED NOWCASTING IN CENTRAL SÃO PAULO HAS SHOWN CLEAR NOWCASTING IN CENTRAL SÃO PAULO HAS SHOWN CLEAR POTENTIAL.POTENTIAL.

•STATISTICAL TEXTURE DESCRIPTORS HAVE PROVEN A STATISTICAL TEXTURE DESCRIPTORS HAVE PROVEN A VALID INPUT TO THE NOWCASTING WITH NEURAL VALID INPUT TO THE NOWCASTING WITH NEURAL NETWORK IN CENTRAL SÃO PAULO.NETWORK IN CENTRAL SÃO PAULO.

•IMPROVEMENTS RESULTING FROM AVERAGING IMPROVEMENTS RESULTING FROM AVERAGING DESCRIPTOR VALUES INDICATES THAT EVEN RELATIVELY DESCRIPTOR VALUES INDICATES THAT EVEN RELATIVELY MINOR OPERATIONS ON IMAGE CHARACTERISTICS CAN MINOR OPERATIONS ON IMAGE CHARACTERISTICS CAN SIGNIFICANTLY IMPACT NETWORK PERFORMANCE.SIGNIFICANTLY IMPACT NETWORK PERFORMANCE.

•FURTHER IMPROVEMENTS SHOULD BE PARTICULARLY FURTHER IMPROVEMENTS SHOULD BE PARTICULARLY EXPECTED FROM TEXTURE CLASSIFICATION THROUGH EXPECTED FROM TEXTURE CLASSIFICATION THROUGH GABOR FILTERING.GABOR FILTERING.

nowcasting with neural network using reflectivity images of meteorological radar

Documents

images values

output values

shownerrors values

average values

neural network approach

output layerline command

area probability of

successsecond network