A tool to dynamically merge corresponding weather data to crop data

Xinwen Yu, Seishi Ninomiya, Atsushi Yamakawa

National Agriculture Research Center, Tsukuba, Japan

Dead Storage Data Issue A lot of digital data sets are continuously produced in

agricultural experimental stations Using ordinal software such as spread sheet applications But they are likely to be kept in local stations and scientist level

The data sets are isolated and hardly integrated among different locations

How to ease data publication for merging and sharing for end users

Different resource integration issue Separated crop data are hard to be

integrated with different resources, e.g. weather data

Heterogeneity e.g. Models constructed using local

data are only applicable locally How to integrate crop data with

weather data under user’s request

If merging and sharing are possible,

End users can freely combine separated data sets from different locations and perform analysis on them

Datamining over the huge amount of data sets becomes real and we can possibly find out unknown facts

Integration with completely different resources e.g. weather data becomes also possible

Constructing global model also becomes possible

With Web-based data sharing & integration

Once you upload experimental data in spreadsheet file (e.g. MS Excel), data automatically become sharable over the Internet among different locations

Crop data can be integrated with weather data No skill is required Just a part of everyday data management Uniformity of tables is not required Low cost

Internet

DBMS

Application Server

Servlet Container

EJB Container

Application

Concept

Multi-location data sharing and integration Integration with weather data

MetBroker

Basic structure of application

Google Map

Client

Brow

ser

Controller

Crop DBCrop Data Service

Model

ServicesWeb application

MetBroker

AMeDAS

AMeDAS

AMeDAS

View

System components

Java Runtime Environment 1.5.04

PostgreSQL7.4

JRE supported OS

JBoss-4.0.3

EJB3.0(DBMS abstraction)

Struts1.2(Web Interface)

・ IE, Firefox, etc.・ Excel2002, newer

Main menu Very simple menu structure Simple and clean user interface

Data upload

Crop Data ServiceEJB3

SourceXML

CropData

HistoryData transforming

Data validating

XSLT style sheet

Data Schema

a

b

c

Crop Database

a. Transformation of source data file using style XSLT style sheet, storing source data file in XML to SourceXML table.

b. Validating transformed data XML using data schema c. Data service binds data XML to data object (JAXB), maps data object to data records t

o CropData table; stores uploaded information including user name, spreadsheet file name, uploading time, count of data record to History table.

Upload history

Web Application

Crop Data Service EJB3

Crop db

Specifying query conditions then executing data query.

Browsing and/or download queried crop data

Data Query

AMeDAS

Web Application

Crop Data

Location Table

Crop Data Service EJB3

MetBroker

ab

AMeDASAMeDAS

cd

Data integration

Location registration

Weather station selection

Discussion

37 spreadsheet files uploaded 151072 records 6486 varieties 184 properties 256 different experiment locations 1980-2004

Available to open society

Data integration is realized in our brokerage based Data Grid MetBroker is the core of the Data Grid

The application demonstrates the power of Data Grid

This approach can be used to all other crop data sharing and integration

Crop modelers are using this application to develop robust crop models

Thank you for your attention