Next Generation Data System & Collaborative Portal

November 2, 2006

James G Bellingham, Michael A GodinDorota Kolber, Lionel Pawlowski

Monterey Bay Aquarium Research Institute

Data/Collaboration Highlights• “Virtual Experiments” preceded the field

work• Most researchers remained at home

institutions.• More real-time model outputs• Standard products for comparing data• A variety of ways to share findings and

data– Collaborative system allowed investigators to

post analyses and/or link to own sites

2006/11/02 Bellingham & Godin

• Automated data collection system – Included data streams from outside the experiment– Often fragile with respect to source data changes– Mostly worked, but still finding some problems

(thanks, Wayne!)

• Non-ASAP data originators often ‘last minute’– Difficult to handle– Some errors in common data format resulted

• Collaborative portal used in lieu of RTOC meetings

2006/11/02 Bellingham & Godin

Data/Collaboration Highlights

Data System Throughput

From July 20 to September 14, 2006:

• 75,406 raw files (160 GB) collected by central server.

• 187,494 common format files (129 GB) created.

• 38 data formats handled

• 2200 GB data served

2006/11/02 Bellingham & Godin

MB2006 Data Collection7/6 7/13 7/20 7/27 8/3 8/10 8/17 8/24 8/31 9/7 9/14

mbari_auv_dorado389codar_mybg

calpoly_remus_auv_locothompson_ctd_loco

whoi_glider_loco_we04whoi_glider_loco_we11

ncom_icon_mbaysio_glider_asap_05_mean

ausi_sauv03_locosio_glider_asap_12_meansio_glider_asap_11_meansio_glider_asap_13_mean

mbari_drifter_stella07nrl_coamps_aosn_ocean

uw_triaxus_aesopnps_aircraft_asap

whoi_glider_asap_we02ant_glider_01_ctd_upsant_glider_02_ctd_ups

whoi_glider_asap_we07whoi_glider_asap_we09whoi_glider_asap_we08whoi_glider_asap_we12

roms_asap_mbaywhoi_glider_asap_we10whoi_glider_asap_we05

pt_sur_ctd_chavezhops_mbay_12h

hops_anonuevo_12hhops_mbay

hops_anonuevosio_seasoar_aesop

mbari_drifter_stella01mbari_drifter_stella02mbari_drifter_stella03mbari_drifter_stella04mbari_drifter_stella05mbari_drifter_stella06mbari_drifter_stella08mbari_drifter_stella09mbari_drifter_stella10

pt_sur_aesopnps_adcp1_asapnps_adcp2_asap

calpoly_glider_ru06_asapuw_seaglider_106_ctd_upsuw_seaglider_022_ctd_upsuw_seaglider_023_ctd_ups

new_horizon_svppt_sur_udas

LOCO ASAP UPS/PLUSNetAESOP Other

2006/11/02 Bellingham & Godin

2006/08 Overall Usage

Teleconference

Dow

nloa

dsV

isits

/Site

sH

its

2006/11/02 Bellingham & Godin

Countries that accessed the site

Argentina Czech Republic Italy Portugal

Australia Denmark JapanRussian Federation

Belgium Estonia Korea (South) Sweden

Brazil Finland Mexico Switzerland

Canada France Netherlands Taiwan

Chile Germany Network Turkey

Cocos Islands Hungary New Zealand

United Kingdom

Colombia India Norway United States

Costa Rica Ireland Poland

Collaborative Portal Usage

• Distributed observatory management achieved

• Usage levels varied from day to day

• Tools other than portal were sometimes used

• More clarity in decisions on days when COOP tool was used heavily.

2006/11/02 Bellingham & Godin

2006/08 Collaborative Portal Usage

Teleconference

Missed Teleconference Press Day No Call Scheduled

Pro

posa

lsP

ostin

gsLo

gins

2006/11/02 Bellingham & Godin

2006 Experiment Lessons

• Standard visualizations are very useful

• Virtual experiments are useful for deciding upon standard visualizations

• Common format data needs formal, continuous QC

• Need versioning on both source (raw) data and common format data

• Collaborative tools should interoperate with familiar tools (e-mail lists, teleconference)

2006/11/02 Bellingham & Godin

Data System Thoughts• Add archive data (MUSE-2000, AOSN-2003)

• Add missing/revised MB2006 data

• Refine access controls

• Continue as “ad hoc observatory”– All available data to 500km west of Monterey

• Apply in other field programs and observatories

• Allow real-time queries/visualizations (Metadata Oriented Query Assistant – MOQUA)

2006/11/02 Bellingham & Godin

User Feedback

• Data System– What worked / did not work for you?– What would you like to see in the future?

• Standard Products – can they be improved?• Portal

– What worked / did not work for you?– What would you like to see in the future?

2006/11/02 Bellingham & Godin

COOP thoughts

• Data system needs be brought into collaboration space– MOQUA as portlet?– Visualization portlet?– Produce movies of standard products?– Ability to share data by passing ‘pointers?’– Ability to set up custom products or alerts?

COOP thoughts• Needs to scale gracefully as duration,

data volume, and number of users grow– Provide overview of COOP activity (e.g.,

when is there lots of discussion, when is there a lot of interrogation of the site, what is the favorite page of the moment, etc. etc.)

– Search capability (when did a particular individual post material to COOP)

– Pivot off of something other than day– Allow users to break into separate

‘experiment spaces?’

COOP thoughts

• Personalize COOP– News update for you when you log on (e.g.

everyone voted on a new plan on the 12th, a particular person posted a message, etc.)

– News flashes email to you (email version of above, or email version of update)?

– Custom comparison products produced for you in your private workspace?

– Alerts triggered off of observations or data (e.g. winds predicted to be from the NW tomorrow)

Entrain talent

• Computer science community very excited about COOP

• Visualization – e.g. Donna Cox, UW effort using World Wind

• Work flow – e.g. KEPLER

• Data systems – e.g. MBARI Shore Side Data System

• Data systems – metadata standards…