hydrologic information use and needs assessment utah state university examples of existing systems...

Hydrologic Information Use and Needs Assessment

Utah State University

• Examples of Existing Systems

• User Needs Assessment Survey Results

David G Tarboton

with input from

Jeff Horsburgh, Rob Gillies, Christina Bandaragoda, John Weeks, Doug Ramsey

Some existing hydrology related information systems at USU

• Bear River Watershed Information System

• Intermountain Region Digital Image Archive (IRDIAC)

• Climate and Weather Data Server

• UNIDATA port

Internet Based WISBear River http://water.usu.edu/wis/bear_river/Great Salt Lake http://greatsaltlake.utah.edu/

• Web browser as client software

• All databases and data analysis managed on master server

• ESRI’s ArcIMS powers the map server and provides visualization of GIS datasets

• SQL Server and Time Series Analyst manage time series data and provide access to analysis and visualization

http://water.usu.edu/wis/bear_river/

http://greatsaltlake.utah.edu/

Clic

k

Servers• Web Server Dell PowerEdge 700, $3000• Database Server Dell PowerEdge 2800, $3000• Base Station for real time data collection, Pentium III 600

MHz, < $1000 now.

Operating Systems• Windows Server 2003 and Windows XP

Software• IIS v6.0• .Net Framework v1.1• ProEssentials v5 (for time series plotting)• ArcIMS v4.0• ArcGIS • Microsoft SQL Server 2000 Standard Edition

Disk Space• 223 GB hard drive/~20 GB used

Communication• 100Mb connection to USU internet

Operators• Jeff Horsburgh – Research Engineer with expertise in

data analysis, data manipulation, GIS, modeling, custom GIS application development, etc.

• Rick Satterfield – Graduate Student system programmer with expertise in developing and deploying Internet applications

• Crystal Yap – data technician (downloading, formatting, etc.)

• Amber Spackman – data technician (downloading, formatting, etc.)

http://earth.gis.usu.edu

http://earth.gis.usu.edu/

http://earth.gis.usu.edu/

Servers• Sunfire 280R server with 2 1.2 GHz processors and 8GB

RAM, ~$20,000• Partnership in Beowulf Linux Cluster• Windows Server to support ArcGIS geodatabase

processing

Operating Systems• Solaris, Linux, Windows Server XP

Software• MSQL• Apache and Tomcat web services• MapServer, PHP, PHP Mapscript• GDAL and IDL/ENVI for image processing• ESRI Geodatabase Server

Disk Space• 2TB at 95% capacity

Communication• 1GB connection to USU internet

Operators• 1 FTE programmer/analyst with knowledge of C, PHP, IDL, GIS

programming languages and scripts. • ½ FTE web developer • ¼ - ½ FTE System Administrator with knowledge in Solaris, Linux,

and PC networking architecture, security, installation and integration if 3rd- party software, etc.

• ½ FTE Project Manager to oversee and manage daily activities.• ¼ – ½ FTE Project Director. • 3 student technicians (1.5 FTE) that organize, process imagery and

data, populate the image database, and provide programming and web support.

• 2 postdoctoral fellows to provide research support

Climate and Weather Data Server

Integrated Weather and Climate Database

Goals• Data Ingestion from Many Sources• WEB Accessible• User Friendly Navigation and Interfaces• User Friendly Data Output• High Quality Reports and Graphs• High Quality Maps• All available climate data in one location

Currently offline - undergoing restructuring!

Servers• One Windows PC, $3000, System Management

Front end • Two Redhat Linux PC, $3000, Apache Web

Servers • 1TB RAID Disk Array $6000 Operating Systems• Windows Server 2003 • RedHat Enterprise Linux

Software• PostgreSQL • Apache Web Services • In house developed applications

Disk Space• 120GB, 30% full • 250GB, 40% full • 1000GB (1TB), 60% full

Communication• 100MB connection to USU internet

Operators• Systems Administrator - MS Windows, RedHat Linux• Programmer to build custom utilities: Java, C, C++, Perl • Software Management and Maintenance - Third Party

or In house • Subject Matter Expert - Climate

UNIDATA- access and visualization of meteorological information

IDD Data Feed

Data is ingested using a client / server program - Local Data Manager.

Scripts are configured to select data through pattern matching techniques which define data requested and amount.

Currently USU ingests 1GB / hour.

Visualization and Analysis

• Several custom software packages are made available for analysis and visualization of data

• Interactive Data Viewer (IDV) - shown on left

• McIDAS• GEMPACK

Servers

• Solaris Ultra 10, < $500 now

Operating Systems

• Solaris

Software

• LDM

• Unidata

• McIdas

Disk Space• 250GB, 90% full

Communication• 10MB connection to USU internet

Operators• Systems Administrator - Solaris• Programmer to build custom utilities: Java, C,

C++, Perl • Software Management and Maintenance - Third

Party or In house • Subject Matter Expert - Climate

User Needs Assessment Survey Results

18 USU Respondents

Respondents

02

46

810

12

Fac

ulty

Gra

d S

tude

nt

Und

ergr

adS

tude

nt

Oth

er

Uni

v. P

ro.

Oth

er

Operating System

02

46

810

12

Win

dow

s

Mac

/OS

X

Linu

x

Sol

aris

Uni

x

Oth

er

Minutes to complete survey

02

46

810

12

5 10 15 20 25 30 35

Mean: 15.5

User Needs Assessment Survey Results

36 CUAHSI Respondents

Respondents

05

1015

2025

30

Fac

ulty

Gra

d S

tude

nt

Und

ergr

adS

tude

nt

Uni

vers

ityP

rofe

ssio

nal

Wor

king

Pro

fess

iona

l

Oth

er

Win

dow

s

MA

C/O

S X

Linu

x

Sol

aris

Uni

x

Oth

er

Operating System %

0

20

40

60

80

100

Minutes to complete survey

02

46

810

12

0 5 10 15 20 25 30

Mean: 15.5

CUASHI Survey Respondent Specialities0 2 4 6 8 10 12 14 16

Engineering:Hydrology/WaterResources

Engineering:Groundwater

Geology

Computer Science

Climate/Meteorology

Biology

GIS

Engineering: Electrical

Water Quality/Chemistry

Geomorphology

Software - USU

Other Software mentioned:

SAS - 3 Surfer- 2R, Groundwater Vistas, STAMMT-L, HEC-RAS, Adobe Illustrator, Kaleidagraph, MS SQL Server, ArcIMS, MapWindow, Macromedia, Minitab, SigmaPlot

Please rate each of the following software packages and programming languages with respect to how important they are for hydrologic analysis in your research.

4.5

3.2

2.4

2.3

2.3

2.2

2.0

1.8

1.8

1.6

1.3

1.1

Excel

ArcGIS/ArcView

C++

Visual Basic

Matlab

MS Access

FORTRAN

SPSS

S-Plus

Java

Mathematica

GRASS

Raw mean scores; n = 18

1=Never use or do not find useful; 2=Have used but do not rely on this; 3=Use occasionally and am comfortable with its use; 4=Use often; 5=Use frequently and find indispensable.

Software - USUPlease rate each of the following software packages and programming languages with respect to how important they are for hydrologic analysis in your research.

Excel

05

1015

1 2 3 4 5

Mean: 4.5

MS Access

05

1015

1 2 3 4 5

Mean: 2.24

Matlab

05

1015

1 2 3 4 5

Mean: 2.28

Mathematica

05

1015

1 2 3 4 5

Mean: 1.29

S-Plus

05

1015

1 2 3 4 5

Mean: 1.76

SPSS

05

1015

1 2 3 4 5

Mean: 1.82

ArcGIS/ArcView

05

1015

1 2 3 4 5

Mean: 3.17

GRASS

05

1015

1 2 3 4 5

Mean: 1.11

FORTRAN

05

1015

1 2 3 4 5

Mean: 2

Visual Basic

05

1015

1 2 3 4 5

Mean: 2.33

C++

05

1015

1 2 3 4 5

Mean: 2.39

Java

05

1015

1 2 3 4 5

Mean: 1.65

Software-CUAHSIRate the following software packages and programming languages with respect to how important they are for hydrologic analysis in your research

4.0

3.9

3.3

3.2

2.7

2.6

2.5

2.4

2.4

2.3

2.2

2.1

2.1

1.9

1.9

1.8

Excel

ArcGIS/ArcView

FORTRAN

C/C++

Java

MS Access

Visual Basic

Matlab

SQL/Server

Modflow

Adobe Illustrator

HEC models

GMS, WMS, SMS

R

SWAT

Sigma Plot

1 =Never use ordo not find

2=Have used, butdo not rely

5=Use frequentlyand find indispensable

4=Use often

3=Use occasionallyand am comfortable with its use


Other Software Mentioned• Rockware (2)• Imagine/IDL (2)• MySQL (2)• Perl (2)• RiverTools • SWS • Tcl/Tk(GUI) • Web – Python “2ope” • Visual/Modeling – IDL/PV-Ware • Global Mapper • TELEMAC• Virtualization – UML/VMware/etc• PHREEQC • Aqua Chem • IPW• GEMPAK• GRADS• Hydrus2D• DSSAT• Topmodel• ArcIMS• Pro Essentials

4.0

3.9

3.3

3.2

2.7

2.6

2.5

2.4

2.4

2.3

2.2

2.1

2.1

1.9

1.9

1.8

Excel

ArcGIS/ArcView

FORTRAN

C/C++

Java

MS Access

Visual Basic

Matlab

SQL/Server

Modflow

Adobe Illustrator

HEC models

GMS, WMS, SMS

R

SWAT

Sigma Plot

1 =Never use ordo not find

2=Have used, butdo not rely


4=Use often

3=Use occasionallyand am comfortable with its use


Software-CUAHSIRate the following software packages and programming languages with respect to how important they are for hydrologic analysis in your research

Rate the following software packages and programming languages with respect to how important they are for hydrologic analysis in

your research - CUAHSI

1.7

1.7

1.7

1.7

1.6

1.6

1.6

1.5

1.5

1.3

1.3

1.2

Surfer

SPSS

SAS

HSPF

S-Plus

GRASS

Visual Modflow

Mathematica

PostgreSQL

Tecplot

Groundwater Vistas

Kaleidagraph

1 =Never use ordo not find useful

2=Have used, butdo not rely on this


4=Use often3=Use occasionallyand am comfortable with its use


Please rate each software functionality with respect to the priority for including this functionality

in a hydrologic information system - USU

4.9

4.2

4.2

4.1

3.8

3.6

3.5

3.3

2.9

2.8

Data storage and retrieval

Visualization of spatial data

Visualization of time series data

Building relational links

Efficient coupling with 3rd party analysis software

Presentation to non-technical audiences

Development of publication quality figures

Numerical analysis

Multivariate statistical analysis

Univariate statistical analysis

1 =Low priority 5=High priority


Rate the software functionality with respect to the priority for including in a hydrologic information system - CUAHSI

4.5

4.4

4.2

4.1

4.1

4.1

4.1

4.0

3.9

3.9

3.9

3.9

3.7

3.4

Store and retrieve digital products from a hydrologicdigital library

Include GIS data on terrain, soils, land cover, geology,stream networks

Allow connection to hydrologic models

Include information from weather and climate models,and Nexrad

Design metadata and develop tools for preparing it

Support intelligent searching for hydrologic data,models, reports and papers

Include remote sensing information

Automatically harvest hydrologic observation data fromagency websites

Couple hydrologic fluxes among atmosphere, surfaceand groundwater

Provide access to real time hydrologic and weatherdata

Allow downloads from this database in delimited asciiformat

Synthesize national, state and local data into a singlespace-time database

Allow connection to statistical analysis systems

Include modeling capabilities for flow and transport

1 =Low priority 5=High priority


Independence - USU:Please indicate the following on a scale of 1-5, where

1=not important, 5=essential

all computer systems

02

46

810

12

1 2 3 4 5

Mean: 4.53

independently of 3rd party software

02

46

810

12

1 2 3 4 5

Mean: 3.82

HIS software should work on all computer systems

HIS software should work independently from any 3rd party software

Independence - CUAHSI:Please indicate the following on a scale of 1-5, where

1=not important, 5=essential

HIS software should work on all computer systems (Windows, Linux, Mac, Unix)

HIS software should leverage commercial software systems (e.g. ArcGIS, SAS)

All

05

1015

2025

30

1 2 3 4 5

Mean: 4.03

leverage0

510

1520

2530

1 2 3 4 5

Mean: 3.46

Open Source

05

1015

2025

30

1 2 3 4 5

Mean: 4.09

HIS should be implemented using open source software

List data standards and file formats that you

think CUAHSI should adopt - USU

• NetCDF

• check the ALMA convention for Land Surface Modeling

• Grib

• USGS

• NASA - hdf

Are there data for which CUAHSI should

define standards - USU• Standards are important, but I do not know what they all are.• Streamflow, Precip, Landuse, Topography, Landcover• Unpublished data in technical reports must be protected, esp when

graduate theses and dissertations are involved• Of course, mostly in terms of denomination of the variables. Check

ALMA standards as an example.• Time series of any sort should have a standard that controls how

they are stored and distributed (this is primarily so that qualifying comments or metadata are not lost).

• Data should probably be available in a variety of export formats.• EPA storet- have good chemical data AND flow from the same

sample site. Otherwise it is pretty useless.

List data standards and file formats that you think CUAHSI should adopt - CUAHSI

• Time series• Space time grids• Some image formats• Hydrologic Observations• ISO• HDF• NetCDF• NASA DIF• GeoTiff• XML based formats• At least conversion from difficult data formats• Flat files• Data model for time series needs more development• How should I know? I’m just a user.• Any raster grid supported by ESRI• Ascii, shapefile, MS Access, XML• XML, XML, and XML

Are there data for which CUAHSI should define standards - CUAHSI

• Probably• Transboundary datasets• Sediment transport• Domain specific data – whatever is going to be

offered to users• Adapt existing standards• In situ observational data – such as SCAN

network data• Metadata format and ontology structure• Vector data, raster data, image data

Rate the ease of access to hydrologic data and information

3.7

3.6

3.3

3.3

3.1

2.9

2.9

2.8

2.8

2.6

2.5

2.5

2.1

2.0

1.8

1.5

USGS Historical Streamflow

USGS Real Time Streamflow

National Elevation Dataset

SNOTEL data

National Land Cover Data

National Hydrography Dataset

NCDC Precipitation

PRISM Precipitation Data

LANDSAT Satellite Imagery

STATSGO soils data

SSURGO soils data

USGS National Geology data

EPA STORET Water Quality Data

NEXRAD Radar precipitation

University of Washington Gridded MeteorologicalData (Maurer, et al., 2002, J. Climate 15, 3237-

NCEP North American Regional Reanalysis(NARR) climate data

1 =Low 5=High


USU

Rate the ease of access to hydrologic data and information - CUAHSI

3.8

3.8

3.8

3.8

3.5

3.4

3.1

3.0

3.0

2.9

2.9

2.9

2.8

2.6

2.6

2.6

2.6

2.5

2.5

2.5

2.4

2.3






USGS Hydrologic Landscape Regions

SSURGO soils data

STATSGO soils data



NCDC Precipitation

Groundwater level


Water quality/ChemistryUniversity of Washington Gridded

Meteorological Data (Maurer, et al., 2002, J.SNOTEL data

EPA STORET Water Quality DataReal-time weather and Nexrad data from

UnidataNCEP North American Regional Reanalysis(NARR) climate data

Water use


Evapotranspiration1 =Low 5=High


Rate the priority for data inclusion in HIS

4.8

4.8

4.7

4.6

4.4

4.4

4.4

4.2

4.2

4.1

3.9

3.9

3.9

3.9

3.7

3.7



NCDC Precipitation



SNOTEL data





STATSGO soils data

SSURGO soils data


University of Washington GriddedMeteorological Data (Maurer, et al.,


NCEP North American RegionalReanalysis (NARR) climate data

1 =Low 5=High


Other data mentioned

Water Use/Diversion data - 2

ET, Groundwater level, Water Quality/Chemistry, Energy balance components (latent and sensible heat)

USU

Rate the priority for data inclusion in HIS - CUAHSI

4.7

4.7

4.6

4.6

4.6

4.6

4.6

4.5

4.5

4.5

4.5

4.4

4.4

4.4

4.4

4.4

4.3

4.3

4.2

4.0

4.0

3.8


Groundwater level

NCDC Precipitation


Water quality/Chemistry





SNOTEL data



Water use

Evapotranspiration


USGS Hydrologic Landscape Regions

SSURGO soils data

STATSGO soils data

LANDSAT Satellite ImageryNCEP North American Regional Reanalysis (NARR)

climate dataReal-time weather and Nexrad data from UnidataUniversity of Washington Gridded MeteorologicalData (Maurer, et al., 2002, J. Climate 15, 3237-

1 =Low 5=High


Other National Datasets you think should be considered - CUAHSI

• Corps of Engineers Stream stage/discharge• WRRI – Institutes• Reservoir storage/elevation/chemistry (Corps of Engineers, Bureau of Reclamation)• National orthophotography• Higher resolution HUCs• Urban/anthropic/water-related infrastructure• General geology• Population density• Various NRCS/WCC & NOHRSC data• NADP precip chemistry• Stream sediment data• Hydrogeologic data (K, Chem, etc.)• Snow courses• Vegetation maps• US Census/population data• Ecological data/ecoregions• Agricultural census data

Local Datasets that you think should be included - USU

• Geomorphic/Channel Characteristics, (.shp)• Fish Distribution and abundance (Excel)• Other biotic data on streams( Excel)• Abiotic data on streams (Excel)• LIS (Gribs)• U Washington (NetCDF)• Unidata• Real time river diversions and reservoir elevations (txt)• Real time water quality monitoring data• Real Time climate/weather observations (like Mesowest) • Local government records indicating land use changes• US govt Population Census data• Meteorological and edaphic state variable data from CSI loggers• Precipitation (ASCII, Excel))• Soil Moisture (ASCII, Excel)• Soil temperature (excel)• Soil characteristics (ascii)• Meteorological data (wind speed, humidity, air temperature, solar radiation) (ASCII)

Local Datasets that you think should be included - CUAHSI

• Borehole data – geologic electric and geologic logs – locations

• Seismic reflection lines• NPDES sites• Geologic surface profiles with original

data points• Groundwater raw water chemistry• Irrigation diversions• Pressure, moisture, concentration (SQL)• USGS FRI – Forest growth data• National rangeland condition data• US breeding bird survey• Airborne Lidar – filtered bare earth and

first return• High resolution orthorectified aerial

photographs• Various mountain datasets• RAWS & other mesonets

• Precipitation isotope data• Calsil (data system), Cdoc (data

system)• Weather Mesonets• High level land use• High resolution topography• Land management practices• Water management practices• Model input for specific watersheds• Met and flux tower• Realtime water quality data• Diversions, water management, water

rights• Water use• GAP analysis

Rate the importance of the following roles of an HIS

4.8

4.4

3.8

3.7

3.7

3.6

3.6

3.4

3.1

Retrieval of relevant National, Community, andHydrologic Observatory datasets

Uploading, archival and sharing of hydrologic datawith collaborators and the CUAHSI community

Source or repository of educational information(e.g. online lectures, seminars or educational

modules)Source or repository of information necessary forthe interpretation of HIS data (e.g. what do soilmoisture measurements mean for a user not an

Development of community data models andstandards for data representation

Interfacing of hydrologic datasets in standardformat with 3rd party analysis software

Sharing of models, software and hydrologicsimulation systems

Development and management of communitymodels and software

Software system for hydrologic analysis


1=Do not include this

2=Low priority for this capability

3=Good idea, include if possible

4=Important, HIS should place high priority on this

5=This is an essential role for HIS

USU

Rate the importance of the following roles of an HIS - CUAHSI

4.7

4.6

3.8

3.8

3.5

3.5

3.5

3.4

2.8

Retrieval of relevant National, Community, andHydrologic Observatory datasets

Uploading, archival and sharing of hydrologic datawith collaborators and the CUAHSI community

Interfacing of hydrologic datasets in standardformat with 3rd party analysis software

Development of community data models andstandards for data representation

Source or repository of educational information(e.g. online lectures, seminars or educational

modules)

Software system for hydrologic analysis

Sharing of models, software and hydrologicsimulation systems

Source or repository of information necessary forthe interpretation of HIS data (e.g. what do soilmoisture measurements mean for a user not an

Development and management of communitymodels and software


1 Never use or do not find useful

2Have used, but do not rely on

5 Use frequently and find indispensable

4 Use often

3Use occasionallyand am comfortable with its use

Additional Comments/Suggestions• Transboundary approach should be included into the HIS project (e.g., all

watersheds along the US/MX border are binational on both surface and GW hydrology

• Need tools to easily register raster data at different scales (e.g., MODIS, Landsat, orthophotos, NED, gridded climate)

• Consider uncertainty• Consider anthropic influences• Prioritize ingestion and dissemination of data, methods for data visualization,

manipulation and, analysis• Need attention to ease of use, intuitive interfaces, responsiveness, etc.• Don’t forget the hydrogeologists, hydrochemists, and stream geomorphologists• Is there more interest in accessing unfiltered data than model results?• Each observation needs quality information (adequacy of time series data

model)• Priority on simple data model for hydrologic observations with derivatives for

groundwater, streamflow, etc. Priority 2 = data fusion and assimilation technology

• Needs – integrated models, data to models, standard models, software for analysis of model output and uncertainty

hydrologic information use and needs assessment utah state university examples of existing systems...

Documents

data analysis

data manipulation

time series data

map server

real time data collection

database server dell

amber spackman data

arcgis microsoft sql