presented to

GIS Committee, Florida Model Task Force

presented by

Albert GanAssociate ProfessorLehman Center for Transportation ResearchFlorida International University

May 14, 2010

FSUTMS Data Framework

Problems with FSUTMS Input Data

Input data are stored in individual files of different file formats

• DBF, CSV, PEN, DAT, LIN, NET, MAT, etc.

Text files do not have variable names

Variable names are inconsistent among different models

Files are not saved in standard file folders

Consequence: Data are more difficult to maintain and understand

Consequence: Tools cannot be developed without customizing for individual models

Project Objective

Design a data framework for FSUTMS that

• facilitates the maintenance of model input data

• allows tools to be developed and applied across all models with minimum or no customization

Three Basic Design Approaches

Use of geodatabase

Use of “master” geodatabase

Include all variables used by all FSUTMS models

Use of Geodatabase

Store all files as a set of tables in a personal geodatabase file

GIS-TM uses this approach more than 10 years ago

2 GB size limit of Personal Geodatabase is not a problem

Geodatabase for all base-year data for SERPM is only 65 MB, or about 3% of the 2 GB limit

Use of Master Database Approach

2010 2020

2030 2040

Traditional Approach

2000Master Database

Scenario ExtractionScenario Merging

2010

Forward or Backward Propagation

Include all Variables for all FSUTMS Models

Each model has all of its variables included in the data framework

There is NOT a standard set of data variables in the data framework

Local models will not be affected as a result of transitioning to the data framework

However, each model will have to do the following:

• modify to standard table (was file) names, variable names, and file folders

• modify input scripts to read from a standard geodatabase instead of individual files

FSUTMS Models Considered

Florida Statewide Model (FLSWM)

Northeast Regional Planning Model (NERPM)

Tampa Bay Regional Planning Model (TBRPM)

Southeast Regional Planning Model (SERPM)

Treasure Coast Regional Planning Model (TCRPM)

Greater Treasure Coast Regional Planning Model (GTCRPM)

Central Florida Regional Planning Model (CFRPM)

Northwest Florida Regional Planning Model (NWFRPM)

Gainesville Urbanized Area Transportation Model (GUATS)

Polk TPO Model (POLK)

Project Tasks

Developing standards for folder names and structure, database names, table names, field names, and field types for all input data as well as setting standards for metadata files.

Designing a master database that is inclusive of all standard input data tables and fields of all modes from all scenarios of all local models.

Developing a master database system that implements the master database approach.

Developing a procedure for converting existing input files into the proposed master database.

Inconsistent Variable Names: An Example

FLSWM NERPM TBRPM SERPM TCRPM GTCRPM CFRPM NWFRPM GUATS POLK

Z Z ZONE TAZ Z Z TAZ Z Z Z

HBWP HBWKP HBWP P1 P1 P1 PHBW P1 HBWP HBWP

HBWA HBWKA HBWA A1 A1 A1 AHBW A1 HBWA HBWA

HBSHP HBSHP HBSHP P2 P2 P2 PHBSH P2 HBSHP HBSHP

HBSHA HBSHA HBSHA A2 A2 A2 AHBSH A2 HBSHA HBSHA

HBSRP HBSRP HBSRP P3 P3 P3 PHBSR P3 HBSRP HBSRP

HBSRA HBSRA HBSRA A3 A3 A3 AHBSR A3 HBSRA HBSRA

HBOP HBOTP HBOP P6 P5 P5 PHBO P4 HBOP HBOP

HBOA HBOTA HBOA A6 A5 A5 AHBO A4 HBOA HBOA

NHBP NHBSP PNHB P5 NHBP NHBP

NHBA NHBSA ANHB A5 NHBA NHBA

NHBWP P7 P6 P6

NHBWA A7 A6 A6

NHBOP P8 P7 P7

NHBOA A8 A7 A7

HBUP

Variable Naming Conventions

10 characters maximum

Use meaningful names

If a variable name contains multiple words, use underscores only if it helps to avoid confusion

If a variable is used by multiple models: choose one that is most commonly used subject to the above criteria

If a variable is used by only a specific model: use the original name unless they do not meet the general criteria above

Reference document: FSUTMS Powered by Cube/Voyager Data Dictionary, December 2005

Applying the Naming Conventions

FLSWM NERPM TBRPM SERPM TCRPM GTCRPM CFRPM NWFRPM GUATS POLK

Standard

Z Z ZONE TAZ Z Z TAZ Z Z Z TAZ

HBWP HBWKP HBWP P1 P1 P1 PHBW P1 HBWP HBWP HBWP

HBWA HBWKA HBWA A1 A1 A1 AHBW A1 HBWA HBWA HBWA

HBSHP HBSHP HBSHP P2 P2 P2 PHBSH P2 HBSHP HBSHP HBSHP

HBSHA HBSHA HBSHA A2 A2 A2 AHBSH A2 HBSHA HBSHA HBSHA

HBSRP HBSRP HBSRP P3 P3 P3 PHBSR P3 HBSRP HBSRP HBSRP

HBSRA HBSRA HBSRA A3 A3 A3 AHBSR A3 HBSRA HBSRA HBSRA

HBOP HBOTP HBOP P6 P5 P5 PHBO P4 HBOP HBOP HBOP

HBOA HBOTA HBOA A6 A5 A5 AHBO A4 HBOA HBOA HBOA

NHBP NHBSP         PNHB P5 NHBP NHBP NHBP

NHBA NHBSA         ANHB A5 NHBA NHBA NHBA

    NHBWP P7 P6 P6         NHBWP

    NHBWA A7 A6 A6         NHBWA

    NHBOP P8 P7 P7         NHBOP

    NHBOA A8 A7 A7         NHBOA

                 HBUP   HBUP

Table Name Conventions

Maintain most traditional names

• SPDCAP, VFACTORS, FF, PANDA, etc.

No need to attach model year and model alternative; they are part of the scenario database file name

Transportation-network related tables

• HwyNodes, HwyLinks, PtNodes, PtLinks, PtLines

Non-transportation network related tables

• See next slide

Non-Transportation Network Related Tables

ZONEDATA

ZONEDATA_LIFESTYLE

SPECGEN

EETRIPS

INT_EXT

PRODRATES

ATTRRATES

DUWEIGHTS

PANDA

SPDCAP

TOLLLINKS

TURN_PENALTIES

VFACTORS

FF

TRANSIT_MODES

TRANSFER_FARES

TRANSIT_OPERATORS

WAIT_CURVES

TRANSIT_SPEED

Master Geodatabase File

Contains all tables for all models, all networks, all scenarios

Scenario Geodatabase

Contains only applicable tables for specific model, specific model year and model alternative

Metadata Standards

Not metadata files currently used by any models

FGDC Plus Stylesheet (Federal Geographic Data Committee)

Template: http://www.fsutmsonline.net/SERPM_MASTER.XML

Edit in Word: C:\FSUTMS\D4\SERPM\MASTER\SERPM_MASTER.DOC

Source: FSUTMS New Standards and Enhancements – A User Oriented Approach, January 2006

File Folder Structure and Naming Conventions

Master Database System

Master Database System Program

Cubeor

ArcGIS Desktop

HwyLinks

ProjectInfo

BaseYearNumber

HNet_Link

Reference Tables

Feature Classes

MNet_MergeInfo

Citilabs_Networks

Reference Tables

In

put

Mer

ge

Ext

ract

Out

put

Save

Open

ScenarioDatabase

Master Database

HwyNodesPtLinesPtLinksPtNodesTAZs

ImpExpInfo

HNet_NodePTNet_PtLinePTNet_PtLinkPTNet_PtNodeTAZ

Attribute Tables

Socioeconomic Data

Socioeconomic Data

…..

Attribute Tables

Socioeconomic Data

Socioeconomic Data

…..

Citilabs_TransitGroups

Feature Classes

Scenario Extraction

Scenario Merging: saving as a new scenario

Scenario Merging: update original scenario

Data Conversion Procedure (from existing files to master geodatabase)

Step 1: Create a geodatabase for the base-year data

• Create an empty Cube geodatabase in Cube

• Import highway network

• Import public transit network

• Import TAZ feature class (optional)

Step 2: Create a master geodatabase

• Create an empty geodatabase in ArcCatalog

• Import feature classes from Cube geodatabase into master geodatabase

• Import other non-transportation network related data

Data Conversion Procedure (continued)

Step 3: Update master geodatabase in Microsoft Access

• Rename variables to standard names

• Add master database system tables (BaseYearNumber, ProjectInfo, ImpExpInfo)

• Add LineID field to PTLines feature class

• Add master geodatabase key fields (MNet_Year, MNet_Status, and MNet_ProjectID) to feature class and data tables

• Fill key fields with appropriate values

Master Database System Install Package

Download: http://lctr.eng.fiu.edu/MasterDBS.zip (beta version, 14 MB)

Include the SERPM master database file as an example (installed to C:\FSUTMS\D4\SEPRM\MASTER\)

Require ArcGIS Engine Runtime or ArcGIS Desktop installed on the computer running this application

Project Contributors

Vidya Mysore, FDOT Project Manager

Albert Gan, FIU Project Manager

Kaiyu Liu and Tao Wang, Programmers, FIU

Lina Kulikowski, Broward MPO

Sarah McKinley, Capton Siluvairajan, and Robert Schiffer, CSI

Minhua Wang, Citilabs, Inc.

David Schmitt, AECOM USA, Inc.

Kenneth Kaltenbach, Corradino Group, Inc.

FDOT Research Center and Staff Members

Questions?

Albert GanLehman Center for Transportation ResearchFlorida International Universitygana@fiu.edu305-348-3116