leveraging modelbuilder for advanced geoprocessingiteration = looping = repeat a process over and...

Leveraging ModelBuilder for Advanced

GeoprocessingSciences Solution Engineer-Kristen Hocutt

khocutt@esri.com

NASA DEVELOP Team-Alison Thieme & Kathleen Moore

alison.n.thieme@nasa.gov

Agenda

Why are we here?

NASA DEVELOP: Southern Arizona Ecological Forecasting2

Iterations3

Branching4

What’s New & Q&A5

1

Why ModelBuilder?

EFFICIENCY

AUTOMATION

FLEXIBILITY

NASA DEVELOP National Program

Capacity Building : www.nasa.gov/applied-sciences/capacitybuilding

Participants + Earth Observations + Decision Makers

DEVELOP bridges the gap between NASA Earth Science and society,

building capacity in both its participants and end-user organizations

to better prepare them to handle the environmental challenges

that face society.

“Shaping the future by integrating Earth

observations into global decision making.”

Spring 2017 Stats

13%

13%

13%

18%4%

4%

31%

4%

Ag

Climate

Disasters

Eco

Energy

Health & AQ

Water

Cross-Cutting

Application Areas Addressed

1stT

ER

M (

17)

2n

dT

ER

M (

6)

23

Projects

87% Domestic

13% International

30 States & 6 Countries Impacted

9

4

26

2

4

6

4

2

0 10 20 30

State Govt

Local Govt

Federal

For-Profit

Non-Profit

International

Academia

Consortium

Partner Total by Type

57

Partners

2017 Spring Portfolio

• Mississippi River Basin Disasters II (MSFC)

• Philippines Disasters (NCEI)• Southern Appalachia Disasters

(UGA)

• Glacier National Park Climate II (LaRC)

• Missouri River Climate II (NCEI)• Santa Monica Mountains Climate

(JPL)

• Intermountain West Eco (ID)• New York Eco (GSFC)• Yellowstone Eco (WC)

• Alabama Ag (MSFC)• Arizona Ag (JPL)• Chesapeake Bay Ag (GSFC)

• Phoenix Health & AQ (AZ)

• Nez Perce-Clearwater NF Energy

(FC)

• Wyoming Cross-Cutting

• Arizona Water (FC)

• Chesapeake Bay Water (LaRC)

• Chile Water (ARC)

• Lake Erie Water (ARC)

• Mississippi Sound Water (LaRC-JPL)

• Southeastern Arizona Water II

(MCHD)

• Southeastern Idaho Water II (ID)

National Aeronautics and

Space Administration

Natalie Queally

Nick Rousseau

Erika Higa

SOUTHERN ARIZONA

ECOLOGICAL

FORECASTINGDetecting and Monitoring Invasive Buffelgrass in

the National Parks of Southwestern Arizona

Molly Spater (Project Lead)

Presented by Alison Thieme

alison.n.thieme@nasa.gov

Invasive shrub grass native to Africa and Eurasia

Introduced to the Southwestern U.S. to improve degraded rangelands

Invaded both disturbed and undisturbed desert environments

Widely-distributed in the Sonoran Desert ecosystem

Pennisetum ciliare

Image Source: Molly Spater

Pennisetum ciliare

Buffelgrass presence

OPCNM Boundary

Estimated National Distribution of Buffelgrass

Data Source: Utah State University Herbarium

Buffelgrass spreads rapidly and fills the normally-barren gaps between native vegetation

Buffelgrass outcompetes native desert plants for water and soil nutrients

Buffelgrass increases wildfire severity and the number of wildfire events

With wildfire frequency, buffelgrass threatens public safety and property

Community Concerns

Image Credit: NPS

Utilize NASA Earth Observations and WorldView-2 data to detect invasive buffelgrass

Investigate the effectiveness of the CLaRe and MTMF metrics used in Wallace et al (2016) for buffelgrass detection

Create classification map of buffelgrasspresence/absence in Organ Pipe Cactus National Monument

Objectives

Image Credit: NPS

Organ Pipe Cactus National Monument

Years: 2009-2012

Focus on months of high precipitation

Study Area and Period

Data Source: NPS

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0

20

40

60

80

100

120

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Rai

nfa

ll (i

nch

es)

Tem

per

atu

re (

˚F)

Organ Pipe Cactus National Monument - Weather

Average Max °F Average Min °F Average Rainfall/(in)

Arizona

Organ Pipe Cactus National Monument

NASA Satellites and Sensors

Image Credit: NASA Eyes on Earth

Image Credit: NASA.gov

Aqua/Terra: Moderate Resolution Imaging Spectroradiometer (MODIS)

MODIS GPP/NPP Project and Global Evapotranspiration Project (University of Montana)

Launched October 2009Commercial satellite with up to

0.46 meter spatial resolution

Image Credit: Satellite Imaging Corporation

WorldView-2

WV-2

Methodology

Spectral Based

Technique

SpatialCorrelationTechnique

Mixture-Tuned

Matched Filtering (MTMF)

Climate-Landscape Response (CLaRe) Model

MODIS

Image Credit: NPS

MTMF Methodology

WorldView-2 Image

Acquisition

Image pre-processing

Target Detection:

MTMF

Matched Filter

Threshold Modeling

Cohen’s Kappa Coefficient

Binary Classification

Presence/Absence Map

MTMF = Mixture-Tuned Matched Filtering

Subpixel, spectral-based analysis

Incorporates field measurements of target spectra

What is MTMF?

Buffelgrass

Presence

Absence

WorldView-2 Image

Acquisition

Image pre-processing

Target Detection:

MTMF

Matched Filter

Threshold Modeling

Cohen’s Kappa Coefficient

Binary Classification

Presence/Absence Map

Matched Filter Threshold Modeling

Matched Filter Threshold Modeling

Matched Filter

Threshold Model

Cohen’s Kappa

Coefficient

Matched Filter Threshold Modeling

Iterate Until High Cohen’s Kappa Coefficient Achieved

Vector Data

Buffelgrasspresence/absence field data points collected by NPS

Points are then buffered and used as polygons

Model Inputs

Raster Data

Imagery with Matched Filter Scores as Pixel Values

Imagery with Infeasibility as Pixel Values

Matched Filter Threshold Modeling

Raster Calculator Combine Matched Filter (MF) and

Infeasibility values derived from satellite imagery

Reclassify Convert calculated raster into

target/non-target raster

ArcMap Tools UtilizedMF Score Infeasibility Value

Target Detected

Low Low

High High

TargetNon-target

Matched Filter Threshold Modeling

Raster to Polygon Converts presence/absence raster into polygons to

compare with NPS field data

Select Layer by Location Compares NPS field data to MF classified

presence/absence

Vector to Raster Converts point or polygon data into a raster

Reclassify Classifies raster into YY YN NY NN layers

Mosaic to New Raster Mosaics all YY YN NY NN layers into one raster

ArcMap Tools Utilized

YY

NYNN

MTMF Presence MTMF Absence

Fiel

d A

bse

nce

Fiel

d P

rese

nce YN

Matched Filter Threshold Modeling

Cohen’s Kappa Coefficient:

Statistic used to measure agreement between two judges in sorting categorical items

Takes into account agreement occurring by chance

MF Threshold & Cohen’s Kappa

Date MF Threshold KappaJanuary 16, 2012 0.1 0.340

0.15 0.4280.2 0.487

0.25 0.4870.3 0.407

February 10, 2012 0.1 0.577

0.15 0.6750.2 0.841

0.25 0.793

October 14, 2012 0.3 0.185

0.4 0.4500.5 0.709

0.55 0.097

Yes No

Yes 1012 31

No 18 157

Classification Map

Presence

OPCNM

Errors & Uncertainties

Image Credit: NPS

Field Data was very inconsistent Point data were supposed to represent plots Plot sizes were not always included Density data and exact location of plots not included either Whether buffelgrass was pulled was only included in 2012 and did not include if

treatment was successful

MF thresholds were based on field data extrapolation Only a few images from 2012 were analyzed Conservative approach was taken for Infeasibility Values and Matched Filter Scores

MTMF Conclusions

Image Credit: NPS

MTMF will be more useful for detecting pixels with at least 20% coverage buffelgrass

Higher MF scores appear to yield higher Kappa coefficients

There may be seasonal effects on classification accuracy

Accurate polygon field data is necessary for validation

Acknowledgements

This material is based upon work supported by NASA through contract NNL11AA00B and cooperative agreement NNX14AB60A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration

Science AdvisorNASA Jet Propulsion Laboratory

Dr. Natasha Stavros

DEVELOP MentorBenjamin Holt

Partner OrganizationNational Park Service, Saguaro National Park Rincon Mountain District

Dana Backer, Park Ecologist

USGS Southwest Biological Science Center

Dr. Cynthia Wallace, Research Scientist

Dr. Seth Munson, Research Ecologist

Northern Arizona University (NAU)

Dr. Temuulen Sankey, Remote Sensing Scientist

Organ Pipe Cactus National Monument

Jeanne Taylor, Botanist

ITERATION

Iteration in ModelBuilder = Run entire model or a

single tool or a set of tools repeatedly

Tools to iterate in ModelBuilder = Iterators

Example - Iterate over a list of feature classes

and project each feature class.

Iterator

Iteration = looping = repeat a process over and

over

Iterators

Iterates over a starting and

ending value by a given value.Iterates "while" a

condition is true or false.Iterates over features in a

feature class.

Iterates over rows in a table. Iterates over each value in a field.

Iterates over a list of values.

Iterates over datasets in a Workspace or Feature Dataset.

Iterates over feature

classes in a Workspace or Feature Dataset.

Iterates over files in a folder.

Iterates over rasters in a

Workspace or a Raster Catalog.Iterates over tables in a workspace.

Iterates over workspaces in a folder.

For Value While Continue

Iterate

Feature

SelectionValue

Selected

Features

Iterate

Row

SelectionValue

Selected

RowsIterate

Field

ValuesValue

Iterate

MultivalueValue

Iterate

Datasets

Name

Dataset Iterate

Feature

ClassesName

Feature

Class Iterate

Files

Name

File

Iterate

Rasters

Name

RasterIterate

Tables

Name

Table

Name

WorkspaceIterate

Workspaces

Accessing the iterators in ModelBuilder

PRECONDITIONEXPLICITLY CONTROL THE ORDER OF

OPERATIONS IN A MODEL

Workflow for Winery Site Suitability at a Local and State Scale

• Scenario: You are a vineyard manager

and your employer owns a large

property in Monterey County, CA.

Much of it is already planted with

grapes, but there are several unused

blocks of land. You will use your

enological knowledge and GIS skills

to select the most suitable block for

planting additional vines.

• Environmental Indicators:

- Sunlight

- Land must be mostly south facing

- Slope = <14%, machine harvestable

- Little shade

Layers Needed/Created:

- Elevation/DEM

- Slope

- Aspect

- Hillshade

Collect data &

run preprocessing

Factor in

Indicators & run

models for specific

Use cases

Overlay dataFind wineries

located in the

best areas

Scheid Vineyards

Kristen Hocutt

Demo 1: Iterating

Rasters and

PreConditions

Model Within a

Model

Model within a

Model within a

Model

Model 1

Model 2

Model 3

Rules for a nested model/model within a model

• Only one iterator can be used per model.

• Add only the tools you want to run as many times as an iterator in the

same model.

• The script will not include the iteration logic if a model with an iterator

is exported to a Python script.

• The output of any tool connected to the iterator can have (if required) a

unique name for each iteration to avoid being overwritten by

• Using the system variable %n%C:\Sctatch\scratch.gdb\output_%n%

• Using the Name or Value output of the iterator C:\Sctatch\scratch.gdb\output_%Name%

C:\Sctatch\scratch.gdb\output_%Value%

• Using any other variable in the model as an inline variable

C:\Scratch\scratch.gdb\output_%XYZ%

Rules for a nested model/model within a model

• Give default values to your sub model for setting and

testing it

• Make model parameters in your sub-model that you want as

variable in main model

Python Equivalent of Iterators in ModelBuilder

Allows you to assemble

processes into logical units

GROUPING

Examples:

Add an Empty Group to the Model

Select an existing model element and make it a Group

IF some condition is true, THEN

perform an action; ELSE

the condition is false,

perform a different action.

File in

Workspace

If file X exists Add a field

Else if file X does not exists

Copy and then add a field

BRANCHING

Feature Class

If has X projection

Do nothing

Else - Project

Examples:

How to branch?

• The condition - If-then-else

1

Calculate Value tool

2

Script tool

• Where to find the tool

• Python Code

• Data types

• Preconditions in a model

• Inline Variable Substitution

• Merge Branch tool

• Creating a script tool

• Python code

• Setting script tool properties

• Preconditions in a model

• Merge Branch tool

OR

INLINE VARIABLE SUBSTITUTIONTHE VALUE OF ANY VARIABLE CAN BE USED IN

THE TOOL PARAMETERS BY ENCLOSING THE

NAME OF THE SUBSTITUTING VARIABLE

BETWEEN THE PERCENT SIGNS (%)

Inline variable substitution: %VariableName%

• In SQL expressions - Name = ‘%Value%’

• In Output name – C:\Scratch\Scratch.gdb\%Value%

• If string put “quotes” around your inline variable substitution

Example: variable name = Wilson

Expression: “NAME” = ‘%Wilson%

Example:

C:\Data\Output.gdb\Clipped_%Name%

Output Feature Class:

C:\Data\Output.gdb\Clipped_Wilson

Workflow for Winery Site Suitability at a Local and State Scale

• Scenario: You are a vineyard manager

and your employer owns a large

property in Monterey County, CA.

Much of it is already planted with

grapes, but there are several unused

blocks of land. You will use your

enological knowledge and GIS skills

to select the most suitable block for

planting additional vines.

• Environmental Indicators:

- Sunlight

- Land must be mostly south facing

- Slope = <14%, machine harvestable

- Little shade

Layers Needed/Created:

- Elevation/DEM

- Slope

- Aspect

- Hillshade

Collect data &

run preprocessing

Factor in

Indicators & run

models for specific

Use cases

Overlay dataFind wineries

located in the

best areas

Scheid Vineyards

Kristen Hocutt

Demo 2:

Grouping

Kristen Hocutt

Demo 3:

Branching Using

CON Tool-

Rasters

Branching-

Feature Layer

Branching Using

Script Tool

Model Only Tools

• Collect Values

- Use to collect values in an iterative model

- Use to combine 2 multi-values into one

- Use to convert a list to a multi-value

• Get Field Value

- Use to get a single value from a table

• Merge Branch

- Use to bring multiple processing branches back together

• Parse path

- Use to parse the path and file from full path names

• Stop

- Use to stop iterative model

- Similar to while

- Can be combined with an iterator

- Example: run 10 times or until some condition is true.

ModelBuilder: UpdatesArcGIS Pro 1.2 to ArcGIS Pro1.4

• You can create a group of any model elements. This group can be labeled, collapsed

and expanded. Use for simplifying complex models.

• Any dataset variable can be marked to ‘Add to Display’. These datasets will be

added to the last active map when the model is run from within ModelBuilder

• You can drag a layer of dataset onto any tool element in your model and

automatically see a list of all valid tool parameters and environments that layer or

dataset can be connected to.

More Information go to the What’s New with ArcGIS Pro page.

ModelBuilder: Migration to ArcGIS ProDifferences in ArcMap vs. ArcGIS Pro

• Compatibility: Models created in Pro can NOT be used in other ArcGIS desktop

applications. You must save the Toolbox in the correct version to do so.

• Selection, Connection and Navigation: Tools to Modes

• Calculate Field: No longer supports VB expressions, must use Python syntax

• Intermediate and Managed Data: These options have been removed

• Colors, Shapes and Images: Different color scheme in Pro

• List and Series: NOT supported in ArcGIS Pro

• Feature and Records Set: Interactive drawing & entering of features/records is NOT

supported

More Information go to the ArcGIS Pro documentation page.

Where to get help? Docs, Blogs, Videos

ModelBuilder for ArcGIS for Desktop• ArcGIS Pro

• Vocabulary• Grouping

• ArcMap• Esri Press• Esri Video

Branching

1.Understanding which if troubles you

2.If you are stuck at "if" – Part 1 – Branching using the Calculate Value tool

3.If you are stuck at "if" – Part 2 – Example of using Script tool to create

branches using if-else logic

4.If you are stuck at "if" – Part 3 – Does Extension Exists model example

5.If you are stuck at "if" – Part 4 – Does Selection Exists model example

6.If you are stuck at "if" – Part 5- Does Projection Exist model example

Survey InformationFebruary 13th – 14th

• Survey information

• Please fill out the session survey in your mobile app

• Select in Leveraging ModelBuilder for Advanced Geoprocessing in

the Mobile App

- Use the Search Feature to quickly find this title

• Click “Technical Workshop Survey”

• Answer a few short questions and enter any comments.

Still to come … February 13th

Overview of Conference Activities

‒ Python: Beyond the Basics: 3:00pm, Room 146B

‒ Web AppBuilder for ArcGIS: Build Web Apps with No Coding: 3:00pm, Room 150A

‒ Managing Imagery & Raster Data: 4:15pm, Room 152A

‒ Exhibits in the lower hall

‒ 12:30 PM – 6:30 PM

‒ Monday Night Social held in the Expo area

‒ 5:00PM - 6:30 PM

‒ If you want to bring your significant other to the Tuesday Night Social there are tickets being sold on the Expo floor.

Still to come … February 14th

Tuesday Night Networking Social @ The Smithsonian

National Air and Space Museum

6:30pm-9:30pm

See You There!!

Questions?

“Learn from yesterday, live for today, hope

for tomorrow. The important thing is not to

stop questioning.”

–Albert Einstein