2013 tips and tricks mashup, from modelbuilder to formal python code, step-by-step by zeke houk
DESCRIPTION
ArcGIS ModelBuilder provides the user with a simple way to capture geoprocessing instructions as a Python script. This presentation explores the path to promote that script into formal production level code. Beginning with the geoprocessing foundation exported from ModelBuilder, we augment the code to: • clarify the command line parameters • organize the directories • set the SDE workspace • insert print commands • check if feature classes exist • use try / catch blocks to trap errors • execute operating system commands • add logging throughout the code The end result of these additions is production level code that is easy to maintain and deploy. It is particularly well-suited to run automatically as a Windows Scheduled Task. This session is intended for users who are getting started with Python, or even just thinking about it.TRANSCRIPT
ModelBuilder to Python:
Step-by-Step
Zeke Houk
GIS Developer
ModelBuilder Model:
Delete Features from
simple_conduits:
Pull snapshot from SDE:
Get Rid of Conduit 0:
Dissolve on Conduit Number:
Guido Van Rossum
BDFL
Where did the name come from?
Python script:
Clean up names:
• >>> # Local variables:
• >>> simple_conduits__3_ = R:\\ARG_Projects\\ConduitMapBook\\DissolvedConduits.gdb\\simple_conduits"
• >>> dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
• >>> ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
• >>> simple_conduits__4_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
• >>> N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"
• >>> dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
• >>> simple_conduits__6_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
>>> simple_conduits_3_,
>>> simple_conduits_4_,
>>> simple_Conduits_6_
>>> redundant…
Replace with simple_conduits:
• >>> # Local variables:
• >>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
• >>> dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
• >>> ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
• >>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
• >>> N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"
• >>> dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
• >>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
Global replace simple_conduits
For simple_conduits_3_,
simple_conduits_4_, and
simple_conduits_6_.
Duplicates dropped:
>>> # Local variables:
>>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
>>> dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
>>> ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
>>> N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"
>>> dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
Global replace simple_conduits
For simple_conduits_3_,
simple_conduits_4_, and
simple_Conduits_6_.
Now fix dissolved_conduits:
# Local variables:
simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"
Becomes…
# Local variables:
simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"
dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
Use File Geodatabase Name:
• # Local variables:
• file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"
• simple_conduits = file_gdb + "simple_conduits"
• dissolved_conduits = file_gdb + "dissolved_conduits"
• N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"
• ARG_CONDUIT = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
Becomes…
• # Local variables:
• simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"
• dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"
• ARG_CONDUIT = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
• N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"
Use File Geodatabase Name:
file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"
simple_conduits = file_gdb + "simple_conduits"
dissolved_conduits = file_gdb + "dissolved_conduits"
N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"
Use SDE Connection Name:
# Local variables:
file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"
simple_conduits = file_gdb + "simple_conduits"
dissolved_conduits = file_gdb + "dissolved_conduits"
N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"
sde_connection = "Database Connections\\Connection to DWGIS3.sde\\"
ARG_CONDUIT = sde_connection + "ARG.CONDUIT"
Becomes…
# Local variables:
file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"
simple_conduits = file_gdb + "simple_conduits"
dissolved_conduits = file_gdb + "dissolved_conduits"
N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"
ARG_CONDUIT = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"
Use SDE Connection Name:
sde_connection = "Database Connections\\Connection to DWGIS3.sde\\"
ARG_CONDUIT = sde_connection + "ARG.CONDUIT"
Cleaned up names:
# Local variables:
file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"
simple_conduits = file_gdb + "simple_conduits"
dissolved_conduits = file_gdb + "dissolved_conduits"
N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"
sde_connection = "Database Connections\\Connection to DWGIS3.sde\\"
ARG_CONDUIT = sde_connection + "ARG.CONDUIT"
Congratulations!
PyWin:
PyWin
Python for Windows extensions
by
Mark Hammond
http://sourceforge.net/projects/pywin32/
pywin32-218.win32-py2.7.exe
Process Steps:
Delete Features from simple_conduits
Append from SDE into simple_conduits,
just COND_NUM and WATER_TYPE
Delete NO_Zero_Conduits
Select from simple_conduits into N0_Zero_Conduits
Delete dissolved_conduits
Dissolve N0_Zero_Conduits into dissolved_conduits
Fix up Process Steps:
• # Process: Append
• arcpy.Append_management(ARG_CONDUIT, simple_conduits, "NO_TEST", "COND_NUM \"COND_NUM\" true true false 2 Short 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE \"WATER_TYPE\" true true false 5 Text 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length \"Shape_Length\" false true true 8 Double 0 0 ,First,#", "")
Becomes…
• # Process: Append
• arcpy.Append_management(
• "'Database Connections\\Connection to DWGIS3.sde\\
• ARG.DistributionSystem\\ARG.CONDUIT'", simple_conduits,
• "NO_TEST", "COND_NUM \"COND_NUM\" true true false 2 Short 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE \"WATER_TYPE\" true true false 5 Text 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length \"Shape_Length\" false true true 8 Double 0 0 ,First,#", "")
Fix up Process Steps:
• # Process: Delete
• arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass")
Becomes…
• # Process: Delete
• arcpy.Delete_management(
• "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\
• NO_Zero_Conduits",
• "FeatureClass")
Fix up Process Steps:
• # Process: Delete
• arcpy.Delete_management(dissolved_conduits, "FeatureClass")
Becomes…
• # Process: Delete
• arcpy.Delete_management(
• "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\
• dissolved_conduits",
• "FeatureClass")
All Scrubbed Up: # Process:
arcpy.DeleteFeatures_management(simple_conduits)
arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass")
arcpy.Delete_management(dissolved_conduits, "FeatureClass")
arcpy.Append_management(ARG_CONDUIT, simple_conduits, "NO_TEST", "COND_NUM \"COND_NUM\" true true false 2 Short 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE \"WATER_TYPE\" true true false 5 Text 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length \"Shape_Length\" false true true 8 Double 0 0 ,First,#", "")
arcpy.Select_analysis(simple_conduits, N0_Zero_Conduits, "\"COND_NUM\" <> 0")
arcpy.Dissolve_management(N0_Zero_Conduits, dissolved_conduits, "COND_NUM;WATER_TYPE", "", "SINGLE_PART", "DISSOLVE_LINES")
Geek Speak - If:
if arcpy.Exists(something):
# do something with it
if not arcpy.Exists(something):
# it does not exist
Avoid Errors:
if arcpy.Exists(dissolved_conduits):
arcpy.Delete_management(dissolved_conduits, "FeatureClass")
Becomes…
arcpy.Delete_management(dissolved_conduits, "FeatureClass")
Delete Processes: if arcpy.Exists(N0_Zero_Conduits):
arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass")
if arcpy.Exists(dissolved_conduits):
arcpy.Delete_management(dissolved_conduits, "FeatureClass")
if arcpy.Exists(simple_conduits):
arcpy.DeleteFeatures_management(simple_conduits)
else:
print "Missing feature class -> " + simple_conduits
Alternative:
>>> from arcpy import env
>>> env.overwriteOutput = True
>>> env.workspace = “C:/EsriPress/Python/Data/”
Either way, explain:
>>> # There feature classes probably already exist.
>>> # They are left over from the last time we ran this script.
>>> # Deleting the old versions makes room for the new ones.
Geek Speak – Try / Catch:
>>> try:
>>> # .. To do something that might blow up
>>> # and it “throws” an Exception
>>> except Exception ex:
>>> print ex
>>>
Catch Exception:
>>> try:
>>> arcpy.Append_management(CONDUIT
>> except Exception as ex:
>>> print ex
Test Exception:
>>> try:
>>> bogus = 1.0 / 0.0 # forces exception
>>> arcpy.Append_management(ARG_CONDUIT,
>>> except Exception as ex:
>>> print ex
Logging: import logging
logfile = "C:/temp/demo.log"
logging.basicConfig(
level=logging.INFO,
filename=logfile)
logger = logging.getLogger()
logger.info("A swallow beats its wings 43 times every second.")
logger.warning("Go away or I shall taunt you a second time.")
logger.error("Run away! Run away!")
print "Done."
C:\temp\demo.log: INFO:A swallow beats its wings 43 times every second.
WARNING:Go away or I shall taunt you a second time.
ERROR:Run away! Run away!
Logging: logging.info(" checking for " + HYDRANT_BRANCH )
if arcpy.Exists(HYDRANT_BRANCH ):
try:
arcpy.Append_management(APPEND_HYD_BR, hyd_branch, ...)
logging.info(" HYDRANT_BRANCH_shp appended to SDE")
except Exception as ex:
logging.error(" HYDRANT_BRANCH_shp append FAILED")
logging.Exeception(ex)
Recap – Do These 3 Things…
1. Organize variable names
2. Trap Errors
3. Write a log file
"First shalt thou take out the Holy Pin.
Then shalt thou count to three, no more, no less.
Three shall be the number thou shalt count, and the number of the counting shall be three.”
Pythonic:
• Explicit is better than implicit.
• Simple is better than complex.
• Readability counts.
The Zen of Python, by Tim Peters, includes:
Geek Speak - Blocks:
>>> if arcpy.Exists(simple_conduits):
>>> arcpy.DeleteFeatures_management(simple_conduits)
>>> # maybe do something else
>>> # some additional processing
>>> else:
>>> print "Missing feature class -> " + simple_conduits
Duck Typing:
If it looks like a duck, quacks like a duck, etc.
Works with strings,
integers,
floating point,
lists,
etc.
Strings:
Enclosed in either ‘single’ or “double” quotes.
Concatenate using + sign.
>>> topic = "GIS"
>>> location = "Rockies“
>>> event = topic + " in the " + location
>>> print event
GIS in the Rockies
Strings: More built-in functions:
>>> event = "GIS in the Rockies"
>>> print event.lower()
>>> print event.upper()
>>> print event.title()
>>> string_length = len(event)
>>> print string_length
>>> print event.find("Rockies",0,string_length)
gis in the rockies
GIS IN THE ROCKIES
Gis In The Rockies
18
11
Triple quotes can span multiple
lines: “””The Dead Collector: Bring out yer dead.
Large Man: Here's one.
The Dead Collector: That'll be ninepence.
The Dead Body: I'm not dead.
The Dead Collector: 'Ere, he says he's not dead.
Large Man: Yes he is.
The Dead Body: I'm not.
The Dead Collector: He isn't.
Large Man: Well, he will be soon, he's very ill.
The Dead Body: I'm getting better. “””
Lists: >>> python_sketch = ['Dead Parrot', 'Confuse-a-Cat', 'Silly Walks', 'Four Yorkshiremen','Fish Slap'];
>>> # notice duck typing
>>> print python_sketch [0]
>>> print python_sketch [1]
>>> python_sketch.append('Limberjack Song')
>>> print len(python_sketch)
Dead Parrot
Confuse-a-Cat
6
For loop:
>>> for funny_bit sketch in python_sketch:
>>> print funny_bit + ' hahaha‘
Dead Parrot hahaha
Confuse-a-Cat hahaha
Silly Walks hahaha
Four Yorkshiremen hahaha
Fish Slap hahaha
Limberjack Song hahaha
Command Line Args:
>>> userid = arcpy.GetParameterAsText(0)
>>> if not userid:
>>> print ‘Usage: python.exe my_script.py userid password ‘
>>> pword = arcpy.GetParameterAsText(1)
>>> if not pword:
>>> print ‘Usage: python.exe my_script.py userid password ‘
Command Line Args:
C:\> D:\python27\python.exe my_script.py zeke password
Python.exe is located on D:\python27
my_script.py is some python code
Userid is zeke
My Oracle password is at the end
Command Line Args:
>>> userid = arcpy.GetParameterAsText(0)
>>> pword = arcpy.GetParameterAsText(1)
Oracle connection:
connection_string = userid + '/' + pword + '@dwsde'
# looks like this -> zeke/password@dwsde
import cx_Oracle
connection = cx_Oracle.connect(connection_string)
cursor = connection.cursor()
cursor.execute('select distinct water_type from conduit')
for result in cursor:
print result
Oracle connection:
cursor.execute('select distinct diameter from conduit')
allresults = cursor.fetchall()
for result in allresults:
print result
cursor.close()
connection.close()
print "Done."
Nee
Here is a shrubbery.
Not Habit Forming:
But is considered a “Gateway” language,
which frequently leads to more nerdy behavior.
Good Book:
Good Book:
GIS Day November 22:
See denverwater.org for directions...
Becomes…