week 14 lecture: advanced python programming, concepetsand...
TRANSCRIPT
Week 14 Lecture:Advanced Python Programming,
Concepets and ExamplesIntroduction to Programming for GIS & Remote Sensing
GEO6938‐1469GEO4938‐147A
Re‐cap
From Past Lectures
•Last week:
–Package managem
ent a
nd Pytho
n extension
•Python
Package Index –pip and pip‐Win
•Exam
ples of u
sing Python
packages
This Week: Advanced Topics
•Ho
w do we take all we know
and
go furthe
r?•Keep
ing it specific to Pytho
n:–Parallel processing
–Ad
vanced
plotting an
d scientific compu
ting
–Spatial data in Pytho
n
Basic
s of P
arallel Processing
•Multip
le cores == workers
•Ho
w do we spread
work am
ong them
?–“M
ulti‐threaded
” or “parallel” processes
•In Pytho
n, th
e sim
plest techn
ique
:–import multiprocessing
Exam
ple 1: M
ultip
rocessing
•Ope
n the Python
file located in th
is week’s
/srcfolder
•Note that we are on
ly usin
g base Pytho
n packages
Multip
rocessing Co
nt’d.
•We can send
data to and
receive data back
from
individu
al workers
–To sh
are data between workers you
can
use
“thread and process s
afe” com
mun
ication:
•The multip
rocessing.Que
ueob
ject
•The orde
r in which th
e ite
ms a
re se
nt to
workers doe
s not correspon
d to th
e orde
r you
get the
m back whe
n using apply_async()
–If it matters, use Pool.map() instead
Multip
rocessing is Co
mplex
•This is barely sc
ratching
the surface of m
ulti‐
threaded
or p
arallel processes
•Piping, cores vs. th
reads, etc.
•Start‐up
and
overhead cost in
CPU
processing
•Learn more:
–http://docs.python
.org/2/library/multip
rocessing.html
–Multip
rocessing and ArcG
IS:
•http://blogs.esri.com
/esri/a
rcgis/20
11/08/29
/multip
rocessing/
Scientific Co
mpu
ting in Pytho
n
•Moving aw
ay from
ArcGISand toward more
gene
ral scientific com
putin
g
•Many packages goo
d for scientific com
putin
g–Often
platfo
rm sp
ecifc
–Be
st way to
get th
em?
•Pre‐bu
ilt platfo
rm
My Suggestio
infor P
ytho
n‐based
Scientific Co
mpu
ting
•An
acon
da–https://store.continuu
m.io/cshop
/anacond
a/–It’s free, based
on all ope
n source so
ftware
–Includ
es Pytho
n, num
py, m
atplotlib, scipy, sci‐kit,
etc.
–Ha
s its own package managem
ent called cond
awhich also
includ
es a C++
distrib
utable:
•e.g. at com
mand prom
pt: conda
install matplotlib
Exam
ples 2‐5, A
nacond
a
•We will use th
e Minicon
dadistrib
ution of
Anacon
da (fou
nd in
/lib of you
r assignm
ent)
and illustrate the use of th
e cond
apackage
managem
ent system
–http://rep
o.continuu
m.io/m
inicon
da/
–Which includ
es pip fo
r non
‐con
dapackages:
•conda
install pip
Installing Co
ndaPackages is Easy
•From
the command prom
pt:
For O
ur Examples:
condainstall scipy
condainstall matplotlib
condainstall pandas
condainstall pip
condainstall seaborn
condainstall husl
condainstall statsmodels
condainstall moss
condainstall scikit-learn
condainstall bokeh
Exam
ple 2 ‐Seabo
rnPlottin
g
•http://stanford.ed
u/~m
waskom/softw
are/seaborn/index.html
Exam
ple 3‐5 ‐B
okeh
Plottin
g
•http://bokeh
.pydata.org/index.html
Exam
ple 6‐7 –Plottin
g Maps a
nd
Spatial D
ata in Pytho
n•Usin
g the Ba
semap
extension to m
atplotlib
–http://m
atplotlib.org/basem
ap/users/examples.htm
l
•Install binaries:
–http://sou
rceforge.net/projects/matplotlib/files/matplotlib
‐too
lkits/
Reading and Writing Shapefilesin
Python
‐pyshp
•… with
out a
rcpy!
•https://code
.goo
gle.com/p/pyshp
/
pyshpInterface for S
hape
file
Creatin
g ShapefilesF
rom Scratch
This isn
’t the en
d…
•… if you
have any qu
estio
ns let’s ta
lk abo
ut
them
next w
eek!
•No specific lecture or to
pics will be covered
unless you
propo
se th
em•Time to work on
you
r projects
•Time to add
ress con
cerns w
ith m
e