Python Code Examples

Word Spotting

import sys

fname1 = "c:\Python Course\ex1.txt"

for line in open(fname1,'r').readlines():

for word in line.split():

if word.endswith('ing'):

print word

Creating a Dictionary of First Names

def createNameDict():dictNameFile=open('project/dictionaries/names.txt','r')dictContent=dictNameFile.read() #read all the filedictWords=dictContent.split(",") #return a list with the wordsnameDict={} # initialize a dictionaryfor word in dictWords:

nameDict[word.strip()]=" " #enters each word to the dctionary.return nameDict

Computing Accuracy Results I# anfiles.py# Program to analyze the results of speaker identification.# Illustrates Python dictionarys

import string, glob, sys

def main(): # read correct file and test file fname1 = sys.argv[1] fname2 = sys.argv[2] text1 = open(fname1,'r').read() text1 = string.lower(text1) words1 = string.split(text1) correct_len = len(words1)

text2 = open(fname2,'r').read() text2 = string.lower(text2) words2 = string.split(text2)

Computing Accuracy Results II # construct a dictionary of correct results correct = {} for w in words1: correct[w] = 1 for i in range(correct_len): in_count = 0 portion2 = words2[:i+1] for w in portion2: if correct.get(w,0) > 0: in_count+=1 accuracy = float(in_count)/float(len(portion2)) print "%5d, %5d,%.2f" % (len(portion2), in_count, accuracy)

if __name__ == '__main__': main()

Word Histograms

import sre, string

pattern = sre.compile( r'[a-zA-Z]+' )

def countwords(text): dict = {} try: iterator = pattern.finditer(text) for match in iterator: word = match.group() try: dict[word] = dict[word] + 1 except KeyError: dict[word] = 1 except sre.error: pass # triggers when first index goes to -1, terminates loop.

Word Histograms

items = [] for word in dict.keys(): items.append( (dict[word], word) ) items.sort() items.reverse() return items

# if run as a script, count words in stdin.if __name__ == "__main__": import sys x = countwords( sys.stdin.read() ) s = map(str, x) t = string.joinfields(s, "\n") print t

Extracting People Names and Company Namesimport string, sre, glob, sys

def createNameDict():dictNameFile=open('names.txt','r')dictContent=dictNameFile.read() #read all the filedictWords=dictContent.split(",") #return a list with the wordsnameDict={} # initialize a dictionaryfor word in dictWords:

nameDict[word.strip()]=" " #enters each word to the dctionary.return nameDict

def main(): # read file fname1 = sys.argv[1] text1 = open(fname1,'r').read() namesDic = createNameDict()

CompanySuffix = sre.compile(r'corp | ltd | inc | corporation | gmbh | ag | sa ', sre.IGNORECASE)

pattern = sre.compile( r'([A-Z]\w+[ .,-]+)+'

Extracting People Names and Company Names r'(corp|CORP|Corp|ltd|Ltd|LTD|inc|Inc|INC|corporation|Corporation|

CORPORATION|gmbh|GMBH|ag|AG|sa|SA)' r'(\.?)') pattern1 = sre.compile( r'([A-Z]\w+[\s.-]*){2,4}' )

#Companies capitalWords=sre.finditer(pattern,text1) for match in capitalWords: CapSeq = match.group() print CapSeq #People capitalWords1=sre.finditer(pattern1,text1) for match in capitalWords1: wordList=match.group().split()

#check name in names dictionaryif namesDic.has_key(wordList[0].strip()):

print match.group() if __name__ == '__main__': main()

NLTK

NLTK defines a basic infrastructure that can be used to build NLP programs in Python. It provides: Basic classes for representing data relevant to natural

language processing. Standard interfaces for performing tasks, such as

tokenization, tagging, and parsing. Standard implementations for each task, which can be

combined to solve complex problems. Extensive documentation, including tutorials and

reference documentation.

RE Show

>>> from nltk.util import re_show>>> string = """... It’s probably worth paying a premium for funds

that invest in markets... that are partially closed to foreign investors, such

as South Korea, ...... """>>> re_show(’t...’, string)I{t’s }probably wor{th p}aying a premium for funds {that}

inves{t in} markets {that} are par{tial}ly closed {to f}oreign inves{tors}, such as Sou{th K}orea, ...

>>>

Classes in Python

Defining Classes

>>> class SimpleClass: ... def __init__(self, initial_value): ... self.data = initial_value ... def set(self, value): ... self.data = value ... def get(self): ... print self.data ... >>> x = SimpleClass(4)

Inheritance

B is a subclass of A

>>> class B(A):

... def __init__(self):

SimpleTokenizer implements the interface of TokenizerI

>>> class SimpleTokenizer(TokenizerI):

... def tokenize(self, str):

... words = str.split()

... return [Token(words[i], Location(i))

... for i in range(len(words))]

Inheritance Example

class point: def __init__(self, x=0, y=0): self.x, self.y = x, y

class cartesian(point): def distanceToOrigin(self):

return floor(sqrt(self.x**2 + self.y**2)) class manhattan(point): def distanceToOrigin(self):

return self.x + self.y

Sets

Sets in Python

The sets module provides classes for constructing and manipulating unordered collections of unique elements. Common uses include: membership testing, removing duplicates from a sequence, and computing standard math operations on sets such

as intersection, union, difference, and symmetric difference.

Like other collections, sets support x in set, len(set), and for x in set. Being an unordered collection, sets do not record element position or order of insertion. Accordingly, sets do not support indexing, slicing, or other sequence-like behavior.

Some Details about Implementation

Most set applications use the Set class which provides every set method except for __hash__(). For advanced applications requiring a hash method, the ImmutableSet class adds a __hash__() method but omits methods which alter the contents of the set.

The set classes are implemented using dictionaries. As a result, sets cannot contain mutable elements such as lists or dictionaries.

However, they can contain immutable collections such as tuples or instances of ImmutableSet.

For convenience in implementing sets of sets, inner sets are automatically converted to immutable form, for example, Set([Set(['dog'])]) is transformed to Set([ImmutableSet(['dog'])]).

Set Operations

Operation  Equivalent  Result len(s)   cardinality of set s   x in s

  test x for membership in sx not in s

  test x for non-membership in ss.issubset(t)

s <= t test whether every element in s is in ts.issuperset(t) s >= t test whether every element in t is in s   s.union(t)

s | t new set with elements from both s and ts.intersection(t)

s & t new set with elements common to s and ts.difference(t)

s - t new set with elements in s but not in ts.symmetric_differe

nce(t) s ^ tnew set with elements in either s or t but

not boths.copy()

  new set with a shallow copy of s

Operations not for ImmutableSet

Operation  Equivalent  Result 

s.union_update(t) s |= t return set s with elements added from t

s.intersection_update(t) s &= t

return set s keeping only elements also found in t

s.difference_update(t) s -= t

return set s after removing elements found in t

s.symmetric_difference_update(t)

s ^= treturn set s with elements from s or t but not

both 

s.add(x)   add element x to set s

s.remove(x) 

remove x from set s; raises KeyError if not present

s.discard(x)   removes x from set s if present

s.pop() 

remove and return an arbitrary element from s; raises KeyError if empty

s.clear()   remove all elements from set s

Set Examples

>>> from sets import Set>>> engineers = Set(['John', 'Jane', 'Jack', 'Janice'])>>> programmers = Set(['Jack', 'Sam', 'Susan', 'Janice'])>>> managers = Set(['Jane', 'Jack', 'Susan', 'Zack'])>>> employees = engineers | programmers | managers #

union>>> engineering_management = engineers & managers #

intersection>>> fulltime_management = managers - engineers - programmers

# difference>>> engineers.add('Marvin') # add element>>> print engineersSet(['Jane', 'Marvin', 'Janice', 'John', 'Jack'])>>> employees.issuperset(engineers) # superset testFalse

Set Examples

>>> employees.union_update(engineers) # update from another set

>>> employees.issuperset(engineers)True>>> for group in [engineers, programmers, managers, employees]:... group.discard('Susan') # unconditionally remove

element... print group...Set(['Jane', 'Marvin', 'Janice', 'John', 'Jack'])Set(['Janice', 'Jack', 'Sam'])Set(['Jane', 'Zack', 'Jack'])Set(['Jack', 'Sam', 'Jane', 'Marvin', 'Janice', 'John', 'Zack'])

Google API

Get it from http://sourceforge.net/projects/pygoogle/

A Python wrapper for the Google web API. Allows you to do Google searches, retrieve pages from the Google cache, and ask Google for spelling suggestions.

Utilizing the Google API - I

import sysimport stringimport codecsimport google

print '<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">'print '<head>'print ' <title>Google with Python</title>'print '</head>'print '<body>'

print '<h1>Google with Python</h1>'

google.LICENSE_KEY = '[YOUR GOOGLE LICENSE KEY]'sys.stdout = codecs.lookup('utf-8')[-1](sys.stdout)query = “Your Query" data = google.doGoogleSearch(query)

Utilizing the Google API - IIprint '<p><strong>1-10 of "' + query + '" total results for 'print str(data.meta.estimatedTotalResultsCount) + '</strong></p>'for result in data.results: title = result.title title = title.replace('<b>', '<strong>') title = title.replace('</b>', '</strong>')

snippet = result.snippet snippet = snippet.replace('<b>','<strong>') snippet = snippet.replace('</b>','</strong>') snippet = snippet.replace('<br>','<br />') print '<h2><a href="' + result.URL + '">' + title + '</a></h2>' print '<p>' + snippet + '</p>'print '</body>‘print '</html>'

Yahoo API

http://pysearch.sourceforge.net/ http://python.codezoo.com/pub/component/

4193?category=198 This project implements a Python API for the

Yahoo Search Webservices API. pYsearch is an OO abstraction of the web services, with emphasis on ease of use and extensibility.

URLLIB

This module provides a high-level interface for fetching data across the World Wide Web.

In particular, the urlopen() function is similar to the built-in function open(), but accepts Universal Resource Locators (URLs) instead of filenames.

Some restrictions apply -- it can only open URLs for reading, and no seek operations are available.

Urllib Syntax

# Use http://www.someproxy.com:3128 for http proxying

proxies = {'http': 'http://www.someproxy.com:3128'}

filehandle = urllib.urlopen(some_url, proxies=proxies)

# Don't use any proxies

filehandle = urllib.urlopen(some_url, proxies={})

URLLIB Examples

Here is an example session that uses the "GET" method to retrieve a URL containing parameters:

>>> import urllib>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s"

% params)>>> print f.read() The following example uses the "POST" method instead: >>> import urllib>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query",

params)>>> print f.read()

What is a Proxy

A proxy server is a computer that offers a computer network service to allow clients to make indirect network connections to other network services.

A client connects to the proxy server, then requests a connection, file, or other resource available on a different server.

The proxy provides the resource either by connecting to the specified server or by serving it from a cache.

In some cases, the proxy may alter the client's request or the server's response for various purposes.

A proxy server can also serve as a firewall.