snap developer tutorials

8/13/2019 Snap Developer Tutorials

http://slidepdf.com/reader/full/snap-developer-tutorials 1/26

SnapLogic® Snap Developer Tutorials

Document Release: May 2012

SnapLogic, Inc.

71 East Third Avenue

San Mateo, California 94401

U.S.A.

www.snaplogic.com

http://www.snaplogic.com/



Copyright Information

© 2012 SnapLogic, Inc. All Rights Reserved. Terms and conditions, pricing, and

other information subject to change without notice. ”SnapLogic” and “SnapStore”

are among the trademarks of SnapLogic, Inc. All other product and company names

and marks mentioned are the property of their respective owners and are men-

tioned for identification purposes only. “SnapLogic” is registered in the U.S. and

Trademark Office.



Table of Contents

SnapLogic® Snap Developer Tutorials 1

Table of Contents 3

Overview 5

The Snap Development Model 5

Types of Snaps 5

Parts of a Snap 6

Outline of Snap Creation 6

Designing Connectivity Snaps 9

Building a Peer Reader C omponent 11

Building a Twitter Timeline Reader Component 19

- 3 -



SnapLogic ® Snap Developer Tutorials

- 4 -



OverviewSnaps are add-ons for the SnapLogic integration framework that are packaged for easy instal-

lation and distribution. These Snaps are usually sold through the SnapStore, but can also be

distributed within an organization as private Snaps.

In this tutorial, you will walk through the steps involved in creating a complete Snap, from the

high level design down to the code and packaging. The Snap will add Twitter capabilities to

SnapLogic. Twitter has a simple API and is easy to understand, while it's complex enough to

highlight common patterns in Snap development.

The Snap Development Model

Developing Snaps for the SnapLogic platform is a straightforward process. Although Snaps

vary in complexity and functionality, all Snaps follow the same pattern, use the same APIs,

and take advantage of the common functions provided by the SnapLogic platform.

However, developing Snaps is not the same as using SnapLogic. As a SnapLogic user, you are

creating resources and assembling Pipelines from existing Components and resources you

have already created. You are focusing on data flow and manipulation, not the lower level

details.

As a Snap developer, you have to view the problem differently. Rather than thinking about

how to use existing capabilities, you have to view things from the perspective of how you can

create Components and resources that others can use.

In order to gain that perspective, a good Snap developer has to be familiar with using Snap-

Logic first, and must understand the main elements of SnapLogic:

Resources and Components

l Pipelines

l Data Services

l Data types

(If you're just getting started with SnapLogic, you should read the SnapLogic user tutorials

first, become familiar with the basic concepts, and then come back here.)

Types of Snaps

There are two general categories of Snaps:

l Connectivity Snaps

l Solution Snaps

- 5 -

1

http://www.snaplogic.com/docs/tutorial/tutorial.htm

http://www.snaplogic.com/docs/tutorial/tutorial.htm




Connectivity Snaps add connectivity to an application or data source. The SalesForce.com,

NetSuite, and SugarCRM Snaps are examples of this type of Snap. These Snaps normally

include new Components that access the application API, and translate it to the SnapLogic API.

Solution Snaps are higher level Snaps which implement the business logic for a specific inte-

gration scenario, such as 'quote to bill' between a CRM system and a financial system. Solu-

tion Snaps normally include Pipeline and resource definitions that implement business logic.They might also include Components, or might depend on connectivity Components provided

by another Snap.

This Twitter Snap will be a connectivity Snap; it will add Twitter read and write capabilities to

SnapLogic, but will not provide any predefined Pipeline logic to process Tweets for a specific

purpose.

Parts of a Snap

A Snap is composed of three parts:

l Pipelines and Resources

l Components

l An installation program

All Snaps include an installation program. Connectivity Snaps will include primarily Com-

ponents and resources, while solutions Snaps will be oriented towards Pipelines and

resources.

Since this tutorial builds a connectivity Snap, you will include an installer, Components, and

some resources to get the users started.

Outline of Snap Creation

l Designing Connectivity Snaps

l Building a Peer Reader Component

l Building a Twitter Timeline Reader Component

l Building a Writer Component

l Defining capabilities

l Defining output views

l Execution

l Building a Transformer Component


l Defining views

l Execution

- 6 -



Overview

l Packaging Snaps for distribution

l Snap installer

l Adding coolness

l Implementing Suggest

l Implementing pass-through

l Error handling

l Advanced packaging

l Including predefined resources

l Generating resources automatically

See Also

l Submitting the Snap to the SnapLogic SnapStore

l Creating a Custom Component in Java

l Creating a Custom Component in Python

- 7 -

http://www.snaplogic.com/docs/snap-developer/create-components-python.htm






http://www.snaplogic.com/docs/snap-developer/create-components-java.htm






http://www.snaplogic.com/docs/snap-developer/snap-packaging.htm









- 8 -




Designing Connectivity SnapsThe first step in designing a Snap is to shift perspective to being a potential user of the Snap.

How will someone use your Snap? Use your knowledge of the application and how it's com-

monly integrated, to determine what should be exposed through your Snap, what should be

handled with the existing platform capabilities, and how the Snap will fit into the data flow

model. Some of the questions you should ask yourself are:

l Which application objects should be exposed as data?

l What data access is needed?

l What data needs to be read?

l What data needs to be written or updated?

l What application functions might need to be called?

l What transformations are likely to be used with the data?

l What utility functions from the application should be exposed?

Based the answers to those questions, you can start creating a model of the Snap. Some of

these capabilities will require new Components, others can be handled by creating Pipelines or

resources.

In general, a connectivity Snap will include a reader Component and a writer Component.

There might also be one or two function Components. For example, in CRM integration, the

'convert' functionality isn't a data source or target, and is best implemented as a trans-

formation or utility function.

In this case, you have to decide how to expose Twitter, and how to translate that into Com-

ponents others can use. Start with the Twitter API documentation to see what can be done.

The Twitter API exposes a lot of functionality. However, much of that functionality is oriented

towards building interactive Twitter clients. Since the users of this Snap are more likely to be

interested in bulk data processing with Twitter, you want to expose the following Twitter data

objects:

l Lists of friends

l Lists of followers

l Featured users

l Public timeline of tweets

l Tweets from a single user

l Tweets from a user's friends

- 9 -

2




Most of this data is intended to be read. You may also want to write (post) tweets, individually

or as a stream, so you will also implement that capability. Since Twitter supports reading user

profile information, this tutorial will support that as well.

Note: This tutorial does not implement any of the functions for managing Twitter

profiles and attributes, since that is limited to an individual Twitter account, and

would rarely be a batch operation.

The operations you'll be exposing from Twitter are basic reading and writing; there are no

application 'functions' that will be exposed (unlike the CRM 'convert' function.) For data trans-

formation, most of the operations the users will want are basic data movement, and string or

date processing. These can easily be handled by the existing transformations in SnapLogic.

However, you will be adding a capability to make Twitter data processing simpler - parsing of

the actual Twitter text into words and categories. This involves some natural language proc-

essing that isn't really data flow oriented, so it's best if you provide this as a separate utility

function.

In the end, this implementation of a Twitter Snap will include four Components:

l A Twitter Peer Reader for friend, follower, and user lists.

l A Twitter Timeline Reader for reading actual tweets.

l A Twitter Search Component for reading search results.

l A Twitter Writer for posting tweets.

l A Tweet profiler for parsing and categorizing tweet text.

The decision to create four Components is influenced by how you want to create views, how

you intend to manage properties, and general organization of the code for maintainability.

Keep reading, and those decisions will become clearer as you drill into the implementation.

- 10 -



Building a Peer Reader ComponentComponents are implemented as Python or Java classes, which are derived from the Snap-

Logic Component API class. The SnapLogic Component API documentation contains all the

information about the interface calls available to a Component author. Java class doc-

umentation can be found in the <version>/doc directory of the SnapLogic installation. Apart

from the necessary language specific differences, the Component API for Java and Python

components is the same. I am going to implement the essentials in this tutorial. You can also

refer to the Creating a Component documentation for more detail on Component devel-

opment. This tutorial uses Python for the Twitter Snap.

Start with the peer reader Component, which will read lists of friends and followers from

Twitter. There are several functions this Component will need to implement, as well as some

boilerplate code needed for all components:


l Defining a resource templates

l Defining properties

l Defining input views

l Validation

l Execution

Start with an empty template for the Component:from decimal import Decimal

from urllib2 import HTTPError

from snaplogic.common.snap_exceptions import SnapComponentError,

SnapComponentConfigError

from snaplogic.cc.component_api import ComponentAPI

import snaplogic.cc.prop as prop

from snaplogic.common import version_info

from snaplogic.common.data_types import SnapString, SnapNumber,

SnapDateTime

from snaplogic.snapi_base import keys

import twitter

class PeerRead(ComponentAPI):

''' SnapLogic component that reads twitter peers - friends and

followers '''

api_version = '1.0'

- 11 -

3

http://www.snaplogic.com/docs/snap-developer/snap-developer.htm#create-component.htm






component_version = '1.0'

component_label = 'Twitter Peer Reader'

component_description = 'Reads Twitter friends and followers

timelines'

#component_doc_uri =

#

'https://www.snaplogic.org/trac/wiki/Documentation/%s/ComponentRef/' %\

# version_info % version_info.doc_uri_version

capabilities = {

ComponentAPI.CAPABILITY_INPUT_VIEW_LOWER_LIMIT : 0,

ComponentAPI.CAPABILITY_INPUT_VIEW_UPPER_LIMIT : 0,

ComponentAPI.CAPABILITY_OUTPUT_VIEW_LOWER_LIMIT : 1,

ComponentAPI.CAPABILITY_OUTPUT_VIEW_UPPER_LIMIT : 1,

}

def __init__(self):

pass

def logmsg(self, msg):

"""Convenience method for logging a message."""

self.log(snap_log.LEVEL_INFO, msg)

def debug(self, msg):

"""Convenience method for logging a debug message."""

self.log(snap_log.LEVEL_DEBUG, msg)

def create_resource_template(self):

pass

def validate(self, err_obj):

pass

def suggest_resource_values(self, err_obj):

pass

def validate_runtime(self):

"""validation checks that are valid at design time and run time

"""

pass

def execute(self, input_views, output_views):

self.output_view.completed()

# $Id: PeerRead.py 291 2009-05-29 01:01:53Z mikeyp $

- 12 -



Building a Peer Reader Component

from decimal import Decimal



SnapComponentConfigError

from snaplogic.cc.component_api import ComponentAPIimport snaplogic.cc.prop as prop

from snaplogic.common import version_info


SnapDateTime


import twitter

class PeerRead(ComponentAPI):

''' SnapLogic component that reads twitter peers - friends and

followers '''

api_version = '1.0'

component_version = '1.0'

component_label = 'Twitter Peer Reader'

component_description = 'Reads Twitter friends and followers

timelines'


#

'https://www.snaplogic.org/trac/wiki/Documentation/%s/ComponentRef/' %

\


capabilities = {





}

def __init__(self):

# The types of peer information we can read, and the functions

to handle them

self.peer_lists = {'Featured': self.readFeaturedPeers,'User': self.readSinglePeer,

'Friends': self.readFriends,

'Followers': self.readFollowers

}

# predefined output view

self.output_view_def = (

- 13 -




('Id', SnapNumber, 'user ID'),

('Name', SnapString, ''),

('ScreenName', SnapString, ''),

('Location', SnapString, ''),

('Description', SnapString, ''),

('ProfileImageURL', SnapString, ''),

('BackgroundTile' , SnapString, ''),('BackgroundImageURL' , SnapString, ''),

('ProfileSidebarColor' , SnapString, ''),

('ProfileBackgroundColor', SnapString, ''),

('ProfileLinkColor' , SnapString, ''),

('ProfileTextColor' , SnapString, ''),

('Protected', SnapString, ''),

('UTCOffset', SnapString, ''),

('TimeZone' , SnapString, ''),

('URL', SnapString, ''),

('StatusCount', SnapNumber, ''),

('FollowersCount', SnapNumber, ''),

('FriendsCount', SnapNumber, ''),('FavouritesCount', SnapNumber, ''),

)








p = prop.SimpleProp('Username', SnapString,

'Twitter user name',

None,

False)

self.set_property_def('AuthUsername', p)

p = prop.SimpleProp('Password', SnapString,

'Twitter password',

{'obfuscate' : 0},

False)

self.set_property_def('AuthPassword', p)

p = prop.SimpleProp('Twitter User', SnapString,'Twitter user of interest, defaults to Username',

None,

False)

self.set_property_def('TwitterUser', p)

p = prop.SimpleProp('Peer Information', SnapString,

'Twitter peer data to read',

- 14 -




{'lov' : self.peer_lists.keys() },

False)

self.set_property_def('PeerInformation', p)

self.set_property_value('PeerInformation', 'Followers')

self.add_record_output_view_def('Output', self.output_view_def,

'Twitter Peers',False)


pass


pass


"""validation checks that are valid at design time and run time

"""pass


self.output_view = output_views.values()[keys.SINGLE_VIEW]

self.auth_user = self.get_property_value('AuthUsername')

self.auth_pw= self.get_property_value('AuthPassword')

self.twitter_user = self.get_property_value('TwitterUser')

self.t = twitter.Api(self.auth_user, self.auth_pw)

self.t.SetXTwitterHeaders('SnapLogic', 'http://www.snaplogic.com',

version_info.server_version)

self.peerlist_type = self.get_property_value('PeerInformation')

self.peer_lists[self.peerlist_type]()

def readFeaturedPeers(self):

raise('NotImplemented')

def readSinglePeer(self):

raise('NotImplemented')

def readFriends(self):

try:

for user in self.t.iterFriends(user=self.twitter_user,

count=None):

- 15 -




out_record = self.output_view.create_record()

out_record['Id'] = Decimal(user.id)

out_record['Name'] = unicode(user.name)

out_record['ScreenName'] = unicode(user.screen_name)

out_record['Location'] = unicode(user.location)

out_record['Description'] = unicode(user.description)out_record['ProfileImageURL'] = unicode(user.profile_image_

url)

out_record['BackgroundTile'] = unicode(user.profile_

background_tile)

out_record['BackgroundImageURL'] =

unicode(user.profile_background_image_url)

out_record['ProfileSidebarColor'] =

unicode(user.profile_sidebar_fill_color)

out_record['ProfileBackgroundColor'] =

unicode(user.profile_background_color)

out_record['ProfileLinkColor'] = unicode(user.profile_link_

color)out_record['ProfileTextColor'] = unicode(user.profile_text_

color)

out_record['Protected'] = unicode(user.protected)

out_record['UTCOffset'] = unicode(user.utc_offset)

out_record['TimeZone'] = unicode(user.time_zone)

out_record['URL'] = unicode(user.url)

out_record['StatusCount'] = Decimal(user.statuses_count)

out_record['FollowersCount'] = Decimal(user.followers_

count)

out_record['FriendsCount'] = Decimal(user.friends_count)

out_record['FavouritesCount'] = Decimal(user.favourites_

count)

self.output_view.write_record(out_record)

except HTTPError, e:

if e.code == 401:

raise SnapComponentError("Failed to log in to twitter ")

else:

raise (e)


def readFollowers(self):

try:

for user in self.t.iterFollowers(user=self.twitter_user,count=None):


out_record['Id'] = Decimal(user.id)

out_record['Name'] = unicode(user.name)

- 16 -




out_record['ScreenName'] = unicode(user.screen_name)

out_record['Location'] = unicode(user.location)

out_record['Description'] = unicode(user.description)

out_record['ProfileImageURL'] = unicode(user.profile_image_

url)

out_record['BackgroundTile'] = unicode(user.profile_

background_tile)out_record['BackgroundImageURL'] =

unicode(user.profile_background_image_url)

out_record['ProfileSidebarColor'] =

unicode(user.profile_sidebar_fill_color)

out_record['ProfileBackgroundColor'] =

unicode(user.profile_background_color)

out_record['ProfileLinkColor'] = unicode(user.profile_link_

color)

out_record['ProfileTextColor'] = unicode(user.profile_text_

color)

out_record['Protected'] = unicode(user.protected)

out_record['UTCOffset'] = unicode(user.utc_offset)out_record['TimeZone'] = unicode(user.time_zone)

out_record['URL'] = unicode(user.url)

out_record['StatusCount'] = Decimal(user.statuses_count)

out_record['FollowersCount'] = Decimal(user.followers_

count)

out_record['FriendsCount'] = Decimal(user.friends_count)

out_record['FavouritesCount'] = Decimal(user.favourites_

count)



if e.code == 401:raise SnapComponentError("Failed to log in to twitter ")

else:

raise (e)


- 17 -



- 18 -




Building a Twitter Timeline ReaderComponent# $Id: StatusRead.py 291 2009-05-29 01:01:53Z mikeyp $

from decimal import Decimal


import datetime

import twitter


SnapComponentConfigErrorfrom snaplogic.cc.component_api import ComponentAPI

import snaplogic.cc.prop as propfrom snaplogic.common import version_

info


SnapDateTime


class StatusRead(ComponentAPI):

''' SnapLogic component that reads twitter status updates '''

api_version = '1.0'

component_version = '1.0'component_label = 'Twitter Status Reader'

component_description = 'Reads Twitter statuses and timelines'


#

'https://www.snaplogic.org/trac/wiki/Documentation/%s/ComponentRef/' %

\


capabilities = {





}

def __init__(self):

self.timelines = {'Public': self.readPublicTimeline,

'User': self.readUserTimeline,

- 19 -

4




'Friends': self.readFriendsTimeline,

'Single': self.readStatus,

}

'The types of timelines we can read, and the functions to handle

them'

self.output_view_def = (

('CreatedAt', SnapDateTime, 'Time of the Tweet '),('TweetId', SnapNumber, 'Tweet ID '),

('Tweet', SnapString, 'The Tweet'),

('UserId', SnapNumber, 'Twitter user ID'),

('UserScreenName', SnapString, 'Twitter user screen name'),

('ReplyToId', SnapNumber, 'Reply to Tweet ID'),

('ToUserId', SnapNumber, 'Reply to Twitter user ID'),

('ToUserScreenName', SnapString, 'Reply to Twitter screen nam

('Source', SnapString, 'Origin of the Tweet'),

('Location', SnapString, 'Location of the Tweet'),

('MinId', SnapNumber, 'The minimum Tweet id returned so far')

('MaxId', SnapNumber, 'The maximum Tweet id returned so far')

)'predefined output view'








p = prop.SimpleProp('Username', SnapString,

'Twitter user name',

None, False)

self.set_property_def('AuthUsername', p)

p = prop.SimpleProp('Password', SnapString,

'Twitter password',

{'obfuscate' : 0},

False)

self.set_property_def('AuthPassword', p)

p = prop.SimpleProp('Timeline', SnapString,

'Twitter timeline to read',

{'lov' : self.timelines.keys() },False)

self.set_property_def('Timeline', p)

self.set_property_value('Timeline', 'Public')

p = prop.SimpleProp('Twitter User', SnapString,

'Twitter user of interest, defaults to Username ',

None,

- 20 -



Building a Twitter Timeline Reader Component

False)

self.set_property_def('TwitterUser', p)

p = prop.SimpleProp('After Id', SnapString,

'Read statuses with id greater than this.',

None,

False)

self.set_property_def('SinceId', p)p = prop.SimpleProp('Before Id', SnapString,

'Read statuses with id less than or equal to this.',

None,

False)

self.set_property_def('MaxId', p)

self.add_record_output_view_def('Output', self.output_view_de

'Twitter Tweets', False)


"""validation checks that are valid at design time and

run time

"""

# validation notes

# public timeline accepts since

# friends, user might require login

# public timeline: since id

# friends timeline: since id, since (date)

# user timeline: : since id , since (date) public

# single : since id ,

# XXX python-twitter supports dates, not documented in API?

# XXX what is the precedence for since/max id, since they are# mutually exclusive

pass


pass


"""validation checks which are deferred to run time

"""

pass


self.output_view = output_views.values()[keys.SINGLE_VIEW]

self.auth_user = self.get_property_value('AuthUsername')

self.auth_pw= self.get_property_value('AuthPassword')

self.twitter_user = self.get_property_value('TwitterUser')

- 21 -




if self.twitter_user is None or len(self.twitter_user) == 0:

self.twitter_user = self.auth_user

self.timeline_type = self.get_property_value('Timeline')

self.since_id = self.get_property_value('SinceId')

self.max_id = self.get_property_value('MaxId')

self.validate_runtime()

self.t = twitter.Api(self.auth_user, self.auth_pw)

self.t.SetXTwitterHeaders('SnapLogic', 'http://www.snaplogic.com'

version_info.server_version)

self.timelines[self.timeline_type]()

def readUserTimeline(self):

""" proces timeline for an individual user """

try:

for stat, min, max in \

self.t.iterUserTimeline(user=self.twitter_user,count=None,

since_id=self.since_id,

max_id = self.max_id):


# <created_at> Wed Apr 22 10:33:40 +0000 2009

out_record['CreatedAt'] =\

datetime.datetime.strptime(stat.created_at,

'%a %b %d %H:%M:%S +0000 %Y')

out_record['TweetId'] = Decimal(stat.id)

out_record['Tweet'] = unicode(stat.text)

out_record['UserId'] = Decimal(stat.user.id)

out_record['UserScreenName'] = unicode(stat.user.screen_nif (stat.in_reply_to_status_id) is None:

out_record['ReplyToId'] = None

else:

out_record['ReplyToId'] = Decimal(stat.in_reply_to_status

if (stat.in_reply_to_user_id is None):

out_record['ToUserId'] = None

else:

out_record['ToUserId'] = Decimal(stat.in_reply_to_user_id

if (stat.in_reply_to_screen_name is None):

out_record['ToUserScreenName'] = None

else:

out_record['ToUserScreenName'] =unicode(stat.in_reply_to_screen_name)

out_record['Source'] = unicode(stat.source)

out_record['Location'] = unicode(stat.user.location)

out_record['MinId'] = Decimal(min)

out_record['MaxId'] = Decimal(max)

- 22 -



Building a Twitter Timeline Reader Component



if e.code == 401:


else:

raise (e)


def readPublicTimeline(self):

""" process the public timeline """

try:


self.t.iterPublicTimeline(count=None,

since_id=self.since_id):




datetime.datetime.strptime(stat.created_at,'%a %b %d %H:%M:%S +0000 %Y')




out_record['UserScreenName'] = unicode(stat.user.screen_n

if (stat.in_reply_to_status_id) is None:


else:

out_record['ReplyToId'] = Decimal(stat.in_reply_to_status



else:out_record['ToUserId'] = Decimal(stat.in_reply_to_user_id



else:

out_record['ToUserScreenName'] =

unicode(stat.in_reply_to_screen_name)

out_record['Source'] = unicode(stat.source)

out_record['Location'] = unicode(stat.user.location)



self.output_view.write_record(out_record)except HTTPError, e:

if e.code == 401:


else:

raise (e)


- 23 -




def readFriendsTimeline(self):

try:


self.t.iterFriendsTimeline(count=None,user=self.twitter_u

since_id=self.since_id):out_record = self.output_view.create_record()



datetime.datetime.strptime(stat.created_at,

'%a %b %d %H:%M:%S +0000 %Y')




out_record['UserScreenName'] = unicode(stat.user.screen_n

if (stat.in_reply_to_status_id) is None:


else:out_record['ReplyToId'] = Decimal(stat.in_reply_to_status



else:

out_record['ToUserId'] = Decimal(stat.in_reply_to_user_id



else:

out_record['ToUserScreenName'] =

unicode(stat.in_reply_to_screen_name)

out_record['Source'] = unicode(stat.source)out_record['Location'] = unicode(stat.user.location)





if e.code == 401:


else:

raise (e)


def readStatus(self):pass

- 24 -



C connectivity Snaps

designing 9

S

Snaps

designing connectivity Snaps 9

development model 5

parts 6types 5

- 25 -

Index

snap developer tutorials

Documents