![Page 1: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/1.jpg)
Back to FME School3-Day webinar series
![Page 2: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/2.jpg)
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
![Page 3: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/3.jpg)
Let’s begin today’s lesson...
![Page 4: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/4.jpg)
Potent Potions for Batch Data
Processing
![Page 5: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/5.jpg)
The Heavy Lifters: Two key Transformers for Bulk Data Processing
Meet the
FeatureReader & FeatureWriter
![Page 6: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/6.jpg)
FeatureReader (since FME 2011)
Solves problems such as:
1. Read data within a selected boundary
2. Read data within a user drawn area3. Process a directory of files
4. Read in data FME has written in the same workspace
![Page 7: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/7.jpg)
FeatureReader
![Page 8: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/8.jpg)
FeatureWriter (since FME 2016)
Do things AFTER the data is written in FME Workbench.
Less need for Shutdown Python or TCL
![Page 9: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/9.jpg)
● copy data after writing● FTP results● Email an attachment● Load into Dropbox or S3● Insert data into HTML file● Database validation● FME Server Notification● 3rd Party Integration● Can Write in Parallel!
![Page 10: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/10.jpg)
FeatureReader and WriterIn the Real World
Web Client with FME
![Page 11: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/11.jpg)
FeatureReader & WriterIn the Real World
![Page 12: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/12.jpg)
FeatureWriter Demo
![Page 13: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/13.jpg)
Now Let’sBatch...
![Page 14: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/14.jpg)
250,000 CAD files & rasterson mobile devices
![Page 15: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/15.jpg)
![Page 16: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/16.jpg)
Tip: Know Your Potions and Choose Wisely
![Page 17: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/17.jpg)
Today’s Potions1. Wildcards
2. Batch Deploy
3. Parent/child Workspaces
4. Parent/child Server Workspaces
![Page 18: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/18.jpg)
Potion 1: One Wild<card>
Dataset
![Page 19: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/19.jpg)
Multi-Dataset Picker
![Page 20: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/20.jpg)
Multi-Dataset Picker
![Page 21: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/21.jpg)
Multi-Dataset Picker
![Page 22: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/22.jpg)
Multi-Dataset Picker
ShapefileMapInfo
Most rasters
DWGDGNSQLite
![Page 23: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/23.jpg)
Dataset Wildcards
Extended glob syntax:
Symbol Matches
? Any single character
* Any sequence of zero or more characters
[chars] Any single character in chars.
[a-d] Any character between a and d inclusive
{a,b,...} Any of the sub-patterns a, b
/**/ 0 or more subdirectories
![Page 24: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/24.jpg)
Time to brewPotion 1
![Page 25: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/25.jpg)
Wildcard Bulk Data Processing Enticements
✓ Simple to set up
✓ Can transform across file boundaries
- Needs memory & time
![Page 26: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/26.jpg)
Wildcard Bulk Data Processing
Pitfalls
x Recovery from data errors difficult
x Feature Type vs File vs Format Issues
x No granular logx No ability to
parallelize
![Page 27: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/27.jpg)
Potion 2: Batch Deploy
![Page 28: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/28.jpg)
Batch Deploy Script Writer
![Page 29: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/29.jpg)
Batch Deploy Script Writer
![Page 30: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/30.jpg)
Batch Deploy Script Writer
![Page 31: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/31.jpg)
Batch Deploy Script Writer
![Page 32: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/32.jpg)
Batch Deploy Script Writer
![Page 33: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/33.jpg)
Batch Deploy Script Writer
![Page 34: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/34.jpg)
Time to brewPotion 2
![Page 35: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/35.jpg)
Batch Deploy Enticements
✓ Simple to set up✓ Runs quickly✓ Can script via
command line✓ Run on demand
![Page 36: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/36.jpg)
Batch Deploy Pitfalls
x Recovery from data errors difficult
x No granular logx Destination dataset
naming can be tricky
![Page 37: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/37.jpg)
Potion 3: Parent/ChildWorkspaces
![Page 38: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/38.jpg)
Parent/Child Workspace Ingredients
• Parent Workspace:– PathReader– WorkspaceRunner
• Child Workspace:– FeatureWriter
![Page 39: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/39.jpg)
Parent Ingredients
![Page 40: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/40.jpg)
Parent Ingredients
![Page 41: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/41.jpg)
Child Ingredients
![Page 42: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/42.jpg)
Time to brewPotion 3
![Page 43: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/43.jpg)
Parent/Child Workspace Enticements
✓ Separate transformation from workflow
✓ Generate audit logs✓ All authored within
Workbench
![Page 44: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/44.jpg)
Parent/ChildWorkspace
Pitfalls
x Not all writers can be used concurrently
x Slow to run each child workspace separately
x Recovery from data errors not easy if concurrent runs used
![Page 45: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/45.jpg)
Potion 4: Parent/Child Server
Workspaces
![Page 46: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/46.jpg)
Parent/Child Server
Workspace Ingredients
• Parent Workspace:– PathReader– FMEServerJobSubmitter– FMEServerJobWaiter
• Child Workspace:– FeatureWriter
![Page 47: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/47.jpg)
Parent Ingredients
![Page 48: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/48.jpg)
Parent Ingredients
![Page 49: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/49.jpg)
Parent Ingredients
![Page 50: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/50.jpg)
Child Ingredients
![Page 51: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/51.jpg)
Time to brewPotion 4
![Page 52: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/52.jpg)
Parent/Child Server
Workspace Enticements
✓ Separate transformation from workflow
✓ Generate audit logs✓ All authored within
Workbench✓ Make full use of
parallelism = FAST
![Page 53: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/53.jpg)
Parent/ChildServer
Workspace Pitfalls
x Not all writers can be used concurrently
x Data needs to be accessible to Server Engines- Consider using Server Data Resources
x Craft your reload/audit plan
![Page 54: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/54.jpg)
Summary● Many ways to handle bulk data moves
● Choose your potion wisely - each has pluses and minuses
● FeatureReader/FeatureWriter increase flexibility
● FME Server is the most robust automation choice
![Page 56: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/56.jpg)
Ultimate Real-Time Monitor Anything, Update Anything
![Page 57: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/57.jpg)
FME Server Notification Service lets you act on events as they happen, and send information as it becomes available.
Publications and Subscriptions let you publish to and monitor different systems and data.
![Page 58: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/58.jpg)
NotificationsWhat they are for
What they are not for
![Page 59: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/59.jpg)
NotificationsWhat they are for
✓ A brief message, usually to trigger an action.
What they are not forx Transmitting large amounts
of spatial data.
![Page 60: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/60.jpg)
NotificationsWhat they are for
✓ A brief message, usually to trigger an action.
✓ Triggering an FME Server response to an event that happened outside of FME.
What they are not forx Transmitting large amounts
of spatial data.x Triggering an FME Server
response to a continuous series of messages (many per second).
![Page 61: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/61.jpg)
NotificationsWhat they are for
✓ A brief message, usually to trigger an action.
✓ Triggering an FME Server response to an event that happened outside of FME.
✓ Sending a message about something that happened in FME.
What they are not forx Transmitting large amounts
of spatial data.x Triggering an FME Server
response to a continuous series of messages (many per second).
x Sending more than one message per second about what’s happening in FME.
![Page 62: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/62.jpg)
FME Server Notification Service
See the list of what built-in publications and subscribers can be: http://fme.ly/protocols
![Page 63: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/63.jpg)
Workflows for“Ultimate Real-Time”
Poll it Push it Update it
![Page 64: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/64.jpg)
Poll it“Are we there yet?”
![Page 65: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/65.jpg)
Polling Workflow1. Change detection
○ GeoRSS reader○ Caching○ Timestamps
2. Run the workspace on a scheduleTimestamp
![Page 66: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/66.jpg)
Scheduling (FME Server)
![Page 67: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/67.jpg)
GeoRSS Reader
![Page 68: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/68.jpg)
GeoRSS Reader
• Contains change detection logic
• Option to only read new features
![Page 69: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/69.jpg)
Caching
• Whole datasets or timestamps
• ChangeDetector
• SQL - match on ID
![Page 70: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/70.jpg)
ChangeDetector Example
![Page 71: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/71.jpg)
SQL to find different features:@Value(uuid) = "uuid" AND @Value(aqi) != "aqi"
SQL Example
![Page 72: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/72.jpg)
Timestamps
● Cache the timestamp
● Last updated attribute
● Filter features newer than timestamp
● Add a timestamp attributes to your dataset
![Page 73: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/73.jpg)
Reads last cached time and replaces with current time
![Page 74: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/74.jpg)
Checks against timestamp and filters out old features
![Page 75: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/75.jpg)
Pros and ConsCaching whole datasets
✓ Easy to set up and detect changes.x Need to store entire
dataset - processing time?
Caching timestamps
✓ No storage needed.x Harder to set up -
variables, datetimes.
Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?
![Page 76: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/76.jpg)
Story: Real-Time Traffic
![Page 77: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/77.jpg)
![Page 78: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/78.jpg)
![Page 79: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/79.jpg)
✓ Poll Waze feed
✓ Parse XML
✓ Filter
✓ Transform
✓ Validate
✓ Email notifications
![Page 80: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/80.jpg)
Workflows for“Ultimate Real-Time”
Poll it Push it Update it
![Page 81: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/81.jpg)
Push It
![Page 82: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/82.jpg)
Pushing Methods• Database triggers• Webhooks
In FME Server:• Direct URL• REST API
![Page 83: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/83.jpg)
Database Triggers
● Databases can trigger events.
● Functions with HTTP requests can push data to FME Server.
![Page 84: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/84.jpg)
Database Triggers
![Page 85: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/85.jpg)
Webhooks
● i.e. Give systems/services a URL that will respond.
● Send the notification to FME Server.
● Queue the received notifications and process the requests on a separate thread.
![Page 86: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/86.jpg)
Pros and ConsDatabase Triggers
✓ Pushes right to FME Server topic.
✓ Can push lots of changes to a holding table.
✓ Perfect for real-time database changes.x Database permissions.
Webhooks
✓ No need to waste effort polling.
✓ Truly real time.x Complex to set up.x Not all systems provide
or accept webhooks.
![Page 87: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/87.jpg)
Direct URLSystems can use HTTP to push data to topics or trigger workspaces to run.
![Page 88: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/88.jpg)
![Page 89: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/89.jpg)
Topic POST URL
![Page 90: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/90.jpg)
REST APIIn-depth interaction
with FME Server
● Send/receive notifications● Run jobs● Manage connections● Manage users● Licensing● Backups● Scheduling● Etc!
![Page 91: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/91.jpg)
Workflows for“Ultimate Real-Time”
Poll it Push it Update it
![Page 92: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/92.jpg)
To Poll or Push?
![Page 93: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/93.jpg)
Poll vs. Push Considerations
● Simplicity vs. “real-time”ness● Can the system push to FME Server?● Are jobs too long for polling?● API restrictions for polling?
![Page 94: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/94.jpg)
Tip: Use FME Server for internal systems and FME Cloud for external systems.
![Page 95: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/95.jpg)
Update It
![Page 96: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/96.jpg)
Updating Methods
• FME Writerso Bulk
drop/load/overwriteo Update option
• SQLExecuter• HTTPCaller
![Page 97: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/97.jpg)
FME Writers: UPDATE operation
![Page 98: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/98.jpg)
INSERT, UPDATE or DELETE in the middle of a workflow.
![Page 99: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/99.jpg)
POST data or changes to systems.
![Page 100: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/100.jpg)
Summary
• Polling workflow:o Change detection + scheduling
• Pushing options:o Database triggers, webhookso FME Server: URL or REST API
• Updating:o Operations in writer
parameterso SQLExecutoro HTTPCaller
![Page 101: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/101.jpg)
Questions?
Database triggers tutorial:fme.ly/dbtriggers
Webhooks etc:fme.ly/web
REST API documentation:fme.ly/restv3
![Page 102: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/102.jpg)
FME Server: Charms for the
Enterprise
![Page 103: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/103.jpg)
Enterprise-Level Charms
★ Security - a new level with new charms
★ Deployment - move solutions easily
★ Automate the enterprise
★ Jobs Control
![Page 104: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/104.jpg)
Security Charm #1Active Directory
![Page 105: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/105.jpg)
✓ Any number of Active Directory domains✓ Any number of users per domain✓ A hybrid model - mix server and AD users
Active Directory
Functionality based on feedback and experience from users! Thank you!
![Page 106: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/106.jpg)
Active Directory Support
Import Active Directory
Users
Hybrid security model
![Page 107: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/107.jpg)
Over to FME
![Page 108: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/108.jpg)
Security Charm #2
Sharing and user control
![Page 109: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/109.jpg)
✓ Users can share with other users or groups.
✓ Sharing defined for everything except “shared resources”.
✓ Users are isolated from each other.
✓ No load on Administrator.
Familiar Sharing Model (Google, Dropbox, etc.)
Modern Sharing Model
![Page 110: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/110.jpg)
User-Controlled Security
✓ Users have control of all FME Server components
✓ Similar to state-of-the-art services like Google Drive.
![Page 111: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/111.jpg)
Security Charm #3Connections & 3rd-Party Applications
![Page 112: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/112.jpg)
Connection Based 3rd Party Integration
Why use connections?
● Encapsulate user credentials in a secure manner.
● Define for Databases and Web Applications
● Share across multiple workspaces.
● Share without revealing credentials.
![Page 113: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/113.jpg)
Connection Based 3rd Party Integration
Many web connections.● Amazon Web Services● OAuth 2.0 (tens of
thousands of services)● Token● HTTP Authentication
All FME-supported databases.
Demo
![Page 114: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/114.jpg)
Database Connections
![Page 115: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/115.jpg)
Web ConnectionsMany web connections are shipped with FME
![Page 116: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/116.jpg)
Web Connections
More are being added toFME Hub all the time
![Page 117: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/117.jpg)
Security Charm #4We secure your infrastructure too.
![Page 118: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/118.jpg)
On Premises vs FME Cloud
On-premise:
● you secure your infrastructure
● you configure FME Server security
FME Cloud:
● you configure FME Server security
![Page 119: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/119.jpg)
FME Cloud SecurityWe manage data, application and transmission security.
● HTTPS only
● Annual security audit
● Automated security patching
● Leverage AWS Compliance - Industry leaders
● Control firewall rules (protocols & ports)
![Page 120: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/120.jpg)
Deployment CharmEasily move solutions
![Page 121: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/121.jpg)
Challenge: How can you move an FME Server
solution from one server to another?
![Page 122: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/122.jpg)
Your FME Server solutions usually consist of many components.
Workspaces Subscriptions Schedules
Publications Resources Connections
Custom Formats Custom Transformers Topics
![Page 123: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/123.jpg)
FME Server Projects
Portable Server SolutionsPlace your screenshot here
![Page 124: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/124.jpg)
FME Server ProjectsPortable way to move, distribute, and share FME Server solutions.
Benefit #1 - Move a project from one machine to another.
Development StagingTest Production
![Page 125: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/125.jpg)
Projects.
Benefit #2 - Backup (export) a project any time for archiving.
FME Server Projects
![Page 126: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/126.jpg)
Projects.
Benefit #3 - Share your projects.
First - Export your project!
Projects.
Project Export
FME Server Projects
![Page 127: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/127.jpg)
Projects.
Benefit #3 - Share your projects
Next - Send to those with whom you want to share it.
Projects.
Project Export
FME Server Projects
![Page 128: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/128.jpg)
Projects.
Benefit #3 - Share your projects Finally - Receivers simply import the project.
Projects.
Project Export
FME Server Projects
![Page 129: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/129.jpg)
Projects.
Benefit #4 - Publish Projects to FME Hub. (Coming Soon)
Share FME Server solutions with the FME community.
Projects.
Project Export
FME Server Projects
![Page 130: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/130.jpg)
Automation Charm
![Page 131: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/131.jpg)
EnterpriseAutomation
This charm is so powerful that we have another presentation
just on this.
![Page 132: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/132.jpg)
Termination Charm
![Page 133: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/133.jpg)
Goal: Ensure FME Engines don’t get stuck on a job.
Workflows are often dependent on 3rd Party applications/services.
An engine may hang or take too long waiting on responses.
![Page 134: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/134.jpg)
For any job you now have a new setting (Schedules and run Workspace)
Max elapsed time in seconds job will run.If time expires job is killed!
Charm: Specify maximum time a job can run.
![Page 135: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/135.jpg)
Charm: Specify maximum time a job can run.
If job takes longer than allocated then the job is cancelled.
![Page 136: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/136.jpg)
Summary - FME Server 2017 Charms✓ Enterprise Level Security: Enable fine-tuned control with
the new security model.
✓ Enterprise Deployment Model: Move server solutions between machines using Projects.
✓ Enterprise Automation: Automate everything.*
✓ Enterprise Level Job Control: Define max job time for your jobs.
![Page 137: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/137.jpg)
That wraps up today’s classes...
![Page 138: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/138.jpg)
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
![Page 139: Back to FME School - Day 2: Your Data and FME](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a64bbf47f8b9ac21c8b5143/html5/thumbnails/139.jpg)
Questions?
knowledge.safe.com