ultimate real-time — monitor anything, update anything

Download Ultimate Real-Time — Monitor Anything, Update Anything

Post on 21-Mar-2017

212 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

  • Ultimate Real-Time Monitor Anything, Update Anything

  • FME Server Notification Service lets you act on events as they happen, and send information as it becomes available. Publications and Subscriptions let you publish to and monitor different systems and data.

  • Notifications What they are for

    What they are not for

  • Notifications What they are for

    A brief message, usually to trigger an action.

    What they are not for x Transmitting large amounts of

    spatial data.

  • Notifications What they are for

    A brief message, usually to trigger an action.

    Triggering an FME Server response to an event that happened outside of FME.

    What they are not for x Transmitting large amounts of

    spatial data. x Triggering an FME Server

    response to a continuous series of messages (many per second).

  • Notifications What they are for

    A brief message, usually to trigger an action.

    Triggering an FME Server response to an event that happened outside of FME.

    Sending a message about something that happened in FME.

    What they are not for x Transmitting large amounts of

    spatial data. x Triggering an FME Server

    response to a continuous series of messages (many per second).

    x Sending more than one message per second about whats happening in FME.

  • FME Server Notification Service

    See the list of what built-in publications and subscribers can be: http://fme.ly/protocols

  • Workflows for Ultimate Real-Time

    Poll it Push it Update it

  • Poll it Are we there yet?

  • Polling Workflow 1. Change detection

    GeoRSS reader Caching Timestamps

    2. Run the workspace on a schedule

    Timestamp

  • Scheduling (FME Server)

  • GeoRSS Reader

  • GeoRSS Reader

    Contains change detection logic

    Option to only read new features

  • Caching

    Whole datasets or timestamps

    ChangeDetector

    SQL - match on ID

  • ChangeDetector Example

  • SQL to find different features: @Value(uuid) = "uuid" AND @Value(aqi) != "aqi"

    SQL Example

  • Timestamps

    Cache the timestamp

    Last updated attribute

    Filter features newer than timestamp

    Add a timestamp attributes to your dataset

  • Reads last cached time and replaces with current time

  • Checks against timestamp and filters out old features

  • Pros and Cons Caching whole datasets

    Easy to set up and detect changes.

    x Need to store entire dataset - processing time?

    Caching timestamps

    No storage needed. x Harder to set up -

    variables, datetimes.

    Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?

  • Story: Real-Time Traffic

  • Poll Waze feed

    Parse XML

    Filter

    Transform

    Validate

    Email notifications

  • Workflows for Ultimate Real-Time

    Poll it Push it Update it

  • Push It

  • Pushing Methods Database triggers Webhooks

    In FME Server:

    Direct URL REST API

  • Database Triggers

    Databases can trigger events.

    Functions with HTTP

    requests can push data to FME Server.

  • Database Triggers

  • Webhooks

    i.e. Give systems/services a URL that will respond.

    Send the notification to

    FME Server. Queue the received

    notifications and process the requests on a separate thread.

  • Pros and Cons Database Triggers

    Pushes right to FME Server topic.

    Can push lots of changes to a holding table.

    Perfect for real-time database changes.

    x Database permissions.

    Webhooks

    No need to waste effort polling.

    Truly real time. x Complex to set up. x Not all systems provide

    or accept webhooks.

  • Direct URL Systems can use HTTP to push data to topics or trigger workspaces to run.

  • Topic POST URL

  • REST API In-depth interaction

    with FME Server

    Send/receive notifications Run jobs Manage connections Manage users Licensing Backups Scheduling Etc!

  • Workflows for Ultimate Real-Time

    Poll it Push it Update it

  • To Poll or Push?

  • Poll vs. Push Considerations Simplicity vs. real-timeness Can the system push to FME Server? Are jobs too long for polling? API restrictions for polling?

  • Tip: Use FME Server for internal systems and FME Cloud for external systems.

  • Update It

  • Updating Methods

    FME Writers o Bulk drop/load/overwrite o Update option

    SQLExecuter HTTPCaller

  • FME Writers: UPDATE operation

  • INSERT, UPDATE or DELETE in the middle of a workflow.

  • POST data or changes to systems.

  • Summary

    Polling workflow: o Change detection + scheduling

    Pushing options: o Database triggers, webhooks o FME Server: URL or REST API

    Updating: o Operations in writer parameters o SQLExecutor o HTTPCaller

  • Questions?

    Database triggers tutorial: fme.ly/dbtriggers

    Webhooks etc:

    fme.ly/web

    REST API documentation: fme.ly/restv3

Recommended

View more >