A Graphical Monitoring and Notification Flow Language (MNFL) for User-

Configurable In-Home Assistive Monitoring

Alex Edgcomb

Oral Qualification

March 2nd, 2011

The Goal: User-configurable notification systems for in-home use

Identify personat front door

Fall down at home

In bathroom for 1 hour

Napping too long

2 of 25

Not enough configurability

3 of 25

GrandCare

BeClose

QuietCare

Too much configurability

4 of 25

X10(Also dangerous) SmartHome

Outside too long at night

Not safe to be outside at night

Grandma has Alzheimer’s

Want to be notified if she is for long

5 of 25

Defining the system

Send a text message if someone leaves, withoutreturning, for more than 10 minutes at night.

Leaving = motion to the right Returning = motion to the left

Night = 10PM to 5AM

6 of 25

Microsoft VPL

7 of 25

Microsoft VPL• If-statements with >1 variable require Join

• Join operator only takes two values at a time• Join waits for both variables to receive data

• Asynchronous data sources leads to raise condition.• If-statements require all variables using in If-statement and

Calculations done afterward.• Hard (if not impossible) to increment on a raising/falling edge

8 of 25

Lego Mindstorms v2.0

• Lacks global variables.• Variables cannot be read/written more than once.• No sense of actual time.

9 of 25

ScratchNew functionality leads to:

1.Redundant code

2.Complex code

Reason: Independent behavior is not separated away.

For example, add a 2nd door:

1.Create second code block

2.Embed new timer/variable

10 of 25

RIBS and RIMS

CS120B (Introduction to Embedded Systems) students take 5+ weeks to correctly describe exact behavior.

inWindow = (time >= 10PM || time <= 5AM)

var Time 10minWait = time

Period length = 1 second

1) Define synch SM in RIBS.

2) Automatically convert to C code in RIMS.

11 of 25

MNFL

System Pros Cons

Microsoft VPL Flow language style is intuitive for streaming applications

If-statements do not handle asynchronous data

Lego Mindstorm Easy to learn for people of age 10+

Variables too restrictive

Scratch Graphical cues indicate purpose/usage of blocks

Temporal programming does not handle new functionality well

RIBS and RIMS Precise behavioral description

Challenging to learn

12 of 25

MNFL Data Types

13 of 25

Boolean – True / False

Tone – A pair of {frequency, amplitude}

Sound – Stream of tones

Picture – 2D array of pixels

Video – Stream of pictures

Number – Negative and positive whole numbers

MNFL Modifiers of Data Types

14 of 25

Snapshot

Stream

Snippet

MNFL Core blocks

15 of 25

Video demo

16 of 25

YouTube version of video:http://www.youtube.com/user/eslucr#p/u/9/xpaOkhYBXT8

In bathroom for over 1 hour

17 of 25

Got up 5 or more times at night

18 of 25

“Got up” = was motion, then no motion for 10 minutes

Take 1 pill in the morning

19 of 25

KPN Execution semantics

Kahn processing networks (KPN) Processes communicate through channels

When all input channels have a token, the process executes and outputs tokens

20 of 25Kahn, G. The Semantics of a Simple Language for Parallel Programming. Information Processing 1974. North-Holland Publishing Company, 1974.

SDF Execution semantics

• Synchronous data flow• Process = KPN process• Channel = KPN channel• Difference:

• Processes read and write a constant number of tokens.

21 of 25

Lee, Edward A. and David G. Messerschmitt. Synchronous Data Flow. Proceedings of the IEEE, vol. 75, no. 9, p 1235-1245, September, 1987.

MNFL Execution semantics

• Monitoring and notification network language (MNFL)• Block = KPN process• Connection = KPN channel• Difference

• Block’s execute when any input channel has a token

22 of 25

MNFL block SM

• Inside each block is a state machine.

• Q = queue of inputs I = current input• P = previous input O = output

23 of 25

Implementation figures(as of 02/27/11)

• 8,000 total lines of ActionScript• 700 lines for discrete event simulation engine• 400 lines for base block class

• 80% related to graphical appearance and behavior• 50 – 600 lines for each block

• Largest two blocks:• Send message - high configurability• Fall extractor - detecting falls from silhouettes

• 26 fully implemented blocks• Block spawn, deletion and connection creation take

less than 1 ms on a single core 3.0GHz PC.

24 of 25

Conclusions and Future work• EasyNotify

• Easy to use compared to currently available systems

• Core blocks contain in-home functionality• MNFL

• Data types tailored to in-home monitoring sensors

• Non-restrictive block communication

1. Human subject trials to improve usability2. Develop mechanism for users to build and

integrate new blocks3. Move computations to cloud

25 of 25