Real World Activity Recognition with Multiple

Goals

Derek Hao Hu

With Sinno Pan, Vincent Zheng, Nathan Liu, Qiang Yang

Hong Kong University of Science and Technology

Ubicomp 2008 Presentation

COEX, Seoul, Korea

September 21, 2008

Activity Recognition Applications

Location-based services, suggest routes via activity / goal recognition

Security-related applications

Assisting the sick and the disabled.

Keywords

Activity Recognition Input: Sensor readings (may be various kinds) across

a number of time slices Output: The actions / goals inferred at each time slice

Concurrent: several goals are pursued in the same time slice

Interleaving: goals are pursued non-consecutively, in that one goal is paused and then resumed after a while during which time another goal is being pursued.

ExamplesIf you are enjoying your breakfast…(Action 1)

With an egg boiling on the stove… (Action 2)

The egg is now boiling!

What should you do now?

PAUSE Action 1 and Start Action 2

After that, just go back and resume Action 1, with your peeled egg and your unfinished breakfast.

Interleaving activities

ExamplesIf you…are enjoying your breakfast…

(Action 1)

[SHOULDN’T IT BE LUNCH? OK OK, If you are having your LUNCH] (Action 1)

And you are watching TV at THE SAME TIME? (Action 2)

Action 1: Having Lunch and Action 2: Watching TV are happening at the same time slice, therefore, it’s…

ConcurrentActivities

In real-world situations, how often do users pursue goals at once in a concurrent and interleaving manner?

If humans often pursue several goals in a concurrent and interleaving manner, are we able to detect such complex social and behavioral patterns from sensors alone?

Do these goals have any deep association with the algorithm complexity of the solutions that are aimed at recognizing the goals?

Questions we plan to answer

Psychologists believe…

“Unlike the sequences directed to a single goal in a simple or technical plan, human intended action is influenced by multiple goals.” [Oatley 1992]

The main characteristic of human planning is to reason about problems arising from multiple goals [Wilensky 1983]

Similar claims can be found in educational psychology, cognitive science and anthropology.

How we answer these questions

Three angles 1. Investigate the dataset, showing that pursuing

multiple goals is commonplace in human activities 2. Propose a solution using Conditional Random

Field (CRF) for this multiple goal recognition problem

3. Analyze the granularity of goal composition graphs, and later show that different accuracies can be achieved in different levels of the hierarchy.

Related Work There has been many papers related to activity recognition and due

to time and space constraints, we can only list a few. [Patterson et al. 2003] Inferring high-level behavior from low-level sensors [Patterson et al. 2004] Opportunity knocks: A system to provide cognitive

assistance with transportation services [Intille et al. 2006] Using a live-in laboratory for ubiquitous computing research [Logan et al. 2007] A long-term evaluation of sensing modalities for activity

recognition [Hodges and Pollack, 2007] The use of electronic sensors for human identification

There are many other related papers in AI conferences, check Henry Kautz, Liao Lin, Jie Yin, Qiang Yang’s paper for more algorithmic solutions.

However, to our best knowledge, no paper before has formally posed out the multiple goal activity recognition problem and try to analyze it with real world datasets as we do.

Analysis of Dataset: Goal Hierarchy

MIT PlaceLab House_n PLIA1 Dataset Recorded on Friday 03/04/2005 from 9am to 12am

with a volunteer in PlaceLab Lowest level of activities are extracted from the

original data, including activities such as “sweeping”, “washing-ingredients”, etc.

Relevant activities are combined together into some higher level activities, such as “dealing-with-clothes”

At the highest level, 9 categories of activities are differentiated from each other.

Goal Hierarchy

Subfigure 1

(Please check Subfigure 2 and 3 in my homepage: http://www.cse.ust.hk/~derekhh/)

Existence of multiple goals

Goal composition types

Previous approaches try to tackle activity recognition / goal recognition problems in these two types only.

Our approach plans to extend the activity recognition and the possible goal composition types to all the five possible types here.

Our Proposed Approach: Skip-Chain Conditional Random Field (SCCRF)

For technical details, please refer to:Derek Hao Hu and Qiang Yang, CIGAR: Concurrent and Interleaving Goal and Activity Recognition, in AAAI 2008.

Two problems we need to solve:1) Q: How do we add the “skip edges” between nodes? A: Learn the posterior probability , then set a

predefined threshold for setting such “skip edges”.2) Q: What is the feature function we are using in this CRF model? A: The feature function is highly domain and application dependent.

We can refer to Liao Lin and Douglas Vail’s work for some suggestions about some good examples of feature functions in location-based activity recognition.

( | , )i j kP A A G

Intuitively speaking, we are using the long-distance dependencies to capture the relatedness between interleaving activities.

Our Proposed Approach: minimizing objective function in goal graph

1

0.32 1

0.93 0.27 1

0.48 0.13 0.72 1

S

Our goal is to learn a similarity matrix between different goals, and use this matrix to minimize an objective function described below.Such an approach is for tackling concurrent goals.

Still two questions we need to answer:1) Why would a similarity matrix be useful?

In real-world examples, it may be more likely that goal A is being pursued if goal B is known to being pursued at the same time, e.g. “eating-dinner” and “sitting-at-table”. Dissimilarity matrix may also help, but we didn’t discuss it here.

2) What is the intuitive explanation of the objective function?

2 ' 2

, {1,..., }

min ( ) ( )i j ij i ii j m

P P S P P

If the similarity between two goals are large enough, the probability inferred would be small, constrained by the first part.

For technical details, please refer to:Derek Hao Hu and Qiang Yang, CIGAR: Concurrent and Interleaving Goal and Activity Recognition, in AAAI 2008.

This part is set as a “regularizer” such that there will not be too much difference between the tuned probability and the initial probability inferred from the CRF.

Do Humans Pursue Multiple Goals? (Question 1) A1: YES

1. Proportion of concurrent goals increase as levels go higher.2. The number is really large ,more than 50% at the highest level3. As the window size get bigger, concurrent goals are more frequently.

Figure Time

Can we accurately predict multiple goals? (Q2) A2: YES

This is our algorithm, compared with the commonly used Naïve Bayes algorithm on a WiFi dataset we had collected in our CSE Department Office.

Two parameters are tuned with different values to show the stability of our algorithm.

Differences in goal hierarchies? (Q3)

Experiment on the MIT PlaceLab dataset, with accuracies tested on different levels of the goal hierarchy we had constructed, still shows promising improvement over Naïve Bayes.

Another small experiment

We designed this experiment since we are curious about the “learnability”, “sensitivity” or “differentiability” of different kinds of sensors. We would like to know, would there be a big change when we use different kinds of sensor readings for training and testing? What would the result be?

The experiment is simple and we plan to look into this problem further in our future discussions.

Future Discussions

Some future topics: 1) Can we do better when there are multiple users?

Multiple-user multiple-goal activity recognition Distinguish between different users with different actions

Actions may collaborate, compete…as in Wilensky’s description

2) Can we construct a goal hierarchy automatically from the observation sequences? Learn which actions are more similar that can be constructed together

toward a higher level of action 3) Can we automatically choose the “best” granularity with a few

labeled sensor readings? Aid the application where a certain accuracy criteria must be met.

Thanks for your attention!

Questions?