Last modified 2 months ago Last modified on 04/17/17 10:37:40

Modified Mon Apr 17 10:37:40 2017 by Eric.Woldridge.

Challenge Problem #8: Desktop Activity Recognition


Desktop knowledge workers perform a wide variety of procedures or activities each day as they carry out their work. Much of this work is repetitive, so there has long been interest in providing intelligent assistance for these tasks. One such effort was the Activity Recognition and Proactive Assistance (ARPA) effort within the CALO project led by SRI International in 2006-7, which is the inspiration for this challenge problem.

Problem Overview

Students were recruited to execute a series of desktop workflows in response to incoming email messages. Windows desktop applications were instrumented to capture important events such as Email Send/Receive?, Save Attachment, Open file, SaveAs? file, visit web page, and so on.

Each desktop workflow is modeled as a sample from a Logical Hidden Markov Model (LoHMM; Kersting, et al, 2006; Natarajan, et al., 2008). Each observed event is labeled with the corresponding state that generated that observation. Teams will be given a labeled training set. Evaluation will be on a test set.

The user is modeled as having a library of parameterized workflows. Workflow executions are interleaved, so at each time step, the user may choose to switch to a previously-active workflow or create an instance of a new workflow and start executing it.

The application goal is to identify which workflow the user is currently executing and offer to perform steps (e.g., to complete the workflow). Our metrics are chosen with this goal in mind, although automated assistance is not part of the challenge problem.

Phase 1

Given entire sequences of observed events, learn the transition probabilities of the LoHMMs. Given a new event sequence, determine the MAP assignment of those events to states of LoHMM instances (including parameter bindings). Metrics: error rate.

Phase 2

Given a prefix from times 1 to 𝑡 of an event stream, determine the marginal posterior distribution over workflow names, parameter bindings, and states for all events in the prefix. Metric: Number of “prediction opportunities” where the posterior probability of the correct state exceeds a threshold 𝜏.

Evaluation Timeline

  • Phase 1:
    • Introduced June 2016
    • Evaluated Jan 2017
  • Phase 2:
    • Introduced January 2017
    • Evaluated July 2017