Reagent: Converting Ordinary Webpages into Interactive Software Agents

We introduce Reagent, a technology that can be used in conjunction with automated speech recognition to allow users to query and manipulate ordinary webpages via speech and pointing. Reagent can be used out-of-the-box with third-party websites, as it requires neither special instrumentation from website developers nor special domain knowledge to capture semantically-meaningful mouse interactions with structured elements such as tables and plots. When it is unable to infer mappings between domain vocabulary and visible webpage content on its own, Reagent proactively seeks help by engaging in a voice-based interaction with the user.

Reference

Matthew Peveler, Jeffrey O. Kephart, Hui Su, "Reagent: Converting Ordinary Webpages into Interactive Software Agents,"

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, July 2019, 6560-562. 

Bibtex

@inproceedings{ijcai2019-956,
  title     = {Reagent: Converting Ordinary Webpages into Interactive Software Agents},
  author    = {Peveler, Matthew and Kephart, Jeffrey O. and Su, Hui},
  booktitle = {Proceedings of the Twenty-Eighth International Joint Conference on
               Artificial Intelligence, {IJCAI-19}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},             
  pages     = {6560--6562},
  year      = {2019},
  month     = {7},
  doi       = {10.24963/ijcai.2019/956},
  url       = {https://doi.org/10.24963/ijcai.2019/956},
}