Google; Google DeepMind
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina N Toutanova
instruction following;web tasks;user interface tasks;vision and language;representation learning;reinforcement learning;imitation learning;tree search;language grounding;web agents;computer control
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces