* [37] N. Bussmann, P. Giudici, D. Marinelli, and J. Papenbrock, “Explainable machine learning in credit risk management,” _Computational Economics_ , vol. 57, no. 1, pp. 203–216, 2021.
  * [38] I. Basu and S. Maji, “Multicollinearity correction and combined feature effect in shapley values,” in _Australasian Joint Conference on Artificial Intelligence_. Springer, 2022, pp. 79–90.
  * [39] M. T. Ribeiro, S. Singh, and C. Guestrin, ““Why should I trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [40] G. Loveleen, B. Mohan, B. S. Shikhar, J. Nz, M. Shorfuzzaman, and M. Masud, “Explanation-driven hci model to examine the mini-mental state for alzheimer’s disease,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 2022.
  * [41] M. S. Kamal, N. Dey, L. Chowdhury, S. I. Hasan, and K. Santosh, “Explainable AI for glaucoma prediction analysis to understand risk factors in treatment planning,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 71, pp. 1–9, 2022.
  * [42] C. M. Viana, M. Santos, D. Freire, P. Abrantes, and J. Rocha, “Evaluation of the factors explaining the use of agricultural land: a machine learning and model-agnostic approach,” _Ecological Indicators_ , vol. 131, p. 108200, 2021\.
  * [43] I. A. Khan, N. Moustafa, I. Razzak, M. Tanveer, D. Pi, Y. Pan, and B. S. Ali, “Xsru-iomt: Explainable simple recurrent units for threat detection in internet of medical things networks,” _Future Generation Computer Systems_ , vol. 127, pp. 181–193, 2022.
  * [44] X. Huang, S. Jamonnak, Y. Zhao, T. H. Wu, and W. Xu, “A visual designer of layer-wise relevance propagation models,” in _Computer Graphics Forum_ , vol. 40, no. 3. Wiley Online Library, 2021, pp. 227–238.
  * [45] J. Sun, S. Lapuschkin, W. Samek, and A. Binder, “Explain and improve: Lrp-inference fine-tuning for image captioning models,” _Information Fusion_ , vol. 77, pp. 233–246, 2022.
  * [46] S. Gholizadeh and N. Zhou, “Model explainability in deep learning based natural language processing,” _arXiv preprint arXiv:2106.07410_ , 2021.
  * [47] A. Nguyen, F. Krause, D. Hagenmayer, and M. Färber, “Quantifying explanations of neural networks in e-commerce based on lrp,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2021, pp. 251–267.
  * [48] R. Achtibat, M. Dreyer, I. Eisenbraun, S. Bosse, T. Wiegand, W. Samek, and S. Lapuschkin, “From” where” to” what”: Towards human-understandable explanations through concept relevance propagation,” _arXiv preprint arXiv:2206.03208_ , 2022.
  * [49] T. Idé, A. Dhurandhar, J. Navrátil, M. Singh, and N. Abe, “Anomaly attribution with likelihood compensation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 5, 2021, pp. 4131–4138.
  * [50] L. Utkin and A. Konstantinov, “An extension of the neural additive model for uncertainty explanation of machine learning survival models,” in _Cyber-Physical Systems: Intelligent Models and Algorithms_. Springer, 2022, pp. 3–13.
  * [51] Q. Chen, G. Pan, W. Chen, and P. Wu, “A novel explainable deep belief network framework and its application for feature importance analysis,” _IEEE Sensors Journal_ , vol. 21, no. 22, pp. 25 001–25 009, 2021.
  * [52] A. Nascita, A. Montieri, G. Aceto, D. Ciuonzo, V. Persico, and A. Pescapé, “Xai meets mobile traffic classification: Understanding and improving multimodal deep learning architectures,” _IEEE Transactions on Network and Service Management_ , vol. 18, no. 4, pp. 4225–4246, 2021.
  * [53] C. Wu, H. Zhang, J. Chen, Z. Gao, P. Zhang, K. Muhammad, and J. Del Ser, “Vessel-GAN: Angiographic reconstructions from myocardial CT perfusion with explainable generative adversarial networks,” _Future Generation Computer Systems_ , vol. 130, pp. 128–139, 2022.
  * [54] M. Zolanvari, Z. Yang, K. Khan, R. Jain, and N. Meskin, “TRUST XAI: Model-agnostic explanations for AI with a case study on IIoT security,” _IEEE Internet of Things Journal_ , 2021.
  * [55] L. Antwarg, R. M. Miller, B. Shapira, and L. Rokach, “Explaining anomalies detected by autoencoders using shapley additive explanations,” _Expert Systems with Applications_ , vol. 186, p. 115736, 2021.
  * [56] K. Zhang, J. Zhang, P.-D. Xu, T. Gao, and D. W. Gao, “Explainable AI in deep reinforcement learning models for power system emergency control,” _IEEE Transactions on Computational Social Systems_ , 2021.
  * [57] M. Liu, J. Shi, K. Cao, J. Zhu, and S. Liu, “Analyzing the training processes of deep generative models,” _IEEE transactions on visualization and computer graphics_ , vol. 24, no. 1, pp. 77–87, 2017.
  * [58] P. Y. Wang, S. Galhotra, R. Pradhan, and B. Salimi, “Demonstration of generating explanations for black-box algorithms using lewis,” _Proceedings of the VLDB Endowment_ , vol. 14, no. 12, pp. 2787–2790, 2021\.
  * [59] J. Peng, K. Zou, M. Zhou, Y. Teng, X. Zhu, F. Zhang, and J. Xu, “An explainable artificial intelligence framework for the deterioration risk prediction of hepatitis patients,” _Journal of Medical Systems_ , vol. 45, no. 5, pp. 1–9, 2021.
  * [60] V. Nagisetty, L. Graves, J. Scott, and V. Ganesh, “xAI-GAN: Enhancing generative adversarial networks via explainable AI systems,” _arXiv preprint arXiv:2002.10438_ , 2020.
  * [61] M. S. Kamal, A. Northcote, L. Chowdhury, N. Dey, R. G. Crespo, and E. Herrera-Viedma, “Alzheimer’s patient analysis using image and gene expression data and explainable-AI to present associated genes,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 70, pp. 1–7, 2021.
  * [62] M. Moradi and M. Samwald, “Post-hoc explanation of black-box classifiers using confident itemsets,” _Expert Systems with Applications_ , vol. 165, p. 113941, 2021.
  * [63] M. Nourani, C. Roy, J. E. Block, D. R. Honeycutt, T. Rahman, E. Ragan, and V. Gogate, “Anchoring bias affects mental model formation and user reliance in explainable AI systems,” in _26th International Conference on Intelligent User Interfaces_ , 2021, pp. 340–350.
  * [64] M. Lee, J. Jeon, and H. Lee, “Explainable AI for domain experts: A post hoc analysis of deep learning for defect classification of TFT–LCD panels,” _Journal of Intelligent Manufacturing_ , pp. 1–13, 2021.
  * [65] S. Knapič, A. Malhi, R. Saluja, and K. Främling, “Explainable artificial intelligence for human decision support system in the medical domain,” _Machine Learning and Knowledge Extraction_ , vol. 3, no. 3, pp. 740–770, 2021.
  * [66] Á. Delgado-Panadero, B. Hernández-Lorca, M. T. García-Ordás, and J. A. Benítez-Andrades, “Implementing local-explainability in gradient boosting trees: Feature contribution,” _Information Sciences_ , 2022\.
  * [67] Y. Zhang, D. Defazio, and A. Ramesh, “Relex: A model-agnostic relational model explainer,” in _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , 2021, pp. 1042–1049.
  * [68] T. Srinath and H. Gururaja, “Explainable machine learning in identifying credit card defaulters,” _Global Transitions Proceedings_ , 2022.
  * [69] N. Prentzas, M. Pitsiali, E. Kyriacou, A. Nicolaides, A. Kakas, and C. S. Pattichis, “Model agnostic explainability techniques in ultrasound image analysis,” in _2021 IEEE 21st International Conference on Bioinformatics and Bioengineering (BIBE)_. IEEE, 2021, pp. 1–6.
  * [70] P. R. Magesh, R. D. Myloth, and R. J. Tom, “An explainable machine learning model for early detection of parkinson’s disease using lime on datscan imagery,” _Computers in Biology and Medicine_ , vol. 126, p. 104041, 2020\.
  * [71] Z. U. Ahmed, K. Sun, M. Shelly, and L. Mu, “Explainable artificial intelligence (XAI) for exploring spatial variability of lung and bronchus cancer (LBC) mortality rates in the contiguous USA,” _Scientific reports_ , vol. 11, no. 1, pp. 1–15, 2021.
  * [72] T. Spinner, U. Schlegel, H. Schäfer, and M. El-Assady, “explainer: A visual analytics framework for interactive and explainable machine learning,” _IEEE transactions on visualization and computer graphics_ , vol. 26, no. 1, pp. 1064–1074, 2019.
  * [73] A. Kuppa and N.-A. Le-Khac, “Adversarial xai methods in cybersecurity,” _IEEE transactions on information forensics and security_ , vol. 16, pp. 4924–4938, 2021.
  * [74] S. Saifullah, D. Mercier, A. Lucieri, A. Dengel, and S. Ahmed, “Privacy meets explainability: A comprehensive impact benchmark,” _arXiv preprint arXiv:2211.04110_ , 2022.
  * [75] Y. Wang, J. Lam, and H. Lin, “Consensus of linear multivariable discrete-time multiagent systems: Differential privacy perspective,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 12, pp. 13 915–13 926, 2022.
  * [76] O. A. Wahab, A. Mourad, H. Otrok, and T. Taleb, “Federated machine learning: Survey, multi-level classification, desirable criteria and future directions in communication and networking systems,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 2, pp. 1342–1397, 2021.
  * [77] G. Rjoub, O. A. Wahab, J. Bentahar, and A. Bataineh, “Trust-driven reinforcement selection strategy for federated learning on iot devices,” _Computing_ , pp. 1–23, 2022.
  * [78] T. T. Huong, T. P. Bac, K. N. Ha, N. V. Hoang, N. X. Hoang, N. T. Hung, and K. P. Tran, “Federated learning-based explainable anomaly detection for industrial control systems,” _IEEE Access_ , vol. 10, pp. 53 854–53 872, 2022.
  * [79] J. L. C. Bárcena, P. Ducange, A. Ercolani, F. Marcelloni, and A. Renda, “An approach to federated learning of explainable fuzzy regression models,” in _2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)_. IEEE, 2022, pp. 1–8.
  * [80] Z. Müftüoğlu, M. Kızrak, and T. Yıldırım, “Privacy-preserving mechanisms with explainability in assistive ai technologies,” _Advances in Assistive Technologies: Selected Papers in Honour of Professor Nikolaos G. Bourbakis–Vol. 3_ , pp. 287–309, 2022.
  * [81] S. Milli, L. Schmidt, A. D. Dragan, and M. Hardt, “Model reconstruction from model explanations,” in _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , 2019, pp. 1–9.
  * [82] F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction $\\{$APIs$\\}$,” in _25th USENIX security symposium (USENIX Security 16)_ , 2016, pp. 601–618.
  * [83] R. Shokri, M. Strobel, and Y. Zick, “Privacy risks of explaining machine learning models,” 2019.
  * [84] F. Harder, M. Bauer, and M. Park, “Interpretable and differentially private predictions,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 4083–4090.
  * [85] S. Löbner, W. B. Tesfay, T. Nakamura, and S. Pape, “Explainable machine learning for default privacy setting prediction,” _IEEE Access_ , vol. 9, pp. 63 700–63 717, 2021.
  * [86] G. Rjoub, O. Abdel Wahab, J. Bentahar, and A. Bataineh, “A trust and energy-aware double deep reinforcement learning scheduling strategy for federated learning on IoT devices,” in _International Conference on Service-Oriented Computing_. Springer, 2020, pp. 319–333.
  * [87] G. Rjoub, O. A. Wahab, J. Bentahar, and A. Bataineh, “Trust-driven reinforcement selection strategy for federated learning on IoT devices,” _Computing_ , pp. 1–23, 2022.
  * [88] N. Drawel, J. Bentahar, A. Laarej, and G. Rjoub, “Formal verification of group and propagated trust in multi-agent systems,” _Autonomous Agents and Multi-Agent Systems_ , vol. 36, no. 1, p. 19, 2022.
  * [89] R. Machlev, M. Perl, J. Belikov, K. Levy, and Y. Levron, “Measuring explainability and trustworthiness of power quality disturbances classifiers using XAI-explainable artificial intelligence,” _IEEE Transactions on Industrial Informatics_ , 2021.
  * [90] A. Kuppa and N.-A. Le-Khac, “Black box attacks on explainable artificial intelligence (XAI) methods in cyber security,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [91] G. Rjoub, J. Bentahar, and O. A. Wahab, “Explainable AI-based federated deep reinforcement learning for trusted autonomous driving,” in _2022 International Wireless Communications and Mobile Computing (IWCMC)_. IEEE, 2022, pp. 318–323.
  * [92] G. Rjoub, J. Bentahar, O. A. Wahab, and N. Drawel, “One-shot federated learning-based model-free reinforcement learning,” in _The International Conference on Deep Learning, Big Data and Blockchain (DBB 2022)_. Springer, 2022, pp. 39–52.
  * [93] G. Rjoub, J. Bentahar, and Y. Joarder, “Active federated yolor model for enhancing autonomous vehicles safety,” in _Mobile Web and Intelligent Information Systems: 18th International Conference, MobiWIS 2022, Rome, Italy, August 22–24, 2022, Proceedings_. Springer, 2022, pp. 49–64.
  * [94] H. Mankodiya, M. S. Obaidat, R. Gupta, and S. Tanwar, “XAI-AV: Explainable artificial intelligence for trust management in autonomous vehicles,” in _2021 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI)_. IEEE, 2021, pp. 1–5.
  * [95] H. Elayan, M. Aloqaily, and M. Guizani, “Internet of behavior (IoB) and explainable AI systems for influencing IoT behavior,” _arXiv preprint arXiv:2109.07239_ , 2021.
  * [96] O. A. Wahab, “Intrusion detection in the IoT under data and concept drifts: Online deep learning approach,” _IEEE Internet of Things Journal_ , vol. 9, no. 20, pp. 19 706–19 716, 2022.
  * [97] R. R. Karn, P. Kudva, H. Huang, S. Suneja, and I. M. Elfadel, “Cryptomining detection in container clouds using system calls and explainable machine learning,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 32, no. 3, pp. 674–691, 2020.
  * [98] M. Wang, K. Zheng, Y. Yang, and X. Wang, “An explainable machine learning framework for intrusion detection systems,” _IEEE Access_ , vol. 8, pp. 73 127–73 141, 2020.
  * [99] G. Baryannis, S. Dani, and G. Antoniou, “Predicting supply chain risks using machine learning: The trade-off between performance and interpretability,” _Future Generation Computer Systems_ , vol. 101, pp. 993–1004, 2019.
  * [100] D. L. Aguilar, M. A. M. Perez, O. Loyola-Gonzalez, K.-K. R. Choo, and E. Bucheli-Susarrey, “Towards an interpretable autoencoder: a decision tree-based autoencoder and its application in anomaly detection,” _IEEE Transactions on Dependable and Secure Computing_ , 2022.
  * [101] G. Iadarola, F. Martinelli, F. Mercaldo, and A. Santone, “Towards an interpretable deep learning model for mobile malware detection and family identification,” _Computers & Security_, vol. 105, p. 102198, 2021.
  * [102] D. L. Marino, C. S. Wickramasinghe, and M. Manic, “An adversarial approach for explainable AI in intrusion detection systems,” in _IECON 2018-44th Annual Conference of the IEEE Industrial Electronics Society_. IEEE, 2018, pp. 3237–3243.
  * [103] A. Y. Al Hammadi, C. Y. Yeun, E. Damiani, P. D. Yoo, J. Hu, H. K. Yeun, and M.-S. Yim, “Explainable artificial intelligence to evaluate industrial internal security using EEG signals in IoT framework,” _Ad Hoc Networks_ , vol. 123, p. 102641, 2021.
  * [104] C. Seibold, A. Hilsmann, and P. Eisert, “Focused LRP: Explainable AI for face morphing attack detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 88–96.
  * [105] W. Garcia, J. I. Choi, S. K. Adari, S. Jha, and K. R. Butler, “Explainable black-box attacks against model-based authentication,” _arXiv preprint arXiv:1810.00024_ , 2018.
  * [106] R. Rocha, D. Carneiro, and P. Novais, “Continuous authentication with a focus on explainability,” _Neurocomputing_ , vol. 423, pp. 697–702, 2021.
  * [107] M. AL-Essa, G. Andresini, A. Appice, and D. Malerba, “Xai to explore robustness of features in adversarial training for cybersecurity,” in _International Symposium on Methodologies for Intelligent Systems_. Springer, 2022, pp. 117–126.
  * [108] S. Hariharan, R. Rejimol Robinson, R. R. Prasad, C. Thomas, and N. Balakrishnan, “Xai for intrusion detection system: comparing explanations based on global and local scope,” _Journal of Computer Virology and Hacking Techniques_ , pp. 1–23, 2022.

|  Gaith Rjoub received his Master’s degree in Quality Systems Engineering
from Concordia Institute for Information Systems Engineering (CIISE), Canada
in 2014, and his PhD degree in Information and Systems Engineering from CIISE,
Canada in 2021. He is an assistant professor in the Data Science department at
Princess Sumaya University for Technology (PSUT). From 2021 to 2022, he was a
Postdoctoral Fellow with University of Montreal and Concordia University. His
research interests include cloud and edge computing, machine and deep
learning, artificial intelligence and big data analytics.
---|---
|  Jamal Bentahar (Member, IEEE) received the Ph.D. degree in computer science
and software engineering from Laval University, Québec, QC, Canada, in
2005.,He is a Professor with Concordia Institute for Information Systems
Engineering, Concordia University, Montreal, QC, Canada. From 2005 to 2006, he
was a Postdoctoral Fellow with Laval University, and then an NSERC
Postdoctoral Fellow with Simon Fraser University, Burnaby, BC, Canada. He is a
Visiting Professor with Khalifa University, Abu Dhabi, UAE. His research
interests include the areas of computational logics, reinforcement learning,
multiagent systems, service computing, game theory, and software
engineering.,Prof. Bentahar was an NSERC Co-Chair for Discovery Grant for
Computer Science from 2016 to 2018. He is an Associate Editor of IEEE TSC.
---|---
|  Omar Abdel Wahab is Assistant Professor in the Department of Computer and
Software Engineering at Polytechnique Montréal, Canada. He holds a Ph.D. in
Information and Systems Engineering from Concordia University, Montréal,
Canada. He received his M.Sc. in computer science in 2013 from the Lebanese
American University (LAU), Lebanon. From 2017 to 2018, he was postdoctoral
fellow at the École de Technologie Supérieure (Canada), where he worked on an
industrial research project in collaboration with Rogers and Ericsson. His
research interests are in the areas of cybersecurity, artificial intelligence,
Internet of Things and big data analytics.
---|---
|  Rabeb Mizouni is an Associate Professor in the Department of Electrical
Engineering and Computer Science at Khalifa University, Abu Dhabi, United Arab
Emirates. She got her M.Sc. and Ph.D. in Electrical and Computer Engineering
from Concordia University, Montreal, Canada in 2002 and 2007 respectively.
Currently, she is interested in the deployment of context aware mobile
applications, crowd sensing, Artificial Intelligence, IoT and Blockchain. Dr.
Mizouni is currently an Associate Editor for IEEE Internet of Things magazine
and the IEEE Transaction on Service Computing.
---|---
|  Alyssa Song is an undergraduate student studying Computer Science at the
University of Waterloo in Waterloo, Ontario Canada. She completed an
undergraduate research assistantship in Spring 2022 under the co-supervision
of Professor Robin Cohen and Professor Jamal Bentahar. She was also a student
that term in the Responsible AI program run under the Natural Sciences and
Engineering Council of Canada’s CREATE (Collaborative Research and Training
Experience) initiative.
---|---
|  Robin Cohen is a Professor in the Cheriton School of Computer Science at
the University of Waterloo in Waterloo, Ontario Canada. She conducts research
in the subfield of Artificial Intelligence known as Multiagent Systems and has
a particular interest in exploring multiagent trust modeling, with
applications to social media. She is a Senior Member of the AAAI and a
recipient of the Canadian Artificial Intelligence Association’s Lifetime
Achievement Award.
---|---
|  Hadi Otrok (senior member, IEEE) received his Ph.D. in ECE from Concordia
University. He holds a Full professor position in the department of Electrical
Engineering and Computer Science (EECS) at Khalifa University, an affiliate
associate professor in the Concordia Institute for Information Systems
Engineering at Concordia University, Montreal, Canada, and an affiliate
associate professor in the electrical department at Ecole de Technologie
Superieure (ETS), Montreal, Canada. He is an associate editor at: IEEE TNSM,
Ad-hoc networks (Elsevier), and IEEE TSC. His research interests include:
Blockchain, reinforcement learning, Federated Learning, crowd sensing and
sourcing, ad hoc networks, and cloud and fog security.
---|---
|  Azzam Mourad Received his M.Sc. in CS from Laval University, Canada (2003)
and Ph.D. in ECE from Concordia University, Canada (2008). He is currently
Professor of Computer Science and Founding Director of the Cyber Security
Systems and Applied AI Research Center with the Lebanese American University,
Visiting Professor of Computer Science with New York University Abu Dhabi and
Affiliate Professor with the Software Engineering and IT Department, Ecole de
Technologie Superieure (ETS), Montreal, Canada. His research interests include
Cyber Security, Federated Machine Learning, Network and Service Optimization
and Management targeting IoT and IoV, Cloud/Fog/Edge Computing, and Vehicular
and Mobile Networks. He has served/serves as an associate editor for IEEE
Transactions on Services Computing, IEEE Transactions on Network and Service
Management, IEEE Network, IEEE Open Journal of the Communications Society, IET
Quantum Communication, and IEEE Communications Letters, the General Chair of
IWCMC2020-2022, the General Co-Chair of WiMob2016, and the Track Chair, a TPC
member, and a reviewer for several prestigious journals and conferences. He is
an IEEE senior member.
---|---


# Rearranging the Environment to Maximize Energy with a Robotic Circuit
Drawing

Xianglong Tan1,2,∗, Zhikang Liu1,3,∗, Chen Yu1 and Andre Rosendo1 *This work
was supported by the NNSFC grant 61850410527.1Living Machines Laboratory,
School of Information Science and Technology, ShanghaiTech University, China.
2Hamlyn Centre, Imperial College London, UK. 3Department of Computer Science,
University of York, UK. ∗These authors contributed equally to this work.

###### Abstract

Robots with the ability to actively acquire power from surroundings will be
greatly beneficial for long-term autonomy and to survive in uncertain
environments. In this work, we present a robot capable of drawing circuits
with conductive ink while also rearranging the visual world to receive maximum
energy from a power source. A range of circuit drawing tasks is designed to
simulate real-world scenarios, including avoiding physical obstacles and
regions that would discontinue drawn circuits. We adopt the state-of-the-art
Transporter networks for pick-and-place manipulation from visual observation.
We conduct experiments in both simulation and real-world settings, and our
results show that, with a small number of demonstrations, the robot learns to
rearrange the placement of objects (removing obstacles and bridging areas
unsuitable for drawing) and to connect a power source with a minimum amount of
conductive ink. As autonomous robots become more present, in our houses and
other planets, our proposed method brings a novel way for machines to keep
themselves functional by rearranging their surroundings to create their own
electric circuits.

##  I INTRODUCTION

Recent developments in the fields of robotic hardware, sensing, and machine
learning have led to great progress in robotic applications in various areas
[1, 2, 3]. Recent advances in Robotics have led to progress in various areas
and naturally raised the demand for fully-autonomous, long-lasting and self-
evolving robots [4], keeping human efforts further out of the loop. As with
living species, robots consume energy to perform different tasks. Industrial
robots deployed in the real world are typically powered by an uninterruptible
power supply without the danger of running out of energy. For mobile robots,
however, have their mobility, operational time and performance limited by the
low storage capacity of batteries [5]. This prompts the benefits of an
adaptable energy acquisition ability for robots operating in inhospitable
environments or away from human aid.

Previous works on self-recharging robots showed a framework for robots to
navigate autonomously to a charging station [6]. They use cyclic genetic
algorithm to optimize the control program in simulation. In another work [7],
researchers built an autonomous robotic system capable of plugging itself into
electrical outlets to recharge. Mayton et al. [8] proposed a mobile
manipulation platform capable of plugging itself into a standard U.S.
electrical outlet. Instead of using vision to assist the process, the plugging
was guided entirely by measurements of electrical emissions from the outlets.
However, outlets have different standards across regions and they are not
always available, especially in the field and unexplored areas. Batteries are
more often used in these scenarios, but the connection of cables to batteries
by robots is difficult and a fairly unexplored field of research. Other
researchers focus on the design of power systems for self-powered robots.
Solar panels are widely used for robots to harvest energy from ambient sources
to recharge batteries. They are normally low-cost and light-weight, but suffer
from low efficiency and restrictive application environment. In a series of
works on energy harvesting [9, 10, 11], researchers demonstrate the powering
of nanobots from ambient mechanical energy. In a more recent work [12], a
novel robotic power system is powered by scavenging energy from external
metals. However, in these works the power harvested from the environment is
very limited, and it can be hardly used to power any standard robot.

Manipulation of objects and rearrangement of environments have a long history
in robotics. Traditional pick-and-place solutions follow a standard pipeline:
object recognition, pose estimation, and then grasp planning. For instance,
Jonschkowski et al. [13] used handcrafted image features for object perception
followed by computation of pose estimation for suction objects. Recent data-
driven solutions for manipulation tasks involve imitation learning (IL) and
reinforcement learning (RL). Finn et al. [14] used meta-learning as a pre-
training procedure that uses demonstrations from different environments. The
method is shown to be able to achieve one-shot imitation learning for novel
scenarios and scale to raw pixel inputs. Duan et al. [15] also achieve one-
shot imitation on a block stacking task. They used soft attention for
processing both the state-action sequence that corresponds to the
demonstration, and for processing the components of the vector specifying the
locations of various blocks. RL algorithms have also shown their ability on
learning reliable policy end-to-end with raw image inputs. Kalashnikov et al.
[16] proposed an RL algorithm that is entirely self-supervised, using only
grasp outcome labels that are obtained automatically by the robot for the
grasping task. Zeng et al. [17] performed RL to learn complementary pushing
and grasping policies that operate end-to-end from visual observations to
actions. Besides using IL or RL, Zeng et al. [18] also proposed a framework
called Transporter Networks that rearranges deep features to infer spatial
displacements from visual input, which can parameterize robot actions.
Experimental results show that it is orders of magnitude more sample efficient
than a variety of end-to-end baselines.

This paper presents a novel approach for robots to leverage ambient electrical
power sources and survive in resource-limited, uncertain environments. Instead
of using wires or cables, the robot learns to rearrange environmental
obstacles and useful resources to build optimal electrical path to a power
supply. We develop a path planning algorithm for 3D circuit drawing and adopt
a novel network, capable of selecting pick and place pose for the robot to
reposition objects. This paper shows that robots can rearrange the physical
world to receive energy from the environment to stay alive.

##  II ROBOTIC CIRCUIT DRAWING

###  II-A Conductive Ink

Graphene-based conductive ink has shown great potential in printing flexible
electronics for its low cost, high connectivity and can be applied directly on
materials like textile[19], paper, and other diverse flexible substrates[20].
Compared to metal-based conductive ink, it is low-toxic, environment friendly,
and easy to make and store [21]. In this work, we follow the instructions in
[22] to fabricate customized graphene-based conductive ink. The ink is made of
$5$ $wt\%$ graphene flakes, $0.5$ $wt\%$ graphene dispersion, and $94.5$
$wt\%$ water, which has a sheet resistance of approximately $2$ $\Omega/sq$.

###  II-B Simulation Setup and Tasks

The simulation is set up with PyBullet [23]. We use a Kinova 6 DoF Jaco robot
arm to perform circuit drawing on a 0.8$\times$0.6m workspace. The two-finger
gripper of the robot enables picking and placing of objects. The electrodes of
the robot and the power source are placed on opposite sides of the workspace.
The main goal is to draw continuous ink to connect the electrode pair of the
robot and the power source.

Circuit drawing tasks are selected based on the possible situation a robot
confronts in complex real-world environments. For example, in search and
rescue missions, the robot needs to automatically plan the drawing path
avoiding obstacles and areas that would discontinue or damage the circuit
(e.g., huge bumps, holes, surface with liquid, sand, etc). We refer to these
areas as forbidden zones in this paper. Meanwhile, drawing longer the circuit
path leads to higher circuit resistance and more ink and energy used. If any
blocking obstacles can be moved by the robot, they should be removed before
the drawing. Moreover, if movable bridge-like structures are presented in the
workspace, reposition them on top of any forbidden zones could enable drawing
above these zones to connect more efficiently. The details of circuit drawing
tasks are shown in Fig. 1.

Figure 1: The four basic circuit drawing tasks in this paper: path planning
for circuit drawing (a), pick and remove obstacles (b), reposition ramps to
draw through a forbidden zone (brown area) (c), reposition ramps to draw above
circuits (d). The workspace is a 0.8$\times$0.6m platform with four electrodes
(two from the robot arm and two from a power source) on each side (a1). The
basic goal for the robot is to connect electrodes with conductive ink (a2),
while avoiding obstacles (a3) and forbidden zones (a4). In task (b), the robot
should pick the obstacle (b2) and place it in a zone away from the center (b3)
to draw the shortest circuit between electrodes (b4). In task (c), when a
forbidden zone is presented (c1), the robot is required to pick a ramp (c2)
and place it in the zone (c3) to allow drawing circuit through the zone (c4).
In task (d), the electrodes of the robot and the power source are in the
opposite position, the robot needs to connect a pair of electrode first (d1),
then pick a ramp (d2) and place on the drawn circuit (d3) to connect the rest
electrodes without shortcircuiting the power source (d4)

##  III METHODS

###  III-A Transporter Networks

We consider the pick-and-place problem with actions $\mathbf{a}_{t}$ defining
the pick and place poses and states $\mathbf{s}_{t}$ represented by visual
observations:

$\pi\left(\mathbf{s}_{t}\right)\rightarrow\mathbf{a}_{t}=\left(\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place
}}\right)\in\mathcal{A},$ (1)

where $\mathcal{T}_{\text{pick}}$ is the pose of the end effector used to pick
an object, and $\mathcal{T}_{\text{place}}$ to place the object, where
$\mathcal{T}_{\text{pick}},\mathcal{T}_{\text{place }}\in\mathbb{R}^{2}$ are
2D coordinates. In our task, the goal is to pick up the bridge-like blocks
then place them over the forbidden zones while picking up other blocks then
placing them out of the working area.

Other than using end-to-end RL for solving the pick-and-place task [24, 25,
26, 17], we use Transporter Networks [18] to recover the distribution of
successful pick poses and the corresponding distribution of successful place
poses . The distributions are trained with pure visual observations, which
allows no assumption of prior information about the picking objects. The state
$\mathbf{s}_{t}$ in our experiment is a grid of pixels reconstructed from
RGB-D information. We use the pick-and-place actions in the space of 2D
translations. We discretize the space of SO(2) rotations into $k$ bins where
the input visual observation $\mathbf{s}_{t}$ are rotated. $\mathbf{s}_{t}$ is
therefore defined on a grid of $\\{(x,y,w)_{i}\\}$ where $w$ describes the
rotation.

Denote such a pose $\\{(x,y,w)i\\}$ as $\xi$. We use Q value function
$\mathcal{Q}_{\mathrm{pick}}(\xi|\mathbf{s}_{t})$ to model the distribution of
the returns among different picking positions and
$\mathcal{Q}_{\mathrm{place}}(\xi|\mathbf{s}_{t},\mathcal{T}_{\text{pick}})$
for placing positions.

Learning Picking and Placing After the Q function
$\mathcal{Q}_{\mathrm{pick}}(\xi|\mathbf{s}_{t})$ is trained by a fully
convolutional network (FCN), the picking position can be obtained by

$\mathcal{T}_{\text{pick
}}=\underset{\xi}{\operatorname{argmax}}\mathcal{Q}_{\text{pick
}}\left(\xi\mid\mathbf{s}_{t}\right).$ (2)

Denote $\mathbf{s}_{t}[\xi]$ as a partial crop centered on a pose $\xi$. We
search for the best position for placing that match with the highest feature
correlation. The Q function for placing can then be written as

$\mathcal{Q}_{\text{place
}}\left(\xi\mid\mathbf{s}_{t},\mathcal{T}_{\text{pick
}}\right)=\psi\left(\mathbf{s}_{t}\left[\mathcal{T}_{\text{pick
}}\right]\right)*\phi\left(\mathbf{s}_{t}\right)[\xi]$ (3)

where $\psi(\cdot)$ and $\phi(\cdot)$ are dense feature embeddings from two
deep models. Similarly, the pose of placing is hence obtained by

$\mathcal{T}_{\text{place
}}=\underset{\xi}{\operatorname{argmax}}\mathcal{Q}_{\text{place
}}\left(\xi\mid\mathbf{s}_{t},\mathcal{T}_{\text{pick }}\right)$ (4)

Network Architecture We use visual observation as the state $\mathbf{s}_{t}$,
which is a $320\times 240$ RGB-D image, where each pixel contains both color
and distance information. This enables our models to learn both texture
features and shape features. The picking model is a FCN that takes the state
$\mathbf{s}_{t}\in\mathbb{R}^{H\times W\times 4}$ as input and outputs dense
pixel-wise state value $\mathcal{V}_{\text{pick }}\in\mathbb{R}^{H\times
W}=\mathrm{softmax}(\mathcal{Q}_{pick}(\xi|\mathbf{s}_{t}))$. Specifically,
the picking model is an hourglass network with encoder-decoder architecture.
It is a 43-layer residual network [27] with 12 residual blocks, 8-stride, and
image-wide softmax. Dilation [28] and ReLU activations [29] are implemented on
all layers except the first. It acts as an attention mechanism. The placing
model is a two-stream FCN that takes the state
$\mathbf{s}_{t}\in\mathbb{R}^{H\times W\times 4}$ as input and outputs two
dense feature maps:
features$\psi\left(\mathbf{s}_{t}\right)\in\mathbb{R}^{H\times W\times d}$ and
$\phi\left(\mathbf{s}_{t}\right)\in\mathbb{R}^{H\times W\times d}$ , where $d$
is the number of feature dimensions. The architecture of the placing model is
similar to the picking one but without ReLU activations in the last layer.
Once the picking pose is obtained by $\mathcal{T}_{\text{pick
}}=\operatorname{argmax}\mathcal{V}_{\text{pick }}$, the partial crop with
size $c$, $\psi\left(\mathbf{s}_{t}\left[\mathcal{T}_{\text{pick
}}\right]\right)\in\mathbb{R}^{c\times c\times d}$ centered around
$\mathcal{T}_{\text{pick }}$ is transformed by $\xi$ and cross-correlated with
the feature map $\phi\left(\mathbf{s}_{t}\right)$ according to (3). The pixel-
wise placing state value can then be obtained by $\mathcal{V}_{\text{place
}}\in\mathbb{R}^{H\times W\times
k}=\operatorname{softmax}\left(\mathcal{Q}_{\text{place
}}\left(\xi\mid\mathbf{s}_{t},\mathcal{T}_{\text{pick }}\right)\right)$.
$\mathcal{T}_{\text{place }}=\operatorname{argmax}\mathcal{V}_{\text{place }}$
is the final placing pose that has the highest correlation with two feature
partial crops.

Learning from Demonstrations We use the rewards $\mathcal{R}$ to collect data
for a dataset $\mathcal{D}$ of $n$ expert-demonstrated trajectories $\tau$,
where each trajectory $\tau_{i}$ is a sequence of state-action pairs
$(\mathbf{s}_{t},\mathbf{a}_{t})$. During the training process, we sample
state-action pairs from the dataset $D$ serving as $\mathcal{T}_{\text{pick}}$
and $\mathcal{T}_{\text{pick}}$ labels. The one-hot pixel maps $Y_{\text{pick
}}\in\mathbb{R}^{H\times W}$ and $Y_{\text{place }}\in\mathbb{R}^{H\times
W\times k}$ the are generated by two training labels. The loss function is the
cross-entropy between these one-hot pixel maps and the state values of picking
and placing:

$\mathcal{L}=-\mathbb{E}_{Y_{\text{pick }}}\left[\log\mathcal{V}_{\text{pick
}}\right]-\mathbb{E}_{Y_{\text{place }}}\left[\log\mathcal{V}_{\text{place
}}\right].$ (5)

We used 3D path planning for calculating the reward ${r_{t}\in\mathcal{R}}$.

###  III-B 3D Path Planning

The path planning is used to not only make the shortest path of the drawing
circuit on the ground, but also be treated as a filter to select the high-
quality data samples for training.

Information Input The terrain information will just be a visual observation,
which is an orthographic top-down view from the agent, and input as a 2D
array, representing the situation of a 3-dimensional working area. It is
different from 2-dimensional space, there is an additional consideration of
displacement in the z index that should be added when we process the
algorithm.

Algorithm In this case, An double layer A* search algorithm, which is an
informed searching algorithm, has been used. Firstly, we start the A* search
from the departure point, but we don’t expand the points to the nearest
points. It expands to the available grooves instead, whatever it is uphill or
downhill, or choose directly to the destination if it is possible and in the
same level. Then, the A* will calculate the distance between two nodes
generated in the previous layer in the second layer.

Figure 2: Our 3D path planning algorithm offers the paths produced several
features: 1. The path can go through 3D dimensional space 2. The path can only
ascend or descend if the height difference of the coming node is lower than a
predefined value. This fits into the situation when we are drawing a circuit
line. 3. The algorithm is based on a double-layer A* algorithm, in the first
layer, it expands to the available grooves, or chooses directly to the
destination if it is possible and in the same level. In the second layer, it
uses A* to calculate the displacement path from each node to the next node
from the previous layer.

This algorithm fits our path planning, which also serves for drawing circuit
lines on the ground. The path is also not allowed to go through any steep
edges, including cliffs and precipitous slopes. This is solved by defining a
minimum and a maximum value to detect slope values. If the depth difference of
two adjacent nodes is less than the minimum slope value, then we can treat it
as they are at the same level. If it is greater than the minimum but less than
the maximum, then it is treated as a slope to move on. And otherwise, it can
not move on at all.

$f_{\left(n\right)}=g_{\left(n\right)}+h_{\left(n\right)}$ (6)

Here is the equation when we use A* algorithm and calculate the cost of
traveling to a node, where n is the node, g(n) is the cost from the parent
node, and h(n), which is known as the Heuristic equation, is the predicted
cost to the final destination. In a 2d path planning problem, the eulerian
equation is just equal to $\sqrt{x^{2}+y^{2}}$, but it is going to be
$\sqrt{x^{2}+y^{2}+z^{2}}$ when it is in 3d space.

###  III-C Simulation and Training Details

We use PyBullet [23] as the simulator and create a customized OpenAi gym [30]
environment for all the tasks. We adopt the above-mentioned transporter
networks as the learning agent. Our visual observation ot is an orthographic
top-down view of a 0.8$\times$0.6 workspace, generated in PyBullet by fusing
640$\times$480 RGB-D images captured with calibrated cameras using known
intrinsics and extrinsics. The top-down image ot has a pixel resolution of
320$\times$240 - each pixel represents a 2.5$\times$2.5mm vertical column of
3D space in the workspace. The number of rotation for kernels in transporter
networks is set to 8. For all experiments, the initial position, color and
type of objects are randomly set. The output action at is the pick pose and
the place pose of the gripper.

Samples for training are automatic collected and filtered only if the reward
feedback of the action meet a threshold. The reward calculation have used the
concept of capacitor discharging equation.

$reward_{onecircuit}={e}^{\frac{mincost-cost}{cost}\times\mu}\\\ $ (7)
$total_{r}eward=reward_{A}\times weight_{A}+reward_{B}\times weight_{B}$ (8)

We train the networks for 1k iterations with 1, 10, and 100 demonstrations
respectively and test the performance with 50 test samples for all the tasks.
In each test, the number of steps for pick and place action equals to the
movable objects in the workspace. If the position of electrodes are inverted,
the robot will connect one of the electrode pairs one step before the maximum
action steps. In other cases, the robot only start drawing circuits after all
action steps.

###  III-D Sim-to-Real Experiment Setup

The experimental setup is shown in Fig. 3. The workspace is a 0.8$\times$0.6m
wooden platform. The cardboard is placed on top of the platform and two metal
bars are fixed on each side of the cardboard, representing the terminals of
the robot and the power source. The conductive ink is prepared and stored in a
glass jar placed on a magnetic stirrer to prevent the graphene from
solidification. The glass jar is linked to a soft pipe that is connected to a
peristaltic pump. The pump pushes the ink towards a nozzle which is held by a
3D-printed dispenser at the end-effector of the Kinova 6DOF Jaco Arm. An
Arduino Uno is used to control the speed of ink flow. The robot arm is
controlled by Moveit (https://github.com/ros-planning/moveit). We use one of
the successful cases from the all-in-one task, which has two obstacles, one
forbidden zone, and inverted electrodes, as the demonstration of the real-
world performance. To achieve sim-to-real control, we use the objects with the
same size and shape trained in the simulation to reconstruct the real-world
workspace. Each pick and place pose is given by the trained networks. We used
the built-in inverse kinematics solver of Moveit to move the robot to the
desire poses.

Figure 3: Experimental setup of the circuit drawing robot. The Kinova 6DOF
Jaco Arm first performs pick-and-place actions at the pose suggested by the
network. Then the arm moves to one of the metal bars at 5cm above the
cardboard. The ROS controller continuously sends the waypoints of the circuit
path calculated from the 3D path planning algorithm. An Arduino then receives
the state of the arm through ROS ad sets the on/off of the peristaltic pump to
control the ink flow. The connection starts to show conductivity after the ink
dries (30 minutes).

##  IV RESULTS

###  IV-A Simulation Tasks

In the simulation environment, we have created and contributed a wide variety
of tasks, which is shown in I. As the difficulty of the task increases, the
number of obstacles and forbidden zones has increased as well. We have trained
the network in these simulations for 1000 iterations, with respectively 1, 10,
and 100 demos.

Trained models There are two types of models trained from the network,
including including the attention model and the transport model, are
responsible for picking and placing actions respectively. As the observation
information inputs, the model will automatically find the maximum Q value and
output the pose, including the position and orientation. Then, these models
are able to act.

Figure 4: In this situation (a) where the task is to pick up an obstacle,
meanwhile try to shorten the length of the circuit path that is going to be
drawn, and place the obstacle to the location we have set up on the top-left
corner. The visual observation of this situation will be input and trained two
models, including the attention model (b) and the transport model (c). The
attention model is used to pick the obstacle, and then the transport one will
find the location to place it. This is done by using using the Q function to
find the place where the maximum of the Q value is. As a result, the obstacle
will be placed at the right place in (d).

Loss Training curves of the picking model (attention model) and the placing
model (transport model) are shown in Fig. 5. The loss value of the attention
model fluctuates during the training process, while the loss value for the
transport model drops significantly. The comparison between these two curves
reveals the different levels of difficulty for the picking and placing tasks
and the complexity of the two models.

The reward We have evaluated their performance based on the average reward
gained during the 100 times test. The reward refers to how far the length of
the path from anode to cathode, and the range is 0-1. The closer the final
path length is to the direct length from the anode to the cathode, the greater
the reward will be.

Performance Evaluation Overall, we can see the performance improves, as the
reward is increasing when we increase the number of demonstrations trained.
The more difficult the task is, the lower reward they will perform if there
are only a few demonstrations for training. However, they all perform
relatively well as the rewards are close to 1 when the network has been
trained with 100 demonstrations.

Among these results, remove-obstacle, which is the simplest task, achieves the
best reward all the time, which means it performs perfectly.

TABLE I: The network is trained with 1, 10, and 100 demos respectively for these tasks. After that, we evaluate their performance based on the average reward gained during 100 tests. The range of reward is from 0-1 |  | forbidden | inverted | average reward with n demos trained
---|---|---|---|---
Task | obstacles | zones | electrodes | n = 1 | n = 10 | n = 100
remove-obstacle | 1 | 0 | ✗ | 1.00 | 1.00 | 1.00
remove-obstacles | 3 | 0 | ✗ | 0.94 | 0.79 | 0.97
bridge-forbidden-zone | 1 | 0 | ✗ | 0.98 | 0.98 | 0.94
bridge-forbidden-zones | 1 | 0 | ✗ | 0.86 | 0.90 | 0.95
draw-above-circuit | 0 | 0 | ✓ | 0.71 | 0.87 | 0.92
remove-obstacle |  |  |  |  |  |
\+ bridge-forbidden-zone | 2 | 1 | ✗ | 0.74 | 0.81 | 0.88
all-in-one | 2 | 1 | ✓ | 0.30 | 0.42 | 0.77
Figure 5: Training curves of the picking model (left) and the placing model
(right). The loss value of the attention model is relatively small but
fluctuates during the training process, while the loss value for the transport
model drops significantly.

###  IV-B Real-World Experiments

The real-world demonstration is shown in Fig. 6, although the final placement
of the objects and circuit are the same as the simulation result, we
confronted difficulties when the robot picked objects that we had to repeat
the pick action several times. Examples of successful and failed cases are
shown in Fig. 7. Overall, the pick action achieves around 60% success rate
from 20 trials.

Figure 6: Demonstrations of real-world performance. The left figure shows the
initial setting of the workspace. Two obstacles (a triangular prism and a
cuboid) and a ramp are placed in the middle area of the cardboard. A forbidden
zone (brown rectangle) is set that no circuit can be drawn directly on it. The
position and size of all elements are set exactly as same as the setting in
the successful simulation test, in which the obstacles are removed to the
waste area (blue square) and the ramp is placed in front of the robot
terminals to allow circuit crossing on top of each other. The right figure
shows the result of real-world actions, where the robot achieves similar
results as in the simulation. The battery connected to the metal bars near the
robot side starts to recharge after around 30 minutes, showing a good
connection of the drawn circuit.

Figure 7: Examples of pick actions. Successful (a), Failed - gripper position
too high, can not pick objects (b), Failed - gripper position too low, objects
will bend and damage the nozzle (c).

##  V DISCUSSION

###  V-A Simulation Gap in Real-World Experiments

The main issue in real-world implementation is the misplacement in z-axis of
the gripper. The original z value is generated from the network and worked
well in simulation tasks. In simulation a constrain between the object and the
gripper is automatically created when the object is within a small distance
threshold from the gripper to guarantee a firm grasp. However, in real cases,
tolerance for displacement in z-axis is much smaller than in the simulation.
In the original paper of Transporter [18], authors use a suction gripper
instead of our two-finger gripper, which is much easier to control but has the
limitation that the object surface should be regular. Although the success
rate for picking in our real-world experiments is around 60%, the use of a
finger gripper is necessary considering irregular objects and structures in
real-world scenarios.

###  V-B Importance of Circuit Drawing for Future Robots

To the best of our knowledge, this is the only work where a robot learns to
rearrange visual world to receive energy from self-printed circuits. In
previous approaches with robots charging themselves, works like [7, 8]
demonstrated the ability of robots plugging themselves to recharge. However,
robots can not always rely on outlets to recharge themselves (e.g. search and
rescue missions and other unstructured environments). The circuits drawn with
conductive ink have much higher flexibility and can be three-dimensionally
printed on walls and ceilings, further increasing the likelihood of robots
connecting themselves to potential power sources and staying functional. In
other works on harvesting energy with specially designed hardware [11, 12],
the accumulation of energy is either slow or low in quantity. Meanwhile, the
learning aspect of Robotics within their approaches was left aside. The
crucial difference between those and the current work is that in our work the
robot needs to learn to rearrange and interact with the environment to access
power, while in the previous works this feature is already embedded to those
robots.

###  V-C Study Limitations

In our experiment, the robotic arm has a limited workspace at which the
terminal of the power supply is accessible at all times. This condition may
differ from a real-case where this robot needs to power itself. Mobile robots
equipped with a drawing system, instead of a liquid ink, could be a better fit
for real-cases. Another limitation is that the robot is not directly powered
by the drawn circuit , and this is because the graphene-based conductive ink
we used has a lower conductivity than metal wires (when our robot moves many
joints at the same time the current requirement exceeds the maximum current).
As material science advances, a conductive ink with a higher conductivity
could directly power robots as an alternative to wires or cables, providing a
creative way for robots to access ambient energy and stay functional.

##  VI CONCLUSION

This paper presents the first robot capable of accessing energy intake through
self-drawn conductive-ink-based electrical connections. With a small number of
demonstrations, the robot learns to rearrange surroundings by picking and
placing objects to create circuit paths with a minimum amount of conductive
ink. Although our experiments are performed in a simplified environment, using
conductive ink (a mean) and a power source (an end), this cable-free approach
enables flexible deployment in complex environments, which is a very useful
proof of concept for autonomous robots (e.g. in search-and-rescue missions).
As robots become more present in our society and even reach other planets,
maximizing the capacity to stay alive is crucial to increase the odds of
success.

## References

  * [1] B. Burger, P. M. Maffettone, V. V. Gusev, C. M. Aitchison, Y. Bai, X. Wang, X. Li, B. M. Alston, B. Li, R. Clowes, et al., “A mobile robotic chemist,” Nature, vol. 583, no. 7815, pp. 237–241, 2020.
  * [2] T. Duckett, S. Pearson, S. Blackmore, B. Grieve, W.-H. Chen, G. Cielniak, J. Cleaversmith, J. Dai, S. Davis, C. Fox, P. From, I. Georgilas, R. Gill, I. Gould, M. Hanheide, A. Hunter, F. Iida, L. Mihalyova, S. Nefti-Meziani, G. Neumann, P. Paoletti, T. Pridmore, D. Ross, M. Smith, M. Stoelen, M. Swainson, S. Wane, P. Wilson, I. Wright, and G.-Z. Yang, “Agricultural robotics: The future of robotic agriculture,” 2018.
  * [3] L. He, N. Herzig, S. d. Lusignan, L. Scimeca, P. Maiolino, F. Iida, and T. Nanayakkara, “An abdominal phantom with tunable stiffness nodules and force sensing capability for palpation training,” IEEE Transactions on Robotics, vol. 37, no. 4, pp. 1051–1064, 2021.
  * [4] A. Rosendo, M. von Atzigen, and F. Iida, “The trade-off between morphology and control in the co-optimized design of robots,” PLOS ONE, vol. 12, pp. 1–14, 10 2017.
  * [5] G.-Z. Yang, J. Bellingham, P. E. Dupont, P. Fischer, L. Floridi, R. Full, N. Jacobstein, V. Kumar, M. McNutt, R. Merrifield, B. J. Nelson, B. Scassellati, M. Taddeo, R. Taylor, M. Veloso, Z. L. Wang, and R. Wood, “The grand challenges of science robotics,” Science Robotics, vol. 3, no. 14, 2018.
  * [6] G. Parker and R. Zbeda, “Learning navigation for recharging a self-sufficient colony robot,” in IEEE International Conference on Systems, 2007.
  * [7] W. Meeussen, M. Wise, S. Glaser, S. Chitta, C. Mcgann, P. Mihelich, E. Marder-Eppstein, M. Muja, V. Eruhimov, and T. Foote, “Autonomous door opening and plugging in with a personal robot,” in IEEE International Conference on Robotics and Automation, 2010.
  * [8] B. Mayton, L. Legrand, and J. R. Smith, “Robot, feed thyself: Plugging in to unmodified electrical outlets by sensing emitted ac electric fields,” in IEEE International Conference on Robotics and Automation, 2010.
  * [9] Z. W. Pan, Z. R. Dai, and Z. L. Wang, “Nanobelts of semiconducting oxides,” Science, vol. 291, no. 5510, pp. 1947–1949, 2001.
  * [10] X. Wang, J. Song, J. Liu, and Z. L. Wang, “Direct-current nanogenerator driven by ultrasonic waves,” Science, vol. 316, no. 5821, pp. 102–105, 2007.
  * [11] S. Xu, Y. Qin, C. Xu, Y. Wei, R. Yang, and Z. L. Wang, “Self-powered nanowire devices,” Nature Nanotechnology, vol. 5, no. 5, pp. 366–373, 2010.
  * [12] M. Wang, U. Joshi, and J. H. Pikul, “Powering electronics by scavenging energy from external metals,” ACS Energy Letters, vol. 5, no. 3, pp. 758–765, 2020.
  * [13] R. Jonschkowski, C. Eppner, S. Höfer, R. Martín-Martín, and O. Brock, “Probabilistic multi-class segmentation for the amazon picking challenge,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–7, IEEE, 2016.
  * [14] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” in Conference on Robot Learning, pp. 357–368, PMLR, 2017.
  * [15] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” in Advances in Neural Information Processing Systems (I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, eds.), vol. 30, Curran Associates, Inc., 2017.
  * [16] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” CoRR, vol. abs/1806.10293, 2018.
  * [17] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser, “Learning synergies between pushing and grasping with self-supervised deep reinforcement learning,” 2018.
  * [18] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee, “Transporter networks: Rearranging the visual world for robotic manipulation,” Conference on Robot Learning (CoRL), 2020.
  * [19] N. Karim, S. Afroj, A. Malandraki, S. Butterworth, C. Beach, M. Rigout, K. S. Novoselov, A. J. Casson, and S. G. Yeates, “All inkjet-printed graphene-based conductive patterns for wearable e-textile applications,” J. Mater. Chem. C, vol. 5, pp. 11640–11648, 2017.
  * [20] L. Huang, Y. Huang, J. Liang, X. Wan, and Y. Chen, “Graphene-based conducting inks for direct inkjet printing of flexible conductive patterns and their applications in electric circuits and chemical sensors,” Nano Research, vol. 4, no. 7, pp. 675–684, 2011.
  * [21] K. Pan, Y. Fan, T. Leng, J. Li, Z. Xin, J. Zhang, L. Hao, J. Gallop, K. S. Novoselov, and Z. Hu, “Sustainable production of highly conductive multilayer graphene ink for wireless connectivity and iot applications,” Nature Communications, vol. 9, no. 1, p. 5197, 2018.
  * [22] D. S. Saidina, N. Eawwiboonthanakit, M. Mariatti, S. Fontana, and C. Hérold, “Recent development of graphene-based ink and other conductive material-based inks for flexible electronics,” Journal of Electronic Materials, vol. 48, no. 6, pp. 3428–3450, 2019.
  * [23] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” in GitHub Repository, 2016.
  * [24] J. Zhang, W. Zhang, R. Song, L. Ma, and Y. Li, “Grasp for stacking via deep reinforcement learning,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 2543–2549, IEEE, 2020.
  * [25] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine, “End-to-end robotic reinforcement learning without reward engineering,” Robotics: Science and Systems, 2019.
  * [26] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual reinforcement learning with imagined goals,” Advances in Neural Information Processing Systems, vol. 31, pp. 9191–9200, 2018.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [28] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (Y. Bengio and Y. LeCun, eds.), 2016.
  * [29] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel (J. Fürnkranz and T. Joachims, eds.), pp. 807–814, Omnipress, 2010.
  * [30] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and andW. Zaremba., “Openai gym,” 2016.


0 Research

# Gen4DS: Workshop on Data Storytelling in an Era of Generative AI

Xingyu Lan
Fudan University  e-mail<EMAIL_ADDRESS>Leni Yang
Hong Kong University of Science and Technology e-mail<EMAIL_ADDRESS>Zezhong Wang
Simon Fraser University  e-mail<EMAIL_ADDRESS>Yun Wang
Microsoft Research Asia
e-mail<EMAIL_ADDRESS>Danqing Shi
Aalto University  e-mail<EMAIL_ADDRESS>Sheelagh Carpendale
Simon Fraser University  e-mail<EMAIL_ADDRESS>
###### Abstract

Storytelling is an ancient and precious human ability that has been
rejuvenated in the digital age. Over the last decade, there has been a notable
surge in the recognition and application of data storytelling, both in
academia and industry. Recently, the rapid development of generative AI has
brought new opportunities and challenges to this field, sparking numerous new
questions. These questions may not necessarily be quickly transformed into
papers, but we believe it is necessary to promptly discuss them to help the
community better clarify important issues and research agendas for the future.
We thus invite you to join our workshop (Gen4DS) to discuss questions such as:
How can generative AI facilitate the creation of data stories? How might
generative AI alter the workflow of data storytellers? What are the pitfalls
and risks of incorporating AI in storytelling? We have designed both paper
presentations and interactive activities (including hands-on creation, group
discussion pods, and debates on controversial issues) for the workshop. We
hope that participants will learn about the latest advances and pioneering
work in data storytelling, engage in critical conversations with each other,
and have an enjoyable, unforgettable, and meaningful experience at the event.

Motivation

Over the last decade, the field of data storytelling has made significant
progress. Researchers in the visualization community have conducted a series
of important theoretical work, such as analyzing the narrative structure,
visual design, and interactive design of data stories [15, 8, 29],
constructing formative design spaces [5, 3, 18], and evaluating the effects of
data storytelling [26, 12]. At the same time, a series of tools to help create
data stories have also been developed, including various interactive design
editors (e.g., Ellipsis [14] for authoring data articles, DataClips [2] for
creating data videos) and intelligent tools to generate data stories
automatically (e.g., DataShot for generating fact sheets [20], Calliope [17]
for structured stories). What is perhaps more exciting is that the application
of data storytelling has achieved significance both in academia and in the
wild. For example, Segal and Heer’s classic paper, Narrative Visualization:
Telling Stories with Data [15], has been cited over two thousand times, with a
considerable portion of the citations coming from other communities such as
human-computer interaction, social science, business studies, and geography,
serving the needs of industries such as news media, education, sports,
politics, and the communication of natural science [19, 30].

In the past, when creating data stories, whether writing narratives, creating
graphics, or editing videos, a large amount of manual involvement was
required. With the emergence and maturity of large models such as GPT, Gemini,
Llama2, and Sora, AI-generated content is becoming the norm, and content
production is undergoing drastic changes. This is full of opportunities and
challenges for the field of data storytelling. On the one hand, with the
emergence of many easy-to-use AI tools, the threshold and cost of creating
data stories are significantly reduced. For example, now, ChatGPT can
efficiently generate text descriptions and even complete stories, Midjourney
can quickly generate illustrations, and tools like Pica and Runaway can
quickly generate videos. Recently, Sora has made further breakthroughs in
cross-modal generation between text, images, and videos, able to generate
longer and clearer videos. More importantly, it demonstrates a strong ability
to understand and narrate the world, such as changing perspectives, leveraging
physical principles, and even expressing emotions [1]. Undoubtedly, the
aforementioned abilities will bring revolutionary changes to the creation and
intelligent generation of data stories, having the potential to provide
breakthroughs in some tough problems in previous research (e.g., weak semantic
understanding of data content in intelligent tools, lack of emotional
expression in generated stories). On the other hand, the involvement of AI
also brings new concerns. For example, the problem of blindly creating content
by large models is a well-known risk [4], which may cause serious problems in
the intelligent generation of data stories. As stories are often presented to
users in the form of insights or conclusions, it is difficult for users to
return to the original data to reanalyze the conclusions if there is nonsense
from AI. Furthermore, for the creators of data stories, what changes will
there be in their workflows? Will there be resistance to AI? How should AI be
integrated into their work and creation legally and ethically? These are all
questions that need to be urgently discussed.

Hence, this workshop will center around the theme of ”Data Storytelling in the
Era of Generative AI (Gen4DS),” offering a platform to delve into the
opportunities and challenges stemming from rapid technological advancements
and illuminate the agenda for future research. In general, we hope to adopt an
open and dialogic approach to stimulate collective wisdom among community
members. In addition to one-to-many presentations, we will also incorporate
hands-on tutorials and group discussions to allow participants to experience
and reflect on the impact of AI on data storytelling through hands-on
activities and critical thinking. This approach aims to spark dynamics among
participants, involving them not only as listeners but also as collaborators
and contributors.

##  1 Why Necessary

The VIS community, with its dedication to visualization and visual analytics
advancements, is well-placed to explore the integration of Generative AI in
data-driven storytelling. The discussion of this integration can contribute to
the current landscape and navigate the future trajectory of the field. The
emergent technology, growing rapidly in 2024, is extending from professional
to everyday applications, necessitating discourse within the VIS sphere on
ethical, practical, and technological implications.

We believe that this workshop is significantly different from the main paper
sessions at the VIS conference, and these two forms of activities can
complement each other. The main reasons include:

  * •

As the technology of generative AI evolves extremely fast, a full paper, which
requires significant effort and time to complete, may struggle to keep pace
with the latest advancements and insights. By welcoming more “light-weighted”
papers, reports, and activities, the workshop can provide a timely platform
for participants to share the most recent insights in generative AI and its
implications for data storytelling.

  * •

Secondly, compared to the formal paper programs, this workshop highly
emphasizes discussion and interaction among participants. We will include a
special hands-on creation session in the workshop. We have also innovatively
designed a debate session focusing on AI controversies. We hope that these
novel activities will enhance the enjoyment and engagement of participants
during VIS and stimulate more meaningful insights.

  * •

Thirdly, generative AI’s cross-disciplinary nature highlights the need for an
informal workshop setting at VIS, encouraging collaboration and idea exchange
across fields like computer science, graphic design, and journalism. Such a
setting promotes hands-on engagement with new tools, offering a platform for
direct experimentation less feasible in formal conference settings.

  * •

Last, this workshop is crucial for fostering a community among researchers,
practitioners, and educators leading the charge in generative AI and data
storytelling. It will facilitate knowledge sharing, debate, and multifaceted
discussions, helping to navigate the ethical and innovative use of AI in
storytelling, thereby making a significant contribution to the field’s
responsible advancement.

##  2 Goals

Firstly, we hope that participants in this workshop will have a fruitful
experience, and this includes:

Exchanging. Through paper presentations, participants will exchange their
cutting-edge research and activities relating to data storytelling.

Exploring. Through interactive activities, participants will explore how to
integrate generative AI into the pipeline of data storytelling. The sessions
are designed to foster an environment of collaborative learning, where
participants can exchange insights on the latest tools, share their practical
experiences, and discuss effective learning methodologies.

Criticizing. By engaging in the discussion and debate on “data storytelling +
generative AI”, participants are poised to uncover novel insights, deepen
reflections, and draw inspiration. We also encourage participants to
critically examine new technologies, guiding future research to utilize AI
cautiously and responsibly.

Connecting. Data storytelling is a highly interdisciplinary field, with
researchers and practitioners coming from diverse backgrounds, organizations,
and countries. By attending this workshop, participants will connect with more
people with common interests and start meaningful conversations.

Secondly, we hope that this workshop can also yield long-term outcomes and
spark continuous dialogues. This includes:

Synthesizing. We plan to synthesize the valuable insights and opinions arising
from the workshop and transform the collective wisdom into a white paper that
suggests the research agenda and grand challenges of data storytelling in the
era of strong artificial intelligence.

Researching. We aim to conduct more in-depth research based on the interesting
ideas and materials collected from the workshop and publish papers surrounding
the topics in Section 3.

Building. We will establish and maintain a long-term forum and
interdisciplinary community centered around data storytelling, open to
researchers, students, and practitioners.

##  3 Scope of topics

The workshop topics include but are not limited to:

  * •

Create multimedia content in data story with generative AI

  * •

Data storytelling design workflows with generative AI

  * •

Scenarios and application of data story with generative AI

  * •

Human-AI collaboration mode in data story creation

  * •

Pedagogy of data story creation with generative AI

  * •

Ethics in generative AI for data storytelling

  * •

Evaluation methods and criteria

  * •

Usage guidelines of generative AI for data storytelling

  * •

Tools/systems leveraging generative AI for data storytelling

  * •

Data storytelling with generative AI for all

  * •

Governance and autonomy

##  4 Submission Formats

The workshop will accept four types of submissions, peer-reviewed by at least
two PC members and one workshop organizer.

  * •

Full papers: 6-8 pages excluding references, intended for publication through
the IEEE Xplore Digital Library with DOI. According to the guidance of IEEE
VIS, accepted workshop papers can contribute to VIS full paper submissions in
a future year. However, such submissions should not contain verbatim copies of
previous content. Furthermore, any full paper submission to VIS must stand as
its own contribution.

  * •

Short papers: 2-4 pages excluding references, not for publication in the IEEE
Xplore Digital Library; short papers will be available on the workshop’s
website.

  * •

Activity reports: practical reports that describe how an activity is conducted
and how it could be reused by others and in other contexts; these activity
reports will be published on the workshop’s website.

  * •

Practical reports: experiences and reflections on data storytelling
experiences that serve the exchange between newly appointed and experienced
faculty; these practical reports will be published on the workshop’s website.

##  5 Workshop activities

We plan to have a half-day workshop with both paper presentations and more
interactive activities.

###  5.1 Intended Format and Housekeeping Issues

As this workshop includes hands-on creation and aims to encourage in-person
deep interactions and discussions, it will be conducted in an offline format.
Some housekeeping issues to consider for the offline format may include:

  * •

Venue Selection. Securing an appropriate venue with adequate space and
facilities for presentations and discussions.

  * •

Desk Arrangement. Arranging the desks or seating into formats (e.g., circles)
that suit group discussions and collaborative activities, fostering an
environment conducive to interaction and idea exchange.

  * •

Material Preparation. Preparing and distributing workshop materials such as
big-sized papers, colored pens, and sticky notes to help write down and
organize ideas offered by group discussions.

  * •

On-site Support. Designating organizers or volunteers to provide assistance
and support to participants throughout the workshop.

###  5.2 Activity Types

####  5.2.1 Paper Presentations

Each accepted paper will have approximately 8-10 minutes for presentation,
with a maximum of 6-8 minutes for introducing the papers and the remaining
time allocated for Q&A. The time left after all accepted papers have been
presented will be flexibly allocated to the following interactive activities.

####  5.2.2 Interactive Activities

After the paper presentation, the interactive activities begin with
experiencing through hands-on creation, then discussing and sharing
experiences and opinions in group discussion pods that focus on different
topics. Finally, a debate on topics of choice to engage in discussion and
exchange opinions.

Hands-on Creation: We plan to engage the participants in experiencing some
generative AI tools that have the potential to help the ideation and authoring
of key elements in data storytelling. We will first have an onboarding session
to introduce some mature and accessible generative AI tools. We will publish
the learning materials ahead on the workshop website to help participants make
necessary preparations. Subsequently, participants will engage in data story
creation to explore the integration of generative AI in the development of
narrative structures and visualizations. Participants will not be asked to
create a whole data story but attempt to design some key elements of data
stories, such as designing the narrative structures and visuals of critical
scenes in data stories. After that, we will collect participants’ feedback on
the challenges they encountered during accomplishing the tasks and how
generative AI may affect their workflows in creating data stories. We expect
this hands-on practice to inspire them for follow-up group discussions, and
help participants leverage generative AI for their future work.

Group Discussion Pods: Based on the workshop topics and the accepted papers,
we will provide a list of distinct aspects of generative AI in data
storytelling for workshop participants to discuss future research
opportunities and challenges regarding the combination of generative AI and
data storytelling (e.g., ethics, collaboration modes, pedagogy). We will have
multiple small groups for the discussion. In the beginning, each group will
look through the list of issues and can freely add new issues they identify as
important. Each discussion group will have a moderator who is expected to be
one of the workshop organizers. The moderator will take charge of the time
control, take notes of the discussion, and guide the discussion procedure.
Using the approach of the Six Thinking Hats [6] encourages thinking from six
distinct perspectives (i.e., Managing, Information, Emotions, Positive, and
Creativity), to explore different aspects of a situation or problem thoroughly
and collaboratively. Participants can choose to change their table during the
discussion, they are encouraged to leave ideas and opinions on sticky notes on
the table during discussion. After the discussion ends, each group will
summarize and report their conclusions in 2 minutes to all workshop
participants.

Debate Arena: The last activity will feature a Debate Arena, where
participants will engage in debates on controversial topics within the domain
of generative AI and data storytelling. Participants can choose the topics
they wish to debate and the side (proponent or opponent) they wish to
represent. Grouping will be based on the number of participants choosing each
topic and side, ensuring a balanced and engaging discussion. This segment is
structured to include defined roles for proponents, opponents, and mediators,
facilitating an informal debate process. The purpose of this arena is to
foster critical discourse on complex issues, encouraging participants to
articulate and examine diverse perspectives. This component of the workshop is
designed to illuminate the multifaceted nature of ethical and practical
considerations in the use of generative AI for data storytelling.

###  5.3 Tentative Schedule

The half-day workshop has the following tentative schedule.

  * •

09:00 - 09:10 Opening & keynote

  * •

09:10 - 09:50 Paper presentation

  * •

09:50 - 10:00 Coffee break

  * •

10:00 - 10:30 Hands-on creation

  * •

10:30 - 11:00 Group discussion pods

  * •

11:00 - 11:10 Coffee break

  * •

11:10 - 11:20 Discussion summary

  * •

11:20 - 12:00 Debate arena

###  5.4 Special Technology or Assistance Needed

Although this workshop does not require special technology, for the hands-on
tasks, we encourage each participant to bring a laptop and ensure smooth
internet connectivity and browser performance. Additionally, since there will
be on-site group discussions, to better track the viewpoints of each group and
provide necessary support (e.g., answering questions), we may need the
assistance of 1-2 student volunteers.

###  5.5 Intended Result and Impact

The workshop will provide a channel for participants to discuss the
opportunities and challenges brought by generative AI to data storytelling.
Participants will present their latest research findings regarding leveraging
generative AI for data storytelling, share their experiences and approaches to
how generative AI can be better utilized, or exchange their perspectives on
future research problems. We expect to receive more than 10 papers after the
call for papers, with 3-6 papers to be presented at our workshop.

We will summarize the lessons learned about utilizing generative AI for data
storytelling revealed in both paper presentations and discussion sessions. We
will develop a website to publish our summary results and provide related
materials for any researchers or data storytelling practitioners to use in
future research and practices. We will conclude with a research agenda based
on the discussion results. Based on this, we will continue to compile the
latest research and cases from academia and industry to form a white paper.
This will contribute to not only data storytelling researchers but also
broader research communities such as AI-support creativity and human-AI
collaboration. We plan to submit new research related to the topics discussed
in this workshop to visualization-related conferences or journals.

##  6 Organization

###  6.1 Workshop Organization Timeline

The timeline for the workshop organization is as follows:

  * •

March 20, 2024: Call for Participation

  * •

July 1, 2024: Deadline for Paper Submission

  * •

July 25, 2024: Reviews Collection

  * •

July 30, 2024: Author Notification

  * •

August 15, 2024: Camera Ready Deadline

###  6.2 Development of the List of Participants

For paper submissions, we plan to advertise on the respective mailing lists
such as ACM CHI, IEEE VIS, and social media (e.g., Twitter). For the
discussion session, in addition to advertising, we will invite both
researchers and practitioners actively involved in data storytelling,
especially those focusing on the combination of AI and data storytelling. The
organizers, primarily young scholars, come from various countries and
institutions and are enthusiastic and passionate about organizing the
workshop. Each of us has published research on data storytelling in top-tier
venues such as IEEE TVCG and ACM CHI, covering topics such as narrative
methods, visual and interactive design, evaluation, and the development of
intelligent tools. Our collaboration network will also aid in identifying
potentially interested participants.

We value cultural diversity and believe that every voice matters. We will
actively encourage individuals from all backgrounds to participate in the
workshop and will strive to create a welcoming and respectful environment for
all participants, regardless of their cultural or ethnic background.

###  6.3 Backup Plan

Our backup plan considers the situation of insufficient paper submissions or
overrun of paper presentations. For the former situation, we will prepare
enough sub-events in the three interactive activities to ensure that they can
cover the whole workshop. For the hands-on activity, we will prepare at least
three cases for participants to experience generative AI tools for data
storytelling designs. At least one case will cover the basic usage of AI tools
and others will involve more advanced usage. For the group discussion pods and
debate arena activities, we will hold a brainstorming among workshop
organizers to prepare topics. The topics in the papers submitted will be added
into consideration further. By manipulating the number of cases and topics for
discussion and debate, we will have a flexibly changeable schedule to ensure
participants can have a fruitful experience in this workshop. On the other
hand, if the schedule is too tight for interactive activities, we will
prioritize the group discussion pods and consider shortening the hands-on
creation and debate arena activities.

##  7 Organizing Committee

Xingyu Lan<EMAIL_ADDRESS>
(https://olivialan.github.io/)

Xingyu Lan is an assistant professor in the School of Journalism, Fudan
University, Shanghai, China. Her research focuses on data storytelling, user
experience, and human-AI interaction. She conducted a series of studies about
users’ affective responses to data stories [9, 10, 12, 13] and received IEEE
VIS Best Paper Award [11]. She was once an award-winning practitioner and has
rich experience in creating data stories. She is now responsible for five
courses concerning data visualization and HCI at Fudan University and has
conducted several hands-on workshops with more than 300 participants for
designing data stories using methods such as brainstorming, storyboarding, and
design hackathon. She was also the co-organizer and host of several conference
events, such as the Visualization Roast panel at the ChinaVis conference
(2022, 2023), and the project manager of the Chinese visualization learning
community, Gradict.

Leni Yang<EMAIL_ADDRESS>
(https://yleni.github.io/)

Leni Yang is a post-doc research fellow in the Department of Computer Science
and Engineering at the Hong Kong University of Science and Technology, Hong
Kong SAR, China. Her research interests include Data Visualization and Human-
Computer Interaction with a focus on investigating effective design theories,
techniques, and authoring tools for turning complex data into understandable
and compelling data stories [27, 28, 29]. Her most recent publications are
about immersive (XR) data storytelling and generative AI-assisted data story
authoring. She has hosted several workshops where participants experience
activities like designing storyboards for data stories, brainstorming, and
group discussions.

Zezhong Wang<EMAIL_ADDRESS>
(https://zezhongwang.com/)

Zezhong Wang is a postdoctoral researcher at the Interactive Experiences Lab,
Simon Fraser University, Canada. His research interests encompass the creation
and examination of data-driven storytelling, harnessing visual communication
and insights derived from data, and employing design innovation to enhance
data accessibility and engagement for the general public. Zezhong earned his
PhD from the School of Informatics, University of Edinburgh, where he explored
the creation of data comics. His research delved into the effectiveness of
data comics [26], their creation [22, 23, 24], and materials for teaching
visualization literacy [25], earning him the VGTC Visualization Dissertation
Award Honorable Mention in 2023.

Yun Wang<EMAIL_ADDRESS>
(https://www.microsoft.com/en-us/research/people/wangyun/)

Yun Wang is a senior researcher in the Data, Knowledge, Intelligence (DKI)
Area at Microsoft Research Asia. Her research interests lie at the
intersection of human-computer interaction and information visualization,
applying the advances in artificial intelligence and data science. Her work
aims to facilitate human-data interaction, human-AI collaboration, visual
storytelling, and natural language interfaces. She has developed techniques
and systems for creating visual communication in diverse forms, such as
infographics, interactive web pages, motion graphics, and animated videos. She
has also explored how to simplify and improve the data interaction workflows
between humans and AI for analysis, ideation, authoring, and storytelling
(e.g., [20, 21]). She has published more than 40 papers in high-impact venues
such as VIS, CHI, UIST, TVCG, and CG&A and serves as a reviewer and program
committee for a variety of venues.

Danqing Shi<EMAIL_ADDRESS>
(https://sdq.github.io/)

Danqing Shi is a postdoc researcher at the Finnish Center for Artificial
Intelligence and Aalto University. His research interests include human-
computer interaction and information visualization. He has strong experience
in building automated visualization tools for data storytelling using AI-based
approaches [17, 16]. Two related visualization systems for data storytelling
developed by him have received IEEE VIS Best Poster Honorable Mention and
ChinaVis Best Poster Honorable Mention [7]. He has also co-hosted several
workshops for data analysts on visualization design and visual data analysis.

Sheelagh Carpendale<EMAIL_ADDRESS>
(https://www.cs.sfu.ca/~sheelagh/)

Professor and Canada Research Chair in Data Visualization, at Simon Fraser
University in the School of Computing Science is a Fellow of the Royal Society
of Canada. Her many awards include the IEEE Visualization Career Award, CS-CAN
Lifetime Achievement, a BAFTA and being inducted into both the ACM CHI
(Computer-Human-Interaction) Academy and the IEEE Visualization Academy. As an
international leader in information visualization and interaction design, she
studies how people interact with information both in work and social settings,
she works towards designing more natural, accessible and understandable
interactive visual representations of data.

## References

  * [1] Sora.  https://openai.com/sora.  Last accessed: Feb 24, 2024.
  * [2] F. Amini, N. H. Riche, B. Lee, A. Monroy-Hernandez, and P. Irani.  Authoring data-driven videos with dataclips.  IEEE transactions on visualization and computer graphics, 23(1):501–510, 2016.
  * [3] B. Bach, Z. Wang, M. Farinella, D. Murray-Rust, and N. Henry Riche.  Design patterns for data comics.  In Proceedings of the 2018 chi conference on human factors in computing systems, pp. 1–12, 2018.
  * [4] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell.  On the dangers of stochastic parrots: Can language models be too big?  In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623, 2021.
  * [5] M. Brehmer, B. Lee, B. Bach, N. H. Riche, and T. Munzner.  Timelines revisited: A design space and considerations for expressive storytelling.  IEEE transactions on visualization and computer graphics, 23(9):2151–2164, 2016.
  * [6] W. J. Carl III.  Six thinking hats: Argumentativeness and response to thinking model.  1996\.
  * [7] Y. Guo, D. Shi, M. Guo, Y. Wu, N. Cao, and Q. Chen.  Talk2data: A natural language interface for exploratory visual analysis via question decomposition.  ACM Transactions on Interactive Intelligent Systems, 2021.
  * [8] J. Hullman, S. Drucker, N. H. Riche, B. Lee, D. Fisher, and E. Adar.  A deeper understanding of sequence in narrative visualization.  IEEE Transactions on visualization and computer graphics, 19(12):2406–2415, 2013.
  * [9] X. Lan, Y. Shi, Y. Wu, X. Jiao, and N. Cao.  Kineticharts: Augmenting affective expressiveness of charts in data stories with animation design.  IEEE Transactions on Visualization and Computer Graphics, 28(1):933–943, 2021.
  * [10] X. Lan, Y. Shi, Y. Zhang, and N. Cao.  Smile or scowl? looking at infographic design through the affective lens.  IEEE Transactions on Visualization and Computer Graphics, 27(6):2796–2807, 2021.
  * [11] X. Lan, Y. Wu, and N. Cao.  Affective visualization design: Leveraging the emotional impact of data.  IEEE Transactions on Visualization and Computer Graphics, 2023.
  * [12] X. Lan, Y. Wu, Y. Shi, Q. Chen, and N. Cao.  Negative emotions, positive outcomes? exploring the communication of negativity in serious data stories.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–14, 2022.
  * [13] X. Lan, X. Xu, and N. Cao.  Understanding narrative linearity for telling expressive time-oriented stories.  In Proceedings of the 2021 CHI conference on human factors in computing systems, pp. 1–13, 2021.
  * [14] A. Satyanarayan and J. Heer.  Authoring narrative visualizations with ellipsis.  Computer Graphics Forum, 33(3):361–370, 2014.
  * [15] E. Segel and J. Heer.  Narrative visualization: Telling stories with data.  IEEE transactions on visualization and computer graphics, 16(6):1139–1148, 2010.
  * [16] D. Shi, F. Sun, X. Xu, X. Lan, D. Gotz, and N. Cao.  Autoclips: An automatic approach to video generation from data facts.  In Computer Graphics Forum, vol. 40, pp. 495–505. Wiley Online Library, 2021.
  * [17] D. Shi, X. Xu, F. Sun, Y. Shi, and N. Cao.  Calliope: Automatic visual data story generation from a spreadsheet.  IEEE Transactions on Visualization and Computer Graphics, 27(2):453–463, 2020.
  * [18] Y. Shi, X. Lan, J. Li, Z. Li, and N. Cao.  Communicating with motion: A design space for animated visual narratives in data videos.  In Proceedings of the 2021 CHI conference on human factors in computing systems, pp. 1–13, 2021.
  * [19] C. Tong, R. Roberts, R. Borgo, S. Walton, R. S. Laramee, K. Wegba, A. Lu, Y. Wang, H. Qu, Q. Luo, et al.  Storytelling and visualization: An extended survey.  Information, 9(3):65, 2018.
  * [20] Y. Wang, Z. Sun, H. Zhang, W. Cui, K. Xu, X. Ma, and D. Zhang.  Datashot: Automatic generation of fact sheets from tabular data.  IEEE transactions on visualization and computer graphics, 26(1):895–905, 2019.
  * [21] Y. Wang, H. Zhang, H. Huang, X. Chen, Q. Yin, Z. Hou, D. Zhang, Q. Luo, and H. Qu.  Infonice: Easy creation of information graphics.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–12, 2018.
  * [22] Z. Wang, H. Dingwall, and B. Bach.  Teaching data visualization and storytelling with data comic workshops.  CHI EA ’19, p. 1–9. Association for Computing Machinery, New York, NY, USA, 2019. doi: 10 . 1145/3290607 . 3299043
  * [23] Z. Wang, J. Ritchie, J. Zhou, F. Chevalier, and B. Bach.  Data comics for reporting controlled user studies in human-computer interaction.  IEEE Transactions on Visualization and Computer Graphics, 27(2):967–977, 2021. doi: 10 . 1109/TVCG . 2020 . 3030433
  * [24] Z. Wang, H. Romat, F. Chevalier, N. H. Riche, D. Murray-Rust, and B. Bach.  Interactive data comics.  IEEE Transactions on Visualization and Computer Graphics, 28(1):944–954, 2022. doi: 10 . 1109/TVCG . 2021 . 3114849
  * [25] Z. Wang, L. Sundin, D. Murray-Rust, and B. Bach.  Cheat sheets for data visualization techniques.  CHI ’20, p. 1–13. Association for Computing Machinery, New York, NY, USA, 2020. doi: 10 . 1145/3313831 . 3376271
  * [26] Z. Wang, S. Wang, M. Farinella, D. Murray-Rust, N. Henry Riche, and B. Bach.  Comparing effectiveness and engagement of data comics and infographics.  CHI ’19, p. 1–12. Association for Computing Machinery, New York, NY, USA, 2019. doi: 10 . 1145/3290605 . 3300483
  * [27] X. Xu, L. Yang, D. Yip, M. Fan, Z. Wei, and H. Qu.  From ‘wow’to ‘why’: Guidelines for creating the opening of a data video with cinematic styles.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–20, 2022.
  * [28] L. Yang, A. Wu, W. Tong, X. Xu, Z. Wei, and H. Qu.  Understanding 3d data videos: From screens to virtual reality.  In 2023 IEEE 16th Pacific Visualization Symposium (PacificVis), pp. 197–206. IEEE, 2023.
  * [29] L. Yang, X. Xu, X. Lan, Z. Liu, S. Guo, Y. Shi, H. Qu, and N. Cao.  A design space for applying the freytag’s pyramid structure to data stories.  IEEE Transactions on Visualization and Computer Graphics, 28(1):922–932, 2021.
  * [30] Z. Zhao and N. Elmqvist.  The stories we tell about data: Surveying data-driven storytelling using visualization.  IEEE Computer Graphics and Applications, 2023.