* [67] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečnỳ, S. Mazzocchi, H. B. McMahan _et al._ , “Towards federated learning at scale: System design,” _arXiv preprint arXiv:1902.01046_ , 2019.
  * [68] Q. Xia, W. Ye, Z. Tao, J. Wu, and Q. Li, “A survey of federated learning for edge computing: Research problems and solutions,” _Journal of High-Confidence Computing_ , p. 100008, 2021.
  * [69] Y. Lu, X. Xu, and L. Wang, “Smart manufacturing process and system automation–a critical review of the standards and envisioned scenarios,” _Journal of Manufacturing Systems_ , vol. 56, pp. 312–325, 2020.
  * [70] N. Sakib, N. U. I. Hossain, F. Nur, S. Talluri, R. Jaradat, and J. M. Lawrence, “An assessment of probabilistic disaster in the oil and gas supply chain leveraging bayesian belief network,” _Journal of Production Economics_ , vol. 235, p. 108107, 2021.
  * [71] H. Lu, L. Guo, M. Azimi, and K. Huang, “Oil and gas 4.0 era: A systematic review and outlook,” _Journal of Computers in Industry_ , vol. 111, pp. 68–90, 2019.
  * [72] A. Hahn, “Operational technology and information technology in industrial control systems,” in _Cyber-security of SCADA and other industrial control systems_ , 2016, pp. 51–68.
  * [73] T. Nguyen, R. G. Gosine, and P. Warrian, “A systematic review of big data analytics for oil and gas industry 4.0,” _Journal of IEEE Access_ , vol. 8, pp. 61 183–61 201, 2020.
  * [74] A. J. Dunnings and T. P. Breckon, “Experimentally defined convolutional neural network architecture variants for non-temporal real-time fire detection,” in _Proceedings of the 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 1558–1562.
  * [75] M. Z. Alom, T. M. Taha, C. Yakopcic, S. Westberg, P. Sidike, M. S. Nasrin, B. C. Van Esesn, A. A. S. Awwal, and V. K. Asari, “The history began from alexnet: A comprehensive survey on deep learning approaches,” _arXiv preprint arXiv:1803.01164_ , 2018.
  * [76] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, “A public domain dataset for human activity recognition using smartphones.” in _Esann_ , 2013.
  * [77] M. Krestenitis, G. Orfanidis, K. Ioannidis, K. Avgerinakis, S. Vrochidis, and I. Kompatsiaris, “Oil spill identification from satellite images using deep neural networks,” _Journal of Remote Sensing_ , vol. 11, no. 15, p. 1762, 2019.
  * [78] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [79] Z. Huang, C. O. Dumitru, Z. Pan, B. Lei, and M. Datcu, “Classification of large-scale high-resolution sar images with deep transfer learning,” _Journal of Geoscience and Remote Sensing Letters_ , 2020.
  * [80] A. Mustafa, M. Alfarraj, and G. AlRegib, “Estimation of acoustic impedance from seismic data using temporal convolutional network,” _arXiv preprint arXiv:1906.02684_ , 2019.
  * [81] R. Versteeg, “The Marmousi experience; velocity model determination on a synthetic complex data set,” _Journal of the Leading Edge_ , vol. 13, no. 9, pp. 927–936, 09 1994.
  * [82] J. Varia, S. Mathew _et al._ , “Overview of amazon web services,” _Amazon Web Services_ , pp. 1–22, 2014.
  * [83] V. Veillon, C. Denninnart, and M. A. Salehi, “F-FDN: Federation of Fog Computing Systems for Low Latency Video Streaming,” in _Proceedings of the 3rd IEEE International Conference on Fog and Edge Computing_ , 2019, pp. 1–9.
  * [84] K. Keahey, P. Riteau, D. Stanzione, T. Cockerill, J. Mambretti, P. Rad, and P. Ruth, “Chameleon: a scalable production testbed for computer science research,” in _Contemporary High Performance Computing: From Petascale toward Exascale_ , May 2019, vol. 3, ch. 5, pp. 123–148.
  * [85] S. S. Ogden and T. Guo, “Characterizing the deep neural networks inference performance of mobile applications,” _arXiv preprint arXiv:1909.04783_ , 2019\.
  * [86] H. Moradi, W. Wang, and D. Zhu, “Adaptive performance modeling and prediction of applications in multi-tenant clouds,” in _Proceedings of the21st International Conference on High Performance Computing and Communications(HPCC)_ , 2019, pp. 638–645.
  * [87] X. Li, M. A. Salehi, Y. Joshi, M. K. Darwich, B. Landreneau, and M. Bayoumi, “Performance analysis and modeling of video transcoding using heterogeneous cloud services,” _Journal of IEEE Trans. on Parallel and Distributed Systems (TPDS)_ , vol. 30, no. 4, pp. 910–922, 2019.
  * [88] Z. Hanusz, J. Tarasinska, and W. Zielinski, “Shapiro-wilk test with known mean,” _Journal of REVSTAT-Statistical_ , vol. 14, no. 1, 2016.
  * [89] I. M. Chakravarti, R. G. Laha, and J. Roy, “Handbook of methods of applied statistics,” _Wiley Series in Probability and Mathematical Statistics (USA) eng_ , 1967.
  * [90] D. J. Lilja, _Measuring computer performance: a practitioner’s guide_. Cambridge university press, 2005\.
  * [91] S. Patil and D. J. Lilja, “Using resampling techniques to compute confidence intervals for the harmonic mean of rate-based performance metrics,” _Journal of IEEE Computer Architecture Letters_ , vol. 9, no. 1, 2010.
  * [92] B. Charyyev, A. Alhussen, H. Sapkota, E. Pouyoul, M. H. Gunes, and E. Arslan, “Towards securing data transfers against silent data corruption,” in _Proceedings of the 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)_ , 2019, pp. 262–271.
  * [93] S. Mukherjee, “Distribution of downlink sinr in heterogeneous cellular networks,” _Journal of Selected Areas in Communications_ , vol. 30, no. 3, pp. 575–585, Apr. 2012.
  * [94] J. Crompton, “The digital oil field hype curve: A current assessment the oil and gas industry’s digital oil field program,” in _Proceedings of the SPE Digital Energy Conference and Exhibition_. OnePetro, 2015.
  * [95] X. Li, M. A. Salehi, M. Bayoumi, and R. Buyya, “Cvss: A cost-efficient and qos-aware video streaming using cloud services,” in _Proceedings of the 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)_ , ser. CCGRID ’16, May 2016, pp. 106–115.
  * [96] C. Denninnart, J. Gentry, and M. A. Salehi, “Improving robustness of heterogeneous serverless computing systems via probabilistic task pruning,” in _Proceedings of the 33rd IEEE International Parallel & Distributed Processing Symposium Workshops_, May. 2019.
  * [97] C. O. Diaz, M. Guzek, J. E. Pecero, G. Danoy, P. Bouvry, and S. U. Khan, “Energy-aware fast scheduling heuristics in heterogeneous computing systems,” in _Proceedings of International Conference on High Performance Computing & Simulation_, Jul. 2011, pp. 478–484.
  * [98] M. Skedsmo, R. Ayasse, N. Soleng, M. Indregard _et al._ , “Oil spill detection and response using satellite imagery, insight to technology and regulatory context,” in _Proceedings of the Society of Petroleum Engineers (SPE) International Conference and Exhibition on Health, Safety, Security, Environment, and Social Responsibility_ , Apr. 2016.
  * [99] R. F. Hussain, M. A. Salehi, A. Kovalenko, S. Salehi, and O. Semiari, “Robust resource allocation using edge computing for smart oil fields,” in _Proceedings of the 24th International Conference on Parallel and Distributed Processing Techniques & Applications_, Aug. 2018.
  * [100] H. A. Bramantyo, B. S. Utomo, and E. M. Khusna, “Data processing for iot in oil and gas refineries,” _Journal of Jaringan Telekomunikasi_ , vol. 12, no. 1, pp. 48–54, 2022.
  * [101] A. M. Titu and A. Stanciu, “Merging operations technology with information technology,” in _Proceedings of the 12th International Conference on Electronics, Computers and Artificial Intelligence (ECAI)_ , 2020, pp. 1–6.
  * [102] J. Daily and J. Peterson, “Predictive maintenance: How big data analysis can improve maintenance,” in _Proceedings of Supply chain integration challenges in commercial aerospace_ , 2017, pp. 267–278.
  * [103] Cisco, “New Realities in Oil and Gas: Data Management and Analytics,” https://www.cisco.com/c/dam/en_us/solutions/industries/energy/docs/OilGasDigitalTransformationWhitePaper.pdf, sep 17 2017, [Online; accessed 2022-08-01].
  * [104] E. Kristiani, C.-T. Yang, C.-Y. Huang, Y.-T. Wang, and P.-C. Ko, “The implementation of a cloud-edge computing architecture using openstack and kubernetes for air quality monitoring application,” _Journal of Mobile Networks and Applications_ , vol. 26, no. 3, pp. 1070–1092, 2021.
  * [105] C. D. Heising and W. S. Grenzebach, “The ocean ranger oil rig disaster: A risk analysis,” _Journal of Risk analysis_ , vol. 9, no. 1, pp. 55–62, 1989.
  * [106] X. Deng, J. Yin, P. Guan, N. N. Xiong, L. Zhang, and S. Mumtaz, “Intelligent delay-aware partial computing task offloading for multi-user industrial internet of things through edge computing,” _Journal of IEEE Internet of Things Journal_ , 2021.
  * [107] L. Cricelli, R. Mauriello, and S. Strazzullo, “Market analysis for innovative services: a case study in the satellite communication industry,” in _Proceedings of The 3rd International Conference on Industrial Engineering and Industrial Management_ , 2022, pp. 92–100.
  * [108] B. Aberle, “Microservice-based machine learning production deployment of an anomaly detection algorithm for smart manufacturing machines,” Master’s thesis, Aalto University. School of Science, 2022.
  * [109] A. J. Calderón Godoy and I. González Pérez, “Integration of sensor and actuator networks and the scada system to promote the migration of the legacy flexible manufacturing system towards the industry 4.0 concept,” _Journal of Sensor and Actuator Networks_ , vol. 7, no. 2, p. 23, 2018.
  * [110] N. Verba, K.-M. Chao, J. Lewandowski, N. Shah, A. James, and F. Tian, “Modeling industry 4.0 based fog computing environments for application analysis and deployment,” _Journal of Future Generation Computer Systems_ , vol. 91, pp. 48–60, 2019.
  * [111] A. Lakhan, Q.-U.-A. Mastoi, M. Elhoseny, M. S. Memon, and M. A. Mohammed, “Deep neural network-based application partitioning and scheduling for hospitals and medical enterprises using iot assisted mobile fog cloud,” _Journal of Enterprise Information Systems_ , vol. 16, no. 7, p. 1883122, 2022\.
  * [112] A. Lochbihler, “A mechanized proof of the max-flow min-cut theorem for countable networks with applications to probability theory,” _Journal of Automated Reasoning_ , pp. 1–26, 2022.
  * [113] Z. M. Çınar, A. Abdussalam Nuhu, Q. Zeeshan, O. Korhan, M. Asmael, and B. Safaei, “Machine learning in predictive maintenance towards sustainable smart manufacturing in industry 4.0,” _Journal of Sustainability_ , vol. 12, no. 19, p. 8211, 2020.
  * [114] M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, and L. Liang, “Self-balancing federated learning with global imbalanced data in mobile systems,” _Journal of IEEE Transactions on Parallel and Distributed Systems_ , vol. 32, no. 1, pp. 59–71, 2020.
  * [115] M. Duan, D. Liu, X. Chen, Y. Tan, J. Ren, L. Qiao, and L. Liang, “Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications,” in _Proceedings of the 37th IEEE International Conference on Computer Design (ICCD)_ , 2019, pp. 246–254.
  * [116] N. Mhaisen, A. Awad, A. Mohamed, A. Erbad, and M. Guizani, “Analysis and optimal edge assignment for hierarchical federated learning on non-iid data,” _Journal of arXiv preprint arXiv:2012.05622_ , 2020.
  * [117] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Proceedings of International Conference on Medical image computing and computer-assisted intervention_ , 2015, pp. 234–241.
  * [118] N. Abraham and N. M. Khan, “A novel focal tversky loss function with improved attention u-net for lesion segmentation,” in _Proceedings of the 16th IEE international symposium on biomedical imaging (ISBI 2019)_ , 2019, pp. 683–687.
  * [119] S. S. M. Salehi, D. Erdogmus, and A. Gholipour, “Tversky loss function for image segmentation using 3d fully convolutional deep networks,” in _International workshop on machine learning in medical imaging_. Springer, 2017, pp. 379–387.
  * [120] R. R. Outeiral, P. Bos, H. J. van der Hulst, A. Al-Mamgani, B. Jasperse, R. Simões, and U. A. van der Heide, “Strategies for tackling the class imbalance problem of oropharyngeal primary tumor segmentation on magnetic resonance imaging,” _Journal of Physics and Imaging in Radiation Oncology_ , vol. 23, pp. 144–149, 2022.
  * [121] S. Jadon, “A survey of loss functions for semantic segmentation,” in _Proceedings of IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB)_ , 2020, pp. 1–7.
  * [122] P. C. Sau, “Retinal blood vessel segmentation using attention module and tversky loss function,” in _Proceedings of International Conference on Communication and Artificial Intelligence_ , 2022, pp. 503–513.
  * [123] A. Ziller, A. Trask, A. Lopardo, B. Szymkow, B. Wagner, E. Bluemke, J.-M. Nounahon, J. Passerat-Palmbach, K. Prakash, N. Rose _et al._ , “Pysyft: A library for easy federated learning,” in _Federated Learning Systems_ , 2021, pp. 111–139.
  * [124] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard _et al._ , “Tensorflow: A system for large-scale machine learning,” in _Proceedings of the 12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, 2016, pp. 265–283.
  * [125] E. Bisong, “Google colaboratory,” in _Building Machine Learning and Deep Learning Models on Google Cloud Platform_ , 2019, pp. 59–64.
  * [126] I. Kilovaty, “Cybersecuring the pipeline,” _Houston Law Review_ , vol. 60, 2023.
  * [127] “Cyberattack causes chaos at key European oil terminals | S&P Global Commodity Insights,” feb 3 2022, [Online; accessed 2022-04-24].
  * [128] “Some of the biggest Cyberattacks on Energy Infrastructure in recent years - Prospero Events Group,” mar 24 2022, [Online; accessed 2022-04-24].
  * [129] “It-OT Convergence in Oil & Gas: Top Strategies & Benefits,” [Online; accessed 2022-04-24].
  * [130] Q. Zhang, Z. Qian, S. Wang, L. Yuan, and B. Gong, “Productivity drain or productivity gain? the effect of new technology adoption in the oilfield market,” _Journal of Energy Economics_ , vol. 108, p. 105930, 2022.
  * [131] “Understand the role of IT/OT convergence in oil and gas,” [Online; accessed 2022-04-26].
  * [132] E. D. Knapp and J. Langill, _Industrial Network Security: Securing critical infrastructure networks for smart grid, SCADA, and other Industrial Control Systems_. Syngress, 2014.
  * [133] N. Jan, A. Nasir, M. S. Alhilal, S. U. Khan, D. Pamucar, and A. Alothaim, “Investigation of cyber-security and cyber-crimes in oil and gas sectors using the innovative structures of complex intuitionistic fuzzy relations,” _Journal of Entropy_ , vol. 23, no. 9, p. 1112, 2021.
  * [134] S. Mandal and A. Chirputkar, “Digital transformation and cyber disruption in the oil and gas industry,” _Journal of PalArch’s Journal of Archaeology of Egypt/Egyptology_ , vol. 17, no. 6, pp. 4402–4416, 2020.
  * [135] Hacquebord and C. Pernet, “Drilling Deep A Look at Cyberattacks on the Oil and Gas Industry,” 2019, [Online] available: https://www.trendmicro.com/vinfo/it/security/news/internet-of-things/drilling-deep-a-look-at-cyberattacks-on-the-oil-and-gas-industry.
  * [136] B. Conde Gallego and J. Drexl, “Iot connectivity standards: how adaptive is the current sep regulatory framework?” _Journal of IIC-International Review of Intellectual Property and Competition Law_ , vol. 50, no. 1, pp. 135–156, 2019.
  * [137] M. T. Hossain, S. Badsha, and H. Shen, “Porch: A novel consensus mechanism for blockchain-enabled future scada systems in smart grids and industry 4.0,” in _Proceedings of IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS)_ , 2020, pp. 1–7.
  * [138] D. Ongaro and J. Ousterhout, “In search of an understandable consensus algorithm,” in _Proceedings of USENIX Annual Technical Conference (Usenix ATC 14)_ , 2014, pp. 305–319.
  * [139] M. B. Mollah, J. Zhao, D. Niyato, K.-Y. Lam, X. Zhang, A. M. Ghias, L. H. Koh, and L. Yang, “Blockchain for future smart grid: A comprehensive survey,” _Journal of IEEE Internet of Things Journal_ , vol. 8, no. 1, pp. 18–43, 2020\.
  * [140] R. Zhang, R. Xue, and L. Liu, “Security and privacy on blockchain,” _Journal of ACM Computing Surveys (CSUR)_ , vol. 52, no. 3, pp. 1–34, 2019\.
  * [141] R. Di Pietro, X. Salleras, M. Signorini, and E. Waisbard, “A blockchain-based trust system for the internet of things,” in _Proceedings of the 23nd ACM on symposium on access control models and technologies_ , 2018, pp. 77–83.
  * [142] S. Abdul-Jabbar, “Council Post: Where Blockchain Technology Can Disrupt The Oil And Gas Industry,” sep 27 2021, [Online; accessed 2022-04-29].
  * [143] J. Grundy, “Human-centric software engineering for next generation cloud-and edge-based smart living applications,” in _Proceedings of the 20th IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGRID)_ , 2020, pp. 1–10.
  * [144] S. Lohr, “Facial recognition is accurate, if you’re a white guy,” in _Ethics of Data and Analytics_. Auerbach Publications, 2018, pp. 143–147.
  * [145] M. Madden, A. Lenhart, S. Cortesi, U. Gasser, M. Duggan, A. Smith, and M. Beaton, “Teens, social media, and privacy,” _Journal of Pew Research Center_ , vol. 21, no. 1055, pp. 2–86, 2013.
  * [146] T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” _Journal of Advances in neural information processing systems_ , vol. 29, 2016\.
  * [147] D. Wesley and L. A. Dau, “Complacency and automation bias in the enbridge pipeline disaster,” _Journal of ergonomics in design_ , vol. 25, no. 1, pp. 17–22, 2017.
  * [148] “Automation Bias,” https://databricks.com/glossary/automation-bias.
  * [149] K. Rick, I. Martén, and U. Von Lonski, “Untapped reserves, promoting gender balance in oil and gas,” _World Petroleum Council, Boston Consulting Group_ , 2017.
  * [150] A. Tversky and D. Kahneman, “Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty.” _Journal of science_ , vol. 185, no. 4157, pp. 1124–1131, 1974\.
  * [151] M. Prause and J. Weigand, “The rig: A leadership practice game to train on debiasing techniques,” in _2017 Winter Simulation Conference (WSC)_ , 2017, pp. 4312–4323.
  * [152] R. F. Hussain, A. Mokhtari, A. Ghalambor, and M. A. Salehi, _IoT for Smart Operations in the Oil and Gas Industry: From Upstream to Downstream_ , 1st ed. Elsevier, 2022.
  * [153] M. L. Rathod, A. Shivaputra, H. Umadevi, K. Nagamani, and S. Periyasamy, “Cloud computing and networking for smartfarm agritech,” _Journal of Nanomaterials_ , vol. 2022, 2022.
  * [154] A. Elhadad, F. Alanazi, A. I. Taloba, and A. Abozeid, “Fog computing service in the healthcare monitoring system for managing the real-time notification,” _Journal of Healthcare Engineering_ , vol. 2022, 2022.
  * [155] Z. Cheng, J. Zhang, T. Song, and J. Hu, “Contract-based scheme for computational resource allocation in cloud-assisted parked vehicular edge computing,” _Journal of Physical Communication_ , p. 101916, 2022.
  * [156] I. Sarkar and S. Kumar, “Deep learning-based energy-efficient computational offloading strategy in heterogeneous fog computing networks,” _Journal of Supercomputing_ , pp. 1–18, 2022.
  * [157] H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep reinforcement learning based resource allocation for v2v communications,” _Journal of IEEE Transactions on Vehicular Technology_ , vol. 68, no. 4, pp. 3163–3173, 2019.
  * [158] H. Zhang, H. Zhou, and M. Erol-Kantarci, “Federated deep reinforcement learning for resource allocation in o-ran slicing,” _Journal of arXiv preprint arXiv:2208.01736_ , 2022.
  * [159] A. Lakhan, M. A. Mohammed, O. I. Obaid, C. Chakraborty, K. H. Abdulkareem, and S. Kadry, “Efficient deep-reinforcement learning aware resource allocation in sdn-enabled fog paradigm,” _Journal of Automated Software Engineering_ , vol. 29, no. 1, pp. 1–25, 2022.
  * [160] G. L. Stavrinides and H. D. Karatza, “Orchestration of real-time workflows with varying input data locality in a heterogeneous fog environment,” in _Proceedings of the 5th International Conference on Fog and Mobile Edge Computing (FMEC)_ , 2020, pp. 202–209.
  * [161] ——, “Data-aware resource allocation of linear pipeline applications in a distributed environment,” in _Proceedings of the 13th International Conference on Information and Communication Systems (ICICS)_ , 2022, pp. 121–126.
  * [162] M. Al Ahmad, S. Alnazer, and S. S. Patra, “Transient analysis of fog nodes with catastrophes,” in _Proceedings of the 3rd International Conference for Emerging Technology (INCET)_ , 2022, pp. 1–5.
  * [163] W. Saeed, Z. Ahmad, A. I. Jehangiri, N. Mohamed, A. I. Umar, and J. Ahmad, “A fault tolerant data management scheme for healthcare internet of things in fog computing,” _Journal of KSII Transactions on Internet and Information Systems (TIIS)_ , vol. 15, no. 1, pp. 35–57, 2021.
  * [164] K. Wang, Y. Shao, L. Xie, J. Wu, and S. Guo, “Adaptive and fault-tolerant data processing in healthcare iot based on fog computing,” _Journal of IEEE transactions on network science and engineering_ , vol. 7, no. 1, pp. 263–273, 2018.
  * [165] P. Zhang, Y. Chen, M. Zhou, G. Xu, W. Huang, Y. Al-Turki, and A. Abusorrah, “A fault-tolerant model for performance optimization of a fog computing system,” _Journal of IEEE Internet of Things Journal_ , vol. 9, no. 3, pp. 1725–1736, 2021.
  * [166] Q. Jia, Z. Shen, W. Song, R. Van Renesse, and H. Weatherspoon, “Supercloud: Opportunities and challenges,” _Journal of ACM SIGOPS Operating Systems Review_ , vol. 49, no. 1, pp. 137–141, 2015.
  * [167] J. Li, J. Jin, D. Yuan, and H. Zhang, “Virtual fog: A virtualization enabled fog computing framework for internet of things,” _Journal of IEEE Internet of Things Journal_ , vol. 5, no. 1, pp. 121–131, 2017.
  * [168] A. Hammoud, M. Kantardjian, A. Najjar, A. Mourad, H. Otrok, Z. Dziong, and N. Guizani, “Dynamic fog federation scheme for internet of vehicles,” _Journal of IEEE Transactions on Network and Service Management_ , 2022.
  * [169] N. K. Giang, R. Lea, and V. C. Leung, “Developing applications in large scale, dynamic fog computing: A case study,” _Journal of Software: Practice and Experience_ , vol. 50, no. 5, pp. 519–532, 2020.
  * [170] F. Ansari, S. Erol, and W. Sihn, “Rethinking human-machine learning in industry 4.0: how does the paradigm shift treat the role of human learning?” _Journal of Procedia manufacturing_ , vol. 23, pp. 117–122, 2018.
  * [171] A. Chacón, C. Angulo, and P. Ponsa, “Developing cognitive advisor agents for operators in industry 4.0,” _Journal of New Trends in the Use of Artificial Intelligence for the Industry_ , vol. 4, p. 127, 2020.
  * [172] M. Madonna, L. Monica, S. Anastasi, and M. Di Nardo, “Evolution of cognitive demand in the human–machine interaction integrated with industry 4.0 technologies,” _Journal of Wit Trans. Built Environ_ , vol. 189, pp. 13–19, 2019.
  * [173] D. Gorecky, M. Schmitt, M. Loskyll, and D. Zühlke, “Human-machine-interaction in the industry 4.0 era,” in _Proceedings of the 12th IEEE international conference on industrial informatics (INDIN)_ , 2014, pp. 289–294.
  * [174] N. Kumar and S. C. Lee, “Human-machine interface in smart factory: A systematic literature review,” _Journal of Technological Forecasting and Social Change_ , vol. 174, p. 121284, 2022.
  * [175] A. Nawaz, A. S. Arora, W. Ali, N. Saxena, M. S. Khan, C. M. Yun, and M. Lee, “Intelligent human–machine interface: An agile operation and decision support for an anammox sbr system at a pilot-scale wastewater treatment plant,” _Journal of IEEE Transactions on Industrial Informatics_ , vol. 18, no. 9, pp. 6224–6232, 2022.
  * [176] H. Shang, C. Sivaparthipan _et al._ , “Interactive teaching using human-machine interaction for higher education systems,” _Journal of Computers and Electrical Engineering_ , vol. 100, p. 107811, 2022.
  * [177] Z. Wang, X. Lou, Z. Yu, B. Guo, and X. Zhou, “Enabling non-invasive and real-time human-machine interactions based on wireless sensing and fog computing,” _Journal of Personal and Ubiquitous Computing_ , vol. 23, no. 1, pp. 29–41, 2019.
  * [178] W. Na, Y. Feng, B. Muthu, and A. Shanthini, “Human machine interaction-assisted smart educational system for rural children,” _Journal of Computers and Electrical Engineering_ , vol. 99, p. 107812, 2022\.
  * [179] M. M. Ahmed, G. Yang, and S. Gaweesh, “Assessment of drivers’ perceptions of connected vehicle–human machine interface for driving under adverse weather conditions: Preliminary findings from wyoming,” _Journal of Frontiers in psychology_ , vol. 11, p. 1889, 2020.

received his Bachelor of Science in computer science and engineering in the
fall of 2011 from Military Institute of Science and Technology, Bangladesh. He
immediately began his career in February of 2012, as a programmer in one of
the software company named “ERA Infotech Ltd”. After nine months of his first
job, he joined one of the top multinational software company “Samsung R&D
Institute” as a software engineer. worked in Samsung for around 4 year and 5
months, and decided to enrich his academic knowledge by pursuing higher
education. As such started his Ph.D. journey in computer science in the fall
of 2017 at the University of Louisiana at Lafayette. While pursuing his Ph.D.
degree received his M.Sc. degree in computer science in the spring of 2019.
His research interests are: cloud computing, resource allocation in a fog
federation, task scheduling, machine learning, DNN-based applications for
Industry 4.0, and federated learning. Currently, is working as a Software
Engineer in one of the software company named “TryCycle Data Systems”,
providing software solutions in healthcare sector of Canada & United States.


# Learning in two-player games between transparent opponents

Adrian Hutter
Google, Switzerland
<EMAIL_ADDRESS>
###### Abstract

We consider a scenario in which two reinforcement learning agents repeatedly
play a matrix game against each other and update their parameters after each
round. The agents’ decision-making is transparent to each other, which allows
each agent to predict how their opponent will play against them. To prevent an
infinite regress of both agents recursively predicting each other
indefinitely, each agent is required to give an opponent-independent response
with some probability at least $\varepsilon$. Transparency also allows each
agent to anticipate and shape the other agent’s gradient step, i.e. to move to
regions of parameter space in which the opponent’s gradient points in a
direction favourable to them. We study the resulting dynamics experimentally,
using two algorithms from previous literature (LOLA and SOS) for opponent-
aware learning. We find that the combination of mutually transparent decision-
making and opponent-aware learning robustly leads to mutual cooperation in a
single-shot prisoner’s dilemma. In a game of chicken, in which both agents try
to manoeuvre their opponent towards their preferred equilibrium, converging to
a mutually beneficial outcome turns out to be much harder, and opponent-aware
learning can even lead to worst-case outcomes for both agents. This highlights
the need to develop opponent-aware learning algorithms that achieve acceptable
outcomes in social dilemmas involving an equilibrium selection problem.

##  1 Introduction

Reinforcement learning is concerned with constructing agents that learn to
achieve high rewards in a diverse set of environments. Multi-agent
reinforcement learning studies learning in environments that contain other
agents, possibly pursuing conflicting goals. The simplest way do deal with the
presence of other learning agents during learning is to treat them as just
another part of the environment. However, such an environment will be non-
static and non-Markovian (Laurent et al.,, 2011; Lanctot et al.,, 2017). The
simplistic treatment also ignores the incentives that our agent’s parameters
and their updates create for the learning process of the other agents. This is
particularly relevant in so-called social dilemmas, in which agents can
benefit from mutual cooperation but have an incentive to unilaterally defect
(Leibo et al.,, 2017). A number of authors recently developed strategies that
allow reinforcement learners to reach mutual cooperation in social dilemmas
(Lerer and Peysakhovich,, 2017; Peysakhovich and Lerer,, 2017; Hughes et al.,,
2018; Wang et al.,, 2018; Foerster et al.,, 2018; Letcher et al.,, 2018;
Baumann et al.,, 2018; Clifton and Riché,, 2020).

The prisoner’s dilemma (PD) is the canonical example of a social dilemma.
Mutual cooperation in the PD is Pareto-superior to mutual defection, but
defecting is a dominant strategy and mutual defection is the only Nash
equilibrium of the game. Strategies like tit-for-tat enable mutual cooperation
in the iterated PD (Axelrod and Hamilton,, 1981; Harper et al.,, 2017), while
cooperation is much more elusive in the single-shot case. Vanilla gradient
descent learners fail to learn tit-for-tat and end up defecting with each
other even when repeatedly interacting in an iterated PD. One strand of recent
work considered learning agents that are aware of each other’s learning
process, and use this to anticipate and shape each other’s parameter updates
(Foerster et al.,, 2018; Letcher et al.,, 2018). When repeatedly playing an
iterated PD, such learners can learn strategies similar to tit-for-tat,
leading to mutual cooperation. In this work, we shall take the idea of mutual
awareness of the two learning agents one step further, as we will now discuss.

Outside of machine learning, different authors have discussed conditions under
which agents might cooperate with each other even in a _single-shot_ PD. What
connects all of these ideas is a condition which we might call _mutual
transparency_ , i.e. agents having some insight into each other’s inner
workings. Hofstadter, (1983) coined the term “superrationality” to describe a
type of player that will cooperate with other superrational players, since
they reason that both players will arrive at the same decision, and hence
mutual defection and mutual cooperation are the only two consistent outcomes.
McAfee, (1984), Howard, (1988), and Tennenholz, (2004) introduced a program
equilibrium formalism, in which players do not directly chose actions in the
game, but rather submit programs which chose actions and are given access to
each other’s source code. Mutual cooperation is enabled by submitting a
program that cooperates if and only if the opponent program is syntactically
exactly identical. However, cooperation is rather fragile if it relies on
programs being exact copies of each other, and so different authors have
worked on making cooperation in a single-shot PD more robust when assuming
mutual transparency of the agents (van der Hoek et al.,, 2013; LaVictoire et
al.,, 2014; Critch,, 2019; Oesterheld,, 2019). These references rely on
logical as opposed to syntactical properties of the agents to achieve
cooperation. Critch, (2019) considers agents that search through all possible
proofs up to length $k$ to prove that their opponent will cooperate with them;
if they find such a proof, they cooperate, otherwise they defect.
Surprisingly, such agents cooperate with each other for large enough but
finite $k$. Oesterheld, (2019) introduces an agent called
$\varepsilon\texttt{GroundedFairBot}$ which cooperates unconditionally with
small probability $\varepsilon$, and otherwise mirrors the opponent’s action
when predicting how the opponent will act when playing against
$\varepsilon\texttt{GroundedFairBot}$. When facing a logically similar agent,
the mutual recursive function calls terminate after a finite number
$O(1/\varepsilon)$ with unit probability and cooperation ensues. When facing a
defector, $\varepsilon\texttt{GroundedFairBot}$ defects with probability
$1-\varepsilon$.

In this work, we bring together the idea of mutual transparency enabling
cooperation with opponent-aware learning as developed in Foerster et al.,
(2018); Letcher et al., (2018). Similar to Oesterheld, (2019), we allow both
agents to predict how their opponent will play against them, while requiring
them to give an opponent-independent response with probability at least
$\varepsilon$. The agents can repeatedly interact in a social dilemma and can
update their parameters after each round. An interesting question is whether
policies similar to $\varepsilon\texttt{GroundedFairBot}$ can emerge as the
result of such a learning process, which we shall answer in the affirmative.

Opponent-aware learning is necessary to learn such mutually cooperating
policies; simple gradient learners will always learn mutual defection. We use
two algorithms developed in previous literature for this, LOLA (Foerster et
al.,, 2018) and SOS (Letcher et al.,, 2018). We will find that both of these
have their own (sometimes unexpected) advantages and drawbacks.

In addition to the PD, we consider another well-known social dilemma, the game
of chicken. A key difference of the game of chicken from the PD is that it has
multiple Nash equilibria, and both players prefer a different one. We find
that unlike in the PD, opponent-aware learning can be used in the game of
chicken to outmaneuver the opponent and navigate them towards one’s preferred
equilibrium. However, both players attempting to do this can result in a
worst-case outcome of both players going straight.

We study this scenario from the perspective of the principals who deploy the
learning agents to interact with each other. From the principals’ perspective,
we can regard this as a game in which a move corresponds to submitting a
certain learning agent. An interesting question will thus be whether in the
game which the principals are playing, a socially optimal outcome is easier to
achieve than in the underlying game which the agents are playing (PD or
chicken). We find that this is the case for the PD, but not for the game of
chicken.

The rest of this work is organized as follows. Sec. 2 briefly introduces
learning with opponent-learning awareness, as developed in recent literature,
and adapt it to our framework. For illustration, we apply it to two simple
two-player games. Sec. 3 formally discusses games in which both players have
the ability to predict how their opponent will play against themselves. Sec. 4
contains our main results, in which we apply the techniques from Secs. 2 and 3
to two well-known social dilemmas, the prisoner’s dilemma and the game of
chicken. Sec. 5 contains our concluding remarks.

##  2 Learning with opponent-learning awareness

The theory of learning while taking the opponent’s learning into account as
used in this work was developed by Zhang and Lesser, (2010); Foerster et al.,
(2018); Letcher et al., (2018). We briefly recapitulate these ideas and adapt
them to our scenario.

Consider a “game” between two players $A$ and $B$, in which expected payoffs
as a function of players’ parameters are given by
$V_{A}(\theta_{A},\theta_{B})$ and $V_{B}(\theta_{A},\theta_{B})$,
respectively. In each round, player $A$ calculates a gradient of $V_{A}$ and
updates their parameters in the direction of this gradient using a learning
rate $\delta_{A}$. Let $\bar{\theta}_{A}$ and $\bar{\theta}_{B}$ denote
players’ current parameters. The “naive” gradient for player $A$ is then given
by

$\displaystyle\xi_{\text{naive},A}$
$\displaystyle=\nabla_{\theta_{A}}V_{A}(\theta_{A},\theta_{B})|_{\theta_{A}=\bar{\theta}_{A},\theta_{B}=\bar{\theta}_{B}}$
$\displaystyle=\nabla_{\theta_{A}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})\ .$
(1)

This “naive” gradient however implicitly assumes $\theta_{B}$ to be static and
so ignores that $B$ is learning as well.

A more sophisticated way of calculating $A$’s gradient might take $B$’s
learning into account. There are two different ways in which $B$’s learning
affects the direction in which $A$’s gradient points: $A$ might want to
calculate their gradient at parameters $\theta_{B}$ that are already updated
by a gradient step of $B$ (in other words, _anticipate_ $B$’s gradient step);
and $A$ might want to move towards regions of parameter space in which $B$’s
gradient points into directions that benefit $A$ (in other words, _shape_
$B$’s gradient step).

In order to arrive at this formally, $A$ considers a gradient step of $B$ in
which $B$ updates their parameters using a learning rate $\eta_{A}$. The
learning rate $\eta_{A}$, used by $A$ in order to anticipate and shape $B$’s
learning, need not be identical to $B$’s actual learning rate $\delta_{B}$.
The LOLA (Learning with Opponent-Learning Awareness (Foerster et al.,, 2018))
gradient of $A$ is then given by

$\displaystyle\xi_{\text{LOLA},A}$
$\displaystyle=\nabla_{\theta_{A}}V_{A}(\theta_{A},\theta_{B}+\eta_{A}\nabla_{\theta_{B}}V_{B}(\theta_{A},\theta_{B}))|_{\theta_{A}=\bar{\theta}_{A},\theta_{B}=\bar{\theta}_{B}}$
$\displaystyle=\nabla_{\theta_{A}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B}+\eta_{A}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B}))$
$\displaystyle\quad+\eta_{A}\nabla_{\theta_{B}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B}+\eta_{A}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B}))\nabla_{\theta_{A}}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B})$
(2)
$\displaystyle=\xi_{\text{naive},A}+\underbrace{\eta_{A}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B})\nabla_{\theta_{B}}\nabla_{\theta_{A}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})}_{\text{``anticipate''}}$
$\displaystyle\quad+\underbrace{\eta_{A}\nabla_{\theta_{B}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})\nabla_{\theta_{A}}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B})}_{\text{``shape''}}+O(\eta_{A}^{2})\
.$ (3)

Zhang and Lesser, (2010) study the first summand in Eq. (2), which (following
Letcher et al., (2018)) we will call Look Ahead (LA). They prove that for two-
action two-player games, LA leads to convergence to a Nash equilibrium for
sufficiently small learning rates. Foerster et al., (2018) study the
combination of the first and third summand in Eq. (3), that is, the naive
gradient and the leading order opponent-shaping correction. Among other
results, they find that it leads to mutual cooperation in an iterated PD. The
version of LOLA we will use in this work is given by Eq. (2), so in contrast
to Foerster et al., (2018) we also incorporate the effects of anticipating the
opponent’s gradient step, and do not perform a leading-order Taylor expansion.

Letcher et al., (2018) introduce SOS (Stable Opponent Shaping), which, when
adapted to our framework, uses a gradient

$\displaystyle\xi_{\text{SOS},A}$
$\displaystyle=\xi_{\text{naive},A}+\eta_{A}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B})\nabla_{\theta_{B}}\nabla_{\theta_{A}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})$
$\displaystyle\quad+p\eta_{A}\nabla_{\theta_{B}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})\nabla_{\theta_{A}}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B})\
.$ (4)

This is identical to Eq. (3), up to a factor $p\in[0,1]$ which is used to tune
the strength of the shaping term and is re-calculated for each gradient step.
Letcher et al., (2018) prove that SOS converges to stable fixed points (SFPs)
in a broad class of games and demonstrate that two SOS learners can avoid
certain cooperation failures that two LOLA learners fall victim to; see Sec.
2.2 for further discussion of this point. The scaling factor $p$ is thereby
chosen as large as possible while guaranteeing that a) $\xi_{\text{SOS},A}$
always has positive inner product with the first two summands of Eq. (2) (that
is, the naive gradient and the leading-order LA correction); and b)
$\xi_{\text{SOS},A}$ goes to zero whenever $\xi_{\text{naive},A}$ does.

In Letcher et al., (2018), the factor $p$ is at each step identical for all
involved learners. This implicitly assumes a certain form of global
coordination which we cannot assume for our purposes. We will thus use a form
of SOS in which each player calculates $p$ independently of all others, as
proposed in Remark 4.7 in Letcher, (2018). As shown in this remark, this
version of SOS inherits convergence guarantees from the version used in
Letcher et al., (2018). The detailed calculation of $p$ is described in
Appendix A.

As discussed, there are some technical differences between the way LOLA and
SOS gradients are calculated in Foerster et al., (2018) and Letcher et al.,
(2018), respectively, and the way they are calculated in this work. A
conceptually more interesting difference is that we do not assume $\eta_{A}$
and $\eta_{B}$ to be fixed and equal to each other. We consider the choice of
$\eta$ (as well as the choice between LOLA and SOS) to be part of the strategy
of the principal who deploys a learning agent, and study the effects of
choosing different values for $\eta$. For instance, $\eta_{A}\rightarrow 0$ is
equivalent to calculating the naive gradient, while choosing $\eta_{A}$ much
larger than $\delta_{B}$ corresponds to $A$ looking multiple steps of $B$
ahead. Larger values of $\eta_{A}$ and $\eta_{B}$ might be particularly
relevant in competitive scenarios, in which both players attempt to outwit
each other.

For illustration, we study the effects of LOLA and SOS on two simple games
that do not yet involve predicting the opponent’s action against oneself.

###  2.1 Ultimatum game

We consider a binary version of the ultimatum game (Güth et al.,, 1982; Sanfey
et al.,, 2003; Henrich et al.,, 2004; Oosterbeek et al.,, 2004) in which
player $A$ (the proposer) receives a pot of $10 and can choose between a
“fair” split ($5 each for them and their opponent) or an “unfair” split ($8
for themselves, $2 for the opponent). Player $B$ (the responder) can choose to
accept the proposed split, or reject it, in which case both players receive
nothing. We assume that $B$ will always accept fair splits, so both players
have a single parameter describing their strategy. Player $A$’s parameter
$\theta_{A}$ describes their probability of proposing a fair split,
$p_{\text{fair}}=\sigma(\theta_{A})$, where $\sigma$ denotes the sigmoid
function. Player $B$’s parameter describes the probability of accepting a
proposed unfair split, $p_{\text{accept}}=\sigma(\theta_{B})$. Payoffs are
given by

$\displaystyle V_{A}$
$\displaystyle=5\,p_{\text{fair}}+8\,(1-p_{\text{fair}})\,p_{\text{accept}}$
$\displaystyle V_{B}$
$\displaystyle=5\,p_{\text{fair}}+2\,(1-p_{\text{fair}})\,p_{\text{accept}}\
,$

and so the naive gradients are

$\displaystyle\xi_{\text{naive},A}$ $\displaystyle=\frac{\partial
V_{A}}{\partial p_{\text{fair}}}\frac{\partial
p_{\text{fair}}}{\partial\theta_{A}}=(5-8\,p_{\text{accept}})\frac{\partial
p_{\text{fair}}}{\partial\theta_{A}}$ $\displaystyle\xi_{\text{naive},B}$
$\displaystyle=\frac{\partial V_{B}}{\partial p_{\text{accept}}}\frac{\partial
p_{\text{accept}}}{\partial\theta_{B}}=2\,(1-p_{\text{fair}})\frac{\partial
p_{\text{accept}}}{\partial\theta_{B}}\ .$ (6)

Player $B$’s naive gradient is thus always positive (since $p_{\text{fair}}<1$
for any finite $\theta_{A}$), while $A$’s naive gradient is positive when
$p_{\text{accept}}\leq\frac{5}{8}$ and negative otherwise.

The full gradient fields for naive learners, SOS learners, and LOLA learners
(with $\eta_{A}=\eta_{B}=1$) are shown in Fig. 1. The field for SOS learners
looks very similar to that for naive learners. In particular, player $B$’s
gradient is always positive, which is expected: Since player $B$’s naive
gradient is always positive, so is their LA gradient; the SOS gradient always
has positive inner product with the leading order LA gradient, meaning for
agents with a single parameter that they always have the same sign. So the SOS
gradient of player $B$ cannot become negative as long as the leading order
approximation is good.

Things change when using LOLA with sufficiently high $\eta_{B}=1$: $B$’s
gradient now becomes negative in certain regions of parameter space, meaning
that the responder decreases their likelihood of accepting an unfair proposal,
since they take into account that a low $p_{\text{accept}}$ will provide an
incentive for the proposer to increase $p_{\text{fair}}$.

Figure 1: Gradient fields in the ultimatum game for naive learners (left), SOS
learners (center), and LOLA learners (right) using $\eta_{A}=\eta_{B}=1$.

###  2.2 Tandem game

Letcher et al., (2018) introduce the tandem game as an example of a game in
which LOLA learners’ attempt at shaping each other’s gradients leads to worse
outcomes for both of them, which is avoided by SOS learners. In this game,
both players have a single parameter, $\theta_{A}=x$ and $\theta_{B}=y$, and
payoffs are given by

$\displaystyle V_{A}$ $\displaystyle=-(x+y)^{2}+2x$ $\displaystyle V_{B}$
$\displaystyle=-(x+y)^{2}+2y\ .$ (7)

The first summand in the payoff functions encourages the players to coordinate
their choice of parameters to satisfy $x\approx-y$, while the second summand
encourages each player to choose their parameter as large as possible.

The overall welfare can be written as

$\displaystyle V_{A}+V_{B}=-2(x+y-\frac{1}{2})^{2}+\frac{1}{8}\ .$ (8)

When both players use LOLA gradients as in Eq. (2) with
$\eta=\eta_{A}=\eta_{B}$, straightforward calculus shows that the set of SFPs
is described by

$\displaystyle x+y=\frac{1-2\eta+4\eta^{2}}{(1-2\eta)^{2}}\ ,$ (9)

which increases monotonically as a function of $\eta\in[0,\frac{1}{2})$ and
diverges as $\eta\rightarrow\frac{1}{2}^{-}$. The resulting overall welfare
$V_{A}+V_{B}$ thus decreases monotonically as a function of
$\eta\in[0,\frac{1}{2})$.

LOLA learners display “arrogant behavior” (Letcher et al.,, 2018) – LOLA
encourages both players to increase their own parameter, expecting that this
will compel (via the first summand in the payoff) their opponent to decrease
their own. SOS by contrast preserves the SFPs described by $x+y=1$ of two
naive learners, leading to higher overall welfare.

Figure 2: Average rewards for different learners (LOLA or SOS) when playing
the tandem game against other learners. Results are sampled over $100$ runs,
shaded regions show one standard deviation.

In the following, we simulate the learning dynamics of LOLA and SOS learners
not only against themselves, but also against each other. We use
$\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=0.1$ and draw initial parameters from
$\mathcal{N}(0,1)$. Results are in Fig. (2). While two SOS learners against
each other receive higher rewards on average than two LOLA learners against
each other, a LOLA learner is able to exploit an SOS learner.

If we consider the “game” in which two principals have to choose a learning
algorithm for their agents, the choice between LOLA and SOS thus turns into a
prisoner’s dilemma: both players choosing SOS is better for both players than
both players choosing LOLA, but choosing LOLA is a dominant strategy (i.e.,
leads to better outcomes than choosing SOS irrespective of the opponent’s
choice). Without some coordination regarding the choice of learning algorithm,
we should thus expect that LOLA will be chosen in a competitive scenario.

##  3 Decision-making with opponent transparency

Consider a normal form game between two players $A$ and $B$. We assume that
both players’ decision making is transparent to each other, and so each player
is able to predict their opponent’s action against an other player, including
themselves. Assume that players $A$ and $B$ have $d_{A}$ and $d_{B}$ actions
available, respectively. The probabilistic reaction of $A$ to predicted
actions of $B$ (when predicting how $B$ will play against $A$) can be
summarized in a $d_{A}\times d_{B}$ matrix $M_{A}\in\mathbb{R}_{\geq
0}^{d_{A}\times d_{B}}$, with non-negative entries and columns that sum to
unity. We further denote $A$’s probability of choosing an action based on
predicting $B$ (as opposed to an opponent-independent action) by $p_{P,A}$,
and their ($d_{A}$-dimensional) probability distribution over actions when
choosing their action independently of their opponent by
$v_{A}\in\mathbb{R}_{\geq 0}^{d_{A}}$. Player $A$’s policy is thus completely
determined by the tuple $(p_{P,A},v_{A},M_{A})$, and analogously for player
$B$. $\varepsilon\texttt{GroundedFairBot}$ introduced by Oesterheld, (2019) is
an example of an agent fitting into this general framework.

Clearly, having both $p_{P,A}\rightarrow 1$ and $p_{P,B}\rightarrow 1$ will
lead to an infinite regress, so we require that both $p_{P,A}\leq
1-\varepsilon$ and $p_{P,B}\leq 1-\varepsilon$, where in the following we will
use $\varepsilon=10^{-3}$. The resulting probability distribution
$w_{A}\in\mathbb{R}_{\geq 0}^{d_{A}}$ over $A$’s actions can then be
calculated analytically. Indeed, it is given by

$\displaystyle w_{A}$
$\displaystyle=(1-p_{P,A})v_{A}+p_{P,A}(1-p_{P,B})M_{A}v_{B}+p_{P,A}p_{P,B}(1-p_{P,A})M_{A}M_{B}v_{A}$
$\displaystyle\quad+p_{P,A}p_{P,B}p_{P,A}(1-p_{P,B})M_{A}M_{B}M_{A}v_{B}+\ldots$
$\displaystyle=(1-p_{P,A})\sum_{k=0}^{\infty}\left(p_{P,A}p_{P,B}M_{A}M_{B}\right)^{k}v_{A}$
$\displaystyle\quad+p_{P,A}(1-p_{P,B})\sum_{k=0}^{\infty}\left(p_{P,B}p_{P,A}M_{A}M_{B}\right)^{k}M_{A}v_{B}$
$\displaystyle=\left(1-p_{P,A}p_{P,B}M_{A}M_{B}\right)^{-1}\left((1-p_{P,A})v_{A}+p_{P,A}(1-p_{P,B})M_{A}v_{B}\right)\
,$ (10)

and analogously for player $B$.

Note that while both players use the opponent’s predicted action against
themselves to chose their action, the actual action probabilities $w_{A}$ and
$w_{B}$ of the players are not correlated.

##  4 Social dilemmas with opponent transparency

In this section, we bring together the techniques introduced in Sec. 2 and
Sec. 3, and apply them to two well-known social dilemmas, the PD and the game
of chicken.

###  4.1 Preliminaries

####  4.1.1 Social dilemmas

The available actions in the PD are cooperate (C) and defect (D). In the game
of chicken they are swerving or going straight, which we will identify with
cooperating and defecting, respectively, for easier comparison with the PD.
Both games are described by four different payoffs, $P$ (punishment for both
players defecting), $R$ (reward for both players cooperating), $T$ (temptation
of defecting against a cooperator), and $S$ (sucker’s payoff for cooperating
with a defector). The PD and the game of chicken share the following
properties: $R>P$ (mutual cooperation is preferable to mutual defection);
$R>S$ (mutual cooperation is preferable to being exploited); and $T>R$ (there
is an incentive to defect against a cooperator). It is often also assumed that
$R>(T+S)/2$, such that mutual cooperation is preferable to alternating rounds
of defection. The only difference between the two games is whether $P$ or $S$
is larger, that is, which action is preferable against a defector. In the PD,
the best response against a defector is to defect oneself, while in the game
of chicken, the best response is to cooperate. This makes defection a dominant
strategy in the PD, while no dominant strategy is available in the game of
chicken.

Tables 1 and 2 show the numerical payoffs we will use in the following. We
choose payoff differences which are sufficiently larger than unity to ensure
that the parameter-to-payoff landscape is sufficiently curved, and the effects
of taking into account e.g. how the opponent’s next gradient step will affect
one’s own gradient become relevant. We note that in iterated games such as in
Foerster et al., (2018) or Letcher et al., (2018), a similar effect is
achieved by adding up all the (discounted) payoffs of an iterated game.

|  $C$ (cooperate) |  $D$ (defect)
---|---|---
$C$ (cooperate) | 30, 30 | 0, 40
$D$ (defect) | 40, 0 | 10, 10
Table 1: Payoff matrix for the prisoner’s dilemma. |  $C$ (swerve) |  $D$ (straight)
---|---|---
$C$ (swerve) | 30, 30 | 0, 40
$D$ (straight) | 40, 0 | -30, -30
Table 2: Payoff matrix for the game of chicken.

####  4.1.2 Parameter initialization and updates

Foerster et al., (2018) and Letcher et al., (2018) consider a set up in which
the two learning agents repeatedly play an _iterated_ PD, and update their
parameters after each iterated PD. There are thus two levels of iteration: an
inner one (the iterated PD), during which parameters are frozen, and an outer
one, in which parameter updates happen. By contrast, we consider a single
level of iteration and update parameters after each single-shot PD.

At the beginning of each experiment, we initialize all parameters from
$\mathcal{N}(0,\sigma^{2})$ with $\sigma=0.1$. In order to calculate player
$A$’s gradients using Eq. (2) or Eq. (2), we need the parameter-to-payoff
function $V_{A}(\theta_{A},\theta_{B})$. This function involves several steps.
First, all parameters $\theta_{A}$ are translated to the probabilities
appearing in $(p_{P,A},v_{A},M_{A})$ using a sigmoid function. The probability
$p_{P,A}$ is clamped to the range $[0,1-\varepsilon]$. Then we use use Eq. (3)
to calculate the probability of each player cooperating, which leads to the
probability of the four possible outcomes $CC$, $CD$, $DC$, and $DD$.
Multiplying the probabilities of the four possible outcomes with the
respective payoffs leads to the expected payoffs $V_{A}$ and $V_{B}$. We use
PyTorch’s autograd functionality to evaluate the gradients appearing in Eq.
(2) and Eq. (2).

###  4.2 Prisoner’s dilemma

####  4.2.1 Cooperativeness as a function of the opponent learning rate

We start by studying the impact of the learning rate $\eta=\eta_{A}=\eta_{B}$
used to take the opponent’s gradient step into account, while holding the
actual learning rates $\delta=\delta_{A}=\delta_{B}=1$ constant. We use values
$\eta\in\\{0.1,0.3,1,3,10,30,100\\}$, each learner performs $1000$ gradient
steps after random initialization, and we perform $n_{\text{sample}}=100$
experiments for each value of $\eta$. Results for two LOLA learners are shown
in Fig. 3 and look substantially the same for SOS vs. SOS and LOLA vs. SOS.
With $\eta\ll\delta$, the LOLA learners are essentially naive learners who end
up always defecting. The probability of mutual cooperation then increases with
increasing $\eta$ up to a sweet spot around $\eta\approx 3\delta$ where mutual
cooperation is almost guaranteed. Increasing $\eta$ even further to
$\eta\gg\delta$ again increases the probability of other outcomes while mutual
cooperation remains the most likely outcome.

Figure 3: Probability of the four different outcomes in the PD after $1000$
gradient steps of two LOLA learners as a function of $\eta=\eta_{A}=\eta_{B}$,
using $\delta_{A}=\delta_{B}=1$. The horizontal axis is logarithmic and spans
three orders of magnitude for $\eta$. Shaded regions show one standard error
calculated as $2\sigma/\sqrt{n_{\text{sample}}}$, where $\sigma$ is the sample
standard deviation and results are sampled over $n_{\text{sample}}=100$
experiments.

####  4.2.2 Fluctuations around equilibrium

We next study the fluctuations around the learning equilibrium in which
players mostly cooperate. Fig. 4 shows the average payoffs and their standard
deviations for LOLA vs. LOLA and SOS vs. SOS learners using
$\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$. Both types of learners achieve
average payoffs close to the Pareto optimal value of $R=30$, but the LOLA
learners show smaller fluctuations around their average rewards.

Figure 4: Average payoff in a PD as a function of learning step for LOLA vs.
LOLA learners and SOS vs. SOS learners, using
$\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$. Shaded regions show one standard
deviation. Results are sampled over $100$ experiments.

####  4.2.3 Parameters after convergence

In the two-player, two-action games we are investigating, agents’ policies are
described by the four probabilities $\Pr[\text{S}]$ (probability of choosing
an action based on predicting (“simulating”) the opponent),
$\Pr[\text{C}|\neg\text{S}]$ (probability of cooperating when giving an
opponent-independent response), $\Pr[\text{C}|\text{C}]$ (probability of
cooperating after opponent’s simulated cooperation), and
$\Pr[\text{C}|\text{D}]$ (probability of cooperating after opponent’s
simulated defection). We will now discuss the values towards which these
probabilities converge.

Figure 5: Final parameters in a PD of LOLA learners with
$\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$ after $1000$ learning steps.
Parameters of the learner who achieves higher expected payoff are at the top.
Error bars show one standard deviation.

Learners with $\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$ achieve final
payoffs (after $1000$ learning steps) which are close to identical: the higher
payoff is $29.19\pm 2.14$ (standard deviation over $100$ experiments), the
lower payoff is $28.74\pm 3.19$. Interestingly, the final parameters are
clearly not identical, see Fig. 5. It is instructive to consider how these
final parameters differ from $\varepsilon\texttt{GroundedFairBot}$
($\Pr[\text{S}]=1-\varepsilon$,
$\Pr[\text{C}|\text{C}]=\Pr[\text{C}|\neg\text{S}]=1$,
$\Pr[\text{C}|\text{D}]=0$). Both learners converge to
$\Pr[\text{C}|\text{C}]\approx 1$, which provides an incentive for their
opponent to cooperate. The learner with higher final payoff differs from
$\varepsilon\texttt{GroundedFairBot}$ by defecting with significant
probability when giving an opponent-independent response, the learner with
lower final payoff differs from $\varepsilon\texttt{GroundedFairBot}$ by
choosing an opponent-independent action with significant probability, and
cooperating after simulated defection with significant probability.

We can provide a heuristic explanation how such an equilibrium can emerge.
When playing against “almost $\varepsilon\texttt{GroundedFairBot}$” (say an
agent that plays like $\varepsilon\texttt{GroundedFairBot}$ $99\%$ of the
time, and randomly $1\%$ of the time), it is better to cooperate
unconditionally than to be $\varepsilon\texttt{GroundedFairBot}$. This can be
seen with the help of Eq. (3), or by considering that cooperating
unconditionally prevents (simulated and real) retribution. Starting from a
hypothetical state in which both agents are “almost
$\varepsilon\texttt{GroundedFairBot}$”, there is thus a gradient towards
cooperating unconditionally, i.e. increasing $\Pr[\text{C}|\text{D}]$ and
lowering $\Pr[\text{S}]$. If $A$ starts moving towards unconditional
cooperation, this invites defection from $B$, i.e. $B$ is invited to lower
their $\Pr[\text{C}|\neg\text{S}]$. The opponent-shaping gradient of $A$ will
thus push against moving too far away from
$\varepsilon\texttt{GroundedFairBot}$. In effect, in the rare cases where $B$
decides to act without simulating $A$, they have a chance to defect against a
cooperating opponent, which yields the highest possible reward $T$; this will
lead to a slightly higher expected payoff for $B$ than for $A$. Which of the
two learners ends in which of the roles just described will be determined by
small differences in the initial paramters.

####  4.2.4 The need for opponent-aware learning

We know that the parameter space available to the learners includes policies
like $\varepsilon\texttt{GroundedFairBot}$, which provide incentives for the
opponent to cooperate while being hard to exploit. We have seen in Fig. 3 that
naive learners (or LOLA learners with $\eta\ll\delta$) do not converge towards
such policies, but learn mutually defecting strategies. Here, we investigate
whether mutual cooperation is at least stable for naive learners when they are
initialized with parameters close to $\varepsilon\texttt{GroundedFairBot}$. We
initialize all parameters to $\theta=\pm 3$ plus small noise from
$\mathcal{N}(0,\sigma^{2})$ with $\sigma=0.1$, such that initially
$\Pr[\text{S}]\approx\Pr[\text{C}|\neg\text{S}]\approx\Pr[\text{C}|\text{C}]\approx
95\%$ and $\Pr[\text{C}|\text{D}]\approx 5\%$.

Figure 6: Prisoner’s dilemma with learners whose parameters are initialized
close to $\varepsilon\texttt{GroundedFairBot}$
($\Pr[\text{S}]\approx\Pr[\text{C}|\neg\text{S}]\approx\Pr[\text{C}|\text{C}]\approx
95\%$ and $\Pr[\text{C}|\text{D}]\approx 5\%$). Both opponents use the same
learning algorithm. They are either naive gradient learners (with
$\delta_{A}=\delta_{B}=1$) or LOLA learners (with
$\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$). Shaded regions show one standard
deviation sampled over $100$ experiments.

Fig. 6 compares the outcomes with and without opponent-aware learning. Both
two naive learners and two LOLA learners initially achieve high levels of
cooperation. However, for the former cooperation starts degrading after a few
hundred learning steps, while for the latter it remains stable over $1000$
steps. As already discussed in Sec. 4.2.3, the reason for this is that when
playing against “almost $\varepsilon\texttt{GroundedFairBot}$”, the naive
gradient points towards cooperating unconditionally, i.e. lowering
$\Pr[\text{S}]$. This can be observed for the naive learners in Fig. 6. As a
consequence, the incentive to reward simulated cooperation with cooperation
decreases, leading to decreasing $\Pr[\text{C}|\text{C}]$. When the opponent-
shaping gradient is present, $\Pr[\text{S}]$ stays close to unity, as seen for
the LOLA learners.

Interestingly, both naive and LOLA learners end up more forgiving than
$\varepsilon\texttt{GroundedFairBot}$ (i.e., both have
$\Pr[\text{C}|\text{D}]>0$). LOLA learners manage to strike a balance between
forgiving simulated defection with significant probability (thus preventing
cascades of simulated and real defection) and keeping $\Pr[\text{C}|\text{D}]$
low enough such that both agents are still incentivized to cooperate with high
probability when choosing an opponent-independent action
($\Pr[\text{C}|\neg\text{S}]\approx 1$).

####  4.2.5 No arms race towards higher opponent learning rates

In a competitive scenario there is no reason to assume that both learners use
the same opponent learning rate $\eta$. Indeed, in a scenario in which
principals submit learning agents to compete with others, the principals might
endow their agents with a high $\eta$ in the hope of their agent outwitting
its opponent. It is thus interesting to study whether a learner with higher
$\eta$ has an advantage over a learner with lower $\eta$ (which might lead to
an “arms race” of learners with ever higher $\eta$), and how stable mutual
cooperation is to learners with unequal opponent learning rates. Fig. 7 shows
results for $\delta_{A}=\delta_{B}=\eta_{A}=1$ and $\eta_{B}=3$. The learner
using the higher opponent learning rate $\eta_{B}=3$ does not manage to
achieve higher payoffs on average. For two LOLA learners, cooperation is
stable over $1000$ learning steps. For two SOS learners, on the other hand,
cooperation slowly starts to deteriorate after a few hundred steps. It is
still the case, however, that the learner with the higher opponent learning
rate does not achieve higher payoffs, and that average payoffs remain clearly
above the baseline of $P=10$ for mutual defection.

Figure 7: Average payoff in a PD as a function of learning step for LOLA vs.
LOLA learners (top) and SOS vs. SOS learners (bottom), using
$\delta_{A}=\delta_{B}=\eta_{A}=1$ and $\eta_{B}=3$. Shown are average payoffs
for the first $1000$ learning steps. Shaded regions show one standard
deviation. Results are sampled over $100$ experiments.

####  4.2.6 The game from the perspective of the principals

We now take the scenario of two principals submitting learning agents one step
further. We can consider the “game” in which a strategy does not correspond to
cooperating with some probability (as in the underlying PD), but to submitting
a certain learning agent that will play multiple rounds of the underlying game
(like the PD) against its opponent while updating its parameters. The
principal then receives the reward of a final single-shot game. (For
sufficiently long training periods, results do not change substantially if the
principals instead receive the sum of all rewards received by their agent
during training.) In such a scenario, each principal might be concerned not
only with the learning of their own agent, but also with how they can shape
the learning of their opponent’s agent.

Within the scope of this work, a learning agent is defined by the algorithm
used to calculate its gradients (naive, LOLA, or SOS), its learning rate
$\delta$, and its opponent learning rate $\eta$ (for non-naive learners). This
still produces a large space of possible learning agents. For simplicity, we
consider just five representatives. We fix $\delta=1$ for all learners and
consider a naive learner, and LOLA and SOS learners with $\eta=1$ or
$\eta=30$.

Fig. 8 shows the results (average reward received in the final single-shot
game) for all possible encounters. A number of insights can be gained from
these results:

  * •

No learner ever gets less than the “safe value” $P=10$ which can be guaranteed
by always defecting.

  * •

LOLA learners with $\eta=1$ manage to get more than $R=30$ against naive
learners.

  * •

Naive learners benefit a lot from playing against (LOLA or SOS) learners with
large opponent learning rate $\eta=30$. In fact, a naive learner is the best
response against an SOS learner with $\eta=30$.

  * •

Larger $\eta$ does not necessarily do better. Against (LOLA or SOS) learners
with $\eta=30$, the best response uses $\eta=0$ or $\eta=1$.

  * •

The game defined by the payoff matrix in Fig. 8 has a Nash equilibrium in
which one principal submits a LOLA learner with $\eta=1$ and the other submits
a LOLA learner with $\eta=30$. Both principals submitting a LOLA learner with
$\eta=1$ is also a Nash equilibrium within error bars. These Nash equilibria
produce payoffs close to $R=30$ for both players.

Submitting a LOLA learner with $\eta=1$ creates an incentive for the opponent
to also submit a LOLA learner, which leads to payoffs much larger than $P=10$
(which are obtained by two naive learners) and in fact close to the Pareto-
efficient value $R=30$ for both players. In some sense, our framework of
learning with opponent transparency thus transforms the PD, with its default
outcome of mutual defection, into a much easier game in which mutual
cooperation is the default outcome.

Figure 8: Average payoff in the final single-shot PD after $1000$ learning
steps. Shown is the payoff of the row strategy versus the column strategy. All
strategies use a learning rate $\delta=1$. Error bars are calculated as
$2\sigma/\sqrt{n_{\text{sample}}}$, where $\sigma$ is the sample standard
deviation and $n_{\text{sample}}=100$ is the number of experiments per
strategy pair. Cells are highlighted if the row strategy is a best response
versus the column strategy within statistical error bars.

###  4.3 Game of chicken

####  4.3.1 Baseline outcome for naive learners

Two naive learners in the PD simply learn to always defect against each other,
even when given the opportunity to predict their opponent against themselves.
In the game of chicken, defection is no longer the best response against a
defecting opponent, and so mutual defection is not a stable outcome for naive
learners. Indeed, one of the two naive learners will learn to always defect
while the other learns to always cooperate, yielding an average payoff of
$(S+T)/2<R$. Which of the two learners becomes the “defector” and which one
the “cooperator” is determined by small differences in the initial parameters.

####  4.3.2 Outwitting the opponent

When using opponent-aware learners instead of naive learners, it becomes
possible for one learner to navigate its opponent towards its preferred
equilibrium. Fig. 9 shows two examples of this. In both of these examples, one
learner manages to get their opponent to cooperate with certainty, while
itself defecting with significant probability. As a result, it ends up
receiving an average reward larger than $R=30$ at test time. The left part of
the figure shows an SOS learner with higher opponent learning rate $\eta$
taking advantage of its opponent with lower $\eta$, suggesting that an “arms
race” of choosing ever higher values of $\eta$ might ensue. The right part of
the figure shows that – somewhat surprisingly – when using equal parameters
($\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$) the more cautious SOS learner
can outmanoeuvre a LOLA learner. The LOLA learner here receives less on
average than the “baseline” of $(S+T)/2=20$ which two naive learners achieve
against each other.

In the game of chicken, one promising opponent-aware strategy might be to move
towards unconditional defection ($\Pr[\text{S}]\rightarrow 0$ and
$\Pr[\text{C}|\neg\text{S}]\rightarrow 0$), forcing the opponent to cooperate
lest they receive the lowest possible payoff $P$. (This strategy in the game
of chicken is sometimes described as “throwing the steering wheel out the
window”, i.e. visibly committing to going straight.) This is not what we
observe in the two examples in Fig. 9. It is the case that in both examples
the player who receives the higher payoff uses
$\Pr[\text{C}|\neg\text{S}]\rightarrow 0$, while moving their opponent to
$\Pr[\text{C}|\neg\text{S}]\rightarrow 1$. However, in both examples the
player who receives the higher payoff actually is the one with the higher
$\Pr[\text{S}]$. So the better performing player manoeuvres their opponent
towards unconditional cooperation, while themselves making their action highly
dependent on the opponent’s predicted action.

Figure 9: Two examples of a learner outmaneuvering their opponent in a game of
chicken. Left: Two SOS learners with $\delta_{A}=\delta_{B}=\eta_{A}=1$ and
$\eta_{B}=3$. Right: A LOLA learner vs. an SOS learner with
$\delta_{A}=\delta_{B}=\eta_{A}=\eta_{B}=1$. In the “Outcome probabilities”
plot, shaded regions show one standard error calculated as
$2\sigma/\sqrt{n_{\text{sample}}}$, where $\sigma$ is the sample standard
deviation; in the other plots, shaded regions show one sample standard
deviation. Results are sampled from $n_{\text{sample}}=100$ experiments.

####  4.3.3 Outcomes as a function of the opponent learning rate

Fig. 10 shows the results of both learners using ever larger values of
$\eta_{A}=\eta_{B}=\eta$ (while holding $\delta_{A}=\delta_{B}=1$ constant).
For sufficiently small values of $\eta$, one of the players will always
cooperate while their opponent will always defect. For some intermediate
values of $\eta$, mutual cooperation becomes the most likely outcome for two
LOLA learners. Two SOS learners by contrast do not achieve a high probability
of mutual cooperation for any value of $\eta$, and for sufficiently large
values of $\eta$, disaster strikes with both players defecting and receiving
the lowest possible payoff $S=-30$.

Figure 10: Outcome probabilities (after $1000$ gradient steps) in a game of
chicken as a function of $\eta_{A}=\eta_{B}=\eta$ (while holding
$\delta_{A}=\delta_{B}=1$ constant), showing LOLA vs. LOLA at the top and SOS
vs. SOS at the bottom. The horizontal axis is logarithmic and spans three
orders of magnitude for $\eta$. Shaded regions show one standard error
calculated as $2\sigma/\sqrt{n_{\text{sample}}}$, where $\sigma$ is the sample
standard deviation and $n_{\text{sample}}=100$.

For large enough $\eta$, SOS learners display “arrogant behavior” in that they
expect that them defecting will compel their opponent to cooperate. This is
especially noteworthy since SOS was specifically designed to avoid such
arrogant behavior. The reason why SOS does not manage to avoid arrogant
behavior here is that the SOS gradient $\xi_{\text{SOS},A}$ is designed to
have non-negative inner product with the Look Ahead gradient
$\xi_{\text{LA},A}$ (see Appendix A), not necessarily with the naive gradient
$\xi_{\text{naive},A}$. If both players are defecting, the naive gradient will
point towards defecting less, but the Look Ahead gradient might (for large
enough $\eta$) not, since it anticipates the opponent defecting less. This
provides a clear example in which incorrectly anticipating one’s opponent
proves harmful.

####  4.3.4 The game from the perspective of the principals

We now consider again the scenario in which two principals submit learning
agents that will play on their behalf. As discussed, against an opponent with
low or intermediate $\eta$, there is an incentive to submit an agent with
higher $\eta$. As we will see, however, the reverse is also true: the best
response to an SOS learner with high $\eta$ is a learner with intermediate
$\eta$, and so the disaster of mutual defection is an unlikely outcome.

Fig. 11 shows the full payoff matrix for the same $5$ learners as used when
discussing the PD. Unlike in the PD, some strategies achieve the lowest
($P=-30$) or highest ($T=40$) possible payoff against certain other
strategies. It is noteworthy that the SOS learners manage to take advantage of
the LOLA learners irrespective of the opponent learning rate $\eta$. There is
a Nash equilibrium in which one principal submits an SOS learners with
$\eta=1$ and the other principal submits an SOS learner with $\eta=30$; these
are mutual best responses within statistical error bars. The average payoffs
that result when choosing between these two learning agents (i.e. the $4$
cells at the bottom-right in Fig. 11) again satisfy the inequalities defining
the game of chicken. The game of chicken thus reproduces itself at a higher
level (with somewhat different payoffs), where “going straight” now
corresponds to choosing a high value of $\eta$ and “swerving” corresponds to
choosing an intermediate value of $\eta$.

Figure 11: Average payoff in the final single-shot game of chicken after
$1000$ learning steps. Shown is the payoff of the row strategy versus the
column strategy. All strategies use a learning rate $\delta=1$. Error bars are
calculated as $2\sigma/\sqrt{n_{\text{sample}}}$, where $\sigma$ is the sample
standard deviation and $n_{\text{sample}}=100$ is the number of experiments
per strategy pair. Cells are highlighted if the row strategy is a best
response versus the column strategy within statistical error bars.

##  5 Discussion

We have studied a scenario in which two principals submit learning agents that
repeatedly interact in a social dilemma. At test time, the agents play a
single round of the social dilemma game, and the resulting rewards are the
principals’ payoffs. Both players’ decision making is mutually transparent,
and both learners are aware of each others’ learning, and take it into account
when updating their own parameters. It is not at all clear how to do this
“optimally”. We have considered two approaches discussed in previous
literature, LOLA and SOS, after adapting them to our set up. We have seen that
LOLA and SOS learners can achieve outcomes which are drastically different
than those achieved by naive gradient descent learners, including mutual
cooperation in the prisoner’s dilemma, yielding Pareto optimal outcomes, but
also mutual defection in the game of chicken, which yields the worst possible
outcome for both players.

We have found that both LOLA and SOS have their own advantages and drawbacks.
In particular:

  * •

Given the choice between LOLA and SOS, choosing LOLA is a dominant strategy in
the tandem game (Fig. 2).

  * •

Mutual cooperation in the prisoner’s dilemma is more robust to using different
opponent learning rates $\eta$ for LOLA than for SOS (Fig. 7 and Fig. 8).

  * •

SOS learners outmanoeuvre LOLA learners in the game of chicken, even if the
LOLA learner uses a higher opponent learning rate $\eta$ than the SOS learner
(Fig. 9 and Fig. 11).

  * •

When two SOS learners with high opponent learning rate $\eta$ face each other
in the game of chicken, they both get the lowest possible payoff (Fig. 10).

This leads to two important open problems. Firstly, develop an opponent-aware
learning algorithm that improves upon LOLA and SOS. Secondly, develop a better
understanding of when and how one learning agent manages to manoeuvre its
opponent towards its preferred equilibrium in the game of chicken, or games
involving an equilibrium selection problem more generally.

From the principals’ perspective, the underlying social dilemma game gets
transformed into a different game: what learning agent is a good response to a
learning agent which their opponent might submit? We have seen that the
prisoner’s dilemma and the game of chicken behave very differently under this
transformation. In the prisoner’s dilemma, there are learning agents which are
a best response against themselves (within the limited set of learning agents
considered in this work), and lead to mutual cooperation with high
probability. The game of chicken, by contrast, offers more room to outwit the
opponent by choosing a higher opponent learning rate $\eta$ than them. The
choice between a “high” and an “intermediate” value of $\eta$ is again a game
of chicken, so in a sense the game of chicken reproduces itself at a higher
level. The scenario of learning under mutual transparency thus facilitates
cooperative outcomes in the prisoner’s dilemma, but not necessarily in the
game of chicken.

In the prisoner’s dilemma, mutual awareness of each other’s inner workings is
helpful for achieving cooperative outcomes. This idea has existed for decades
(at least since Hofstadter, (1983)), and we have now validated it in the
paradigm of gradient-based machine learning. On the other hand, we have seen
that in the game of chicken, in which both learning agents attempt to navigate
the opponent towards their preferred equilibrium, it is hard to predict what
kind of learning agent succeeds against which other. It is even possible that
opponent-aware learners produce the worst possible outcome for themselves and
so do worse than more naive learning agents would. Developing techniques that
are guaranteed to achieve acceptable outcomes in games that involve an
equilibrium selection problem is thus an important open problem if we want to
avoid worst case outcomes in multi agent interactions.

## Acknowledgements

Many thanks to Jakob Foerster and Alistair Letcher for answering questions
about opponent-aware learning, to Jesse Clifton and Daniel Kokotajlo for
helpful discussions, and to Caspar Oesterheld for careful reading of the
manuscript.

##  Appendix A Calculating the scaling parameter in SOS

This appendix describes how to calculate the scaling parameter $p\in[0,1]$ in
SOS. It is based on Letcher et al., (2018) and Remark 4.7 in Letcher, (2018)
and adapted to our framework. The calculation involves two hyperparameters
$a,b\in(0,1)$, where we follow the recommendation of Letcher et al., (2018) to
use $a=0.5$ and $b=0.1$.

Let us define the Look Ahead (LA) gradient

$\displaystyle\xi_{\text{LA},A}=\xi_{\text{naive},A}+\eta_{A}\nabla_{\theta_{B}}V_{B}(\bar{\theta}_{A},\bar{\theta}_{B})\nabla_{\theta_{B}}\nabla_{\theta_{A}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})$
(11)

given by the first two summands of Eq. (3). Further let

$\displaystyle\chi_{A}=\eta_{A}\nabla_{\theta_{B}}V_{A}(\bar{\theta}_{A},\bar{\theta}_{B})$
(12)

denote the leading-order opponent-shaping correction, i.e. the third summand
in Eq. (3). The SOS gradient can then be written as

$\displaystyle\xi_{\text{SOS},A}=\xi_{\text{LA},A}+p\chi_{A}\ .$ (13)

Let $p_{1}=1$ if $\langle\chi_{A},\xi_{\text{LA},A}\rangle\geq 0$ and
$p_{1}=\min\\{1,-a\|\xi_{\text{LA},A}\|^{2}/\langle\chi_{A},\xi_{\text{LA},A}\rangle\\}$
otherwise. Let $p_{2}=\|\xi_{\text{naive},A}\|^{2}$ if
$\|\xi_{\text{naive},A}\|<b$ and $p_{2}=1$ otherwise. Choosing $p\leq p_{1}$
ensures that $\xi_{\text{SOS},A}$ always has non-negative inner product with
$\xi_{\text{LA},A}$. Choosing $p\leq p_{2}$ ensures that SOS converges to
SFPs. In order to guarantee both of these while making the opponent-shaping
correction as strong as possible, choose $p=\min\\{p_{1},p_{2}\\}$.

## References

  * Axelrod and Hamilton, (1981) Axelrod, R. and Hamilton, W. D. (1981).  The evolution of cooperation.  Science, 211(4489):1390–1396.
  * Baumann et al., (2018) Baumann, T., Graepel, T., and Shawe-Taylor, J. (2018).  Adaptive mechanism design: Learning to promote cooperation.  arXiv preprint arXiv:1806.04067.
  * Clifton and Riché, (2020) Clifton, J. and Riché, M. (2020).  Towards cooperative learning equilibrium in reinforcement learning (working draft).
  * Critch, (2019) Critch, A. (2019).  A parametric, resource-bounded generalization of löb’s theorem, and a robust cooperation criterion for open-source game theory.  The Journal of Symbolic Logic, pages 1–15.
  * Foerster et al., (2018) Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. (2018).  Learning with opponent-learning awareness.  In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 122–130. International Foundation for Autonomous Agents and Multiagent Systems.
  * Güth et al., (1982) Güth, W., Schmittberger, R., and Schwarze, B. (1982).  An experimental analysis of ultimatum bargaining.  Journal of Economic Behavior & Organization, 3(4):367–388.
  * Harper et al., (2017) Harper, M., Knight, V., Jones, M., Koutsovoulos, G., Glynatsi, N. E., and Campbell, O. (2017).  Reinforcement learning produces dominant strategies for the iterated prisoner’s dilemma.  arXiv preprint arXiv:1707.06307.
  * Henrich et al., (2004) Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E., and Gintis, H. (2004).  Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-Scale Societies.  Oxford University Press.
  * Hofstadter, (1983) Hofstadter, D. (1983).  Dilemmas for superrational thinkers, leading up to a luring lottery.  Scientific American, 248(6).
  * Howard, (1988) Howard, J. V. (1988).  Cooperation in the prisoner’s dilemma.  Theory and Decision, 24(3):203–213.
  * Hughes et al., (2018) Hughes, E., Leibo, J., Phillips, M., Tuyls, K., Duenez-Guzman, E., Castaneda, A., Dunning, I., Zhu, T., McKee, K., Koster, R., Roff, H., and Graepel, T. (2018).  Inequity aversion improves cooperation in intertemporal social dilemmas.
  * Lanctot et al., (2017) Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Pérolat, J., Silver, D., and Graepel, T. (2017).  A unified game-theoretic approach to multiagent reinforcement learning.  arXiv preprint arXiv:1711.00832.
  * Laurent et al., (2011) Laurent, G. J., Matignon, L., , and Le Fort-Piat, N. (2011).  The world of independent learners is not markovian.  International Journal of Knowledge-Based and Intelligent Engineering Systems, 15(1):55–64.
  * LaVictoire et al., (2014) LaVictoire, P., Fallenstein, B., Yudkowsky, E., Barasz, M., Christiano, P., and Herreshoff, M. (2014).  Program equilibrium in the prisoner’s dilemma via löb’s theorem.  In Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence.
  * Leibo et al., (2017) Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017).  Multi-agent reinforcement learning in sequential social dilemmas.  In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 464–473. International Foundation for Autonomous Agents and Multiagent Systems.
  * Lerer and Peysakhovich, (2017) Lerer, A. and Peysakhovich, A. (2017).  Maintaining cooperation in complex social dilemmas using deep reinforcement learning.  arXiv preprint arXiv:1707.01068.
  * Letcher, (2018) Letcher, A. (2018).  Stability and exploitation in differentiable games.  Master’s thesis, University of Oxford.  Accessed: 2020-06-23.
  * Letcher et al., (2018) Letcher, A., Foerster, J., Balduzzi, D., Rocktäschel, T., and Whiteson, S. (2018).  Stable opponent shaping in differentiable games.  arXiv preprint arXiv:1811.08469.
  * McAfee, (1984) McAfee, R. P. (1984).  Effective computability in economic decisions.
  * Oesterheld, (2019) Oesterheld, C. (2019).  Robust program equilibrium.  Theory and Decision, 86(1):143–159.
  * Oosterbeek et al., (2004) Oosterbeek, H., Sloof, R., and van de Kuilen, G. (2004).  Cultural differences in ultimatum game experiments: Evidence from a meta-analysis.  Experimental Economics, 7(2):171–188.
  * Peysakhovich and Lerer, (2017) Peysakhovich, A. and Lerer, A. (2017).  Consequentialist conditional cooperation in social dilemmas with imperfect information.  arXiv preprint arXiv:1710.06975.
  * Sanfey et al., (2003) Sanfey, A., Rilling, A., and Nystrom, C. (2003).  The neural basis of economic decision-making in the ultimatum game.  Science, 300(5626):1755–1758.
  * Tennenholz, (2004) Tennenholz, M. (2004).  Program equilibrium.  Games and Economic Behavior, 49(2):363–373.
  * van der Hoek et al., (2013) van der Hoek, W., Witteveen, C., and Wooldridge, M. (2013).  Program equilibrium — a program reasoning approach.  International Journal of Game Theory, 42(3):639–671.
  * Wang et al., (2018) Wang, W., Hao, J., Wang, Y., and Taylor, M. (2018).  Towards cooperation in sequential prisoner’s dilemmas: a deep multiagent reinforcement learning approach.  arXiv preprint arXiv:1803.00162.
  * Zhang and Lesser, (2010) Zhang, C. and Lesser, V. (2010).  Multi-agent learning with policy prediction.  In Twenty-Fourth AAAI Conference on Artificial Intelligence.