blog posts from google's ai blog about NLP
https://ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and-high.html
LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything
2022-01-21
None
<div><div>
<p>Posted by Heng-Tze Cheng, Senior Staff Software Engineer and Romal Thoppilan, Senior Software Engineer, Google Research, Brain Team</p><p>
Language models are becoming more capable than ever before and are helpful in a variety of tasks — <a href="https://ai.googleblog.com/2020/06/recent-advances-in-google-translate.html">translating</a> one language into another, <a href="https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html">summarizing a long document</a> into a brief highlight, or <a href="https://ai.googleblog.com/2021/03/progress-and-challenges-in-long-form.html">answering information-seeking questions</a>. Among these, <a href="https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html">open-domain dialog</a>, where a model needs to be able to converse about any topic, is probably one of the most difficult, with a wide range of potential applications and open challenges. In addition to producing responses that humans judge as sensible, interesting, and specific to the context, dialog models should adhere to <a href="https://ai.google/responsibilities/responsible-ai-practices/">Responsible AI practices</a>, and avoid making factual statements that are not supported by external information sources.
</p>
<p>
Today we’re excited to share recent advances in our “<a href="https://arxiv.org/abs/2201.08239">LaMDA: Language Models for Dialog Applications</a>” project. In this post, we’ll give an overview on how we’re making progress towards safe, grounded, and high-quality dialog applications. <a href="https://blog.google/technology/ai/lamda/">LaMDA</a> is built by fine-tuning a family of <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">Transformer</a>-based neural language models specialized for dialog, with up to 137B model parameters, and teaching the models to leverage external knowledge sources. 
</p>
<h2>Objectives &amp; Metrics</h2>
<p>
Defining objectives and metrics is critical to guide training dialog models. LaMDA has three key objectives — Quality, Safety, and Groundedness — each of which we measure using carefully designed metrics:
</p>
<p>
<i>Quality</i>: We decompose Quality into three dimensions, Sensibleness, Specificity, and Interestingness (SSI), which are evaluated by human raters. Sensibleness refers to whether the model produces responses that make sense in the dialog context (e.g., no common sense mistakes, no absurd responses, and no contradictions with earlier responses). Specificity is measured by judging whether the system's response is specific to the preceding dialog context, and not a generic response that could apply to most contexts (e.g., “ok” or “I don’t know”). Finally, Interestingness measures whether the model produces responses that are also insightful, unexpected or witty, and are therefore more likely to create better dialog.
</p>
<p>
<i>Safety:</i> We’re also making progress towards addressing important questions related to the development and deployment of Responsible AI. Our Safety metric is composed of an illustrative set of safety objectives that captures the behavior that the model should exhibit in a dialog. These objectives attempt to constrain the model’s output to avoid any unintended results that create risks of harm for the user, and to avoid reinforcing unfair bias. For example, these objectives train the model to avoid producing outputs that contain violent or gory content, promote slurs or hateful stereotypes towards groups of people, or contain profanity. Our research towards developing a practical Safety metric represents very early work, and there is still a great deal of progress for us to make in this area.
</p>
<p>
<i>Groundedness</i>: The current generation of language models often generate statements that seem plausible, <a href="https://arxiv.org/abs/2112.12870">but actually contradict facts</a> established in known external sources. This motivates our study of groundedness in LaMDA. Groundedness is defined as the percentage of responses with claims about the external world that can be supported by authoritative external sources, as a share of all responses containing claims about the external world. A related metric, Informativeness, is defined as the percentage of responses with information about the external world that can be supported by known sources, as a share of all responses. Therefore, casual responses that do not carry any real world information (e.g., “That’s a great idea”), affect Informativeness but not Groundedness. While grounding LaMDA generated responses in known sources does not in itself guarantee factual accuracy, it allows users or external systems to judge the validity of a response based on the reliability of its source.
</p>
<h2>LaMDA Pre-Training</h2>
<p>
With the objectives and metrics defined, we describe LaMDA’s two-stage training: pre-training and fine-tuning. In the pre-training stage, we first created a dataset of 1.56T words — nearly 40 times more words than what were used to train <a href="https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html">previous dialog models</a> — from public dialog data and other public web documents. After tokenizing the dataset into 2.81T <a href="https://arxiv.org/abs/1808.06226">SentencePiece</a> tokens, we pre-train the model using <a href="https://ai.googleblog.com/2021/12/general-and-scalable-parallelization.html">GSPMD</a> to predict every next token in a sentence, given the previous tokens. The pre-trained LaMDA model has also been widely used for natural language processing research across Google, including <a href="https://arxiv.org/abs/2108.07732">program synthesis</a>, <a href="https://arxiv.org/pdf/2109.01652.pdf">zero-shot learning</a>, <a href="https://arxiv.org/abs/2109.03910">style transfer,</a> as well as in the <a href="https://welmworkshop.github.io/">BIG-bench workshop</a>. 
</p>
<h2>LaMDA Fine-Tuning</h2>
<p>
In the fine-tuning stage, we train LaMDA to perform a mix of <a href="https://en.wikipedia.org/wiki/Generative_model">generative</a> tasks to generate natural-language responses to given contexts, and classification tasks on whether a response is safe and high-quality, resulting in a single multi-task model that can do both. The LaMDA generator is trained to predict the next token on a dialog dataset restricted to back-and-forth dialog between two authors, while the LaMDA classifiers are trained to predict the Safety and Quality (SSI) ratings for the response in context using annotated data. During a dialog, the LaMDA generator first generates several candidate responses given the current multi-turn dialog context, and the LaMDA classifiers predict the SSI and Safety scores for every response candidate. Candidate responses with low Safety scores are first filtered out. Remaining candidates are re-ranked by their SSI scores, and the top result is selected as the response. We further filter the training data used for the generation task with LaMDA classifiers to increase the density of high-quality response candidates.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiUxNhpD_bycIsHoFrzmLGQyk7WNBeunUtqL-0mGrm85jm-gddkj9-pb9RssdntQbRmJIB_BeXhIz4gt4C_eqj5IcPxes7eS1Yi7EqrPZUCCtKxY57LhSDwQlJBDVPVnly-xTXNNrMf_TpGCwp3ggPdkbWYdjlN7yFtU_nd0pGWmn_SPYJ-uJBVgDzexQ=s624"></a></td></tr><tr><td>LaMDA generates and then scores a response candidate.</td></tr></tbody></table>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhbdzB6K3ip35giaEeE9Rw6zt8AgO1dKm2p9g1uboX0ixuatJOxXsnXL5Y5dtdHhNaPvygMe93Zw5C-kJrwMWo6Cjxqom9uZGw2OTsvjKyVtc2GDy_Ad1r9XkxipBJgSj4dklwqAr4IaYRGTkzhUHSTC_3FvjY7cqGs8uaWIb8dCBJM6-lApr1MRflkog=s629"></a></td></tr><tr><td>LaMDA handles arbitrary user input in a way that is sensible, specific, and interesting. Only LaMDA’s very first statement “Hello, I’m a friendly...” was hard coded to set the purpose of the dialog.</td></tr></tbody></table>
<br />
<h2>Factual Grounding</h2>
<p>
While people are capable of checking their facts by using tools and referencing established knowledge bases, many language models draw their knowledge on their internal model parameters only. To improve the groundedness of LaMDA’s original response, we collect a dataset of dialogs between people and LaMDA, which are annotated with information retrieval queries and the retrieved results where applicable. We then fine-tune LaMDA’s generator and classifier on this dataset to learn to call an external information retrieval system during its interaction with the user to improve the groundedness of its responses. While this is very early work, we’re seeing promising results.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEh-h02QIChSfFvjYEucKAYtK2qCQ39KVz8Xb7b7XTVGPq6O7f0vgNl-OVFJ4Sr0WX7iZoxIiAS9VJxfqbr361hF7AeZr2zLpYVlKQzQJyrPrVri8POEKl8cHIvdPPhulRooaoVmkUPz9JFGitY3QebvOZ_6_rXKQXw3DRBLGfKZLLK6zlOgwHJOZgt42Q=s612"></a></td></tr><tr><td>Zero-shot domain adaptation: cherry-picked, but real example of LaMDA pretending to be Mount Everest, by simply setting its initial message to be “Hi I’m Mount Everest. What would you like me to know about me?” Everest LaMDA is shown providing educational and factually correct responses.</td></tr></tbody></table>
<br />
<h2>Evaluation</h2>
<p>
In order to quantify progress against our key metrics, we collect responses from the pre-trained model, fine-tuned model, and human raters (i.e., human-generated responses) to multi-turn two-author dialogs, and then ask a different set of human raters a series of questions to evaluate these responses against the Quality, Safety, and Groundedness metrics. 
</p>
<p>
We observe that LaMDA significantly outperforms the pre-trained model in every dimension and across all model sizes. Quality metrics (Sensibleness, Specificity, and Interestingness, in the first column below) generally improve with the number of model parameters, with or without fine-tuning. Safety does not seem to benefit from model scaling alone, but it does improve with fine-tuning. Groundedness improves as model size increases, perhaps because larger models have a greater capacity to memorize uncommon knowledge, but fine-tuning allows the model to access external knowledge sources and effectively shift some of the load of remembering knowledge to an external knowledge source. With fine-tuning, the quality gap to human levels can be narrowed, though the model’s performance remains below human levels in safety and groundedness.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEids_HMHT2KBlDz-2thMZY8qzfamQFdKrspwbHPGCJtKHvKdTLzfJ6sX97gvXeMqeNJi-0DAjCGHBiJqTH0Zo5-8D2LBIydxY02vRSs3Y8ZaxIeIBEMaBIcspVnL2XsAcczCFW-x2qfqbixdvWcE4USG0xe7gs1A7Sbd-_lBaRy5tdJY0lsDpuCzLis1Q=s699"></a></td></tr><tr><td>Comparing the pre-trained model (PT), fine-tuned model (LaMDA) and human-rater-generated dialogs (Human) across Sensibleness, Specificity, Interestingness, Safety, Groundedness, and Informativeness. The test sets used to measure Safety and Groundedness were designed to be especially difficult.</td></tr></tbody></table>
<br />
<h2>Future Research &amp; Challenges</h2>
<p>
LaMDA’s level of Sensibleness, Specificity and Interestingness unlocks new avenues for understanding the benefits and risks of open-ended dialog agents. It also presents encouraging evidence that key challenges with neural language models, such as using a safety metric and improving groundedness, can improve with larger models and fine-tuning with more well-labeled data. However, this is very early work, and there are significant limitations. Exploring new ways to improve our Safety metric and LaMDA's groundedness, aligned with our <a href="https://ai.google/principles/">AI Principles</a>, will continue to be our main areas of focus going forward.
</p>
<h2>Acknowledgements </h2>
<p>
<i>We'd to like to thank everyone for contributing to the project and paper, including: Blaise Aguera-Arcas, Javier Alberca, Thushan Amarasiriwardena, Lora Aroyo, Martin Baeuml, Leslie Baker, Rachel Bernstein, Taylor Bos, Maarten Bosma, Jonas Bragagnolo, Alena Butryna, Bill Byrne, Chung-Ching Chang, Zhifeng Chen, Dehao Chen, Heng-Tze Cheng, Ed Chi, Aaron Cohen, Eli Collins, Marian Croak, Claire Cui, Andrew Dai, Dipanjan Das, Daniel De Freitas, Jeff Dean, Rajat Dewan, Mark Diaz, Tulsee Doshi, Yu Du, Toju Duke, Doug Eck, Joe Fenton, Noah Fiedel, Christian Frueh, Harish Ganapathy, Saravanan Ganesh, Amin Ghafouri, Zoubin Ghahramani, Kourosh Gharachorloo, Jamie Hall, Erin Hoffman-John, Sissie Hsiao, Yanping Huang, Ben Hutchinson, Daphne Ippolito, Alicia Jin, Thomas Jurdi, Ashwin Kakarla, Nand Kishore, Maxim Krikun, Karthik Krishnamoorthi, Igor Krivokon, Apoorv Kulshreshtha, Ray Kurzweil, Viktoriya Kuzmina, Vivek Kwatra, Matthew Lamm, Quoc Le, Max Lee, Katherine Lee, Hongrae Lee, Josh Lee, Dmitry Lepikhin, YaGuang Li, Yifeng Lu, David Luan, Daphne Luong, Laichee Man, Jianchang (JC) Mao, Yossi Matias, Kathleen Meier-Hellstern, Marcelo Menegali, Muqthar Mohammad,, Muqthar Mohammad, Alejandra Molina, Erica Moreira, Meredith Ringel Morris, Maysam Moussalem, Jiaqi Mu, Tyler Mullen, Tyler Mullen, Eric Ni, Kristen Olson, Alexander Passos, Fernando Pereira, Slav Petrov, Marc Pickett, Roberto Pieraccini, Christian Plagemann, Sahitya Potluri, Vinodkumar Prabhakaran, Andy Pratt, James Qin, Ravi Rajakumar, Adam Roberts, Will Rusch, Renelito Delos Santos, Noam Shazeer, RJ Skerry-Ryan, Grigori Somin, Johnny Soraker, Pranesh Srinivasan, Amarnag Subramanya, Mustafa Suleyman, Romal Thoppilan, Song Wang, Sheng Wang, Chris Wassman, Yuanzhong Xu, Yuanzhong Xu, Ni Yan, Ben Zevenbergen, Vincent Zhao, Huaixiu Steven Zheng, Denny Zhou, Hao Zhou, Yanqi Zhou, and more.</i>
</p>
</div>||||I||||Skip to main content
 LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything
 LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything

 Friday, January 21, 2022

 Posted by Heng-Tze Cheng, Senior Staff Software Engineer and Romal Thoppilan, Senior Software Engineer, Google Research, Brain Team

 Language models are becoming more capable than ever before and are helpful in a variety of tasks — translating one language into another, summarizing a long document into a brief highlight, or answering information-seeking questions. Among these, open-domain dialog, where a model needs to be able to converse about any topic, is probably one of the most difficult, with a wide range of potential applications and open challenges. In addition to producing responses that humans judge as sensible, interesting, and specific to the context, dialog models should adhere to Responsible AI practices, and avoid making factual statements that are not supported by external information sources.

 Today we’re excited to share recent advances in our “LaMDA: Language Models for Dialog Applications” project. In this post, we’ll give an overview on how we’re making progress towards safe, grounded, and high-quality dialog applications. LaMDA is built by fine-tuning a family of Transformer-based neural language models specialized for dialog, with up to 137B model parameters, and teaching the models to leverage external knowledge sources.


 Objectives & Metrics

 Defining objectives and metrics is critical to guide training dialog models. LaMDA has three key objectives — Quality, Safety, and Groundedness — each of which we measure using carefully designed metrics:

 Quality: We decompose Quality into three dimensions, Sensibleness, Specificity, and Interestingness (SSI), which are evaluated by human raters. Sensibleness refers to whether the model produces responses that make sense in the dialog context (e.g., no common sense mistakes, no absurd responses, and no contradictions with earlier responses). Specificity is measured by judging whether the system's response is specific to the preceding dialog context, and not a generic response that could apply to most contexts (e.g., “ok” or “I don’t know”). Finally, Interestingness measures whether the model produces responses that are also insightful, unexpected or witty, and are therefore more likely to create better dialog.

 Safety: We’re also making progress towards addressing important questions related to the development and deployment of Responsible AI. Our Safety metric is composed of an illustrative set of safety objectives that captures the behavior that the model should exhibit in a dialog. These objectives attempt to constrain the model’s output to avoid any unintended results that create risks of harm for the user, and to avoid reinforcing unfair bias. For example, these objectives train the model to avoid producing outputs that contain violent or gory content, promote slurs or hateful stereotypes towards groups of people, or contain profanity. Our research towards developing a practical Safety metric represents very early work, and there is still a great deal of progress for us to make in this area.

 Groundedness: The current generation of language models often generate statements that seem plausible, but actually contradict facts established in known external sources. This motivates our study of groundedness in LaMDA. Groundedness is defined as the percentage of responses with claims about the external world that can be supported by authoritative external sources, as a share of all responses containing claims about the external world. A related metric, Informativeness, is defined as the percentage of responses with information about the external world that can be supported by known sources, as a share of all responses. Therefore, casual responses that do not carry any real world information (e.g., “That’s a great idea”), affect Informativeness but not Groundedness. While grounding LaMDA generated responses in known sources does not in itself guarantee factual accuracy, it allows users or external systems to judge the validity of a response based on the reliability of its source.


 LaMDA Pre-Training

 With the objectives and metrics defined, we describe LaMDA’s two-stage training: pre-training and fine-tuning. In the pre-training stage, we first created a dataset of 1.56T words — nearly 40 times more words than what were used to train previous dialog models — from public dialog data and other public web documents. After tokenizing the dataset into 2.81T SentencePiece tokens, we pre-train the model using GSPMD to predict every next token in a sentence, given the previous tokens. The pre-trained LaMDA model has also been widely used for natural language processing research across Google, including program synthesis, zero-shot learning, style transfer, as well as in the BIG-bench workshop.


 LaMDA Fine-Tuning

 In the fine-tuning stage, we train LaMDA to perform a mix of generative tasks to generate natural-language responses to given contexts, and classification tasks on whether a response is safe and high-quality, resulting in a single multi-task model that can do both. The LaMDA generator is trained to predict the next token on a dialog dataset restricted to back-and-forth dialog between two authors, while the LaMDA classifiers are trained to predict the Safety and Quality (SSI) ratings for the response in context using annotated data. During a dialog, the LaMDA generator first generates several candidate responses given the current multi-turn dialog context, and the LaMDA classifiers predict the SSI and Safety scores for every response candidate. Candidate responses with low Safety scores are first filtered out. Remaining candidates are re-ranked by their SSI scores, and the top result is selected as the response. We further filter the training data used for the generation task with LaMDA classifiers to increase the density of high-quality response candidates.

  
 LaMDA generates and then scores a response candidate.
 
    
 LaMDA handles arbitrary user input in a way that is sensible, specific, and interesting. Only LaMDA’s very first statement “Hello, I’m a friendly...” was hard coded to set the purpose of the dialog.
 



 Factual Grounding

 While people are capable of checking their facts by using tools and referencing established knowledge bases, many language models draw their knowledge on their internal model parameters only. To improve the groundedness of LaMDA’s original response, we collect a dataset of dialogs between people and LaMDA, which are annotated with information retrieval queries and the retrieved results where applicable. We then fine-tune LaMDA’s generator and classifier on this dataset to learn to call an external information retrieval system during its interaction with the user to improve the groundedness of its responses. While this is very early work, we’re seeing promising results.

     
 Zero-shot domain adaptation: cherry-picked, but real example of LaMDA pretending to be Mount Everest, by simply setting its initial message to be “Hi I’m Mount Everest. What would you like me to know about me?” Everest LaMDA is shown providing educational and factually correct responses.
 



 Evaluation

 In order to quantify progress against our key metrics, we collect responses from the pre-trained model, fine-tuned model, and human raters (i.e., human-generated responses) to multi-turn two-author dialogs, and then ask a different set of human raters a series of questions to evaluate these responses against the Quality, Safety, and Groundedness metrics.

 We observe that LaMDA significantly outperforms the pre-trained model in every dimension and across all model sizes. Quality metrics (Sensibleness, Specificity, and Interestingness, in the first column below) generally improve with the number of model parameters, with or without fine-tuning. Safety does not seem to benefit from model scaling alone, but it does improve with fine-tuning. Groundedness improves as model size increases, perhaps because larger models have a greater capacity to memorize uncommon knowledge, but fine-tuning allows the model to access external knowledge sources and effectively shift some of the load of remembering knowledge to an external knowledge source. With fine-tuning, the quality gap to human levels can be narrowed, though the model’s performance remains below human levels in safety and groundedness.

      
 Comparing the pre-trained model (PT), fine-tuned model (LaMDA) and human-rater-generated dialogs (Human) across Sensibleness, Specificity, Interestingness, Safety, Groundedness, and Informativeness. The test sets used to measure Safety and Groundedness were designed to be especially difficult.
 



 Future Research & Challenges

 LaMDA’s level of Sensibleness, Specificity and Interestingness unlocks new avenues for understanding the benefits and risks of open-ended dialog agents. It also presents encouraging evidence that key challenges with neural language models, such as using a safety metric and improving groundedness, can improve with larger models and fine-tuning with more well-labeled data. However, this is very early work, and there are significant limitations. Exploring new ways to improve our Safety metric and LaMDA's groundedness, aligned with our AI Principles, will continue to be our main areas of focus going forward.


 Acknowledgements

 We'd to like to thank everyone for contributing to the project and paper, including: Blaise Aguera-Arcas, Javier Alberca, Thushan Amarasiriwardena, Lora Aroyo, Martin Baeuml, Leslie Baker, Rachel Bernstein, Taylor Bos, Maarten Bosma, Jonas Bragagnolo, Alena Butryna, Bill Byrne, Chung-Ching Chang, Zhifeng Chen, Dehao Chen, Heng-Tze Cheng, Ed Chi, Aaron Cohen, Eli Collins, Marian Croak, Claire Cui, Andrew Dai, Dipanjan Das, Daniel De Freitas, Jeff Dean, Rajat Dewan, Mark Diaz, Tulsee Doshi, Yu Du, Toju Duke, Doug Eck, Joe Fenton, Noah Fiedel, Christian Frueh, Harish Ganapathy, Saravanan Ganesh, Amin Ghafouri, Zoubin Ghahramani, Kourosh Gharachorloo, Jamie Hall, Erin Hoffman-John, Sissie Hsiao, Yanping Huang, Ben Hutchinson, Daphne Ippolito, Alicia Jin, Thomas Jurdi, Ashwin Kakarla, Nand Kishore, Maxim Krikun, Karthik Krishnamoorthi, Igor Krivokon, Apoorv Kulshreshtha, Ray Kurzweil, Viktoriya Kuzmina, Vivek Kwatra, Matthew Lamm, Quoc Le, Max Lee, Katherine Lee, Hongrae Lee, Josh Lee, Dmitry Lepikhin, YaGuang Li, Yifeng Lu, David Luan, Daphne Luong, Laichee Man, Jianchang (JC) Mao, Yossi Matias, Kathleen Meier-Hellstern, Marcelo Menegali, Muqthar Mohammad,, Muqthar Mohammad, Alejandra Molina, Erica Moreira, Meredith Ringel Morris, Maysam Moussalem, Jiaqi Mu, Tyler Mullen, Tyler Mullen, Eric Ni, Kristen Olson, Alexander Passos, Fernando Pereira, Slav Petrov, Marc Pickett, Roberto Pieraccini, Christian Plagemann, Sahitya Potluri, Vinodkumar Prabhakaran, Andy Pratt, James Qin, Ravi Rajakumar, Adam Roberts, Will Rusch, Renelito Delos Santos, Noam Shazeer, RJ Skerry-Ryan, Grigori Somin, Johnny Soraker, Pranesh Srinivasan, Amarnag Subramanya, Mustafa Suleyman, Romal Thoppilan, Song Wang, Sheng Wang, Chris Wassman, Yuanzhong Xu, Yuanzhong Xu, Ni Yan, Ben Zevenbergen, Vincent Zhao, Huaixiu Steven Zheng, Denny Zhou, Hao Zhou, Yanqi Zhou, and more.

 Labels: Natural Language Processing Natural Language Understanding NLP
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Heng-Tze Cheng, Senior Staff Software Engineer and Romal Thoppilan, Senior Software Engineer, Google Research, Brain Team 
Language models are becoming more capable than ever before and are helpful in a variety of tasks — translating one language into another, summarizing a long document into a brief highlight, or answering information-seeking questions. Among these, open-domain dialog, where a model needs to be able to converse about any topic, is probably one of the most difficult, with a wide range of potential applications and open challenges. In addition to producing responses that humans judge as sensible, interesting, and specific to the context, dialog models should adhere to Responsible AI practices, and avoid making factual statements that are not supported by external information sources.
Today we’re excited to share recent advances in our “LaMDA: Language Models for Dialog Applications” project. In this post, we’ll give an overview on how we’re making progress towards safe, grounded, and high-quality dialog applications. LaMDA is built by fine-tuning a family of Transformer-based neural language models specialized for dialog, with up to 137B model parameters, and teaching the models to leverage external knowledge sources. 
Objectives &amp; Metrics
Defining objectives and metrics is critical to guide training dialog models. LaMDA has three key objectives — Quality, Safety, and Groundedness — each of which we measure using carefully designed metrics:
Quality: We decompose Quality into three dimensions, Sensibleness, Specificity, and Interestingness (SSI), which are evaluated by human raters. Sensibleness refers to whether the model produces responses that make sense in the dialog context (e.g., no common sense mistakes, no absurd responses, and no contradictions with earlier responses). Specificity is measured by judging whether the system's response is specific to the preceding dialog context, and not a generic response that could apply to most contexts (e.g., “ok” or “I don’t know”). Finally, Interestingness measures whether the model produces responses that are also insightful, unexpected or witty, and are therefore more likely to create better dialog.
Safety: We’re also making progress towards addressing important questions related to the development and deployment of Responsible AI. Our Safety metric is composed of an illustrative set of safety objectives that captures the behavior that the model should exhibit in a dialog. These objectives attempt to constrain the model’s output to avoid any unintended results that create risks of harm for the user, and to avoid reinforcing unfair bias. For example, these objectives train the model to avoid producing outputs that contain violent or gory content, promote slurs or hateful stereotypes towards groups of people, or contain profanity. Our research towards developing a practical Safety metric represents very early work, and there is still a great deal of progress for us to make in this area.
Groundedness: The current generation of language models often generate statements that seem plausible, but actually contradict facts established in known external sources. This motivates our study of groundedness in LaMDA. Groundedness is defined as the percentage of responses with claims about the external world that can be supported by authoritative external sources, as a share of all responses containing claims about the external world. A related metric, Informativeness, is defined as the percentage of responses with information about the external world that can be supported by known sources, as a share of all responses. Therefore, casual responses that do not carry any real world information (e.g., “That’s a great idea”), affect Informativeness but not Groundedness. While grounding LaMDA generated responses in known sources does not in itself guarantee factual accuracy, it allows users or external systems to judge the validity of a response based on the reliability of its source.
LaMDA Pre-Training
With the objectives and metrics defined, we describe LaMDA’s two-stage training: pre-training and fine-tuning. In the pre-training stage, we first created a dataset of 1.56T words — nearly 40 times more words than what were used to train previous dialog models — from public dialog data and other public web documents. After tokenizing the dataset into 2.81T SentencePiece tokens, we pre-train the model using GSPMD to predict every next token in a sentence, given the previous tokens. The pre-trained LaMDA model has also been widely used for natural language processing research across Google, including program synthesis, zero-shot learning, style transfer, as well as in the BIG-bench workshop. 
LaMDA Fine-Tuning
In the fine-tuning stage, we train LaMDA to perform a mix of generative tasks to generate natural-language responses to given contexts, and classification tasks on whether a response is safe and high-quality, resulting in a single multi-task model that can do both. The LaMDA generator is trained to predict the next token on a dialog dataset restricted to back-and-forth dialog between two authors, while the LaMDA classifiers are trained to predict the Safety and Quality (SSI) ratings for the response in context using annotated data. During a dialog, the LaMDA generator first generates several candidate responses given the current multi-turn dialog context, and the LaMDA classifiers predict the SSI and Safety scores for every response candidate. Candidate responses with low Safety scores are first filtered out. Remaining candidates are re-ranked by their SSI scores, and the top result is selected as the response. We further filter the training data used for the generation task with LaMDA classifiers to increase the density of high-quality response candidates.
         LaMDA generates and then scores a response candidate.   
         LaMDA handles arbitrary user input in a way that is sensible, specific, and interesting. Only LaMDA’s very first statement “Hello, I’m a friendly...” was hard coded to set the purpose of the dialog.   
Factual Grounding
While people are capable of checking their facts by using tools and referencing established knowledge bases, many language models draw their knowledge on their internal model parameters only. To improve the groundedness of LaMDA’s original response, we collect a dataset of dialogs between people and LaMDA, which are annotated with information retrieval queries and the retrieved results where applicable. We then fine-tune LaMDA’s generator and classifier on this dataset to learn to call an external information retrieval system during its interaction with the user to improve the groundedness of its responses. While this is very early work, we’re seeing promising results.
         Zero-shot domain adaptation: cherry-picked, but real example of LaMDA pretending to be Mount Everest, by simply setting its initial message to be “Hi I’m Mount Everest. What would you like me to know about me?” Everest LaMDA is shown providing educational and factually correct responses.   
Evaluation
In order to quantify progress against our key metrics, we collect responses from the pre-trained model, fine-tuned model, and human raters (i.e., human-generated responses) to multi-turn two-author dialogs, and then ask a different set of human raters a series of questions to evaluate these responses against the Quality, Safety, and Groundedness metrics. 
We observe that LaMDA significantly outperforms the pre-trained model in every dimension and across all model sizes. Quality metrics (Sensibleness, Specificity, and Interestingness, in the first column below) generally improve with the number of model parameters, with or without fine-tuning. Safety does not seem to benefit from model scaling alone, but it does improve with fine-tuning. Groundedness improves as model size increases, perhaps because larger models have a greater capacity to memorize uncommon knowledge, but fine-tuning allows the model to access external knowledge sources and effectively shift some of the load of remembering knowledge to an external knowledge source. With fine-tuning, the quality gap to human levels can be narrowed, though the model’s performance remains below human levels in safety and groundedness.
         Comparing the pre-trained model (PT), fine-tuned model (LaMDA) and human-rater-generated dialogs (Human) across Sensibleness, Specificity, Interestingness, Safety, Groundedness, and Informativeness. The test sets used to measure Safety and Groundedness were designed to be especially difficult.   
Future Research &amp; Challenges
LaMDA’s level of Sensibleness, Specificity and Interestingness unlocks new avenues for understanding the benefits and risks of open-ended dialog agents. It also presents encouraging evidence that key challenges with neural language models, such as using a safety metric and improving groundedness, can improve with larger models and fine-tuning with more well-labeled data. However, this is very early work, and there are significant limitations. Exploring new ways to improve our Safety metric and LaMDA's groundedness, aligned with our AI Principles, will continue to be our main areas of focus going forward.
Acknowledgements 
We'd to like to thank everyone for contributing to the project and paper, including: Blaise Aguera-Arcas, Javier Alberca, Thushan Amarasiriwardena, Lora Aroyo, Martin Baeuml, Leslie Baker, Rachel Bernstein, Taylor Bos, Maarten Bosma, Jonas Bragagnolo, Alena Butryna, Bill Byrne, Chung-Ching Chang, Zhifeng Chen, Dehao Chen, Heng-Tze Cheng, Ed Chi, Aaron Cohen, Eli Collins, Marian Croak, Claire Cui, Andrew Dai, Dipanjan Das, Daniel De Freitas, Jeff Dean, Rajat Dewan, Mark Diaz, Tulsee Doshi, Yu Du, Toju Duke, Doug Eck, Joe Fenton, Noah Fiedel, Christian Frueh, Harish Ganapathy, Saravanan Ganesh, Amin Ghafouri, Zoubin Ghahramani, Kourosh Gharachorloo, Jamie Hall, Erin Hoffman-John, Sissie Hsiao, Yanping Huang, Ben Hutchinson, Daphne Ippolito, Alicia Jin, Thomas Jurdi, Ashwin Kakarla, Nand Kishore, Maxim Krikun, Karthik Krishnamoorthi, Igor Krivokon, Apoorv Kulshreshtha, Ray Kurzweil, Viktoriya Kuzmina, Vivek Kwatra, Matthew Lamm, Quoc Le, Max Lee, Katherine Lee, Hongrae Lee, Josh Lee, Dmitry Lepikhin, YaGuang Li, Yifeng Lu, David Luan, Daphne Luong, Laichee Man, Jianchang (JC) Mao, Yossi Matias, Kathleen Meier-Hellstern, Marcelo Menegali, Muqthar Mohammad,, Muqthar Mohammad, Alejandra Molina, Erica Moreira, Meredith Ringel Morris, Maysam Moussalem, Jiaqi Mu, Tyler Mullen, Tyler Mullen, Eric Ni, Kristen Olson, Alexander Passos, Fernando Pereira, Slav Petrov, Marc Pickett, Roberto Pieraccini, Christian Plagemann, Sahitya Potluri, Vinodkumar Prabhakaran, Andy Pratt, James Qin, Ravi Rajakumar, Adam Roberts, Will Rusch, Renelito Delos Santos, Noam Shazeer, RJ Skerry-Ryan, Grigori Somin, Johnny Soraker, Pranesh Srinivasan, Amarnag Subramanya, Mustafa Suleyman, Romal Thoppilan, Song Wang, Sheng Wang, Chris Wassman, Yuanzhong Xu, Yuanzhong Xu, Ni Yan, Ben Zevenbergen, Vincent Zhao, Huaixiu Steven Zheng, Denny Zhou, Hao Zhou, Yanqi Zhou, and more.
LaMDA: Towards Safe, Grounded, and High-Quality Dialog Models for Everything
https://ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and-high.html
https://ai.googleblog.com/2022/01/learning-to-route-by-task-for-efficient.html
Learning to Route by Task for Efficient Inference
2022-01-14
Sneha Kudugunta; Research Software Engineer; Orhan Firat; Research Scientist; Google Research
<div><div>
<p>Posted by Sneha Kudugunta, Research Software Engineer and Orhan Firat, Research Scientist, Google Research
</p><p>
Scaling large language models has resulted in significant quality improvements in natural language understanding (<a href="https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html">T5</a>), generation (<a href="https://arxiv.org/abs/2005.14165">GPT-3</a>) and multilingual neural machine translation (<a href="https://ai.googleblog.com/2019/10/exploring-massively-multilingual.html">M4</a>). One common approach to building a larger model is to increase the depth (number of layers) and width (layer dimensionality), simply enlarging existing dimensions of the network. Such dense models take an input sequence (divided into smaller components, called tokens) and pass every token through the full network, activating every layer and parameter. While these large, dense models have achieved state-of-the-art results on multiple natural language processing (NLP) tasks, their training cost increases linearly with model size. 
</p>
<p>
An alternative, and increasingly popular, approach is to build sparsely activated models based on a <a href="https://en.wikipedia.org/wiki/Mixture_of_experts">mixture of experts</a> (MoE) (e.g., <a href="https://ai.googleblog.com/2021/01/google-research-looking-back-at-2020.html#Translation">GShard-M4</a> or <a href="https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html">GLaM</a>), where each token passed to the network follows a separate subnetwork by skipping some of the model parameters. The choice of how to distribute the input tokens to each subnetwork (the “experts”) is determined by small router networks that are trained together with the rest of the network. This allows researchers to increase model size (and hence, performance) without a proportional increase in training cost.
</p>
<p>
While this is an effective strategy at training time, sending tokens of a long sequence to multiple experts, again makes inference computationally expensive because the experts have to be distributed among a large number of accelerators. For example, serving the 1.2T parameter GLaM model requires 256 TPU-v3 chips. Much like dense models, the number of processors needed to serve an MoE model still scales linearly with respect to the model size, increasing compute requirements while also resulting in significant communication overhead and added engineering complexity.
</p>
<p>
In “<a href="https://arxiv.org/pdf/2110.03742.pdf">Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference</a>”, we introduce a method called Task-level Mixture-of-Experts (TaskMoE), that takes advantage of the quality gains of model scaling while still being efficient to serve. Our solution is to train a large multi-task model from which we then extract smaller, stand-alone per-task subnetworks suitable for inference with no loss in model quality and with significantly reduced inference latency. We demonstrate the effectiveness of this method for multilingual neural machine translation (NMT) compared to other mixture of experts models and to models compressed using <a href="https://arxiv.org/abs/1503.02531">knowledge distillation</a>. 
</p>
<h2>Training Large Sparsely Activated Models with Task Information</h2>
<p>
We train a sparsely activated model, where router networks learn to send tokens of each task-specific input to different subnetworks of the model associated with the task of interest. For example, in the case of multilingual NMT, every token of a given language is routed to the same subnetwork. This differs from other recent approaches, such as the <a href="https://arxiv.org/abs/1701.06538">sparsely gated mixture of expert models</a> (e.g., <a href="https://arxiv.org/abs/2006.16668">TokenMoE</a>), where router networks learn to send different tokens in an input to different subnetworks independent of task.
</p>
<h2>Inference: Bypassing Distillation by Extracting Subnetworks</h2>
<p>
A consequence of this difference in training between TaskMoE and models like TokenMoE is in how we approach inference. Because TokenMoE follows the practice of distributing tokens of the same task to many experts at both training and inference time, it is still computationally expensive at inference.
</p>
<p>
For TaskMoE, we dedicate a smaller subnetwork to a single task identity during training <em>and</em> inference. At inference time, we extract subnetworks by discarding unused experts for each task. TaskMoE and its variants enable us to train a single large multi-task network and then use a separate subnetwork at inference time for each task without using any additional compression methods post-training. We illustrate the process of training a TaskMoE network and then extracting per-task subnetworks for inference below.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgkqREeRRMeieObGensmatGJ4I-5aovQT3PcTgjBNBSWoIPPFB58p99uK2NxApXq7RHMIUPdpaEQsK1UhrhXQK58GOaev6gm0KfAe0Zm2miZwRRm_S7qAmVADwOhOAJeqz6cRVdds4yTj5GpvQVf-gRY4CjoOGkwfOGgNPLhpOwwu9Dwx0pLeQNdX2kJw=s1400"></a></td></tr><tr><td>During training, tokens of the same language are routed to the same expert based on language information (either source, target or both) in task-based MoE. Later, during inference we extract subnetworks for each task and discard unused experts.</td></tr></tbody></table>
<p>
To demonstrate this approach, we train models based on the <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">Transformer</a> architecture. Similar to <a href="https://arxiv.org/abs/2006.16668">GShard-M4</a> and <a href="https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html">GLaM</a>, we replace the <a href="https://en.wikipedia.org/wiki/Feedforward_neural_network">feedforward network</a> of every other transformer layer with a Mixture-of-Experts (MoE) layer that consists of multiple identical feedforward networks, the “experts”. For each task, the routing network, trained along with the rest of the model, keeps track of the task identity for all input tokens and chooses a certain number of experts per layer (two in this case) to form the task-specific subnetwork. The baseline dense Transformer model has 143M parameters and 6 layers on both the encoder and decoder. The TaskMoE and TokenMoE that we train are also both 6 layers deep but with 32 experts for every MoE layer and have a total of 533M parameters. We train our models using publicly available <a href="https://www.statmt.org/wmt21/">WMT</a> datasets, with over 431M sentences across 30 language pairs from different language families and scripts. We point the reader to the <a href="https://arxiv.org/abs/2110.03742">full paper</a> for further details.
</p>
<h2>Results</h2>
<p>
In order to demonstrate the advantage of using TaskMoE at inference time, we compare the throughput, or the number of tokens decoded per second, for TaskMoE, <a href="https://arxiv.org/abs/2006.16668">TokenMoE</a>, and a baseline dense model. Once the subnetwork for each task is extracted, TaskMoE is 7x smaller than the 533M parameter TokenMoE model, and it can be served on a single TPUv3 core, instead of 64 cores required for TokenMoE. We see that TaskMoE has a peak throughput twice as high as that of TokenMoE models. In addition, on inspecting the TokenMoE model, we find that 25% of the inference time has been spent in inter-device communication, while virtually no time is spent in communication by TaskMoE.
</p><table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiz2e-3_mYs-KXdk6YvDcrYoX5eR9cg9RzFhS7BLvHZPFI3fZO2OJ2schmKCK_hOZNwu_o7koZzpcb4IfN4QGWyaYuT3bTNglGIPI6uSl-emDJilcvBVzebwhQIxo_2aqjpZxGR36foIUlZQ3U4JKtupKA44Q2inPe6tH7VRhrgTy-FH6EtHtiIjXf4Yw=s1200"></a></td></tr><tr><td>Comparing the throughput of TaskMoE with TokenMoE across different batch sizes. The maximum batch size for TokenMoE is 1024 as opposed to 4096 for TaskMoE and the dense baseline model. Here, TokenMoE has one instance distributed across 64 TPUv3 cores, while TaskMoE and the baseline model have one instance on each of the 64 cores.</td></tr></tbody></table>
<p>
A popular approach to building a smaller network that still performs well is through knowledge distillation, in which a large teacher model trains a smaller student model with the goal of matching the teacher’s performance. However, this method comes at the cost of additional computation needed to train the student from the teacher. So, we also compare TaskMoE to a baseline TokenMoE model that we compress using knowledge distillation. The compressed TokenMoE model has a size comparable to the per-task subnetwork extracted from TaskMoE. 
</p>
<p>
We find that in addition to being a simpler method that does not need any additional training, TaskMoE improves upon a distilled TokenMoE model by 2.1 <a href="https://en.wikipedia.org/wiki/BLEU">BLEU</a> on average across all languages in our multilingual translation model. We note that distillation retains 43% of the performance gains achieved from scaling a dense multilingual model to a TokenMoE, whereas extracting the smaller subnetwork from the TaskMoE model results in no loss of quality.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjUXaneWOTWV5TTmngavcsT9F9pEqCxEPvUWgcyWZtPhQE9R8uXtNJHVgHDCmCxSmlnSbFyNXOWxWMgf8JPCkum9Mh8TKshDSQRnSUbMaK2a3zEq7ajNtWaiYyowLKQmrJaX2zi0kzjEUWqwh1KRtR9aOLWhB30Z5qIsL7k8vAtD6dt4XbutW0TDVhhuQ=s1558"></a></td></tr><tr><td><a href="https://en.wikipedia.org/wiki/BLEU">BLEU</a> scores (higher is better) comparing a distilled TokenMoE model to the TaskMoE and TokenMoE models with 12 layers (6 on the encoder and 6 on the decoder) and 32 experts. While both approaches improve upon a multilingual dense baseline, TaskMoE improves upon the baseline by 3.1 BLEU on average while distilling from TokenMoE improves upon the baseline by 1.0 BLEU on average.</td></tr></tbody></table>
<br />
<h2>Next Steps</h2>
<p>
The quality improvements often seen with scaling machine learning models has incentivized the research community to work toward advancing scaling technology to enable efficient training of large models. The emerging need to train models capable of <a href="https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/">generalizing to multiple tasks and modalities</a> only increases the need for scaling models even further. However, the practicality of serving these large models remains a major challenge. Efficiently deploying large models is an important direction of research, and we believe TaskMoE is a promising step towards more inference friendly algorithms that retain the quality gains of scaling. 
</p>
 
<h2>Acknowledgements</h2>
<p>
<i>We would like to first thank our coauthors - Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin and Minh-Thang Luong. We would also like to thank Wolfgang Macherey, Yuanzhong Xu, Zhifeng Chen and Macduff Richard Hughes for their helpful feedback. Special thanks to the Translate and Brain teams for their useful input and discussions, and the entire GShard development team for their foundational contributions to this project. We would also like to thank Tom Small for creating the animations for the blog post. </i>
</p>
</div>||||I||||Skip to main content
 Learning to Route by Task for Efficient Inference
 Learning to Route by Task for Efficient Inference
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Learning to Route by Task for Efficient Inference

 Friday, January 14, 2022

 Posted by Sneha Kudugunta, Research Software Engineer and Orhan Firat, Research Scientist, Google Research

 Scaling large language models has resulted in significant quality improvements in natural language understanding (T5), generation (GPT-3) and multilingual neural machine translation (M4). One common approach to building a larger model is to increase the depth (number of layers) and width (layer dimensionality), simply enlarging existing dimensions of the network. Such dense models take an input sequence (divided into smaller components, called tokens) and pass every token through the full network, activating every layer and parameter. While these large, dense models have achieved state-of-the-art results on multiple natural language processing (NLP) tasks, their training cost increases linearly with model size.

 An alternative, and increasingly popular, approach is to build sparsely activated models based on a mixture of experts (MoE) (e.g., GShard-M4 or GLaM), where each token passed to the network follows a separate subnetwork by skipping some of the model parameters. The choice of how to distribute the input tokens to each subnetwork (the “experts”) is determined by small router networks that are trained together with the rest of the network. This allows researchers to increase model size (and hence, performance) without a proportional increase in training cost.

 While this is an effective strategy at training time, sending tokens of a long sequence to multiple experts, again makes inference computationally expensive because the experts have to be distributed among a large number of accelerators. For example, serving the 1.2T parameter GLaM model requires 256 TPU-v3 chips. Much like dense models, the number of processors needed to serve an MoE model still scales linearly with respect to the model size, increasing compute requirements while also resulting in significant communication overhead and added engineering complexity.

 In “Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference”, we introduce a method called Task-level Mixture-of-Experts (TaskMoE), that takes advantage of the quality gains of model scaling while still being efficient to serve. Our solution is to train a large multi-task model from which we then extract smaller, stand-alone per-task subnetworks suitable for inference with no loss in model quality and with significantly reduced inference latency. We demonstrate the effectiveness of this method for multilingual neural machine translation (NMT) compared to other mixture of experts models and to models compressed using knowledge distillation.


 Training Large Sparsely Activated Models with Task Information

 We train a sparsely activated model, where router networks learn to send tokens of each task-specific input to different subnetworks of the model associated with the task of interest. For example, in the case of multilingual NMT, every token of a given language is routed to the same subnetwork. This differs from other recent approaches, such as the sparsely gated mixture of expert models (e.g., TokenMoE), where router networks learn to send different tokens in an input to different subnetworks independent of task.


 Inference: Bypassing Distillation by Extracting Subnetworks

 A consequence of this difference in training between TaskMoE and models like TokenMoE is in how we approach inference. Because TokenMoE follows the practice of distributing tokens of the same task to many experts at both training and inference time, it is still computationally expensive at inference.

 For TaskMoE, we dedicate a smaller subnetwork to a single task identity during training and inference. At inference time, we extract subnetworks by discarding unused experts for each task. TaskMoE and its variants enable us to train a single large multi-task network and then use a separate subnetwork at inference time for each task without using any additional compression methods post-training. We illustrate the process of training a TaskMoE network and then extracting per-task subnetworks for inference below.

     
 During training, tokens of the same language are routed to the same expert based on language information (either source, target or both) in task-based MoE. Later, during inference we extract subnetworks for each task and discard unused experts.
 


 To demonstrate this approach, we train models based on the Transformer architecture. Similar to GShard-M4 and GLaM, we replace the feedforward network of every other transformer layer with a Mixture-of-Experts (MoE) layer that consists of multiple identical feedforward networks, the “experts”. For each task, the routing network, trained along with the rest of the model, keeps track of the task identity for all input tokens and chooses a certain number of experts per layer (two in this case) to form the task-specific subnetwork. The baseline dense Transformer model has 143M parameters and 6 layers on both the encoder and decoder. The TaskMoE and TokenMoE that we train are also both 6 layers deep but with 32 experts for every MoE layer and have a total of 533M parameters. We train our models using publicly available WMT datasets, with over 431M sentences across 30 language pairs from different language families and scripts. We point the reader to the full paper for further details.


 Results

 In order to demonstrate the advantage of using TaskMoE at inference time, we compare the throughput, or the number of tokens decoded per second, for TaskMoE, TokenMoE, and a baseline dense model. Once the subnetwork for each task is extracted, TaskMoE is 7x smaller than the 533M parameter TokenMoE model, and it can be served on a single TPUv3 core, instead of 64 cores required for TokenMoE. We see that TaskMoE has a peak throughput twice as high as that of TokenMoE models. In addition, on inspecting the TokenMoE model, we find that 25% of the inference time has been spent in inter-device communication, while virtually no time is spent in communication by TaskMoE.

      
 Comparing the throughput of TaskMoE with TokenMoE across different batch sizes. The maximum batch size for TokenMoE is 1024 as opposed to 4096 for TaskMoE and the dense baseline model. Here, TokenMoE has one instance distributed across 64 TPUv3 cores, while TaskMoE and the baseline model have one instance on each of the 64 cores.
 


 A popular approach to building a smaller network that still performs well is through knowledge distillation, in which a large teacher model trains a smaller student model with the goal of matching the teacher’s performance. However, this method comes at the cost of additional computation needed to train the student from the teacher. So, we also compare TaskMoE to a baseline TokenMoE model that we compress using knowledge distillation. The compressed TokenMoE model has a size comparable to the per-task subnetwork extracted from TaskMoE.

 We find that in addition to being a simpler method that does not need any additional training, TaskMoE improves upon a distilled TokenMoE model by 2.1 BLEU on average across all languages in our multilingual translation model. We note that distillation retains 43% of the performance gains achieved from scaling a dense multilingual model to a TokenMoE, whereas extracting the smaller subnetwork from the TaskMoE model results in no loss of quality.

       
 BLEU scores (higher is better) comparing a distilled TokenMoE model to the TaskMoE and TokenMoE models with 12 layers (6 on the encoder and 6 on the decoder) and 32 experts. While both approaches improve upon a multilingual dense baseline, TaskMoE improves upon the baseline by 3.1 BLEU on average while distilling from TokenMoE improves upon the baseline by 1.0 BLEU on average.
 



 Next Steps

 The quality improvements often seen with scaling machine learning models has incentivized the research community to work toward advancing scaling technology to enable efficient training of large models. The emerging need to train models capable of generalizing to multiple tasks and modalities only increases the need for scaling models even further. However, the practicality of serving these large models remains a major challenge. Efficiently deploying large models is an important direction of research, and we believe TaskMoE is a promising step towards more inference friendly algorithms that retain the quality gains of scaling.


 Acknowledgements

 We would like to first thank our coauthors - Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin and Minh-Thang Luong. We would also like to thank Wolfgang Macherey, Yuanzhong Xu, Zhifeng Chen and Macduff Richard Hughes for their helpful feedback. Special thanks to the Translate and Brain teams for their useful input and discussions, and the entire GShard development team for their foundational contributions to this project. We would also like to thank Tom Small for creating the animations for the blog post.

 Labels: Deep Learning EMNLP Google Translate Natural Language Processing
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Sneha Kudugunta, Research Software Engineer and Orhan Firat, Research Scientist, Google Research
 
Scaling large language models has resulted in significant quality improvements in natural language understanding (T5), generation (GPT-3) and multilingual neural machine translation (M4). One common approach to building a larger model is to increase the depth (number of layers) and width (layer dimensionality), simply enlarging existing dimensions of the network. Such dense models take an input sequence (divided into smaller components, called tokens) and pass every token through the full network, activating every layer and parameter. While these large, dense models have achieved state-of-the-art results on multiple natural language processing (NLP) tasks, their training cost increases linearly with model size. 
An alternative, and increasingly popular, approach is to build sparsely activated models based on a mixture of experts (MoE) (e.g., GShard-M4 or GLaM), where each token passed to the network follows a separate subnetwork by skipping some of the model parameters. The choice of how to distribute the input tokens to each subnetwork (the “experts”) is determined by small router networks that are trained together with the rest of the network. This allows researchers to increase model size (and hence, performance) without a proportional increase in training cost.
While this is an effective strategy at training time, sending tokens of a long sequence to multiple experts, again makes inference computationally expensive because the experts have to be distributed among a large number of accelerators. For example, serving the 1.2T parameter GLaM model requires 256 TPU-v3 chips. Much like dense models, the number of processors needed to serve an MoE model still scales linearly with respect to the model size, increasing compute requirements while also resulting in significant communication overhead and added engineering complexity.
In “Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference”, we introduce a method called Task-level Mixture-of-Experts (TaskMoE), that takes advantage of the quality gains of model scaling while still being efficient to serve. Our solution is to train a large multi-task model from which we then extract smaller, stand-alone per-task subnetworks suitable for inference with no loss in model quality and with significantly reduced inference latency. We demonstrate the effectiveness of this method for multilingual neural machine translation (NMT) compared to other mixture of experts models and to models compressed using knowledge distillation. 
Training Large Sparsely Activated Models with Task Information
We train a sparsely activated model, where router networks learn to send tokens of each task-specific input to different subnetworks of the model associated with the task of interest. For example, in the case of multilingual NMT, every token of a given language is routed to the same subnetwork. This differs from other recent approaches, such as the sparsely gated mixture of expert models (e.g., TokenMoE), where router networks learn to send different tokens in an input to different subnetworks independent of task.
Inference: Bypassing Distillation by Extracting Subnetworks
A consequence of this difference in training between TaskMoE and models like TokenMoE is in how we approach inference. Because TokenMoE follows the practice of distributing tokens of the same task to many experts at both training and inference time, it is still computationally expensive at inference.
For TaskMoE, we dedicate a smaller subnetwork to a single task identity during training and inference. At inference time, we extract subnetworks by discarding unused experts for each task. TaskMoE and its variants enable us to train a single large multi-task network and then use a separate subnetwork at inference time for each task without using any additional compression methods post-training. We illustrate the process of training a TaskMoE network and then extracting per-task subnetworks for inference below.
         During training, tokens of the same language are routed to the same expert based on language information (either source, target or both) in task-based MoE. Later, during inference we extract subnetworks for each task and discard unused experts.   
To demonstrate this approach, we train models based on the Transformer architecture. Similar to GShard-M4 and GLaM, we replace the feedforward network of every other transformer layer with a Mixture-of-Experts (MoE) layer that consists of multiple identical feedforward networks, the “experts”. For each task, the routing network, trained along with the rest of the model, keeps track of the task identity for all input tokens and chooses a certain number of experts per layer (two in this case) to form the task-specific subnetwork. The baseline dense Transformer model has 143M parameters and 6 layers on both the encoder and decoder. The TaskMoE and TokenMoE that we train are also both 6 layers deep but with 32 experts for every MoE layer and have a total of 533M parameters. We train our models using publicly available WMT datasets, with over 431M sentences across 30 language pairs from different language families and scripts. We point the reader to the full paper for further details.
Results
In order to demonstrate the advantage of using TaskMoE at inference time, we compare the throughput, or the number of tokens decoded per second, for TaskMoE, TokenMoE, and a baseline dense model. Once the subnetwork for each task is extracted, TaskMoE is 7x smaller than the 533M parameter TokenMoE model, and it can be served on a single TPUv3 core, instead of 64 cores required for TokenMoE. We see that TaskMoE has a peak throughput twice as high as that of TokenMoE models. In addition, on inspecting the TokenMoE model, we find that 25% of the inference time has been spent in inter-device communication, while virtually no time is spent in communication by TaskMoE.
          Comparing the throughput of TaskMoE with TokenMoE across different batch sizes. The maximum batch size for TokenMoE is 1024 as opposed to 4096 for TaskMoE and the dense baseline model. Here, TokenMoE has one instance distributed across 64 TPUv3 cores, while TaskMoE and the baseline model have one instance on each of the 64 cores.   
A popular approach to building a smaller network that still performs well is through knowledge distillation, in which a large teacher model trains a smaller student model with the goal of matching the teacher’s performance. However, this method comes at the cost of additional computation needed to train the student from the teacher. So, we also compare TaskMoE to a baseline TokenMoE model that we compress using knowledge distillation. The compressed TokenMoE model has a size comparable to the per-task subnetwork extracted from TaskMoE. 
We find that in addition to being a simpler method that does not need any additional training, TaskMoE improves upon a distilled TokenMoE model by 2.1 BLEU on average across all languages in our multilingual translation model. We note that distillation retains 43% of the performance gains achieved from scaling a dense multilingual model to a TokenMoE, whereas extracting the smaller subnetwork from the TaskMoE model results in no loss of quality.
          BLEU scores (higher is better) comparing a distilled TokenMoE model to the TaskMoE and TokenMoE models with 12 layers (6 on the encoder and 6 on the decoder) and 32 experts. While both approaches improve upon a multilingual dense baseline, TaskMoE improves upon the baseline by 3.1 BLEU on average while distilling from TokenMoE improves upon the baseline by 1.0 BLEU on average.   
Next Steps
The quality improvements often seen with scaling machine learning models has incentivized the research community to work toward advancing scaling technology to enable efficient training of large models. The emerging need to train models capable of generalizing to multiple tasks and modalities only increases the need for scaling models even further. However, the practicality of serving these large models remains a major challenge. Efficiently deploying large models is an important direction of research, and we believe TaskMoE is a promising step towards more inference friendly algorithms that retain the quality gains of scaling. 
 
Acknowledgements
We would like to first thank our coauthors - Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin and Minh-Thang Luong. We would also like to thank Wolfgang Macherey, Yuanzhong Xu, Zhifeng Chen and Macduff Richard Hughes for their helpful feedback. Special thanks to the Translate and Brain teams for their useful input and discussions, and the entire GShard development team for their foundational contributions to this project. We would also like to thank Tom Small for creating the animations for the blog post.
Learning to Route by Task for Efficient Inference
https://ai.googleblog.com/2022/01/learning-to-route-by-task-for-efficient.html
https://ai.googleblog.com/2021/01/learning-to-reason-over-tables-from.html
Learning to Reason Over Tables from Less Data
2021-01-29
Julian Eisenschlos; AI Resident; Google Research; Zürich
<div><div>
<p>Posted by Julian Eisenschlos, AI Resident, Google Research, Zürich</p><p>
The task of recognizing<em> </em><a href="https://en.wikipedia.org/wiki/Textual_entailment">textual entailment</a>, also known as natural language inference, consists of determining whether a piece of text (a <em>premise)</em>, can be implied or contradicted (or neither) by another piece of text (the <em>hypothesis)</em>. While this problem is often considered an important test for the reasoning skills of machine learning (ML) systems and has been studied in depth for plain text inputs, much less effort has been put into applying such models to structured data, such as websites, tables, databases, etc. Yet, recognizing textual entailment is especially relevant whenever the contents of a table need to be accurately summarized and presented to a user, and is essential for high fidelity <a href="https://en.wikipedia.org/wiki/Question_answering">question answering</a> systems and <a href="https://en.wikipedia.org/wiki/Virtual_assistant">virtual assistants</a>.
</p><p>
In "<a href="https://www.aclweb.org/anthology/2020.findings-emnlp.27/">Understanding tables with intermediate pre-training</a>", published in <a href="https://2020.emnlp.org/papers/findings">Findings of EMNLP 2020</a>, we introduce the first pre-training tasks customized for table parsing, enabling models to learn better, faster and from less data. We build upon our earlier <a href="https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html">TAPAS</a> model, which was an extension of the <a href="https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html">BERT</a> bi-directional <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">Transformer</a> model with special embeddings to find answers in tables. Applying our new pre-training objectives to TAPAS yields a new state of the art on multiple datasets involving tables. On <a href="https://tabfact.github.io/">TabFact</a>, for example, it reduces the gap between model and human performance by ~50%. We also systematically benchmark methods of selecting relevant input for higher efficiency, achieving 4x gains in speed and memory, while retaining 92% of the results. All the models for different tasks and sizes are released on <a href="https://github.com/google-research/tapas">GitHub repo</a>, where you can try them out yourself in a <a href="http://tiny.cc/tapas-tabfact-colab">colab</a> Notebook.
</p>
<h2>Textual Entailment</h2><p>
The task of textual entailment is more challenging when applied to tabular data than plain text. Consider, for example, a table from Wikipedia with some sentences derived from its associated table content. Assessing if the content of the table entails or contradicts the sentence may require looking over multiple columns and rows, and possibly performing simple numeric computations, like averaging, summing, differencing, etc.
</p>
<table><tbody><tr><td><a href="https://1.bp.blogspot.com/-TOSTbdc9ndk/YBRIuPjyCZI/AAAAAAAAHEw/NYIj9usvgaQu1Fm9UfBJPL_12M6Y0F5fgCLcBGAsYHQ/s1335/image5.png"></a></td></tr><tr><td>A table together with some statements from <a href="https://tabfact.github.io/">TabFact</a>. The content of the table can be used to support or contradict the statements.</td></tr></tbody></table>
<p>
Following the methods used by TAPAS, we encode the content of a statement and a table together, pass them through a Transformer model, and obtain a single number with the probability that the statement is entailed or refuted by the table.
</p>
<table><tbody><tr><td><a href="https://1.bp.blogspot.com/-zaz8_g9XvxI/YBRJAP21w4I/AAAAAAAAHE4/CvLDXDybaiQs9QNwEdcwW6KaLvyUeJUewCLcBGAsYHQ/s588/TAPAS.jpg"></a></td></tr><tr><td>The <a href="https://www.aclweb.org/anthology/2020.acl-main.398">TAPAS model</a> architecture uses a BERT model to encode the statement and the flattened table, read row by row. Special embeddings are used to encode the table structure. The vector output of the first token is used to predict the probability of entailment.</td></tr></tbody></table>
<p>
Because the only information in the training examples is a binary value (i.e., "correct" or "incorrect"), training a model to understand whether a statement is entailed or not is challenging and highlights the difficulty in achieving generalization in deep learning, especially when the provided training signal is scarce. Seeing isolated entailed or refuted examples, a model can easily pick-up on spurious patterns in the data to make a prediction, for example the presence of the word "tie" in "Greg Norman and Billy Mayfair tie in rank", instead of truly comparing their ranks, which is what is needed to successfully apply the model beyond the original training data. 
</p>
<h2>Pre-training Tasks</h2><p>
Pre-training tasks can be used to “warm-up” models by providing them with large amounts of readily available unlabeled data. However, pre-training typically includes primarily plain text and not tabular data. In fact, TAPAS was originally pre-trained using a simple masked language modelling objective that was not designed for tabular data applications. In order to improve the model performance on tabular data, we introduce two novel pretraining binary-classification tasks called <em>counterfactual</em> and <em>synthetic</em>, which can be applied as a second stage of pre-training (often called <em>intermediate pre-training</em>). 
</p>
<p>
In the counterfactual task, we source sentences from Wikipedia that mention an entity (person, place or thing) that also appears in a given table. Then, 50% of the time, we modify the statement by swapping the entity for another alternative. To make sure the statement is realistic, we choose a replacement among the entities in the same column in the table. The model is trained to recognize whether the statement was modified or not. This pre-training task includes millions of such examples, and although the reasoning about them is not complex, they typically will still sound natural.
</p>
<p>
For the synthetic task, we follow a method similar to <a href="https://www.aclweb.org/anthology/P15-1129/">semantic parsing</a> in which we generate statements using a simple set of grammar rules that require the model to understand basic mathematical operations, such as sums and averages (e.g., "the sum of earnings"), or to understand how to filter the elements in the table using some condition (e.g.,"the country is Australia"). Although these statements are artificial, they help improve the numerical and logical reasoning skills of the model.
</p>
<table><tbody><tr><td><a href="https://1.bp.blogspot.com/-kLdqZMbPSEY/YBRJIn_88tI/AAAAAAAAHE8/MBhRtGDgqKIATplbDWbad6sRniHgjmLhgCLcBGAsYHQ/s1214/image1.png"></a></td></tr><tr><td>Example instances for the two novel pre-training tasks. <b>Counterfactual</b> examples swap entities mentioned in a sentence that accompanies the input table for a plausible alternative. <b>Synthetic</b> statements use grammar rules to create new sentences that require combining the information of the table in complex ways.</td></tr></tbody></table>
<h2>Results</h2><p>
We evaluate the success of the counterfactual and synthetic pre-training objectives on the TabFact dataset by comparing to the baseline TAPAS model and to two prior models that have exhibited success in the textual entailment domain, <a href="https://www.aclweb.org/anthology/2020.acl-main.539/">LogicalFactChecker</a> (LFC) and <a href="https://www.aclweb.org/anthology/2020.emnlp-main.126/">Structure Aware Transformer</a> (SAT). The baseline TAPAS model exhibits improved performance relative to LFC and SAT, but the pre-trained model (TAPAS+CS) performs significantly better, achieving a new state of the art.
</p>
<p>
We also apply TAPAS+CS to question answering tasks on the <a href="https://www.microsoft.com/en-us/download/details.aspx?id=54253">SQA dataset</a>, which requires that the model find answers from the content of tables in a dialog setting. The inclusion of CS objectives improves the previous best performance by more than 4 points, demonstrating that this approach also generalizes performance beyond just textual entailment.
</p>
<table><tbody><tr><td><a href="https://1.bp.blogspot.com/-gRLgwMzDwyQ/YBRJcOPxUFI/AAAAAAAAHFI/cVl0tAGeEX4TLCLt-NvUS7Nfq_FBHy81QCLcBGAsYHQ/s1170/Accuracy.png"></a></td></tr><tr><td>Results on TabFact (<b>left</b>) and SQA (<b>right</b>). Using the synthetic and counterfactual datasets, we achieve new state-of-the-art results in both tasks by a large margin.</td></tr></tbody></table>
<h2>Data and Compute Efficiency</h2><p>
Another aspect of the counterfactual and synthetic pre-training tasks is that since the models are already tuned for binary classification, they can be applied without any fine-tuning to TabFact. We explore what happens to each of the models when trained only on a subset (or even none) of the data. Without looking at a single example, the TAPAS+CS model is competitive with a strong baseline Table-Bert, and when only 10% of the data are included, the results are comparable to the previous state-of-the-art.
</p>
<table><tbody><tr><td><a href="https://1.bp.blogspot.com/-ij2gSWPrt8U/YBRJkWHIfVI/AAAAAAAAHFM/NZWkx9y0f94Gg_FjBLH0-MOXFHYiJANGgCLcBGAsYHQ/s1046/image7.png"></a></td></tr><tr><td>Dev accuracy on TabFact relative to the fraction of the training data used.</td></tr></tbody></table>
<p>
A general concern when trying to use large models such as this to operate on tables, is that their high computational requirements makes it difficult for them to parse very large tables. To address this, we investigate whether one can heuristically select subsets of the input to pass through the model in order to optimize its computational efficiency. 
</p>
<p>
We conducted a systematic study of different approaches to filter the input and discovered that simple methods that select for word overlap between a full column and the subject statement give the best results. By dynamically selecting which tokens of the input to include, we can use fewer resources or work on larger inputs at the same cost. The challenge is doing so without losing important information and hurting accuracy. </p><p>For instance, the models discussed above all use sequences of 512 tokens, which is around the normal limit for a transformer model (although recent efficiency methods like the <a href="https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html">Reformer</a> or <a href="https://ai.googleblog.com/2020/10/rethinking-attention-with-performers.html">Performer</a> are proving effective in scaling the input size). The column selection methods we propose here can allow for faster training while still achieving high accuracy on TabFact. For 256 input tokens we get a very small drop in accuracy, but the model can now be pre-trained, fine-tuned and make predictions up to two times faster. With 128 tokens the model still outperforms the previous state-of-the-art model, with an even more significant speed-up — 4x faster across the board.</p>
<table><tbody><tr><td><a href="https://1.bp.blogspot.com/-LjZgvsrnh7E/YBRJpZUYYUI/AAAAAAAAHFQ/-b-_V4JCJq0S4S_k6vNcqVlRSvfPx6WwwCLcBGAsYHQ/s1046/image6.png"></a></td></tr><tr><td>Accuracy on TabFact using different sequence lengths, by shortening the input with our column selection method.</td></tr></tbody></table>
<p>
Using both the column selection method we proposed and the novel pre-training tasks, we can create table parsing models that need fewer data and less compute power to obtain better results. 
</p>
<p>
We have made available the new models and pre-training techniques at <a href="https://github.com/google-research/tapas">our GitHub repo</a>, where you can try it out yourself in <a href="http://tiny.cc/tapas-tabfact-colab">colab</a>. In order to make this approach more accessible, we also shared models of varying sizes all the way down to “<a href="https://arxiv.org/abs/1908.08962">tiny</a>”. It is our hope that these results will help spur development of table reasoning among the broader research community.
</p>
<h2>Acknowledgements</h2><p>
<em>This work was carried out by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller from our <a href="https://research.google/teams/language/">Language Team</a> in Zürich. We would like to thank Jordan Boyd-Graber, Yasemin Altun, Emily Pitler, Benjamin Boerschinger, Srini Narayanan, Slav Petrov, William Cohen and Jonathan Herzig for their useful comments and suggestions.</em>
</p>
</div>||||I||||Skip to main content
 Learning to Reason Over Tables from Less Data
 Learning to Reason Over Tables from Less Data
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Learning to Reason Over Tables from Less Data

 Friday, January 29, 2021

 Posted by Julian Eisenschlos, AI Resident, Google Research, Zürich

 The task of recognizing textual entailment, also known as natural language inference, consists of determining whether a piece of text (a premise), can be implied or contradicted (or neither) by another piece of text (the hypothesis). While this problem is often considered an important test for the reasoning skills of machine learning (ML) systems and has been studied in depth for plain text inputs, much less effort has been put into applying such models to structured data, such as websites, tables, databases, etc. Yet, recognizing textual entailment is especially relevant whenever the contents of a table need to be accurately summarized and presented to a user, and is essential for high fidelity question answering systems and virtual assistants.

 In "Understanding tables with intermediate pre-training", published in Findings of EMNLP 2020, we introduce the first pre-training tasks customized for table parsing, enabling models to learn better, faster and from less data. We build upon our earlier TAPAS model, which was an extension of the BERT bi-directional Transformer model with special embeddings to find answers in tables. Applying our new pre-training objectives to TAPAS yields a new state of the art on multiple datasets involving tables. On TabFact, for example, it reduces the gap between model and human performance by ~50%. We also systematically benchmark methods of selecting relevant input for higher efficiency, achieving 4x gains in speed and memory, while retaining 92% of the results. All the models for different tasks and sizes are released on GitHub repo, where you can try them out yourself in a colab Notebook.


 Textual Entailment

 The task of textual entailment is more challenging when applied to tabular data than plain text. Consider, for example, a table from Wikipedia with some sentences derived from its associated table content. Assessing if the content of the table entails or contradicts the sentence may require looking over multiple columns and rows, and possibly performing simple numeric computations, like averaging, summing, differencing, etc.

   
 A table together with some statements from TabFact. The content of the table can be used to support or contradict the statements.
 

 Following the methods used by TAPAS, we encode the content of a statement and a table together, pass them through a Transformer model, and obtain a single number with the probability that the statement is entailed or refuted by the table.

     
 The TAPAS model architecture uses a BERT model to encode the statement and the flattened table, read row by row. Special embeddings are used to encode the table structure. The vector output of the first token is used to predict the probability of entailment.
 

 Because the only information in the training examples is a binary value (i.e., "correct" or "incorrect"), training a model to understand whether a statement is entailed or not is challenging and highlights the difficulty in achieving generalization in deep learning, especially when the provided training signal is scarce. Seeing isolated entailed or refuted examples, a model can easily pick-up on spurious patterns in the data to make a prediction, for example the presence of the word "tie" in "Greg Norman and Billy Mayfair tie in rank", instead of truly comparing their ranks, which is what is needed to successfully apply the model beyond the original training data.


 Pre-training Tasks

 Pre-training tasks can be used to “warm-up” models by providing them with large amounts of readily available unlabeled data. However, pre-training typically includes primarily plain text and not tabular data. In fact, TAPAS was originally pre-trained using a simple masked language modelling objective that was not designed for tabular data applications. In order to improve the model performance on tabular data, we introduce two novel pretraining binary-classification tasks called counterfactual and synthetic, which can be applied as a second stage of pre-training (often called intermediate pre-training).

 In the counterfactual task, we source sentences from Wikipedia that mention an entity (person, place or thing) that also appears in a given table. Then, 50% of the time, we modify the statement by swapping the entity for another alternative. To make sure the statement is realistic, we choose a replacement among the entities in the same column in the table. The model is trained to recognize whether the statement was modified or not. This pre-training task includes millions of such examples, and although the reasoning about them is not complex, they typically will still sound natural.

 For the synthetic task, we follow a method similar to semantic parsing in which we generate statements using a simple set of grammar rules that require the model to understand basic mathematical operations, such as sums and averages (e.g., "the sum of earnings"), or to understand how to filter the elements in the table using some condition (e.g.,"the country is Australia"). Although these statements are artificial, they help improve the numerical and logical reasoning skills of the model.

      
 Example instances for the two novel pre-training tasks. Counterfactual examples swap entities mentioned in a sentence that accompanies the input table for a plausible alternative. Synthetic statements use grammar rules to create new sentences that require combining the information of the table in complex ways.
 


 Results

 We evaluate the success of the counterfactual and synthetic pre-training objectives on the TabFact dataset by comparing to the baseline TAPAS model and to two prior models that have exhibited success in the textual entailment domain, LogicalFactChecker (LFC) and Structure Aware Transformer (SAT). The baseline TAPAS model exhibits improved performance relative to LFC and SAT, but the pre-trained model (TAPAS+CS) performs significantly better, achieving a new state of the art.

 We also apply TAPAS+CS to question answering tasks on the SQA dataset, which requires that the model find answers from the content of tables in a dialog setting. The inclusion of CS objectives improves the previous best performance by more than 4 points, demonstrating that this approach also generalizes performance beyond just textual entailment.

    
 Results on TabFact (left) and SQA (right). Using the synthetic and counterfactual datasets, we achieve new state-of-the-art results in both tasks by a large margin.
 


 Data and Compute Efficiency

 Another aspect of the counterfactual and synthetic pre-training tasks is that since the models are already tuned for binary classification, they can be applied without any fine-tuning to TabFact. We explore what happens to each of the models when trained only on a subset (or even none) of the data. Without looking at a single example, the TAPAS+CS model is competitive with a strong baseline Table-Bert, and when only 10% of the data are included, the results are comparable to the previous state-of-the-art.

  
 Dev accuracy on TabFact relative to the fraction of the training data used.
 

 A general concern when trying to use large models such as this to operate on tables, is that their high computational requirements makes it difficult for them to parse very large tables. To address this, we investigate whether one can heuristically select subsets of the input to pass through the model in order to optimize its computational efficiency.

 We conducted a systematic study of different approaches to filter the input and discovered that simple methods that select for word overlap between a full column and the subject statement give the best results. By dynamically selecting which tokens of the input to include, we can use fewer resources or work on larger inputs at the same cost. The challenge is doing so without losing important information and hurting accuracy.

 For instance, the models discussed above all use sequences of 512 tokens, which is around the normal limit for a transformer model (although recent efficiency methods like the Reformer or Performer are proving effective in scaling the input size). The column selection methods we propose here can allow for faster training while still achieving high accuracy on TabFact. For 256 input tokens we get a very small drop in accuracy, but the model can now be pre-trained, fine-tuned and make predictions up to two times faster. With 128 tokens the model still outperforms the previous state-of-the-art model, with an even more significant speed-up — 4x faster across the board.

   
 Accuracy on TabFact using different sequence lengths, by shortening the input with our column selection method.
 

 Using both the column selection method we proposed and the novel pre-training tasks, we can create table parsing models that need fewer data and less compute power to obtain better results.

 We have made available the new models and pre-training techniques at our GitHub repo, where you can try it out yourself in colab. In order to make this approach more accessible, we also shared models of varying sizes all the way down to “tiny”. It is our hope that these results will help spur development of table reasoning among the broader research community.


 Acknowledgements

 This work was carried out by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller from our Language Team in Zürich. We would like to thank Jordan Boyd-Graber, Yasemin Altun, Emily Pitler, Benjamin Boerschinger, Srini Narayanan, Slav Petrov, William Cohen and Jonathan Herzig for their useful comments and suggestions.

 Labels: EMNLP Natural Language Processing open source
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Julian Eisenschlos, AI Resident, Google Research, Zürich 
The task of recognizing  textual entailment, also known as natural language inference, consists of determining whether a piece of text (a premise), can be implied or contradicted (or neither) by another piece of text (the hypothesis). While this problem is often considered an important test for the reasoning skills of machine learning (ML) systems and has been studied in depth for plain text inputs, much less effort has been put into applying such models to structured data, such as websites, tables, databases, etc. Yet, recognizing textual entailment is especially relevant whenever the contents of a table need to be accurately summarized and presented to a user, and is essential for high fidelity question answering systems and virtual assistants.
 
In "Understanding tables with intermediate pre-training", published in Findings of EMNLP 2020, we introduce the first pre-training tasks customized for table parsing, enabling models to learn better, faster and from less data. We build upon our earlier TAPAS model, which was an extension of the BERT bi-directional Transformer model with special embeddings to find answers in tables. Applying our new pre-training objectives to TAPAS yields a new state of the art on multiple datasets involving tables. On TabFact, for example, it reduces the gap between model and human performance by ~50%. We also systematically benchmark methods of selecting relevant input for higher efficiency, achieving 4x gains in speed and memory, while retaining 92% of the results. All the models for different tasks and sizes are released on GitHub repo, where you can try them out yourself in a colab Notebook.
Textual Entailment 
The task of textual entailment is more challenging when applied to tabular data than plain text. Consider, for example, a table from Wikipedia with some sentences derived from its associated table content. Assessing if the content of the table entails or contradicts the sentence may require looking over multiple columns and rows, and possibly performing simple numeric computations, like averaging, summing, differencing, etc.
         A table together with some statements from TabFact. The content of the table can be used to support or contradict the statements.   
Following the methods used by TAPAS, we encode the content of a statement and a table together, pass them through a Transformer model, and obtain a single number with the probability that the statement is entailed or refuted by the table.
         The TAPAS model architecture uses a BERT model to encode the statement and the flattened table, read row by row. Special embeddings are used to encode the table structure. The vector output of the first token is used to predict the probability of entailment.   
Because the only information in the training examples is a binary value (i.e., "correct" or "incorrect"), training a model to understand whether a statement is entailed or not is challenging and highlights the difficulty in achieving generalization in deep learning, especially when the provided training signal is scarce. Seeing isolated entailed or refuted examples, a model can easily pick-up on spurious patterns in the data to make a prediction, for example the presence of the word "tie" in "Greg Norman and Billy Mayfair tie in rank", instead of truly comparing their ranks, which is what is needed to successfully apply the model beyond the original training data. 
Pre-training Tasks 
Pre-training tasks can be used to “warm-up” models by providing them with large amounts of readily available unlabeled data. However, pre-training typically includes primarily plain text and not tabular data. In fact, TAPAS was originally pre-trained using a simple masked language modelling objective that was not designed for tabular data applications. In order to improve the model performance on tabular data, we introduce two novel pretraining binary-classification tasks called counterfactual and synthetic, which can be applied as a second stage of pre-training (often called intermediate pre-training). 
In the counterfactual task, we source sentences from Wikipedia that mention an entity (person, place or thing) that also appears in a given table. Then, 50% of the time, we modify the statement by swapping the entity for another alternative. To make sure the statement is realistic, we choose a replacement among the entities in the same column in the table. The model is trained to recognize whether the statement was modified or not. This pre-training task includes millions of such examples, and although the reasoning about them is not complex, they typically will still sound natural.
For the synthetic task, we follow a method similar to semantic parsing in which we generate statements using a simple set of grammar rules that require the model to understand basic mathematical operations, such as sums and averages (e.g., "the sum of earnings"), or to understand how to filter the elements in the table using some condition (e.g.,"the country is Australia"). Although these statements are artificial, they help improve the numerical and logical reasoning skills of the model.
         Example instances for the two novel pre-training tasks. Counterfactual examples swap entities mentioned in a sentence that accompanies the input table for a plausible alternative. Synthetic statements use grammar rules to create new sentences that require combining the information of the table in complex ways.   
Results 
We evaluate the success of the counterfactual and synthetic pre-training objectives on the TabFact dataset by comparing to the baseline TAPAS model and to two prior models that have exhibited success in the textual entailment domain, LogicalFactChecker (LFC) and Structure Aware Transformer (SAT). The baseline TAPAS model exhibits improved performance relative to LFC and SAT, but the pre-trained model (TAPAS+CS) performs significantly better, achieving a new state of the art.
We also apply TAPAS+CS to question answering tasks on the SQA dataset, which requires that the model find answers from the content of tables in a dialog setting. The inclusion of CS objectives improves the previous best performance by more than 4 points, demonstrating that this approach also generalizes performance beyond just textual entailment.
         Results on TabFact (left) and SQA (right). Using the synthetic and counterfactual datasets, we achieve new state-of-the-art results in both tasks by a large margin.   
Data and Compute Efficiency 
Another aspect of the counterfactual and synthetic pre-training tasks is that since the models are already tuned for binary classification, they can be applied without any fine-tuning to TabFact. We explore what happens to each of the models when trained only on a subset (or even none) of the data. Without looking at a single example, the TAPAS+CS model is competitive with a strong baseline Table-Bert, and when only 10% of the data are included, the results are comparable to the previous state-of-the-art.
         Dev accuracy on TabFact relative to the fraction of the training data used.   
A general concern when trying to use large models such as this to operate on tables, is that their high computational requirements makes it difficult for them to parse very large tables. To address this, we investigate whether one can heuristically select subsets of the input to pass through the model in order to optimize its computational efficiency. 
We conducted a systematic study of different approaches to filter the input and discovered that simple methods that select for word overlap between a full column and the subject statement give the best results. By dynamically selecting which tokens of the input to include, we can use fewer resources or work on larger inputs at the same cost. The challenge is doing so without losing important information and hurting accuracy.  For instance, the models discussed above all use sequences of 512 tokens, which is around the normal limit for a transformer model (although recent efficiency methods like the Reformer or Performer are proving effective in scaling the input size). The column selection methods we propose here can allow for faster training while still achieving high accuracy on TabFact. For 256 input tokens we get a very small drop in accuracy, but the model can now be pre-trained, fine-tuned and make predictions up to two times faster. With 128 tokens the model still outperforms the previous state-of-the-art model, with an even more significant speed-up — 4x faster across the board.
         Accuracy on TabFact using different sequence lengths, by shortening the input with our column selection method.   
Using both the column selection method we proposed and the novel pre-training tasks, we can create table parsing models that need fewer data and less compute power to obtain better results. 
We have made available the new models and pre-training techniques at our GitHub repo, where you can try it out yourself in colab. In order to make this approach more accessible, we also shared models of varying sizes all the way down to “tiny”. It is our hope that these results will help spur development of table reasoning among the broader research community.
Acknowledgements 
This work was carried out by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller from our Language Team in Zürich. We would like to thank Jordan Boyd-Graber, Yasemin Altun, Emily Pitler, Benjamin Boerschinger, Srini Narayanan, Slav Petrov, William Cohen and Jonathan Herzig for their useful comments and suggestions.
Learning to Reason Over Tables from Less Data
https://ai.googleblog.com/2021/01/learning-to-reason-over-tables-from.html
https://ai.googleblog.com/2022/08/towards-helpful-robots-grounding.html
Towards Helpful Robots: Grounding Language in Robotic Affordances
2022-08-16
Brian Ichter; Karol Hausman; Research Scientists; Google Research; Brain Team
<div><div>
<p>Posted by Brian Ichter and Karol Hausman, Research Scientists, Google Research, Brain Team</p><p>
Over the last several years, we have seen significant progress in applying machine learning to robotics. However, robotic systems today are capable of executing <a href="https://ai.googleblog.com/2021/04/multi-task-robotic-reinforcement.html">only very short, hard-coded commands</a>, such as “Pick up an apple,” because they tend to perform best with clear tasks and rewards. They struggle with learning to perform long-horizon tasks and reasoning about abstract goals, such as a user prompt like “I just worked out, can you get me a healthy snack?”
</p><p>
Meanwhile, recent progress in training language models (LMs) has led to systems that can perform a wide range of language understanding and generation tasks with <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">impressive results</a>. However, these language models are inherently not grounded in the physical world due to the nature of their training process: a language model generally does not interact with its environment nor observe the outcome of its responses. This can result in it generating instructions that may be illogical, impractical or unsafe for a robot to complete in a physical context. For example, when prompted with “I spilled my drink, can you help?” the language model <a href="https://arxiv.org/abs/2005.14165">GPT-3</a> responds with “You could try using a vacuum cleaner,” a suggestion that may be unsafe or impossible for the robot to execute. When asking the <a href="https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.html">FLAN</a> language model the same question, it apologizes for the spill with "I'm sorry, I didn't mean to spill it,” which is not a very useful response. Therefore, we asked ourselves, is there an effective way to combine advanced language models with robot learning algorithms to leverage the benefits of both?
</p>
<p>
In “<a href="https://arxiv.org/pdf/2204.01691.pdf">Do As I Can, Not As I Say: Grounding Language in Robotic Affordances</a>”, we present a novel approach, developed in partnership with <a href="https://everydayrobots.com/">Everyday Robots</a>, that leverages advanced language model knowledge to enable a physical agent, such as a robot, to follow high-level textual instructions for physically-grounded tasks, while grounding the language model in tasks that are feasible within a specific real-world context. We evaluate our method, which we call PaLM-SayCan, by placing robots in a real kitchen setting and giving them tasks expressed in natural language. We observe highly interpretable results for temporally-extended complex and abstract tasks, like “I just worked out, please bring me a snack and a drink to recover.” Specifically, we demonstrate that grounding the language model in the real world nearly halves errors over non-grounded baselines. We are also excited to release a <a href="https://github.com/google-research/google-research/tree/master/saycan">robot simulation setup</a> where the research community can test this approach. 
</p>
<table><tbody><tr><td> </td></tr>
<tr></tr><tr><td>With PaLM-SayCan, the robot acts as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task.</td></tr></tbody></table>
<h2>A Dialog Between User and Robot, Facilitated by the Language Model</h2>
<p>
Our approach uses the knowledge contained in language models (Say) to determine and score actions that are useful towards high-level instructions. It also uses an <a href="https://en.wikipedia.org/wiki/Affordance#Affordance_in_robotics">affordance</a> function (Can) that enables real-world-grounding and determines which actions are possible to execute in a given environment. Using the the PaLM language model, we call this PaLM-SayCan.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKjorngIdDFxJRDCG_doQd9MDZ1Ljca12Z_lZz1DUodErBFbFYgJi95fuSf1JR5A-sXpEa_w7iDbigjiBHD0J1cyR6L0t77_N9L5Hb2NU4ReizmUY79ca0CfqTx4uNleoNKv32PcHMV-SefK3UrK4EolSq6R_o0C0Z_aUvJnz_T0ioSGyDCqv-pGr61Q/s1600/image3.gif"></a></td></tr><tr><td>Our approach selects skills based on what the language model scores as useful to the high level instruction and what the affordance model scores as possible.</td></tr></tbody></table>
<p>
Our system can be seen as a dialog between the user and robot, facilitated by the language model. The user starts by giving an instruction that the language model turns into a sequence of steps for the robot to execute. This sequence is filtered using the robot’s skillset to determine the most feasible plan given its current state and environment. The model determines the probability of a specific skill successfully making progress toward completing the instruction by multiplying two probabilities: (1) <em>task-grounding</em> (i.e., a skill language description) and (2) <em>world-grounding</em> (i.e., skill feasibility in the current state).
</p>
<p>
There are additional benefits of our approach in terms of its safety and interpretability. First, by allowing the LM to score different options rather than generate the most likely output, we effectively constrain the LM to only output one of the pre-selected responses. In addition, the user can easily understand the decision making process by looking at the separate language and affordance scores, rather than a single output.
</p>
<br />
<br />
<p>PaLM-SayCan is also interpretable: at each step, we can see the top options it considers based on their language score (blue), affordance score (red), and combined score (green). </p>
<h2>Training Policies and Value Functions</h2>
<p>
Each skill in the agent’s skillset is defined as a policy with a short language description (e.g., “pick up the can”), represented as embeddings, and an affordance function that indicates the probability of completing the skill from the robot’s current state. To learn the affordance functions, we use sparse reward functions set to 1.0 for a successful execution, and 0.0 otherwise.
</p>
<p>
We use <a href="https://ai.googleblog.com/2022/02/can-robots-follow-instructions-for-new.html">image-based behavioral cloning</a> (BC) to train the language-conditioned policies and <a href="https://en.wikipedia.org/wiki/Temporal_difference_learning">temporal-difference-based</a> (TD) reinforcement learning (RL) to train the value functions. To train the policies, we collected data from 68,000 demos performed by 10 robots over 11 months and added 12,000 successful episodes, filtered from a set of autonomous episodes of learned policies. We then learned the language conditioned value functions using <a href="https://arxiv.org/abs/2104.08212">MT-Opt</a> in the <a href="https://everydayrobots.com/">Everyday Robots</a> simulator. The simulator complements our real robot fleet with a simulated version of the skills and environment, which is transformed using <a href="https://arxiv.org/abs/2011.03148">RetinaGAN</a> to <a href="https://everydayrobots.com/thinking/shortening-the-sim-to-real-gap">reduce the simulation-to-real gap</a>. We bootstrapped simulation policies’ performance by using demonstrations to provide initial successes, and then continuously improved RL performance with online data collection in simulation.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNGkij7rct7Ii2UuaeA0us81LjvU6hul2yyUzHiAABUT24GfmYU0RtoAtwhIr-0vgQKG_zQejwPCggvaU4ulk-sXdbxVeeDn-PeeZI9ccaT3hdeDzjIM0S3DoldUzKEsX7J7BGkoM8G14d_vPUTqFRtdUJ0yR-Mq8kWke-xVvL7VWMgr_NvfDaD1vUpA/s1312/image2.png"></a></td></tr><tr><td>Given a high-level instruction, our approach combines the probabilities from the language model with the probabilities from the value function (VF) to select the next skill to perform. This process is repeated until the high-level instruction is successfully completed.</td></tr></tbody></table>
<h2>Performance on Temporally-Extended, Complex, and Abstract Instructions</h2>
<p>
To test our approach, we use robots from Everyday Robots paired with <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">PaLM</a>. We place the robots in a kitchen environment containing common objects and evaluate them on 101 instructions to test their performance across various robot and environment states, instruction language complexity and time horizon. Specifically, these instructions were designed to showcase the ambiguity and complexity of language rather than to provide simple, imperative queries, enabling queries such as “I just worked out, how would you bring me a snack and a drink to recover?” instead of “Can you bring me water and an apple?”
</p>
<p>
We use two metrics to evaluate the system’s performance: (1) the <em>plan success rate</em>, indicating whether the robot chose the right skills for the instruction, and (2) the <em>execution success rate</em>, indicating whether it performed the instruction successfully. We compare two language models, <a href="https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf">PaLM</a> and <a href="https://arxiv.org/abs/2109.01652">FLAN</a> (a smaller language model fine-tuned on instruction answering) with and without the affordance grounding as well as the underlying policies running directly with natural language (Behavioral Cloning in the table below). 
 
The results show that the system using PaLM with affordance grounding (PaLM-SayCan) chooses the correct sequence of skills 84% of the time and executes them successfully 74% of the time, reducing errors by 50% compared to FLAN and compared to PaLM without robotic grounding. This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics. This result indicates a potential future where robotics is able to ride the wave of progress that we have been observing in language models, bringing these subfields of research closer together.
</p>
<br />
<table><tbody><tr>
 <td><b>Algorithm</b>
 </td>
 <td>   </td>
 <td><b>Plan</b>
 </td>
 <td>   </td>
 <td><b>Execute</b>
 </td>
 </tr>
 <tr>
 <td><i>PaLM-SayCan</i>
 </td>
 <td>   </td>
 <td>84%
 </td>
 <td>   </td>
 <td>74%
 </td>
 </tr>
 <tr>
 <td><i>PaLM</i>
 </td>
 <td>   </td>
 <td>67%
 </td>
 <td>   </td>
 <td>-
 </td>
 </tr>
 <tr>
 <td><i>FLAN-SayCan</i>
 </td>
 <td>   </td>
 <td>70%
 </td>
 <td>   </td>
 <td>61%
 </td>
 </tr>
 <tr>
 <td><i>FLAN</i>
 </td>
 <td>   </td>
 <td>38%
 </td>
 <td>   </td>
 <td>-
 </td>
 </tr>
 <tr>
 <td><i>Behavioral Cloning</i>
 </td>
 <td>   </td>
 <td>0%
 </td>
 <td>   </td>
 <td>0%
 </td>
 </tr>
</tbody></table>
<br />
<p>PaLM-SayCan halves errors compared to PaLM without affordances and compared to FLAN over 101 tasks.</p>
<br />
<br />
<table><tbody>
 <tr></tr><tr><td>SayCan demonstrated successful planning for 84% of the 101 test instructions when combined with PaLM.</td></tr></tbody></table>
<p>If you're interested in learning more about this project from the researchers themselves, please check out the video below:</p>
<p></p>
<br />
<h2>Conclusion and Future Work</h2>
<p>
We’re excited about the progress that we’ve seen with PaLM-SayCan, an interpretable and general approach to leveraging knowledge from language models that enables a robot to follow high-level textual instructions to perform physically-grounded tasks. Our experiments on a number of real-world robotic tasks demonstrate the ability to plan and complete long-horizon, abstract, natural language instructions at a high success rate. We believe that PaLM-SayCan’s interpretability allows for safe real-world user interaction with robots. As we explore future directions for this work, we hope to better understand how information gained via the robot’s real-world experience could be leveraged to improve the language model and to what extent natural language is the right ontology for programming robots. We have open-sourced a <a href="https://github.com/google-research/google-research/tree/master/saycan">robot simulation setup</a>, which we hope will provide researchers with a valuable resource for future research that combines robotic learning with advanced language models. The research community can visit the project’s <a href="http://say-can.github.io/">GitHub page</a> and <a href="//g.co/research/palm-saycan">website</a> to learn more.
</p>
<h2>Acknowledgements</h2>
<p>
<em>We’d like to thank our coauthors Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Kelly Fu, Keerthana Gopalakrishnan, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. We’d also like to thank Yunfei Bai, Matt Bennice, Maarten Bosma, Justin Boyd, Bill Byrne, Kendra Byrne, Noah Constant, Pete Florence, Laura Graesser, Rico Jonschkowski, Daniel Kappler, Hugo Larochelle, Benjamin Lee, Adrian Li, Maysam Moussalem, Suraj Nair, Krista Reymann, Jeff Seto, Dhruv Shah, Ian Storz, Razvan Surdulescu, and Vincent Zhao for their help and support in various aspects of the project. And we’d like to thank Tom Small for creating many of the animations in this post.</em>
</p>
</div></div>||||I||||Skip to main content
 Towards Helpful Robots: Grounding Language in Robotic Affordances
 Towards Helpful Robots: Grounding Language in Robotic Affordances
 Philosophy
 Research Areas
 Publications
 People
 Resources
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Towards Helpful Robots: Grounding Language in Robotic Affordances

 Tuesday, August 16, 2022

 Posted by Brian Ichter and Karol Hausman, Research Scientists, Google Research, Brain Team

 Over the last several years, we have seen significant progress in applying machine learning to robotics. However, robotic systems today are capable of executing only very short, hard-coded commands, such as “Pick up an apple,” because they tend to perform best with clear tasks and rewards. They struggle with learning to perform long-horizon tasks and reasoning about abstract goals, such as a user prompt like “I just worked out, can you get me a healthy snack?”

 Meanwhile, recent progress in training language models (LMs) has led to systems that can perform a wide range of language understanding and generation tasks with impressive results. However, these language models are inherently not grounded in the physical world due to the nature of their training process: a language model generally does not interact with its environment nor observe the outcome of its responses. This can result in it generating instructions that may be illogical, impractical or unsafe for a robot to complete in a physical context. For example, when prompted with “I spilled my drink, can you help?” the language model GPT-3 responds with “You could try using a vacuum cleaner,” a suggestion that may be unsafe or impossible for the robot to execute. When asking the FLAN language model the same question, it apologizes for the spill with "I'm sorry, I didn't mean to spill it,” which is not a very useful response. Therefore, we asked ourselves, is there an effective way to combine advanced language models with robot learning algorithms to leverage the benefits of both?

 In “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, we present a novel approach, developed in partnership with Everyday Robots, that leverages advanced language model knowledge to enable a physical agent, such as a robot, to follow high-level textual instructions for physically-grounded tasks, while grounding the language model in tasks that are feasible within a specific real-world context. We evaluate our method, which we call PaLM-SayCan, by placing robots in a real kitchen setting and giving them tasks expressed in natural language. We observe highly interpretable results for temporally-extended complex and abstract tasks, like “I just worked out, please bring me a snack and a drink to recover.” Specifically, we demonstrate that grounding the language model in the real world nearly halves errors over non-grounded baselines. We are also excited to release a robot simulation setup where the research community can test this approach.

   
 
 With PaLM-SayCan, the robot acts as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task.
 


 A Dialog Between User and Robot, Facilitated by the Language Model

 Our approach uses the knowledge contained in language models (Say) to determine and score actions that are useful towards high-level instructions. It also uses an affordance function (Can) that enables real-world-grounding and determines which actions are possible to execute in a given environment. Using the the PaLM language model, we call this PaLM-SayCan.

   
 Our approach selects skills based on what the language model scores as useful to the high level instruction and what the affordance model scores as possible.
 

 Our system can be seen as a dialog between the user and robot, facilitated by the language model. The user starts by giving an instruction that the language model turns into a sequence of steps for the robot to execute. This sequence is filtered using the robot’s skillset to determine the most feasible plan given its current state and environment. The model determines the probability of a specific skill successfully making progress toward completing the instruction by multiplying two probabilities: (1) task-grounding (i.e., a skill language description) and (2) world-grounding (i.e., skill feasibility in the current state).

 There are additional benefits of our approach in terms of its safety and interpretability. First, by allowing the LM to score different options rather than generate the most likely output, we effectively constrain the LM to only output one of the pre-selected responses. In addition, the user can easily understand the decision making process by looking at the separate language and affordance scores, rather than a single output.



 PaLM-SayCan is also interpretable: at each step, we can see the top options it considers based on their language score (blue), affordance score (red), and combined score (green).
 


 Training Policies and Value Functions

 Each skill in the agent’s skillset is defined as a policy with a short language description (e.g., “pick up the can”), represented as embeddings, and an affordance function that indicates the probability of completing the skill from the robot’s current state. To learn the affordance functions, we use sparse reward functions set to 1.0 for a successful execution, and 0.0 otherwise.

 We use image-based behavioral cloning (BC) to train the language-conditioned policies and temporal-difference-based (TD) reinforcement learning (RL) to train the value functions. To train the policies, we collected data from 68,000 demos performed by 10 robots over 11 months and added 12,000 successful episodes, filtered from a set of autonomous episodes of learned policies. We then learned the language conditioned value functions using MT-Opt in the Everyday Robots simulator. The simulator complements our real robot fleet with a simulated version of the skills and environment, which is transformed using RetinaGAN to reduce the simulation-to-real gap. We bootstrapped simulation policies’ performance by using demonstrations to provide initial successes, and then continuously improved RL performance with online data collection in simulation.

     
 Given a high-level instruction, our approach combines the probabilities from the language model with the probabilities from the value function (VF) to select the next skill to perform. This process is repeated until the high-level instruction is successfully completed.
 


 Performance on Temporally-Extended, Complex, and Abstract Instructions

 To test our approach, we use robots from Everyday Robots paired with PaLM. We place the robots in a kitchen environment containing common objects and evaluate them on 101 instructions to test their performance across various robot and environment states, instruction language complexity and time horizon. Specifically, these instructions were designed to showcase the ambiguity and complexity of language rather than to provide simple, imperative queries, enabling queries such as “I just worked out, how would you bring me a snack and a drink to recover?” instead of “Can you bring me water and an apple?”

 We use two metrics to evaluate the system’s performance: (1) the plan success rate, indicating whether the robot chose the right skills for the instruction, and (2) the execution success rate, indicating whether it performed the instruction successfully. We compare two language models, PaLM and FLAN (a smaller language model fine-tuned on instruction answering) with and without the affordance grounding as well as the underlying policies running directly with natural language (Behavioral Cloning in the table below). The results show that the system using PaLM with affordance grounding (PaLM-SayCan) chooses the correct sequence of skills 84% of the time and executes them successfully 74% of the time, reducing errors by 50% compared to FLAN and compared to PaLM without robotic grounding. This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics. This result indicates a potential future where robotics is able to ride the wave of progress that we have been observing in language models, bringing these subfields of research closer together.


 Algorithm Plan Execute
 PaLM-SayCan 84% 74% 
 PaLM 67% - 
 FLAN-SayCan 70% 61% 
 FLAN 38% - 
 Behavioral Cloning 0% 0% 
 

 PaLM-SayCan halves errors compared to PaLM without affordances and compared to FLAN over 101 tasks.
 


 
 SayCan demonstrated successful planning for 84% of the 101 test instructions when combined with PaLM.
 

 If you're interested in learning more about this project from the researchers themselves, please check out the video below:


 Conclusion and Future Work

 We’re excited about the progress that we’ve seen with PaLM-SayCan, an interpretable and general approach to leveraging knowledge from language models that enables a robot to follow high-level textual instructions to perform physically-grounded tasks. Our experiments on a number of real-world robotic tasks demonstrate the ability to plan and complete long-horizon, abstract, natural language instructions at a high success rate. We believe that PaLM-SayCan’s interpretability allows for safe real-world user interaction with robots. As we explore future directions for this work, we hope to better understand how information gained via the robot’s real-world experience could be leveraged to improve the language model and to what extent natural language is the right ontology for programming robots. We have open-sourced a robot simulation setup, which we hope will provide researchers with a valuable resource for future research that combines robotic learning with advanced language models. The research community can visit the project’s GitHub page and website to learn more.


 Acknowledgements

 We’d like to thank our coauthors Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Kelly Fu, Keerthana Gopalakrishnan, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. We’d also like to thank Yunfei Bai, Matt Bennice, Maarten Bosma, Justin Boyd, Bill Byrne, Kendra Byrne, Noah Constant, Pete Florence, Laura Graesser, Rico Jonschkowski, Daniel Kappler, Hugo Larochelle, Benjamin Lee, Adrian Li, Maysam Moussalem, Suraj Nair, Krista Reymann, Jeff Seto, Dhruv Shah, Ian Storz, Razvan Surdulescu, and Vincent Zhao for their help and support in various aspects of the project. And we’d like to thank Tom Small for creating many of the animations in this post.

 Labels: Machine Learning NLP Robotics
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Brian Ichter and Karol Hausman, Research Scientists, Google Research, Brain Team 
Over the last several years, we have seen significant progress in applying machine learning to robotics. However, robotic systems today are capable of executing only very short, hard-coded commands, such as “Pick up an apple,” because they tend to perform best with clear tasks and rewards. They struggle with learning to perform long-horizon tasks and reasoning about abstract goals, such as a user prompt like “I just worked out, can you get me a healthy snack?”
 
Meanwhile, recent progress in training language models (LMs) has led to systems that can perform a wide range of language understanding and generation tasks with impressive results. However, these language models are inherently not grounded in the physical world due to the nature of their training process: a language model generally does not interact with its environment nor observe the outcome of its responses. This can result in it generating instructions that may be illogical, impractical or unsafe for a robot to complete in a physical context. For example, when prompted with “I spilled my drink, can you help?” the language model GPT-3 responds with “You could try using a vacuum cleaner,” a suggestion that may be unsafe or impossible for the robot to execute. When asking the FLAN language model the same question, it apologizes for the spill with "I'm sorry, I didn't mean to spill it,” which is not a very useful response. Therefore, we asked ourselves, is there an effective way to combine advanced language models with robot learning algorithms to leverage the benefits of both?
In “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, we present a novel approach, developed in partnership with Everyday Robots, that leverages advanced language model knowledge to enable a physical agent, such as a robot, to follow high-level textual instructions for physically-grounded tasks, while grounding the language model in tasks that are feasible within a specific real-world context. We evaluate our method, which we call PaLM-SayCan, by placing robots in a real kitchen setting and giving them tasks expressed in natural language. We observe highly interpretable results for temporally-extended complex and abstract tasks, like “I just worked out, please bring me a snack and a drink to recover.” Specifically, we demonstrate that grounding the language model in the real world nearly halves errors over non-grounded baselines. We are also excited to release a robot simulation setup where the research community can test this approach. 
     
   With PaLM-SayCan, the robot acts as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task.   
A Dialog Between User and Robot, Facilitated by the Language Model
Our approach uses the knowledge contained in language models (Say) to determine and score actions that are useful towards high-level instructions. It also uses an affordance function (Can) that enables real-world-grounding and determines which actions are possible to execute in a given environment. Using the the PaLM language model, we call this PaLM-SayCan.
         Our approach selects skills based on what the language model scores as useful to the high level instruction and what the affordance model scores as possible.   
Our system can be seen as a dialog between the user and robot, facilitated by the language model. The user starts by giving an instruction that the language model turns into a sequence of steps for the robot to execute. This sequence is filtered using the robot’s skillset to determine the most feasible plan given its current state and environment. The model determines the probability of a specific skill successfully making progress toward completing the instruction by multiplying two probabilities: (1) task-grounding (i.e., a skill language description) and (2) world-grounding (i.e., skill feasibility in the current state).
There are additional benefits of our approach in terms of its safety and interpretability. First, by allowing the LM to score different options rather than generate the most likely output, we effectively constrain the LM to only output one of the pre-selected responses. In addition, the user can easily understand the decision making process by looking at the separate language and affordance scores, rather than a single output.
PaLM-SayCan is also interpretable: at each step, we can see the top options it considers based on their language score (blue), affordance score (red), and combined score (green). 
Training Policies and Value Functions
Each skill in the agent’s skillset is defined as a policy with a short language description (e.g., “pick up the can”), represented as embeddings, and an affordance function that indicates the probability of completing the skill from the robot’s current state. To learn the affordance functions, we use sparse reward functions set to 1.0 for a successful execution, and 0.0 otherwise.
We use image-based behavioral cloning (BC) to train the language-conditioned policies and temporal-difference-based (TD) reinforcement learning (RL) to train the value functions. To train the policies, we collected data from 68,000 demos performed by 10 robots over 11 months and added 12,000 successful episodes, filtered from a set of autonomous episodes of learned policies. We then learned the language conditioned value functions using MT-Opt in the Everyday Robots simulator. The simulator complements our real robot fleet with a simulated version of the skills and environment, which is transformed using RetinaGAN to reduce the simulation-to-real gap. We bootstrapped simulation policies’ performance by using demonstrations to provide initial successes, and then continuously improved RL performance with online data collection in simulation.
         Given a high-level instruction, our approach combines the probabilities from the language model with the probabilities from the value function (VF) to select the next skill to perform. This process is repeated until the high-level instruction is successfully completed.   
Performance on Temporally-Extended, Complex, and Abstract Instructions
To test our approach, we use robots from Everyday Robots paired with PaLM. We place the robots in a kitchen environment containing common objects and evaluate them on 101 instructions to test their performance across various robot and environment states, instruction language complexity and time horizon. Specifically, these instructions were designed to showcase the ambiguity and complexity of language rather than to provide simple, imperative queries, enabling queries such as “I just worked out, how would you bring me a snack and a drink to recover?” instead of “Can you bring me water and an apple?”
We use two metrics to evaluate the system’s performance: (1) the plan success rate, indicating whether the robot chose the right skills for the instruction, and (2) the execution success rate, indicating whether it performed the instruction successfully. We compare two language models, PaLM and FLAN (a smaller language model fine-tuned on instruction answering) with and without the affordance grounding as well as the underlying policies running directly with natural language (Behavioral Cloning in the table below). 
 
The results show that the system using PaLM with affordance grounding (PaLM-SayCan) chooses the correct sequence of skills 84% of the time and executes them successfully 74% of the time, reducing errors by 50% compared to FLAN and compared to PaLM without robotic grounding. This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics. This result indicates a potential future where robotics is able to ride the wave of progress that we have been observing in language models, bringing these subfields of research closer together.
  
  Algorithm
 
    
  Plan
 
    
  Execute
 
 
 
  PaLM-SayCan
 
    
 84%
 
    
 74%
 
 
 
  PaLM
 
    
 67%
 
    
 -
 
 
 
  FLAN-SayCan
 
    
 70%
 
    
 61%
 
 
 
  FLAN
 
    
 38%
 
    
 -
 
 
 
  Behavioral Cloning
 
    
 0%
 
    
 0%
 
 
 
PaLM-SayCan halves errors compared to PaLM without affordances and compared to FLAN over 101 tasks.
 
    SayCan demonstrated successful planning for 84% of the 101 test instructions when combined with PaLM.   
If you're interested in learning more about this project from the researchers themselves, please check out the video below:
 
Conclusion and Future Work
We’re excited about the progress that we’ve seen with PaLM-SayCan, an interpretable and general approach to leveraging knowledge from language models that enables a robot to follow high-level textual instructions to perform physically-grounded tasks. Our experiments on a number of real-world robotic tasks demonstrate the ability to plan and complete long-horizon, abstract, natural language instructions at a high success rate. We believe that PaLM-SayCan’s interpretability allows for safe real-world user interaction with robots. As we explore future directions for this work, we hope to better understand how information gained via the robot’s real-world experience could be leveraged to improve the language model and to what extent natural language is the right ontology for programming robots. We have open-sourced a robot simulation setup, which we hope will provide researchers with a valuable resource for future research that combines robotic learning with advanced language models. The research community can visit the project’s GitHub page and website to learn more.
Acknowledgements
We’d like to thank our coauthors Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Kelly Fu, Keerthana Gopalakrishnan, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. We’d also like to thank Yunfei Bai, Matt Bennice, Maarten Bosma, Justin Boyd, Bill Byrne, Kendra Byrne, Noah Constant, Pete Florence, Laura Graesser, Rico Jonschkowski, Daniel Kappler, Hugo Larochelle, Benjamin Lee, Adrian Li, Maysam Moussalem, Suraj Nair, Krista Reymann, Jeff Seto, Dhruv Shah, Ian Storz, Razvan Surdulescu, and Vincent Zhao for their help and support in various aspects of the project. And we’d like to thank Tom Small for creating many of the animations in this post.
Towards Helpful Robots: Grounding Language in Robotic Affordances
https://ai.googleblog.com/2022/08/towards-helpful-robots-grounding.html
https://ai.googleblog.com/2022/02/federated-learning-with-formal.html
Federated Learning with Formal Differential Privacy Guarantees
2022-02-28
Brendan McMahan; Abhradeep Thakurta; Research Scientists; Google Research
<div><div>
<p>Posted by Brendan McMahan and Abhradeep Thakurta, Research Scientists, Google Research</p><p>
In 2017, Google <a href="https://ai.googleblog.com/2017/04/federated-learning-collaborative.html">introduced federated learning</a> (FL), an approach that enables mobile devices to collaboratively train machine learning (ML) models while keeping the raw training data on each user's device, decoupling the ability to do ML from the need to store the data in the cloud. Since its introduction, Google has continued to <a href="https://www.youtube.com/watch?v=0gSc_9QqXJg&amp;list=PLSIUOFhnxEiD9uihG5t9ABdPhSVqQ3HWA">actively engage in FL research</a> and deployed FL to power many features in <a href="https://play.google.com/store/apps/details?id=com.google.android.inputmethod.latin&amp;hl=en_US&amp;gl=US">Gboard</a>, including next word prediction, emoji suggestion and out-of-vocabulary word discovery. Federated learning is improving the <a href="https://support.google.com/assistant/answer/10176224">“Hey Google”</a> detection models in Assistant, <a href="https://support.google.com/messages/answer/9327902?hl=en">suggesting replies</a> in Google Messages, <a href="https://ai.googleblog.com/2021/11/predicting-text-selections-with.html">predicting text selections</a>, and more.
</p><p>
While FL allows ML without raw data collection, <a href="https://en.wikipedia.org/wiki/Differential_privacy">differential privacy</a> (DP) provides a quantifiable measure of data anonymization, and when applied to ML can address concerns about models memorizing sensitive user data. This too has been a top research priority, and has yielded one of the first production uses of DP for analytics with <a href="https://security.googleblog.com/2014/10/learning-statistics-with-privacy-aided.html">RAPPOR</a> in 2014, <a href="https://developers.googleblog.com/2019/09/enabling-developers-and-organizations.html">our open-source DP library</a>, <a href="https://developers.googleblog.com/2022/01/expanding-access-to-differential-privacy.html">Pipeline DP</a>, and <a href="https://www.tensorflow.org/responsible_ai/privacy/guide">TensorFlow Privacy</a>. 
</p>
<p>
Through a multi-year, multi-team effort spanning fundamental research and product integration, today we are excited to announce that we have deployed a production ML model using federated learning with a rigorous differential privacy guarantee. For this proof-of-concept deployment, we utilized <a href="https://arxiv.org/abs/2103.00039">the DP-FTRL algorithm</a> to train a recurrent neural network to power next-word-prediction for Spanish-language Gboard users. To our knowledge, this is the first production neural network trained directly on user data announced with a formal DP guarantee (technically ρ=0.81 <a href="https://arxiv.org/pdf/1605.02065.pdf">zero-Concentrated-Differential-Privacy</a>, zCDP, discussed in detail below). Further, the federated approach offers complimentary data minimization advantages, and the DP guarantee protects all of the data on each device, not just individual training examples. 
</p>
<h2>Data Minimization and Anonymization in Federated Learning</h2><p>
Along with fundamentals like transparency and consent, the <a href="https://queue.acm.org/detail.cfm?id=3501293">privacy principles of data minimization and anonymization</a> are important in ML applications that involve sensitive data.
</p>
<p>
Federated learning systems structurally incorporate the principle of <em>data minimization.</em> FL only transmits minimal updates for a specific model training task (focused collection), limits access to data at all stages, processes individuals’ data as early as possible (early aggregation), and discards both collected and processed data as soon as possible (minimal retention).
</p>
<p>
Another principle that is important for models trained on user data is <em>anonymization</em>, meaning that the final model should not <a href="https://ai.googleblog.com/2020/12/privacy-considerations-in-large.html">memorize</a> information unique to a particular individual's data, e.g., phone numbers, addresses, credit card numbers. However, FL on its own does not directly tackle this problem.
</p>
<p>
The mathematical concept of DP allows one to formally quantify this principle of anonymization. Differentially private training algorithms add random noise during training to produce a probability distribution over output models, and ensure that this distribution doesn't change too much given a small change to the training data; ρ-zCDP quantifies how much the distribution could possibly change. We call this <em>example-level</em> DP when adding or removing a single training example changes the output distribution on models in a provably minimal way. 
</p>
<p>
Showing that deep learning with example-level differential privacy was even possible in the simpler setting of centralized training was <a href="https://research.google.com/pubs/pub45428.html">a major step forward in 2016</a>. Achieved by the DP-SGD algorithm, the key was amplifying the privacy guarantee by leveraging the randomness in sampling training examples ("amplification-via-sampling").
</p>
<p>
However, when users can contribute multiple examples to the training dataset, example-level DP is not necessarily strong enough to ensure the users’ data isn't memorized. Instead, we have designed algorithms for <em>user-level</em> DP, which requires that the output distribution of models doesn't change even if we add/remove <em>all</em> of the training examples from any one user (or all the examples from any one device in our application). Fortunately, because FL summarizes all of a user's training data as a single model update, federated algorithms are well-suited to offering user-level DP guarantees.
</p>
<p>
Both limiting the contributions from one user and adding noise can come at the expense of model accuracy, however, so maintaining model quality while also providing strong DP guarantees is a key research focus. 
</p>
<h2>The Challenging Path to Federated Learning with Differential Privacy</h2><p>
In 2018, we introduced <a href="https://arxiv.org/abs/1710.06963">the DP-FedAvg algorithm</a>, which extended the DP-SGD approach to the federated setting with user-level DP guarantees, and in 2020 we <a href="https://arxiv.org/abs/2009.10031">deployed this algorithm</a> to mobile devices for the first time. This approach ensures the training mechanism is not too sensitive to any one user's data, and <a href="https://arxiv.org/abs/2006.07490">empirical privacy auditing techniques</a> rule out some forms of memorization. 
</p>
<p>
However, the amplification-via-samping argument is essential to providing a strong DP guarantee for DP-FedAvg, but in <a href="https://arxiv.org/abs/1902.01046">a real-world cross-device FL system</a> ensuring devices are subsampled precisely and uniformly at random from a large population would be complex and hard to verify. One challenge is that devices choose when to connect (or "check in") based on many external factors (e.g., requiring the device is idle, on unmetered WiFi, and charging), and the number of available devices can vary substantially.
</p>
<p>
Achieving a formal privacy guarantee requires a protocol that does <em>all</em> of the following:
</p>
<ul>
<li>Makes progress on training even as the set of devices available varies significantly with time. 
</li><li>Maintains privacy guarantees even in the face of unexpected or arbitrary changes in device availability. 
</li><li>For efficiency, allows client devices to locally decide whether they will check in to the server in order to participate in training, independent of other devices.
</li>
</ul>
<p>
<a href="https://arxiv.org/abs/2007.06605">Initial work on privacy amplification via random check-ins</a> highlighted these challenges and introduced a feasible protocol, but it would have required complex changes to our production infrastructure to deploy. Further, as with the amplification-via-sampling analysis of DP-SGD, the privacy amplification possible with random check-ins depends on a large number of devices being available. For example, if only 1000 devices are available for training, and participation of at least 1000 devices is needed in each training step, that requires either 1) including all devices currently available and paying a large privacy cost since there is no randomness in the selection, or 2) pausing the protocol and not making progress until more devices are available.
</p>
<h2>Achieving Provable Differential Privacy for Federated Learning with DP-FTRL</h2><p>
To address this challenge, the DP-FTRL algorithm is built on two key observations: 1) the convergence of <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">gradient-descent-style</a> algorithms depends primarily not on the accuracy of individual gradients, but the accuracy of cumulative sums of gradients; and 2) we can provide accurate estimates of cumulative sums with a strong DP guarantee by utilizing negatively correlated noise, added by the aggregating server: essentially, adding noise to one gradient and subtracting that same noise from a later gradient. DP-FTRL accomplishes this efficiently using the Tree Aggregation algorithm [<a href="https://eprint.iacr.org/2010/076.pdf">1</a>, <a href="http://www.wisdom.weizmann.ac.il/~naor/PAPERS/continual_observation.pdf">2</a>].
</p>
<p>
The graphic below illustrates how estimating cumulative sums rather than individual gradients can help. We look at how the noise introduced by DP-FTRL and DP-SGD influence model training, compared to the true gradients (without added noise; in black) which step one unit to the right on each iteration. The individual DP-FTRL gradient estimates (blue), based on cumulative sums, have larger mean-squared-error than the individually-noised DP-SGD estimates (orange), but because the DP-FTRL noise is negatively correlated, some of it cancels out from step to step, and the overall learning trajectory stays closer to the true gradient descent steps.
</p>
<p><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiYdSEjaePn72t3iIp-OEAHoBA_VrZsTCDt7KGMnTf3cZC_D8AAvUXbidCwzJe54CccZFuNLBLZ6uEYGUy67F5fCQk9vVAwpXQUDjl_pJnL1kXy5fw2vkrx7S93UEuoLlhXLcid9KanVJ-8pdUnco9wciGVp-9iJFIjRkzn5y-Am2caCSL5BSQEjOjdDw=s1999"></a></p>
<p>
To provide a strong privacy guarantee, we limit the number of times a user contributes an update. Fortunately, sampling-without-replacement is relatively easy to implement in production FL infrastructure: each device can remember locally which models it has contributed to in the past, and choose to not connect to the server for any later rounds for those models.
</p>
<h2>Production Training Details and Formal DP Statements</h2><p>
For the production DP-FTRL deployment introduced above, each eligible device maintains a local training cache consisting of user keyboard input, and when participating computes an update to the model which makes it more likely to suggest the next word the user actually typed, based on what has been typed so far. We ran DP-FTRL on this data to train a recurrent neural network with ~1.3M parameters. Training ran for 2000 rounds over six days, with 6500 devices participating per round. To allow for the DP guarantee, devices participated in training at most once every 24 hours. Model quality improved over the <a href="https://arxiv.org/abs/2009.10031">previous DP-FedAvg trained model</a>, which offered empirically-tested privacy advantages over non-DP models, but lacked a meaningful formal DP guarantee.
</p>
<p>
The training mechanism we used is available in open-source in <a href="https://www.tensorflow.org/federated/">TensorFlow Federated</a> and <a href="https://github.com/tensorflow/privacy">TensorFlow Privacy</a>, and with the parameters used in our production deployment it provides a meaningfully strong privacy guarantee. <a href="https://colab.sandbox.google.com/github/google-research/federated/blob/master/dp_ftrl/blogpost_supplemental_privacy_accounting.ipynb">Our analysis</a> gives ρ=0.81 zCDP at the user level (treating all the data on each device as a different user), where smaller numbers correspond to better privacy in a mathematically precise way. As a comparison, this is stronger than the <a href="https://www2.census.gov/programs-surveys/decennial/2020/program-management/data-product-planning/2010-demonstration-data-products/ppmf20210608/2021-06-08-privacy-loss_budgetallocation.pdf">ρ=2.63 zCDP guarantee chosen by the 2020 US Census</a>. 
</p>
<h2>Next Steps</h2><p>
While we have reached the milestone of deploying a production FL model using a mechanism that provides a meaningfully small zCDP, our research journey <a href="https://arxiv.org/abs/2202.08312">continues</a>. We are still far from being able to say this approach is possible (let alone practical) for most ML models or product applications, and other approaches to private ML exist. For example, membership inference tests and other empirical privacy auditing techniques can provide complimentary safeguards against leakage of users’ data. Most importantly, we see training models with user-level DP with even a very large zCDP as a substantial step forward, because it requires training with a DP mechanism that bounds the sensitivity of the model to any one user's data. Further, it smooths the road to later training models with improved privacy guarantees as better algorithms or more data become available. We are excited to continue the journey toward maximizing the value that ML can deliver while minimizing potential privacy costs to those who contribute training data.
</p>
<h2>Acknowledgements</h2><p>
<em>The authors would like to thank Alex Ingerman and Om Thakkar for significant impact on the blog post itself, as well as the teams at Google that helped develop these ideas and bring them to practice:<br />
</em></p><ul><em>
 <li><b>Core research team</b>: Galen Andrew, Borja Balle, Peter Kairouz, Daniel Ramage, Shuang Song, Thomas Steinke, Andreas Terzis, Om Thakkar, Zheng Xu
</li><li><b>FL infrastructure team</b>: Katharine Daly, Stefan Dierauf, Hubert Eichner, Timon Van Overveldt, Chunxiang Zheng
</li><li><b>Gboard team</b>: Angana Ghosh, Xu Liu, Yuanbo Zhang
</li><li><b>Speech team</b>: Françoise Beaufays, Mingqing Chen, Rajiv Mathews, Vidush Mukund, Igor Pisarev, Swaroop Ramaswamy, Dan Zivkovic</li></em></ul>
</div>||||I||||Skip to main content
 Federated Learning with Formal Differential Privacy Guarantees
 Federated Learning with Formal Differential Privacy Guarantees
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Federated Learning with Formal Differential Privacy Guarantees

 Monday, February 28, 2022

 Posted by Brendan McMahan and Abhradeep Thakurta, Research Scientists, Google Research

 In 2017, Google introduced federated learning (FL), an approach that enables mobile devices to collaboratively train machine learning (ML) models while keeping the raw training data on each user's device, decoupling the ability to do ML from the need to store the data in the cloud. Since its introduction, Google has continued to actively engage in FL research and deployed FL to power many features in Gboard, including next word prediction, emoji suggestion and out-of-vocabulary word discovery. Federated learning is improving the “Hey Google” detection models in Assistant, suggesting replies in Google Messages, predicting text selections, and more.

 While FL allows ML without raw data collection, differential privacy (DP) provides a quantifiable measure of data anonymization, and when applied to ML can address concerns about models memorizing sensitive user data. This too has been a top research priority, and has yielded one of the first production uses of DP for analytics with RAPPOR in 2014, our open-source DP library, Pipeline DP, and TensorFlow Privacy.

 Through a multi-year, multi-team effort spanning fundamental research and product integration, today we are excited to announce that we have deployed a production ML model using federated learning with a rigorous differential privacy guarantee. For this proof-of-concept deployment, we utilized the DP-FTRL algorithm to train a recurrent neural network to power next-word-prediction for Spanish-language Gboard users. To our knowledge, this is the first production neural network trained directly on user data announced with a formal DP guarantee (technically ρ=0.81 zero-Concentrated-Differential-Privacy, zCDP, discussed in detail below). Further, the federated approach offers complimentary data minimization advantages, and the DP guarantee protects all of the data on each device, not just individual training examples.


 Data Minimization and Anonymization in Federated Learning

 Along with fundamentals like transparency and consent, the privacy principles of data minimization and anonymization are important in ML applications that involve sensitive data.

 Federated learning systems structurally incorporate the principle of data minimization. FL only transmits minimal updates for a specific model training task (focused collection), limits access to data at all stages, processes individuals’ data as early as possible (early aggregation), and discards both collected and processed data as soon as possible (minimal retention).

 Another principle that is important for models trained on user data is anonymization, meaning that the final model should not memorize information unique to a particular individual's data, e.g., phone numbers, addresses, credit card numbers. However, FL on its own does not directly tackle this problem.

 The mathematical concept of DP allows one to formally quantify this principle of anonymization. Differentially private training algorithms add random noise during training to produce a probability distribution over output models, and ensure that this distribution doesn't change too much given a small change to the training data; ρ-zCDP quantifies how much the distribution could possibly change. We call this example-level DP when adding or removing a single training example changes the output distribution on models in a provably minimal way.

 Showing that deep learning with example-level differential privacy was even possible in the simpler setting of centralized training was a major step forward in 2016. Achieved by the DP-SGD algorithm, the key was amplifying the privacy guarantee by leveraging the randomness in sampling training examples ("amplification-via-sampling").

 However, when users can contribute multiple examples to the training dataset, example-level DP is not necessarily strong enough to ensure the users’ data isn't memorized. Instead, we have designed algorithms for user-level DP, which requires that the output distribution of models doesn't change even if we add/remove all of the training examples from any one user (or all the examples from any one device in our application). Fortunately, because FL summarizes all of a user's training data as a single model update, federated algorithms are well-suited to offering user-level DP guarantees.

 Both limiting the contributions from one user and adding noise can come at the expense of model accuracy, however, so maintaining model quality while also providing strong DP guarantees is a key research focus.


 The Challenging Path to Federated Learning with Differential Privacy

 In 2018, we introduced the DP-FedAvg algorithm, which extended the DP-SGD approach to the federated setting with user-level DP guarantees, and in 2020 we deployed this algorithm to mobile devices for the first time. This approach ensures the training mechanism is not too sensitive to any one user's data, and empirical privacy auditing techniques rule out some forms of memorization.

 However, the amplification-via-samping argument is essential to providing a strong DP guarantee for DP-FedAvg, but in a real-world cross-device FL system ensuring devices are subsampled precisely and uniformly at random from a large population would be complex and hard to verify. One challenge is that devices choose when to connect (or "check in") based on many external factors (e.g., requiring the device is idle, on unmetered WiFi, and charging), and the number of available devices can vary substantially.

 Achieving a formal privacy guarantee requires a protocol that does all of the following:

 * Makes progress on training even as the set of devices available varies significantly with time.
 * Maintains privacy guarantees even in the face of unexpected or arbitrary changes in device availability.
 * For efficiency, allows client devices to locally decide whether they will check in to the server in order to participate in training, independent of other devices.

 Initial work on privacy amplification via random check-ins highlighted these challenges and introduced a feasible protocol, but it would have required complex changes to our production infrastructure to deploy. Further, as with the amplification-via-sampling analysis of DP-SGD, the privacy amplification possible with random check-ins depends on a large number of devices being available. For example, if only 1000 devices are available for training, and participation of at least 1000 devices is needed in each training step, that requires either 1) including all devices currently available and paying a large privacy cost since there is no randomness in the selection, or 2) pausing the protocol and not making progress until more devices are available.


 Achieving Provable Differential Privacy for Federated Learning with DP-FTRL

 To address this challenge, the DP-FTRL algorithm is built on two key observations: 1) the convergence of gradient-descent-style algorithms depends primarily not on the accuracy of individual gradients, but the accuracy of cumulative sums of gradients; and 2) we can provide accurate estimates of cumulative sums with a strong DP guarantee by utilizing negatively correlated noise, added by the aggregating server: essentially, adding noise to one gradient and subtracting that same noise from a later gradient. DP-FTRL accomplishes this efficiently using the Tree Aggregation algorithm [1, 2].

 The graphic below illustrates how estimating cumulative sums rather than individual gradients can help. We look at how the noise introduced by DP-FTRL and DP-SGD influence model training, compared to the true gradients (without added noise; in black) which step one unit to the right on each iteration. The individual DP-FTRL gradient estimates (blue), based on cumulative sums, have larger mean-squared-error than the individually-noised DP-SGD estimates (orange), but because the DP-FTRL noise is negatively correlated, some of it cancels out from step to step, and the overall learning trajectory stays closer to the true gradient descent steps.

 
 

 To provide a strong privacy guarantee, we limit the number of times a user contributes an update. Fortunately, sampling-without-replacement is relatively easy to implement in production FL infrastructure: each device can remember locally which models it has contributed to in the past, and choose to not connect to the server for any later rounds for those models.


 Production Training Details and Formal DP Statements

 For the production DP-FTRL deployment introduced above, each eligible device maintains a local training cache consisting of user keyboard input, and when participating computes an update to the model which makes it more likely to suggest the next word the user actually typed, based on what has been typed so far. We ran DP-FTRL on this data to train a recurrent neural network with ~1.3M parameters. Training ran for 2000 rounds over six days, with 6500 devices participating per round. To allow for the DP guarantee, devices participated in training at most once every 24 hours. Model quality improved over the previous DP-FedAvg trained model, which offered empirically-tested privacy advantages over non-DP models, but lacked a meaningful formal DP guarantee.

 The training mechanism we used is available in open-source in TensorFlow Federated and TensorFlow Privacy, and with the parameters used in our production deployment it provides a meaningfully strong privacy guarantee. Our analysis gives ρ=0.81 zCDP at the user level (treating all the data on each device as a different user), where smaller numbers correspond to better privacy in a mathematically precise way. As a comparison, this is stronger than the ρ=2.63 zCDP guarantee chosen by the 2020 US Census.


 Next Steps

 While we have reached the milestone of deploying a production FL model using a mechanism that provides a meaningfully small zCDP, our research journey continues. We are still far from being able to say this approach is possible (let alone practical) for most ML models or product applications, and other approaches to private ML exist. For example, membership inference tests and other empirical privacy auditing techniques can provide complimentary safeguards against leakage of users’ data. Most importantly, we see training models with user-level DP with even a very large zCDP as a substantial step forward, because it requires training with a DP mechanism that bounds the sensitivity of the model to any one user's data. Further, it smooths the road to later training models with improved privacy guarantees as better algorithms or more data become available. We are excited to continue the journey toward maximizing the value that ML can deliver while minimizing potential privacy costs to those who contribute training data.


 Acknowledgements

 The authors would like to thank Alex Ingerman and Om Thakkar for significant impact on the blog post itself, as well as the teams at Google that helped develop these ideas and bring them to practice:

 * Core research team: Galen Andrew, Borja Balle, Peter Kairouz, Daniel Ramage, Shuang Song, Thomas Steinke, Andreas Terzis, Om Thakkar, Zheng Xu
 * FL infrastructure team: Katharine Daly, Stefan Dierauf, Hubert Eichner, Timon Van Overveldt, Chunxiang Zheng
 * Gboard team: Angana Ghosh, Xu Liu, Yuanbo Zhang
 * Speech team: Françoise Beaufays, Mingqing Chen, Rajiv Mathews, Vidush Mukund, Igor Pisarev, Swaroop Ramaswamy, Dan Zivkovic

 Labels: Deep Learning Natural Language Understanding On-device Learning Security and Privacy
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Brendan McMahan and Abhradeep Thakurta, Research Scientists, Google Research 
In 2017, Google introduced federated learning (FL), an approach that enables mobile devices to collaboratively train machine learning (ML) models while keeping the raw training data on each user's device, decoupling the ability to do ML from the need to store the data in the cloud. Since its introduction, Google has continued to actively engage in FL research and deployed FL to power many features in Gboard, including next word prediction, emoji suggestion and out-of-vocabulary word discovery. Federated learning is improving the “Hey Google” detection models in Assistant, suggesting replies in Google Messages, predicting text selections, and more.
 
While FL allows ML without raw data collection, differential privacy (DP) provides a quantifiable measure of data anonymization, and when applied to ML can address concerns about models memorizing sensitive user data. This too has been a top research priority, and has yielded one of the first production uses of DP for analytics with RAPPOR in 2014, our open-source DP library, Pipeline DP, and TensorFlow Privacy. 
Through a multi-year, multi-team effort spanning fundamental research and product integration, today we are excited to announce that we have deployed a production ML model using federated learning with a rigorous differential privacy guarantee. For this proof-of-concept deployment, we utilized the DP-FTRL algorithm to train a recurrent neural network to power next-word-prediction for Spanish-language Gboard users. To our knowledge, this is the first production neural network trained directly on user data announced with a formal DP guarantee (technically ρ=0.81 zero-Concentrated-Differential-Privacy, zCDP, discussed in detail below). Further, the federated approach offers complimentary data minimization advantages, and the DP guarantee protects all of the data on each device, not just individual training examples. 
Data Minimization and Anonymization in Federated Learning 
Along with fundamentals like transparency and consent, the privacy principles of data minimization and anonymization are important in ML applications that involve sensitive data.
Federated learning systems structurally incorporate the principle of data minimization. FL only transmits minimal updates for a specific model training task (focused collection), limits access to data at all stages, processes individuals’ data as early as possible (early aggregation), and discards both collected and processed data as soon as possible (minimal retention).
Another principle that is important for models trained on user data is anonymization, meaning that the final model should not memorize information unique to a particular individual's data, e.g., phone numbers, addresses, credit card numbers. However, FL on its own does not directly tackle this problem.
The mathematical concept of DP allows one to formally quantify this principle of anonymization. Differentially private training algorithms add random noise during training to produce a probability distribution over output models, and ensure that this distribution doesn't change too much given a small change to the training data; ρ-zCDP quantifies how much the distribution could possibly change. We call this example-level DP when adding or removing a single training example changes the output distribution on models in a provably minimal way. 
Showing that deep learning with example-level differential privacy was even possible in the simpler setting of centralized training was a major step forward in 2016. Achieved by the DP-SGD algorithm, the key was amplifying the privacy guarantee by leveraging the randomness in sampling training examples ("amplification-via-sampling").
However, when users can contribute multiple examples to the training dataset, example-level DP is not necessarily strong enough to ensure the users’ data isn't memorized. Instead, we have designed algorithms for user-level DP, which requires that the output distribution of models doesn't change even if we add/remove all of the training examples from any one user (or all the examples from any one device in our application). Fortunately, because FL summarizes all of a user's training data as a single model update, federated algorithms are well-suited to offering user-level DP guarantees.
Both limiting the contributions from one user and adding noise can come at the expense of model accuracy, however, so maintaining model quality while also providing strong DP guarantees is a key research focus. 
The Challenging Path to Federated Learning with Differential Privacy 
In 2018, we introduced the DP-FedAvg algorithm, which extended the DP-SGD approach to the federated setting with user-level DP guarantees, and in 2020 we deployed this algorithm to mobile devices for the first time. This approach ensures the training mechanism is not too sensitive to any one user's data, and empirical privacy auditing techniques rule out some forms of memorization. 
However, the amplification-via-samping argument is essential to providing a strong DP guarantee for DP-FedAvg, but in a real-world cross-device FL system ensuring devices are subsampled precisely and uniformly at random from a large population would be complex and hard to verify. One challenge is that devices choose when to connect (or "check in") based on many external factors (e.g., requiring the device is idle, on unmetered WiFi, and charging), and the number of available devices can vary substantially.
Achieving a formal privacy guarantee requires a protocol that does all of the following:
Makes progress on training even as the set of devices available varies significantly with time. 
 Maintains privacy guarantees even in the face of unexpected or arbitrary changes in device availability. 
 For efficiency, allows client devices to locally decide whether they will check in to the server in order to participate in training, independent of other devices.
Initial work on privacy amplification via random check-ins highlighted these challenges and introduced a feasible protocol, but it would have required complex changes to our production infrastructure to deploy. Further, as with the amplification-via-sampling analysis of DP-SGD, the privacy amplification possible with random check-ins depends on a large number of devices being available. For example, if only 1000 devices are available for training, and participation of at least 1000 devices is needed in each training step, that requires either 1) including all devices currently available and paying a large privacy cost since there is no randomness in the selection, or 2) pausing the protocol and not making progress until more devices are available.
Achieving Provable Differential Privacy for Federated Learning with DP-FTRL 
To address this challenge, the DP-FTRL algorithm is built on two key observations: 1) the convergence of gradient-descent-style algorithms depends primarily not on the accuracy of individual gradients, but the accuracy of cumulative sums of gradients; and 2) we can provide accurate estimates of cumulative sums with a strong DP guarantee by utilizing negatively correlated noise, added by the aggregating server: essentially, adding noise to one gradient and subtracting that same noise from a later gradient. DP-FTRL accomplishes this efficiently using the Tree Aggregation algorithm [1, 2].
The graphic below illustrates how estimating cumulative sums rather than individual gradients can help. We look at how the noise introduced by DP-FTRL and DP-SGD influence model training, compared to the true gradients (without added noise; in black) which step one unit to the right on each iteration. The individual DP-FTRL gradient estimates (blue), based on cumulative sums, have larger mean-squared-error than the individually-noised DP-SGD estimates (orange), but because the DP-FTRL noise is negatively correlated, some of it cancels out from step to step, and the overall learning trajectory stays closer to the true gradient descent steps.
   
To provide a strong privacy guarantee, we limit the number of times a user contributes an update. Fortunately, sampling-without-replacement is relatively easy to implement in production FL infrastructure: each device can remember locally which models it has contributed to in the past, and choose to not connect to the server for any later rounds for those models.
Production Training Details and Formal DP Statements 
For the production DP-FTRL deployment introduced above, each eligible device maintains a local training cache consisting of user keyboard input, and when participating computes an update to the model which makes it more likely to suggest the next word the user actually typed, based on what has been typed so far. We ran DP-FTRL on this data to train a recurrent neural network with ~1.3M parameters. Training ran for 2000 rounds over six days, with 6500 devices participating per round. To allow for the DP guarantee, devices participated in training at most once every 24 hours. Model quality improved over the previous DP-FedAvg trained model, which offered empirically-tested privacy advantages over non-DP models, but lacked a meaningful formal DP guarantee.
The training mechanism we used is available in open-source in TensorFlow Federated and TensorFlow Privacy, and with the parameters used in our production deployment it provides a meaningfully strong privacy guarantee. Our analysis gives ρ=0.81 zCDP at the user level (treating all the data on each device as a different user), where smaller numbers correspond to better privacy in a mathematically precise way. As a comparison, this is stronger than the ρ=2.63 zCDP guarantee chosen by the 2020 US Census. 
Next Steps 
While we have reached the milestone of deploying a production FL model using a mechanism that provides a meaningfully small zCDP, our research journey continues. We are still far from being able to say this approach is possible (let alone practical) for most ML models or product applications, and other approaches to private ML exist. For example, membership inference tests and other empirical privacy auditing techniques can provide complimentary safeguards against leakage of users’ data. Most importantly, we see training models with user-level DP with even a very large zCDP as a substantial step forward, because it requires training with a DP mechanism that bounds the sensitivity of the model to any one user's data. Further, it smooths the road to later training models with improved privacy guarantees as better algorithms or more data become available. We are excited to continue the journey toward maximizing the value that ML can deliver while minimizing potential privacy costs to those who contribute training data.
Acknowledgements 
The authors would like to thank Alex Ingerman and Om Thakkar for significant impact on the blog post itself, as well as the teams at Google that helped develop these ideas and bring them to practice:
   
  Core research team: Galen Andrew, Borja Balle, Peter Kairouz, Daniel Ramage, Shuang Song, Thomas Steinke, Andreas Terzis, Om Thakkar, Zheng Xu
  FL infrastructure team: Katharine Daly, Stefan Dierauf, Hubert Eichner, Timon Van Overveldt, Chunxiang Zheng
  Gboard team: Angana Ghosh, Xu Liu, Yuanbo Zhang
  Speech team: Françoise Beaufays, Mingqing Chen, Rajiv Mathews, Vidush Mukund, Igor Pisarev, Swaroop Ramaswamy, Dan Zivkovic
Federated Learning with Formal Differential Privacy Guarantees
https://ai.googleblog.com/2022/02/federated-learning-with-formal.html
https://ai.googleblog.com/2022/02/4d-net-learning-multi-modal-alignment.html
4D-Net: Learning Multi-Modal Alignment for 3D and Image Inputs in Time
2022-02-23
AJ Piergiovanni; Anelia Angelova; Research Scientists; Google Research
<div><div>
<p>Posted by AJ Piergiovanni and Anelia Angelova, Research Scientists, Google Research</p><p>
While not immediately obvious, all of us experience the world in four dimensions (4D). For example, when walking or driving down the street we observe a stream of visual inputs, snapshots of the 3D world, which, when taken together in time, creates a 4D visual input. Today’s <a href="https://blog.waymo.com/2020/03/introducing-5th-generation-waymo-driver.html">autonomous vehicles</a> and <a href="https://x.company/projects/everyday-robots/">robots</a> are able to capture much of this information through various onboard sensing mechanisms, such as <a href="https://en.wikipedia.org/wiki/Lidar">LiDAR</a> and cameras.
</p><p>
LiDAR is a ubiquitous sensor that uses light pulses to reliably measure the 3D coordinates of objects in a scene, however, it is also sparse and has a limited range — the farther one is from a sensor, the fewer points will be returned. This means that far-away objects might only get a handful of points, or none at all, and might not be seen by LiDAR alone. At the same time, images from the onboard camera, which is a dense input, are incredibly useful for semantic understanding, such as detecting and segmenting objects. With high resolution, cameras can be very effective at detecting objects far away, but are less accurate in measuring the distance. 
</p>
<p>
Autonomous vehicles collect data from both LiDAR and onboard camera sensors. Each sensor measurement is recorded at regular time intervals, providing an accurate representation of the 4D world. However, very few research algorithms use both of these in combination, especially when taken “in time”, i.e., as a temporally ordered sequence of data, mostly due to two major challenges. When using both sensing modalities simultaneously, 1) it is difficult to maintain computational efficiency, and 2) pairing the information from one sensor to another adds further complexity since there is not always a direct correspondence between LiDAR points and onboard camera RGB image inputs.
</p>
<p>
In “<a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Piergiovanni_4D-Net_for_Learned_Multi-Modal_Alignment_ICCV_2021_paper.pdf">4D-Net for Learned Multi-Modal Alignment</a>”, published at <a href="https://iccv2021.thecvf.com/home">ICCV 2021</a>, we present a neural network that can process 4D data, which we call <i> 4D-Net</i>. This is the first attempt to effectively combine both types of sensors, 3D LiDAR <a href="https://en.wikipedia.org/wiki/Point_cloud">point clouds</a> and onboard camera RGB images, when both are in time. We also introduce a <i>dynamic connection learning</i> method, which incorporates 4D information from a scene by performing connection learning across both feature representations. Finally, we demonstrate that 4D-Net is better able to use motion cues and dense image information to detect distant objects while maintaining computational efficiency. 
</p>
<h2>4D-Net</h2>
<p>
In our scenario, we use 4D inputs (3D point clouds and onboard camera image data in time) to solve a very popular visual understanding task, the <a href="https://ai.googleblog.com/2021/02/3d-scene-understanding-with-tensorflow.html">3D box detection of objects</a>. We study the question of how one can combine the two sensing modalities, which come from different domains and have features that do not necessarily match — i.e., sparse LiDAR inputs span the 3D space and dense camera images only produce 2D projections of a scene. The exact correspondence between their respective features is unknown, so we seek to learn the connections between these two sensor inputs and their feature representations. We consider neural network representations where each of the feature layers can be combined with other potential layers from other sensor inputs, as shown below.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgC_X-XlaMV0j1AIAI4rxsLgDY9H74_XxxJ30uGHzUorliBilo1GJoTSJkj2o8USF1sQK_o-ESvI37Q6blDaOl8U78I_NgWOQGYchTQJ1xS4GMBvsYvvXcfbCy9dMNoOZulrMMP4OODNVy2MOhyF_pHKYcR_dvalcceVCEYBKLeqy7hZKBgHTqlxu8jfw=s1600"></a></td></tr><tr><td>4D-Net effectively combines 3D LiDAR point clouds in time with RGB images, also streamed in time as video, learning the connections between different sensors and their feature representations. </td></tr></tbody></table>
<h2>Dynamic Connection Learning Across Sensing Modalities</h2>
<p>
We use a light-weight <a href="https://en.wikipedia.org/wiki/Neural_architecture_search">neural architecture search</a> to learn the connections between both types of sensor inputs and their feature representations, to obtain the most accurate 3D box detection. In the autonomous driving domain it is especially important to reliably detect objects at highly variable distances, with modern LiDAR sensors reaching several hundreds of meters in range. This implies that more distant objects will appear smaller in the images and the most valuable features for detecting them will be in earlier layers of the network, which better capture fine-scale features, as opposed to close-by objects represented by later layers. Based on this observation, we modify the connections to be dynamic and select among features from all layers using <a href="https://arxiv.org/abs/1706.03762">self-attention mechanisms</a>. We apply a learnable linear layer, which is able to apply attention-weighting to all other layer weights and learn the best combination for the task at hand.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjDAQSULYTBgpVu_ji2ip_KNJyUtUro-48RNRAGyArLEzdfQ6xpBhJ_Wj5w5nI9DPUUzWk7fOsurB7h1fY1vuf5YWpPnpZq0POyoQbv7D4WT-uyNDZEGUsUOM830-wImsnksnLPnvz5CUAlCYKhxbqw6wQUlwC-d6gJFUU6Vnjzkw8VrTS1I4YUmONzgw=s1914"></a></td></tr><tr><td>Connection learning approach schematic, where connections between features from the 3D point cloud inputs are combined with the features from the RGB camera video inputs. Each connection learns the weighting for the corresponding inputs. </td></tr></tbody></table>
<h2>Results </h2>
<p>
We evaluate our results against state-of-the-art approaches on the <a href="https://waymo.com/open">Waymo Open Dataset</a> benchmark, for which previous models have only leveraged 3D point clouds in time or a combination of a single point cloud and camera image data. 4D-Net uses both sensor inputs efficiently, processing 32 point clouds in time and 16 RGB frames within 164 milliseconds, and performs well compared to other methods. In comparison, the next best approach is less efficient and accurate because its neural net computation takes 300 milliseconds, and uses fewer sensor inputs than 4D-Net.
</p>
<p><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg9nFlwdxY-6RvW8mF4YKvvXZQU1bP8zX7_GIIvXcrKoaQ-zVzhDbVJx7cqEzZmwNXy7UA1n16sm_YgSU5ZLJ_oaVhgPY_D343HgcdA2XNZjkhj_xcB4x3dG8LKhT87HPv8vrIC-79VcmRWGfXaBaL0jgbSriKvnD58RhWmz2-29bf76NeXZtlxwRiQsQ=s1999"></a></p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiNAJXYM3CXrgoKvP2mqz20eYhv2tTTBXFfX1zBtZCzMSOhclUE0s_qH7Z9y9OrI2I2WOMz4qeOxCo6VM8PZo41ikjK9ZP0mxgHCbNMsXrexOK7CuVLoabP7vWtVBLfD-Cww_5kr_U_olQlVUVKTxGzmD3x_mcHumBE8jLfCct206r1dpnHwUjcTfPfbQ=s1967"></a></td></tr><tr><td>Results on a 3D scene. <b>Top</b>: 3D boxes, corresponding to detected vehicles, are shown in different colors; dotted line boxes are for objects that were missed. <b>Bottom</b>: The boxes are shown in the corresponding camera images for visualization purposes.</td></tr></tbody></table>
<h2>Detecting Far-Away Objects</h2>
<p>
Another benefit of 4D-Net is that it takes advantage of both the high resolution provided by RGB, which can accurately detect objects on the image plane, and the accurate depth that the point cloud data provides. As a result, objects at a greater distance that were previously missed by point cloud-only approaches can be detected by a 4D-Net. This is due to the fusion of camera data, which is able to detect distant objects, and efficiently propagate this information to the 3D part of the network to produce accurate detections. 
</p>
<h2>Is Data in Time Valuable?</h2>
<p>
To understand the value of the 4D-Net, we perform a series of ablation studies. We find that substantial improvements in detection accuracy are obtained if at least one of the sensor inputs is streamed in time. Considering both sensor inputs in time provides the largest improvements in performance.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjd8Md_xdWtFs4cSPWOq19ZVR3s3Z4HHvu2Wr_61XRK5hVCIx-ZZqSsroJ7UxGPNFvzAss46DPCmUXRbpoC4ZvKu9CKxMrQ4kpcF9QTyR7vuAmaJcm-jIVyqSxeSWnhHGGV4RvwjUJultGqIr5D1_ZDuK57HMG5LQOXSnasUrIv-NM9Awnz_y7txS-KGg=s386"></a></td></tr><tr><td>4D-Net performance for 3D object detection measured in <a href="https://waymo.com/intl/en_us/dataset-whats-next/">average precision</a> (AP) when using point clouds (PC), Point Clouds in Time (PC + T), RGB image inputs (RGB) and RGB images in Time (RGB + T). Combining both sensor inputs in time is best (rightmost columns in blue) compared to the left-most columns (green) which use a PC without RGB inputs. All joint methods use our 4D-Net multi-modal learning. </td></tr></tbody></table>
<h2>Multi-stream 4D-Net</h2>
<p>
Since the 4D-Net dynamic connection learning mechanism is general, we are not limited to only combining a point cloud stream with an RGB video stream. In fact, we find that it is very cost-effective to provide a large resolution single-image stream, and a low-resolution video stream in conjunction with 3D point cloud stream inputs. Below, we demonstrate examples of a four-stream architecture, which performs better than the two-stream one with point clouds in time and images in time.
</p>
<p>
Dynamic connection learning selects specific feature inputs to connect together. With multiple input streams, 4D-Net has to learn connections between multiple target feature representations, which is straightforward as the algorithm does not change and simply selects specific features from the union of inputs. This is an incredibly light-weight process that uses a <a href="https://arxiv.org/abs/1806.09055">differentiable architecture search</a>, which can discover new wiring within the model architecture itself and thus effectively find new 4D-Net models.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhvsm87pGM9RITFGtMzqSjGJqkfB6Ef5CU61_TdFuuH1kH3Nr7smQnDmtMNcr905aRqGqazO3enNwEKvoIXk06mAdapOIHm5IIvmSwe-oG064h5caZ4wIKcy9bHi5hS3Mv9YAsmYnUahOKVSkQYBwN1cjh5Cf_gWziNkVZxdicesFFGMoJ6jO-HKPVtUg=s812"></a></td></tr><tr><td>Example multi-stream 4D-Net which consists of a stream of 3D point clouds in time (PC+T), and multiple image streams: a high-resolution single image stream, a medium-resolution single image stream and a video stream (of even lower resolution) images.</td></tr></tbody></table>
<h2>Summary</h2>
<p>
While deep learning has made tremendous advances in real-life applications, the research community is just beginning to explore learning from multiple sensing modalities. We present 4D-Net which learns how to combine 3D point clouds in time and RGB camera images in time, for the popular application of 3D object detection in autonomous driving. We demonstrate that 4D-Net is an effective approach for detecting objects, especially at distant ranges. We hope this work will provide researchers with a valuable resource for future 4D data research.
</p>
<h2>Acknowledgements</h2>
<p>
<i>This work is done by AJ Piergiovanni, Vincent Casser, Michael Ryoo and Anelia Angelova. We thank our collaborators, Vincent Vanhoucke, Dragomir Anguelov and our colleagues at Waymo and Robotics at Google for their support and discussions. We also thank Tom Small for the graphics animation.</i>
</p>
</div>||||I||||Skip to main content
 4D-Net: Learning Multi-Modal Alignment for 3D and Image Inputs in Time
 4D-Net: Learning Multi-Modal Alignment for 3D and Image Inputs in Time
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 4D-Net: Learning Multi-Modal Alignment for 3D and Image Inputs in Time

 Wednesday, February 23, 2022

 Posted by AJ Piergiovanni and Anelia Angelova, Research Scientists, Google Research

 While not immediately obvious, all of us experience the world in four dimensions (4D). For example, when walking or driving down the street we observe a stream of visual inputs, snapshots of the 3D world, which, when taken together in time, creates a 4D visual input. Today’s autonomous vehicles and robots are able to capture much of this information through various onboard sensing mechanisms, such as LiDAR and cameras.

 LiDAR is a ubiquitous sensor that uses light pulses to reliably measure the 3D coordinates of objects in a scene, however, it is also sparse and has a limited range — the farther one is from a sensor, the fewer points will be returned. This means that far-away objects might only get a handful of points, or none at all, and might not be seen by LiDAR alone. At the same time, images from the onboard camera, which is a dense input, are incredibly useful for semantic understanding, such as detecting and segmenting objects. With high resolution, cameras can be very effective at detecting objects far away, but are less accurate in measuring the distance.

 Autonomous vehicles collect data from both LiDAR and onboard camera sensors. Each sensor measurement is recorded at regular time intervals, providing an accurate representation of the 4D world. However, very few research algorithms use both of these in combination, especially when taken “in time”, i.e., as a temporally ordered sequence of data, mostly due to two major challenges. When using both sensing modalities simultaneously, 1) it is difficult to maintain computational efficiency, and 2) pairing the information from one sensor to another adds further complexity since there is not always a direct correspondence between LiDAR points and onboard camera RGB image inputs.

 In “4D-Net for Learned Multi-Modal Alignment”, published at ICCV 2021, we present a neural network that can process 4D data, which we call 4D-Net. This is the first attempt to effectively combine both types of sensors, 3D LiDAR point clouds and onboard camera RGB images, when both are in time. We also introduce a dynamic connection learning method, which incorporates 4D information from a scene by performing connection learning across both feature representations. Finally, we demonstrate that 4D-Net is better able to use motion cues and dense image information to detect distant objects while maintaining computational efficiency.


 4D-Net

 In our scenario, we use 4D inputs (3D point clouds and onboard camera image data in time) to solve a very popular visual understanding task, the 3D box detection of objects. We study the question of how one can combine the two sensing modalities, which come from different domains and have features that do not necessarily match — i.e., sparse LiDAR inputs span the 3D space and dense camera images only produce 2D projections of a scene. The exact correspondence between their respective features is unknown, so we seek to learn the connections between these two sensor inputs and their feature representations. We consider neural network representations where each of the feature layers can be combined with other potential layers from other sensor inputs, as shown below.

    
 4D-Net effectively combines 3D LiDAR point clouds in time with RGB images, also streamed in time as video, learning the connections between different sensors and their feature representations.
 


 Dynamic Connection Learning Across Sensing Modalities

 We use a light-weight neural architecture search to learn the connections between both types of sensor inputs and their feature representations, to obtain the most accurate 3D box detection. In the autonomous driving domain it is especially important to reliably detect objects at highly variable distances, with modern LiDAR sensors reaching several hundreds of meters in range. This implies that more distant objects will appear smaller in the images and the most valuable features for detecting them will be in earlier layers of the network, which better capture fine-scale features, as opposed to close-by objects represented by later layers. Based on this observation, we modify the connections to be dynamic and select among features from all layers using self-attention mechanisms. We apply a learnable linear layer, which is able to apply attention-weighting to all other layer weights and learn the best combination for the task at hand.

     
 Connection learning approach schematic, where connections between features from the 3D point cloud inputs are combined with the features from the RGB camera video inputs. Each connection learns the weighting for the corresponding inputs.
 


 Results

 We evaluate our results against state-of-the-art approaches on the Waymo Open Dataset benchmark, for which previous models have only leveraged 3D point clouds in time or a combination of a single point cloud and camera image data. 4D-Net uses both sensor inputs efficiently, processing 32 point clouds in time and 16 RGB frames within 164 milliseconds, and performs well compared to other methods. In comparison, the next best approach is less efficient and accurate because its neural net computation takes 300 milliseconds, and uses fewer sensor inputs than 4D-Net.

 
 
     
 Results on a 3D scene. Top: 3D boxes, corresponding to detected vehicles, are shown in different colors; dotted line boxes are for objects that were missed. Bottom: The boxes are shown in the corresponding camera images for visualization purposes.
 


 Detecting Far-Away Objects

 Another benefit of 4D-Net is that it takes advantage of both the high resolution provided by RGB, which can accurately detect objects on the image plane, and the accurate depth that the point cloud data provides. As a result, objects at a greater distance that were previously missed by point cloud-only approaches can be detected by a 4D-Net. This is due to the fusion of camera data, which is able to detect distant objects, and efficiently propagate this information to the 3D part of the network to produce accurate detections.


 Is Data in Time Valuable?

 To understand the value of the 4D-Net, we perform a series of ablation studies. We find that substantial improvements in detection accuracy are obtained if at least one of the sensor inputs is streamed in time. Considering both sensor inputs in time provides the largest improvements in performance.

       
 4D-Net performance for 3D object detection measured in average precision (AP) when using point clouds (PC), Point Clouds in Time (PC + T), RGB image inputs (RGB) and RGB images in Time (RGB + T). Combining both sensor inputs in time is best (rightmost columns in blue) compared to the left-most columns (green) which use a PC without RGB inputs. All joint methods use our 4D-Net multi-modal learning.
 


 Multi-stream 4D-Net

 Since the 4D-Net dynamic connection learning mechanism is general, we are not limited to only combining a point cloud stream with an RGB video stream. In fact, we find that it is very cost-effective to provide a large resolution single-image stream, and a low-resolution video stream in conjunction with 3D point cloud stream inputs. Below, we demonstrate examples of a four-stream architecture, which performs better than the two-stream one with point clouds in time and images in time.

 Dynamic connection learning selects specific feature inputs to connect together. With multiple input streams, 4D-Net has to learn connections between multiple target feature representations, which is straightforward as the algorithm does not change and simply selects specific features from the union of inputs. This is an incredibly light-weight process that uses a differentiable architecture search, which can discover new wiring within the model architecture itself and thus effectively find new 4D-Net models.

     
 Example multi-stream 4D-Net which consists of a stream of 3D point clouds in time (PC+T), and multiple image streams: a high-resolution single image stream, a medium-resolution single image stream and a video stream (of even lower resolution) images.
 


 Summary

 While deep learning has made tremendous advances in real-life applications, the research community is just beginning to explore learning from multiple sensing modalities. We present 4D-Net which learns how to combine 3D point clouds in time and RGB camera images in time, for the popular application of 3D object detection in autonomous driving. We demonstrate that 4D-Net is an effective approach for detecting objects, especially at distant ranges. We hope this work will provide researchers with a valuable resource for future 4D data research.


 Acknowledgements

 This work is done by AJ Piergiovanni, Vincent Casser, Michael Ryoo and Anelia Angelova. We thank our collaborators, Vincent Vanhoucke, Dragomir Anguelov and our colleagues at Waymo and Robotics at Google for their support and discussions. We also thank Tom Small for the graphics animation.

 Labels: Multimodal Learning
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by AJ Piergiovanni and Anelia Angelova, Research Scientists, Google Research 
While not immediately obvious, all of us experience the world in four dimensions (4D). For example, when walking or driving down the street we observe a stream of visual inputs, snapshots of the 3D world, which, when taken together in time, creates a 4D visual input. Today’s autonomous vehicles and robots are able to capture much of this information through various onboard sensing mechanisms, such as LiDAR and cameras.
 
LiDAR is a ubiquitous sensor that uses light pulses to reliably measure the 3D coordinates of objects in a scene, however, it is also sparse and has a limited range — the farther one is from a sensor, the fewer points will be returned. This means that far-away objects might only get a handful of points, or none at all, and might not be seen by LiDAR alone. At the same time, images from the onboard camera, which is a dense input, are incredibly useful for semantic understanding, such as detecting and segmenting objects. With high resolution, cameras can be very effective at detecting objects far away, but are less accurate in measuring the distance. 
Autonomous vehicles collect data from both LiDAR and onboard camera sensors. Each sensor measurement is recorded at regular time intervals, providing an accurate representation of the 4D world. However, very few research algorithms use both of these in combination, especially when taken “in time”, i.e., as a temporally ordered sequence of data, mostly due to two major challenges. When using both sensing modalities simultaneously, 1) it is difficult to maintain computational efficiency, and 2) pairing the information from one sensor to another adds further complexity since there is not always a direct correspondence between LiDAR points and onboard camera RGB image inputs.
In “4D-Net for Learned Multi-Modal Alignment”, published at ICCV 2021, we present a neural network that can process 4D data, which we call  4D-Net. This is the first attempt to effectively combine both types of sensors, 3D LiDAR point clouds and onboard camera RGB images, when both are in time. We also introduce a dynamic connection learning method, which incorporates 4D information from a scene by performing connection learning across both feature representations. Finally, we demonstrate that 4D-Net is better able to use motion cues and dense image information to detect distant objects while maintaining computational efficiency. 
4D-Net
In our scenario, we use 4D inputs (3D point clouds and onboard camera image data in time) to solve a very popular visual understanding task, the 3D box detection of objects. We study the question of how one can combine the two sensing modalities, which come from different domains and have features that do not necessarily match — i.e., sparse LiDAR inputs span the 3D space and dense camera images only produce 2D projections of a scene. The exact correspondence between their respective features is unknown, so we seek to learn the connections between these two sensor inputs and their feature representations. We consider neural network representations where each of the feature layers can be combined with other potential layers from other sensor inputs, as shown below.
         4D-Net effectively combines 3D LiDAR point clouds in time with RGB images, also streamed in time as video, learning the connections between different sensors and their feature representations.    
Dynamic Connection Learning Across Sensing Modalities
We use a light-weight neural architecture search to learn the connections between both types of sensor inputs and their feature representations, to obtain the most accurate 3D box detection. In the autonomous driving domain it is especially important to reliably detect objects at highly variable distances, with modern LiDAR sensors reaching several hundreds of meters in range. This implies that more distant objects will appear smaller in the images and the most valuable features for detecting them will be in earlier layers of the network, which better capture fine-scale features, as opposed to close-by objects represented by later layers. Based on this observation, we modify the connections to be dynamic and select among features from all layers using self-attention mechanisms. We apply a learnable linear layer, which is able to apply attention-weighting to all other layer weights and learn the best combination for the task at hand.
         Connection learning approach schematic, where connections between features from the 3D point cloud inputs are combined with the features from the RGB camera video inputs. Each connection learns the weighting for the corresponding inputs.    
Results 
We evaluate our results against state-of-the-art approaches on the Waymo Open Dataset benchmark, for which previous models have only leveraged 3D point clouds in time or a combination of a single point cloud and camera image data. 4D-Net uses both sensor inputs efficiently, processing 32 point clouds in time and 16 RGB frames within 164 milliseconds, and performs well compared to other methods. In comparison, the next best approach is less efficient and accurate because its neural net computation takes 300 milliseconds, and uses fewer sensor inputs than 4D-Net.
   
         Results on a 3D scene. Top: 3D boxes, corresponding to detected vehicles, are shown in different colors; dotted line boxes are for objects that were missed. Bottom: The boxes are shown in the corresponding camera images for visualization purposes.   
Detecting Far-Away Objects
Another benefit of 4D-Net is that it takes advantage of both the high resolution provided by RGB, which can accurately detect objects on the image plane, and the accurate depth that the point cloud data provides. As a result, objects at a greater distance that were previously missed by point cloud-only approaches can be detected by a 4D-Net. This is due to the fusion of camera data, which is able to detect distant objects, and efficiently propagate this information to the 3D part of the network to produce accurate detections. 
Is Data in Time Valuable?
To understand the value of the 4D-Net, we perform a series of ablation studies. We find that substantial improvements in detection accuracy are obtained if at least one of the sensor inputs is streamed in time. Considering both sensor inputs in time provides the largest improvements in performance.
         4D-Net performance for 3D object detection measured in average precision (AP) when using point clouds (PC), Point Clouds in Time (PC + T), RGB image inputs (RGB) and RGB images in Time (RGB + T). Combining both sensor inputs in time is best (rightmost columns in blue) compared to the left-most columns (green) which use a PC without RGB inputs. All joint methods use our 4D-Net multi-modal learning.    
Multi-stream 4D-Net
Since the 4D-Net dynamic connection learning mechanism is general, we are not limited to only combining a point cloud stream with an RGB video stream. In fact, we find that it is very cost-effective to provide a large resolution single-image stream, and a low-resolution video stream in conjunction with 3D point cloud stream inputs. Below, we demonstrate examples of a four-stream architecture, which performs better than the two-stream one with point clouds in time and images in time.
Dynamic connection learning selects specific feature inputs to connect together. With multiple input streams, 4D-Net has to learn connections between multiple target feature representations, which is straightforward as the algorithm does not change and simply selects specific features from the union of inputs. This is an incredibly light-weight process that uses a differentiable architecture search, which can discover new wiring within the model architecture itself and thus effectively find new 4D-Net models.
         Example multi-stream 4D-Net which consists of a stream of 3D point clouds in time (PC+T), and multiple image streams: a high-resolution single image stream, a medium-resolution single image stream and a video stream (of even lower resolution) images.   
Summary
While deep learning has made tremendous advances in real-life applications, the research community is just beginning to explore learning from multiple sensing modalities. We present 4D-Net which learns how to combine 3D point clouds in time and RGB camera images in time, for the popular application of 3D object detection in autonomous driving. We demonstrate that 4D-Net is an effective approach for detecting objects, especially at distant ranges. We hope this work will provide researchers with a valuable resource for future 4D data research.
Acknowledgements
This work is done by AJ Piergiovanni, Vincent Casser, Michael Ryoo and Anelia Angelova. We thank our collaborators, Vincent Vanhoucke, Dragomir Anguelov and our colleagues at Waymo and Robotics at Google for their support and discussions. We also thank Tom Small for the graphics animation.
4D-Net: Learning Multi-Modal Alignment for 3D and Image Inputs in Time
https://ai.googleblog.com/2022/02/4d-net-learning-multi-modal-alignment.html
https://ai.googleblog.com/2022/02/robust-routing-using-electrical-flows.html
Robust Routing Using Electrical Flows
2022-02-18
Ali Kemal Sinop; Kostas Kollias; Research Scientists; Google Research
<div><div>
<p>Posted by Ali Kemal Sinop and Kostas Kollias, Research Scientists, Google Research</p><p>
In the world of <a href="https://en.wikipedia.org/wiki/Network_theory">networks</a>, there are models that can explain observations across a diverse collection of applications. These include simple tasks such as computing the shortest path, which has obvious applications to routing networks but also applies in biology, e.g., where the slime mold <a href="https://en.wikipedia.org/wiki/Physarum_polycephalum">Physarum</a> is able to <a href="https://en.wikipedia.org/wiki/Physarum_polycephalum#Situational_behavior">find shortest paths</a> in mazes. Another example is <a href="https://en.wikipedia.org/wiki/Braess%27s_paradox">Braess’s paradox</a> — the observation that adding resources to a network can have an effect opposite to the one expected — which manifests not only in road networks but also in mechanical and electrical systems. For instance, constructing a new road can increase traffic congestion or adding a new link in an electrical circuit can increase voltage. Such connections between electrical circuits and other types of networks have been exploited for various tasks, such as <a href="https://arxiv.org/abs/1711.06530">partitioning networks</a>, and <a href="http://www.mit.edu/~har/routing.pdf">routing flows</a>.
</p><p>
In “<a href="https://dl.acm.org/doi/abs/10.1145/3474717.3483961">Robust Routing Using Electrical Flows</a>”, which won the Best Paper Award at <a href="https://sigspatial2021.sigspatial.org/">SIGSPATIAL 2021</a>, we present another interesting application of electrical flows in the context of road network routing. Specifically, we utilize ideas from electrical flows for the problem of constructing multiple alternate routes between a given source and destination. Alternate routes are important for many use cases, including finding routes that best match user preferences and for <em>robust routing, </em>e.g., routing that guarantees finding a good path in the presence of traffic jams. Along the way, we also describe how to quickly model electrical flows on road networks.
</p>
<h2>Existing Approaches to Alternate Routing</h2><p>
Computing alternate routes on road networks is a relatively new area of research and most techniques rely on one of two main templates: the <em>penalty method </em>and the <em>plateau method. </em>In the former, alternate routes are iteratively computed by running a <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm">shortest path algorithm</a> and then, in subsequent runs, adding a penalty to those segments already included in the shortest paths that have been computed, to encourage further exploration. In the latter, two shortest path trees are built simultaneously, one starting from the origin and one from the destination, which are used to identify sequences of road segments that are common to both trees. Each such common sequence (which are expected to be important arterial streets for example) is then treated as a visit point on the way from the origin to the destination, thus potentially producing an alternate route. The penalty method is known to produce results of high quality (i.e., average travel time, diversity and robustness of the returned set of alternate routes) but is very slow in practice, whereas the plateau method is much faster but results in lower quality solutions.
</p>
<h2>An Alternate to Alternate Routing: Electrical Flows</h2><p>
Our approach is different and assumes that a routing problem on a road network is in many ways analogous to the flow of <a href="https://en.wikipedia.org/wiki/Electric_current">electrical current</a> through a <a href="https://en.wikipedia.org/wiki/Electrical_network">resistor network</a>. Though the electrical current travels through many different paths, it is weaker along paths of higher resistance and stronger on low resistance ones, all else being equal. 
</p>
<p>
We view the road network as a <a href="https://ai.googleblog.com/2021/09/efficient-partitioning-of-road-networks.html#:~:text=road%20networks%20and-,graphs,-%2C%20where%20intersections%20become">graph</a>, where intersections are nodes and roads are edges. Our method then models the graph as an electrical circuit by replacing the edges with <a href="https://en.wikipedia.org/wiki/Resistor">resistors</a>, whose resistances equal the road traversal time, and then connecting a <a href="https://en.wikipedia.org/wiki/Electric_battery">battery</a> to the origin and destination, which results in electrical current between those two points. In this analogy, the resistance models how time-consuming it is to traverse a segment. In this sense, long and congested segments have high resistances. Intuitively speaking, the flow of electrical current will be spread around the entire network but concentrated on the routes that have lower resistance, which correspond to faster routes. By identifying the primary routes taken by the current, we can construct a viable set of alternates from origin to destination. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgEWe1fU4Q10xXePykclSodna0nnPheiaYXuKgDbivgPrede-kr4TIfyvdKDXQvwc7LGlkTekFwLwMC8b7AGaUo6-92jKKMj87EDNgX4mRzQ3WkA2hz1LvUMwkAR6h092OAxmsupLHzRAn2r5CfHhVhSbplWhtSopdCl6kWGZGINdeDNIMIM8MhsZzUkw=s1250"></a></td></tr><tr><td>Example of how we construct the electrical circuit corresponding to the road network. The current can be decomposed into three flows, <em>i</em><sub>1</sub>, <em>i</em><sub>2</sub> and <em>i</em><sub>3</sub>; each of which corresponds to a viable alternate path from Fremont to San Rafael.</td></tr></tbody></table>
<p>
In order to compute the electrical flow, we use <a href="https://en.wikipedia.org/wiki/Kirchhoff%27s_circuit_laws">Kirchhoff’s</a> and <a href="https://en.wikipedia.org/wiki/Ohm%27s_law">Ohm’s</a> laws, which say respectively: 1) the algebraic sum of currents at each junction is equal to zero, meaning that the traffic that enters any intersection also exits it (for instance if three cars enter an intersection from one street and another car enters the same intersection from another street, a total of four cars need to exit the intersection); and 2) the current is directly proportional to the voltage difference between endpoints. If we write down the resulting equations, we end up with a <em><a href="https://en.wikipedia.org/wiki/System_of_linear_equations">linear system</a></em> with <em>n</em> equations over <em>n</em> variables, which correspond to the <a href="https://en.wikipedia.org/wiki/Electric_potential">potentials</a> (i.e, the voltage) at each intersection. While voltage has no direct analogy to road networks, it can be used to help compute the flow of electrical current and thus find alternate routes as described above. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEglpwvQmsKYb3dXjdGfIZ3nqnPcrwEVCvXgYRouQaaiuowUidJvlhwlJRB1krNxmzjh3TBrawec_0KVMksYMf79Yg11u4iKy2MREMvxjWtNEEQD5Jl25413FPhWLYeTJLiOVZgRTHqayJt3Fzax9tJlPtyD3JeNISg7FHmAX1lUHEzes4OJpg7pYr9UyA=s1999"></a></td></tr><tr><td>In order to find the electrical current <em>i</em> (or flow) on each wire, we can use Kirchhoff’s law and Ohm’s law to obtain a linear system of equations in terms of voltages (or potentials) <em>v</em>. This yields a linear system with three equations (representing Kirchhoff’s law) and three unknowns (voltages at each intersection).</td></tr></tbody></table>
<p>
So the computation boils down to computing values for the variables of this linear system involving a very special matrix called <a href="https://en.wikipedia.org/wiki/Laplacian_matrix">Laplacian matrix</a>. Such matrices have many useful properties, e.g., they are symmetric and <em>sparse — </em>the number of off-diagonal non-zero entries is equal to twice the number of edges. Even though there are many existing near-linear time solvers for such systems of linear equations, they are still too slow for the purposes of quickly responding to routing requests with low latency. Thus we devised a new algorithm that solves these linear systems much faster for the special case of road networks<sup><a href="#fn1">1</a></sup>. 
</p>
<h2>Fast Electrical Flow Computation</h2><p>
The first key part of this new algorithm involves <a href="https://en.wikipedia.org/wiki/Gaussian_elimination">Gaussian elimination</a>, which is possibly the most well-known method for solving linear systems. When performed on a Laplacian matrix corresponding to some resistor network, it corresponds to the <a href="https://en.wikipedia.org/wiki/Y-%CE%94_transform">Y-Δ transformation</a>, which reduces the number of nodes, while preserving the voltages. The only downside is that the number of edges may increase, which would make the linear system even slower to solve. For example, if a node with 10 connections is eliminated using the Y-Δ transformation, the system would end up with 35 new connections! 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjbLZlUWlXaUCzrXY1Iqzc4grOUULX1xEgoVbv7U9eivR_6BMGujWTGz2eLwJ7ziLULk4gClJvS2HEG-IHJcH4bR1PXCaJEVvZfzcNPIO5gB9bgaee6JkkRSTY5GqTQfzE7WtziPNJbuL68MSU2FIxiXr6Q-aAWlFBde5hSiaZsaq79xEUBzTE9cFi5EQ=s1226"></a></td></tr><tr><td>The Y-Δ transformation allows us to remove the middle junction and replace it with three connections (<em>R<sub>a</sub></em>, <em>R<sub>b</sub></em> and <em>R<sub>c</sub></em>) between <em>N</em><sub>1</sub>, <em>N</em><sub>2</sub> and <em>N</em><sub>3</sub>. (Image from <a href="https://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg">Wikipedia</a>)</td></tr></tbody></table>
<p>
However if one can identify parts of the network that are connected to the rest through very few nodes (lets call these connections <em>bottlenecks</em>), and perform elimination on everything else while leaving the bottleneck nodes, the new edges formed at the end will only be between bottleneck nodes. Provided that the number of bottleneck nodes is much smaller than the number of nodes eliminated with Y-Δ — which is true in the case of road networks since bottleneck nodes, such as bridges and tunnels, are much less common than regular intersections — this will result in a large net decrease (e.g., ~100x) in terms of graph size. Fortunately, identifying such bottlenecks in road networks can be done easily by <a href="https://ai.googleblog.com/2021/09/efficient-partitioning-of-road-networks.html">partitioning such a network</a>. By applying Y-Δ transformation to all nodes except the bottlenecks<sup><a href="#fn2">2</a></sup>, the result is a much smaller graph for which the voltages can be solved faster. 
</p>
<p>
But what about computing the currents on the rest of the network, which is not made up of bottleneck nodes? A useful property about electrical flows is that once the voltages on bottleneck nodes are known, one can easily compute the electrical flow for the rest of the network. The electrical flow inside a part of the network only depends on the voltage of bottleneck nodes that separate that part from the rest of the network. In fact, it’s possible to precompute a small matrix so that one can recover the electrical flow by a single matrix-vector multiplication, which is a very fast operation that can be run in parallel. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjyph3rNJb2kVHix3z2-4Ws2_XRCS7RNXdZM0sxQ66VhvRknl90AB7l4nbOwS9-o1zRwtyyClSmpYY1suobZYV4il7UjvfXoPVWzLjZAVxiRKYWnewK5FNiKaGlsExHQjmosvwWj-996ILDm5WqYZFkFi9cr5kdaKcp_0eNOHs4kF1QpeFyzLDOSj-L_g=s1999"></a></td></tr><tr><td>Consider the imposed conceptual road network on Staten Island (<strong>left</strong>), for which directly computing the electrical flow would be slow. The bridges (red nodes) are the bottleneck points, and we can eliminate the whole road network inside the island by repeatedly applying Gaussian Elimination (or Y-Δ transformation). The resulting network (<strong>middle</strong>) is a much smaller graph, which allows for faster computation. The potentials inside the eliminated part are always a fixed linear combination of the bottleneck nodes (<strong>right</strong>).</td></tr></tbody></table>
<p>
Once we obtain a solution that gives the electrical flow in our model network, we can observe the routes that carry the highest amount of electrical flow and output those as alternate routes for the road network.
</p>
<h2>Results</h2><p>
Here are some results depicting the alternates computed by the above algorithm. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjs282idpf1AKJpWaR6SWBpdDVL1Ffn9RpDE-W9iA16KfxQWhmoHNR54Q93tBM-kbJX6JQnHyNZ1iTflSMULcEGD_SaEDr4fyFiPF5e345eCqu75nTixvd6RfURW0V1ohWffELlHjRCavri9lmYrv1Te7eNTIwUZtttsbwG53BQJQJSvq29LZpnAydviA=s1066"></a></td></tr><tr><td>Different alternates found for the Bay Area. Different colors correspond to different routes from the origin (red icon toward the bottom) to the destination (blue icon toward the top).</td></tr></tbody></table>
<h2>Conclusion</h2><p>
In this post we describe a novel approach for computing alternate routes in road networks. Our approach is fundamentally different from the main techniques applied in decades of research in the area and provides high quality alternate routes in road networks by studying the problem through the lens of electrical circuits. This is an approach that can prove very useful in practical systems and we hope inspires more research in the area of alternate route computation and related problems. Interested readers can find a more detailed discussion of this work in our SIGSPATIAL 2021 <a href="https://www.youtube.com/watch?v=SnqXKDRceDc&amp;list=PLBqIcp8uAbkMNDIGk9LE9LD1-C6lvs6ut&amp;index=96">talk recording</a>.
</p>
<h2>Acknowledgements</h2><p>
<em>We thank our collaborators Lisa Fawcett, Sreenivas Gollapudi, Ravi Kumar, Andrew Tomkins and Ameya Velingker from Google Research.</em>
</p>
<hr />
<p>
 <sup><a><b>1</b></a></sup>Our techniques work for any network that can be broken down to smaller components with the removal of a few nodes. <a href="#fnref1"><sup>↩</sup></a><br />
 <sup><a><b>2</b></a></sup> Performing Y-Δ transformation one-by-one for each node will be too slow. Instead we eliminate whole groups of nodes by taking advantage of the algebraic properties of Y-Δ transformation. <a href="#fnref2"><sup>↩</sup></a></p>
</div>||||I||||Skip to main content
 Robust Routing Using Electrical Flows
 Robust Routing Using Electrical Flows
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Robust Routing Using Electrical Flows

 Friday, February 18, 2022

 Posted by Ali Kemal Sinop and Kostas Kollias, Research Scientists, Google Research

 In the world of networks, there are models that can explain observations across a diverse collection of applications. These include simple tasks such as computing the shortest path, which has obvious applications to routing networks but also applies in biology, e.g., where the slime mold Physarum is able to find shortest paths in mazes. Another example is Braess’s paradox — the observation that adding resources to a network can have an effect opposite to the one expected — which manifests not only in road networks but also in mechanical and electrical systems. For instance, constructing a new road can increase traffic congestion or adding a new link in an electrical circuit can increase voltage. Such connections between electrical circuits and other types of networks have been exploited for various tasks, such as partitioning networks, and routing flows.

 In “Robust Routing Using Electrical Flows”, which won the Best Paper Award at SIGSPATIAL 2021, we present another interesting application of electrical flows in the context of road network routing. Specifically, we utilize ideas from electrical flows for the problem of constructing multiple alternate routes between a given source and destination. Alternate routes are important for many use cases, including finding routes that best match user preferences and for robust routing, e.g., routing that guarantees finding a good path in the presence of traffic jams. Along the way, we also describe how to quickly model electrical flows on road networks.


 Existing Approaches to Alternate Routing

 Computing alternate routes on road networks is a relatively new area of research and most techniques rely on one of two main templates: the penalty method and the plateau method. In the former, alternate routes are iteratively computed by running a shortest path algorithm and then, in subsequent runs, adding a penalty to those segments already included in the shortest paths that have been computed, to encourage further exploration. In the latter, two shortest path trees are built simultaneously, one starting from the origin and one from the destination, which are used to identify sequences of road segments that are common to both trees. Each such common sequence (which are expected to be important arterial streets for example) is then treated as a visit point on the way from the origin to the destination, thus potentially producing an alternate route. The penalty method is known to produce results of high quality (i.e., average travel time, diversity and robustness of the returned set of alternate routes) but is very slow in practice, whereas the plateau method is much faster but results in lower quality solutions.


 An Alternate to Alternate Routing: Electrical Flows

 Our approach is different and assumes that a routing problem on a road network is in many ways analogous to the flow of electrical current through a resistor network. Though the electrical current travels through many different paths, it is weaker along paths of higher resistance and stronger on low resistance ones, all else being equal.

 We view the road network as a graph, where intersections are nodes and roads are edges. Our method then models the graph as an electrical circuit by replacing the edges with resistors, whose resistances equal the road traversal time, and then connecting a battery to the origin and destination, which results in electrical current between those two points. In this analogy, the resistance models how time-consuming it is to traverse a segment. In this sense, long and congested segments have high resistances. Intuitively speaking, the flow of electrical current will be spread around the entire network but concentrated on the routes that have lower resistance, which correspond to faster routes. By identifying the primary routes taken by the current, we can construct a viable set of alternates from origin to destination.

    
 Example of how we construct the electrical circuit corresponding to the road network. The current can be decomposed into three flows, i1, i2 and i3; each of which corresponds to a viable alternate path from Fremont to San Rafael.
 

 In order to compute the electrical flow, we use Kirchhoff’s and Ohm’s laws, which say respectively: 1) the algebraic sum of currents at each junction is equal to zero, meaning that the traffic that enters any intersection also exits it (for instance if three cars enter an intersection from one street and another car enters the same intersection from another street, a total of four cars need to exit the intersection); and 2) the current is directly proportional to the voltage difference between endpoints. If we write down the resulting equations, we end up with a linear system with n equations over n variables, which correspond to the potentials (i.e, the voltage) at each intersection. While voltage has no direct analogy to road networks, it can be used to help compute the flow of electrical current and thus find alternate routes as described above.

      
 In order to find the electrical current i (or flow) on each wire, we can use Kirchhoff’s law and Ohm’s law to obtain a linear system of equations in terms of voltages (or potentials) v. This yields a linear system with three equations (representing Kirchhoff’s law) and three unknowns (voltages at each intersection).
 

 So the computation boils down to computing values for the variables of this linear system involving a very special matrix called Laplacian matrix. Such matrices have many useful properties, e.g., they are symmetric and sparse — the number of off-diagonal non-zero entries is equal to twice the number of edges. Even though there are many existing near-linear time solvers for such systems of linear equations, they are still too slow for the purposes of quickly responding to routing requests with low latency. Thus we devised a new algorithm that solves these linear systems much faster for the special case of road networks 1 .


 Fast Electrical Flow Computation

 The first key part of this new algorithm involves Gaussian elimination, which is possibly the most well-known method for solving linear systems. When performed on a Laplacian matrix corresponding to some resistor network, it corresponds to the Y-Δ transformation, which reduces the number of nodes, while preserving the voltages. The only downside is that the number of edges may increase, which would make the linear system even slower to solve. For example, if a node with 10 connections is eliminated using the Y-Δ transformation, the system would end up with 35 new connections!

   
 The Y-Δ transformation allows us to remove the middle junction and replace it with three connections (Ra, Rb and Rc) between N1, N2 and N3. (Image from Wikipedia)
 

 However if one can identify parts of the network that are connected to the rest through very few nodes (lets call these connections bottlenecks), and perform elimination on everything else while leaving the bottleneck nodes, the new edges formed at the end will only be between bottleneck nodes. Provided that the number of bottleneck nodes is much smaller than the number of nodes eliminated with Y-Δ — which is true in the case of road networks since bottleneck nodes, such as bridges and tunnels, are much less common than regular intersections — this will result in a large net decrease (e.g., ~100x) in terms of graph size. Fortunately, identifying such bottlenecks in road networks can be done easily by partitioning such a network. By applying Y-Δ transformation to all nodes except the bottlenecks 2 , the result is a much smaller graph for which the voltages can be solved faster.

 But what about computing the currents on the rest of the network, which is not made up of bottleneck nodes? A useful property about electrical flows is that once the voltages on bottleneck nodes are known, one can easily compute the electrical flow for the rest of the network. The electrical flow inside a part of the network only depends on the voltage of bottleneck nodes that separate that part from the rest of the network. In fact, it’s possible to precompute a small matrix so that one can recover the electrical flow by a single matrix-vector multiplication, which is a very fast operation that can be run in parallel.

         
 Consider the imposed conceptual road network on Staten Island (left), for which directly computing the electrical flow would be slow. The bridges (red nodes) are the bottleneck points, and we can eliminate the whole road network inside the island by repeatedly applying Gaussian Elimination (or Y-Δ transformation). The resulting network (middle) is a much smaller graph, which allows for faster computation. The potentials inside the eliminated part are always a fixed linear combination of the bottleneck nodes (right).
 

 Once we obtain a solution that gives the electrical flow in our model network, we can observe the routes that carry the highest amount of electrical flow and output those as alternate routes for the road network.


 Results

 Here are some results depicting the alternates computed by the above algorithm.

    
 Different alternates found for the Bay Area. Different colors correspond to different routes from the origin (red icon toward the bottom) to the destination (blue icon toward the top).
 


 Conclusion

 In this post we describe a novel approach for computing alternate routes in road networks. Our approach is fundamentally different from the main techniques applied in decades of research in the area and provides high quality alternate routes in road networks by studying the problem through the lens of electrical circuits. This is an approach that can prove very useful in practical systems and we hope inspires more research in the area of alternate route computation and related problems. Interested readers can find a more detailed discussion of this work in our SIGSPATIAL 2021 talk recording.


 Acknowledgements

 We thank our collaborators Lisa Fawcett, Sreenivas Gollapudi, Ravi Kumar, Andrew Tomkins and Ameya Velingker from Google Research.

 1 Our techniques work for any network that can be broken down to smaller components with the removal of a few nodes. ↩
 2 Performing Y-Δ transformation one-by-one for each node will be too slow. Instead we eliminate whole groups of nodes by taking advantage of the algebraic properties of Y-Δ transformation. ↩

 Labels: Algorithms optimization
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Ali Kemal Sinop and Kostas Kollias, Research Scientists, Google Research 
In the world of networks, there are models that can explain observations across a diverse collection of applications. These include simple tasks such as computing the shortest path, which has obvious applications to routing networks but also applies in biology, e.g., where the slime mold Physarum is able to find shortest paths in mazes. Another example is Braess’s paradox — the observation that adding resources to a network can have an effect opposite to the one expected — which manifests not only in road networks but also in mechanical and electrical systems. For instance, constructing a new road can increase traffic congestion or adding a new link in an electrical circuit can increase voltage. Such connections between electrical circuits and other types of networks have been exploited for various tasks, such as partitioning networks, and routing flows.
 
In “Robust Routing Using Electrical Flows”, which won the Best Paper Award at SIGSPATIAL 2021, we present another interesting application of electrical flows in the context of road network routing. Specifically, we utilize ideas from electrical flows for the problem of constructing multiple alternate routes between a given source and destination. Alternate routes are important for many use cases, including finding routes that best match user preferences and for robust routing, e.g., routing that guarantees finding a good path in the presence of traffic jams. Along the way, we also describe how to quickly model electrical flows on road networks.
Existing Approaches to Alternate Routing 
Computing alternate routes on road networks is a relatively new area of research and most techniques rely on one of two main templates: the penalty method and the plateau method. In the former, alternate routes are iteratively computed by running a shortest path algorithm and then, in subsequent runs, adding a penalty to those segments already included in the shortest paths that have been computed, to encourage further exploration. In the latter, two shortest path trees are built simultaneously, one starting from the origin and one from the destination, which are used to identify sequences of road segments that are common to both trees. Each such common sequence (which are expected to be important arterial streets for example) is then treated as a visit point on the way from the origin to the destination, thus potentially producing an alternate route. The penalty method is known to produce results of high quality (i.e., average travel time, diversity and robustness of the returned set of alternate routes) but is very slow in practice, whereas the plateau method is much faster but results in lower quality solutions.
An Alternate to Alternate Routing: Electrical Flows 
Our approach is different and assumes that a routing problem on a road network is in many ways analogous to the flow of electrical current through a resistor network. Though the electrical current travels through many different paths, it is weaker along paths of higher resistance and stronger on low resistance ones, all else being equal. 
We view the road network as a graph, where intersections are nodes and roads are edges. Our method then models the graph as an electrical circuit by replacing the edges with resistors, whose resistances equal the road traversal time, and then connecting a battery to the origin and destination, which results in electrical current between those two points. In this analogy, the resistance models how time-consuming it is to traverse a segment. In this sense, long and congested segments have high resistances. Intuitively speaking, the flow of electrical current will be spread around the entire network but concentrated on the routes that have lower resistance, which correspond to faster routes. By identifying the primary routes taken by the current, we can construct a viable set of alternates from origin to destination. 
         Example of how we construct the electrical circuit corresponding to the road network. The current can be decomposed into three flows, i 1, i 2 and i 3; each of which corresponds to a viable alternate path from Fremont to San Rafael.   
In order to compute the electrical flow, we use Kirchhoff’s and Ohm’s laws, which say respectively: 1) the algebraic sum of currents at each junction is equal to zero, meaning that the traffic that enters any intersection also exits it (for instance if three cars enter an intersection from one street and another car enters the same intersection from another street, a total of four cars need to exit the intersection); and 2) the current is directly proportional to the voltage difference between endpoints. If we write down the resulting equations, we end up with a  linear system  with n equations over n variables, which correspond to the potentials (i.e, the voltage) at each intersection. While voltage has no direct analogy to road networks, it can be used to help compute the flow of electrical current and thus find alternate routes as described above. 
         In order to find the electrical current i (or flow) on each wire, we can use Kirchhoff’s law and Ohm’s law to obtain a linear system of equations in terms of voltages (or potentials) v. This yields a linear system with three equations (representing Kirchhoff’s law) and three unknowns (voltages at each intersection).   
So the computation boils down to computing values for the variables of this linear system involving a very special matrix called Laplacian matrix. Such matrices have many useful properties, e.g., they are symmetric and sparse — the number of off-diagonal non-zero entries is equal to twice the number of edges. Even though there are many existing near-linear time solvers for such systems of linear equations, they are still too slow for the purposes of quickly responding to routing requests with low latency. Thus we devised a new algorithm that solves these linear systems much faster for the special case of road networks 1 . 
Fast Electrical Flow Computation 
The first key part of this new algorithm involves Gaussian elimination, which is possibly the most well-known method for solving linear systems. When performed on a Laplacian matrix corresponding to some resistor network, it corresponds to the Y-Δ transformation, which reduces the number of nodes, while preserving the voltages. The only downside is that the number of edges may increase, which would make the linear system even slower to solve. For example, if a node with 10 connections is eliminated using the Y-Δ transformation, the system would end up with 35 new connections! 
         The Y-Δ transformation allows us to remove the middle junction and replace it with three connections (Ra , Rb  and Rc ) between N 1, N 2 and N 3. (Image from Wikipedia)   
However if one can identify parts of the network that are connected to the rest through very few nodes (lets call these connections bottlenecks), and perform elimination on everything else while leaving the bottleneck nodes, the new edges formed at the end will only be between bottleneck nodes. Provided that the number of bottleneck nodes is much smaller than the number of nodes eliminated with Y-Δ — which is true in the case of road networks since bottleneck nodes, such as bridges and tunnels, are much less common than regular intersections — this will result in a large net decrease (e.g., ~100x) in terms of graph size. Fortunately, identifying such bottlenecks in road networks can be done easily by partitioning such a network. By applying Y-Δ transformation to all nodes except the bottlenecks 2 , the result is a much smaller graph for which the voltages can be solved faster. 
But what about computing the currents on the rest of the network, which is not made up of bottleneck nodes? A useful property about electrical flows is that once the voltages on bottleneck nodes are known, one can easily compute the electrical flow for the rest of the network. The electrical flow inside a part of the network only depends on the voltage of bottleneck nodes that separate that part from the rest of the network. In fact, it’s possible to precompute a small matrix so that one can recover the electrical flow by a single matrix-vector multiplication, which is a very fast operation that can be run in parallel. 
         Consider the imposed conceptual road network on Staten Island (left), for which directly computing the electrical flow would be slow. The bridges (red nodes) are the bottleneck points, and we can eliminate the whole road network inside the island by repeatedly applying Gaussian Elimination (or Y-Δ transformation). The resulting network (middle) is a much smaller graph, which allows for faster computation. The potentials inside the eliminated part are always a fixed linear combination of the bottleneck nodes (right).   
Once we obtain a solution that gives the electrical flow in our model network, we can observe the routes that carry the highest amount of electrical flow and output those as alternate routes for the road network.
Results 
Here are some results depicting the alternates computed by the above algorithm. 
         Different alternates found for the Bay Area. Different colors correspond to different routes from the origin (red icon toward the bottom) to the destination (blue icon toward the top).   
Conclusion 
In this post we describe a novel approach for computing alternate routes in road networks. Our approach is fundamentally different from the main techniques applied in decades of research in the area and provides high quality alternate routes in road networks by studying the problem through the lens of electrical circuits. This is an approach that can prove very useful in practical systems and we hope inspires more research in the area of alternate route computation and related problems. Interested readers can find a more detailed discussion of this work in our SIGSPATIAL 2021 talk recording.
Acknowledgements 
We thank our collaborators Lisa Fawcett, Sreenivas Gollapudi, Ravi Kumar, Andrew Tomkins and Ameya Velingker from Google Research.
   1  Our techniques work for any network that can be broken down to smaller components with the removal of a few nodes.  ↩  
   2   Performing Y-Δ transformation one-by-one for each node will be too slow. Instead we eliminate whole groups of nodes by taking advantage of the algebraic properties of Y-Δ transformation.  ↩
Robust Routing Using Electrical Flows
https://ai.googleblog.com/2022/02/robust-routing-using-electrical-flows.html
https://ai.googleblog.com/2022/02/machine-learning-for-mechanical.html
Machine Learning for Mechanical Ventilation Control
2022-02-17
None
<div><div>
<p>Posted by Daniel Suo, Software Engineer and Elad Hazan, Research Scientist, Google Research, on behalf of the Google AI Princeton Team</p><p>
 Mechanical ventilators provide critical support for patients who have difficulty breathing or are unable to breathe on their own. They see frequent use in scenarios ranging from routine <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4057231/">anesthesia</a>, to <a href="https://pubmed.ncbi.nlm.nih.gov/31785591/">neonatal intensive care</a> and life support during the <a href="https://pubmed.ncbi.nlm.nih.gov/32195705/some%20citation">COVID-19 pandemic</a>. A typical ventilator consists of a compressed air source, valves to control the flow of air into and out of the lungs, and a "respiratory circuit" that connects the ventilator to the patient. In some cases, a sedated patient may be connected to the ventilator via a tube inserted through the trachea to their lungs, a process called <em>invasive ventilation</em>.
</p><table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiRy-2wY4sdQnDbi1LiAg90l1Txupj3tJcRH18o_aIAWgz8jL4yAX3IwMzwAZ5Z66o7E1sBAjJsS7aTqKCzFkFDIg9Dk4_yQH2CEbVQIrhbM_M_UWzY8wkTCWzHjNtHTciovDysw4_01MnHAy_w3WtaUUWmw3iOjfQqyD2uBdtNqLRfxJErnjOn84fB5Q=s861"></a></td></tr><tr><td>A mechanical ventilator takes breaths for patients who are not fully capable of doing so on their own. In invasive ventilation, a controllable, compressed air source is connected to a sedated patient via tubing called a respiratory circuit.</td></tr></tbody></table>
<p>
 In both invasive and non-invasive ventilation, the ventilator follows a clinician-prescribed breathing waveform based on a respiratory measurement from the patient (e.g., airway pressure, <a href="https://en.wikipedia.org/wiki/Tidal_volume">tidal volume</a>). In order to prevent harm, this demanding task requires both robustness to differences or changes in patients' lungs and adherence to the desired waveform. Consequently, ventilators require <a href="https://www.nytimes.com/2020/11/22/health/Covid-ventilators-stockpile.html">significant attention</a> from highly-trained clinicians in order to ensure that their performance matches the patients’ needs and that they do not cause <a href="https://www.nejm.org/doi/full/10.1056/NEJMra1208707">lung damage</a>.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEihd-zaYGxyG8fPVY6YUXkXhvdXLcY_AV3WjUu2MYzEMT8zbOxm7w5r4kQCJeW5Tx5UhCN8EuCQGF12imeYoh6X7opopmbsTktu9b16rWi8tq1Bl3i66UweCbmUbwWZUxD8Jkz4gJP_BUl8GyvqQY6EHGvRLOeqkkT975M7MTApLECri_N53zY7MPctXg=s1999"></a></td></tr><tr><td>Example of a clinician-prescribed breathing waveform (orange) in units of airway pressure and the actual pressure (blue), given some controller algorithm.</td></tr></tbody></table>
<p>
In “<a href="https://arxiv.org/abs/2102.06779">Machine Learning for Mechanical Ventilation Control</a>”, we present exploratory research into the design of a deep learning–based algorithm to improve medical ventilator control for invasive ventilation. Using signals from an artificial lung, we design a control algorithm that measures airway pressure and computes necessary adjustments to the airflow to better and more consistently match prescribed values. Compared to other approaches, we demonstrate improved robustness and better performance while requiring less manual intervention from clinicians, which suggests that this approach could reduce the likelihood of harm to a patient’s lungs.
</p>
<h2>Current Methods</h2><p>
Today, ventilators are controlled with methods belonging to the <a href="https://en.wikipedia.org/wiki/PID_controller">PID family</a> (i.e., Proportional, Integral, Differential), which control a system based on the history of errors between the observed and desired states. A PID controller uses three characteristics for ventilator control: proportion (“P”) — a comparison of the measured and target pressure; integral (“I”) — the sum of previous measurements; and differential (“D”) — the difference between two previous measurements. Variants of PID have been used since the 17th century and today form the basis of many controllers in both industrial (e.g., controlling heat or fluids) and consumer (e.g., controlling espresso pressure) applications. 
</p>
<p>
PID control forms a solid baseline, relying on the sharp reactivity of P control to rapidly increase lung pressure when breathing in and the stability of I control to hold the breath in before exhaling. However, operators must tune the ventilator for specific patients, often repeatedly, to balance the “ringing” of overzealous P control against the ineffectually slow rise in lung pressure of dominant I control.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgSWYF5L_79S39MhBJV4SLHMu9iA0W0dNFyET3umj9USdN3Rx83K8Ds-lO54Vz08UJWX8IArEKPdNmJTdXHieInFNWPPI4q6ksyJBBAIRUSNpqh5VVRBDpyS8MFjas1yWMRFpNRHzgusgyCoqdtRU9KrM2UUxrlRmJ7W1RbPdRCB_uehJ9MceuVD7S2dg=s1999"></a></td></tr><tr><td>Current PID methods are prone to over- and then under-shooting their target (ringing). Because patients differ in their physiology and may even change during treatment, highly-trained clinicians must constantly monitor and adjust existing methods to ensure such violent ringing as in the above example does not occur.</td></tr></tbody></table>
<p>
To more effectively balance these characteristics, we propose a neural network–based controller to create a set of control signals that are more broad and adaptable than PID-generated controls.
</p>
<h2>A Machine-Learned Ventilator Controller</h2><p>
While one could tune<em> </em>the coefficients of a PID controller (either manually or via an exhaustive grid search) through a limited number of repeated trials, it is impossible to apply such a direct approach towards a deep controller, as deep neural networks (DNNs) are often parameter-rich and require significant training data. Similarly, popular model-free approaches, such as <a href="https://en.wikipedia.org/wiki/Q-learning">Q-Learning</a> or <a href="http://www.scholarpedia.org/article/Policy_gradient_methods">Policy Gradient</a>, are data-intensive and therefore unsuitable for the physical system at hand. Further, these approaches don't take into account the intrinsic differentiability of the ventilator dynamical system, which is deterministic, continuous and contact-free. 
</p>
<p>
We therefore adopt a model-based approach, where we first learn a DNN-based simulator of the ventilator-patient dynamical system. An advantage of learning such a simulator is that it provides a more accurate data-driven alternative to physics-based models, and can be more widely distributed for controller research. 
</p>
<p>
To train a faithful simulator, we built a dataset by exploring the space of controls and the resulting pressures, while balancing against physical safety, e.g., not over-inflating a test lung and causing damage. Though PID control can exhibit ringing behavior, it performs well enough to use as a baseline for generating training data. To safely explore and to faithfully capture the behavior of the system, we use PID controllers with varied control coefficients to generate the control-pressure trajectory data for simulator training. Further, we add random deviations to the PID controllers to capture the dynamics more robustly.
</p>
<p>
We collect data for training by running mechanical ventilation tasks on a <a href="https://www.ingmarmed.com/product/quicklung/">physical test lung</a> using an open-source ventilator designed by Princeton University's <a href="https://www.peoplesvent.org/en/latest/">People's Ventilator Project</a>. We built a ventilator farm housing ten ventilator-lung systems on a server rack, which captures multiple airway resistance and compliance settings that span a spectrum of patient lung conditions, <a href="https://www.iso.org/standard/68844.html">as required for practical applications of ventilator systems</a>. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhMJDK4Moy9Gism34PcmJ1Ws9F2TwkZ5pbk0DVu44rTZkBpU9N_mmG7r6b0zf8nEq5I9diwkP5T0R3xfSGj3GluFvx-60_TKPnzPOIRXRs92WKKeofSNNKfdvT78U3lqOs_N8SmXs-JuhFk88vi1Ux95wCBE_wUOsiuiseHBnKzVJuhweZ9waCF_HIYhA=s1319"></a></td></tr><tr><td>We use a rack-based ventilator farm (10 ventilators / artificial lungs) to collect training data for a ventilator-lung simulator. Using this simulator, we train a DNN controller that we then validate on the physical ventilator farm.</td></tr></tbody></table>
<p>
The true underlying state of the dynamical system is not available to the model directly, but rather only through observations of the airway pressure in the system. In the simulator we model the state of the system at any time as a collection of previous pressure observations and the control actions applied to the system (up to a limited lookback window). These inputs are fed into a DNN that predicts the subsequent pressure in the system. We <em>train</em> this simulator on the control-pressure trajectory data collected through interactions with the test lung.
</p>
<p>
The performance of the simulator is measured via the sum of deviations of the simulator’s predictions (under self-simulation) from the ground truth.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg9ael5kBG2bvZVlg04tkHSGRz-FToaZnoakF6DjPupzv0oowVhV3FDkISDCTwwh7cDqMD2dqk824tLFONsccNPQCpP272UNpUnxjXHCImeA6XeSgODZs3qjWSwzQNnCvUFGizAqgsp7rbd5RMyD81zO8cl0MWs-iMm5OG0Kge0BnuVsfrMisOZrx-Abg=s1999"></a></td></tr><tr><td>While it is infeasible to compare real dynamics with their simulated counterparts over all possible trajectories and control inputs, we measure the distance between simulation and the known safe trajectories. We introduce some random exploration around these safe trajectories for robustness.</td></tr></tbody></table>
<p>
Having learned an accurate simulator, we then use it to train a DNN-based controller completely offline. This approach allows us to rapidly apply updates during controller training. Furthermore, the differentiable nature of the simulator allows for the stable use of the direct policy gradient, where we analytically compute the gradient of the loss with respect to the DNN parameters.  We find this method to be significantly more efficient than model-free approaches. <br /> 
</p>
<h2>Results</h2><p>
To establish a baseline, we run an exhaustive grid of PID controllers for multiple lung settings and select the best performing PID controller as measured by average absolute deviation between the desired pressure waveform and the actual pressure waveform. We compare these to our controllers and provide evidence that our DNN controllers are better performing and more robust.
</p>
<ol>
<li><em>Breathing waveform tracking performance:</em>
<p>
 We compare the best PID controller for a given lung setting against our controller trained on the learned simulator for the same setting. Our learned controller shows a 22% lower mean absolute error (MAE) between target and actual pressure waveforms. </p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEguTFJxPUwCYYuq2dPBn79zSmxEWOglr0zvJ5X3_BT4b4jw3qrtgKauY9JtK8BXLSioFyLEhB4LVZkVAtZ_0MweYENdzmF8PLHg-oO2-czNRlIUh04yCMXEe95so1Fzi_ra8jc7U3R5at7CMV2ktFHOT1o-vsNIpVpxyUaIsU7ozmo57tGwM7gYfKdJYw=s895"></a></td></tr><tr><td>Comparison of the MAE between target and actual pressure waveforms (lower is better) for the best PID controller (orange) for a given lung setting (shown for two settings, R=5 and R=20) against our controller (blue) trained on the learned simulator for the same setting. The learned controller performs up to 22% better.</td></tr></tbody></table>
</li>
 <li><em>Robustness:</em> 
<p> Further, we compare the performance of the single best PID controller across the entire set of lung settings with our controller trained on a set of learned simulators over the same settings. Our controller performs up to 32% better in MAE between target and actual pressure waveforms, suggesting that it could require less manual intervention between patients or even as a patient's condition changes.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhs5Ggjt_RI75vxf-foUeXE0n9ySNYSZvAqCMfHlwLibxmciibeyACfwWd7W0xfrbC8jIlR5mPFKjFv2PINKMjHOooA4Monu9JpX3U7daZcIpZA6v-6NaRKso3zfvB5vpxJQYA6wRxZVrTO8IXzf3PdHPFdK16g2wRZ2L_YGKWAuPJ3mRzLZUdtd67aew=s895"></a></td></tr><tr><td>As above, but comparing the single best PID controller across the entire set of lung settings against our controller trained over the same settings. The learned controller performs up to 32% better, suggesting that it may require less manual intervention.</td></tr></tbody></table>
 </li>
</ol>
<p>
Finally, we investigated the feasibility of using model-free and other popular RL algorithms (<a href="https://arxiv.org/abs/1707.06347">PPO</a>, <a href="https://arxiv.org/abs/1312.5602v1">DQN</a>), in comparison to a direct policy gradient trained on the simulator. We find that the simulator-trained direct policy gradient achieves slightly better scores and does so with a more stable training process that uses orders of magnitude fewer training samples and a significantly smaller hyperparameter search space.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhPTAdcBONvWuDyBLgLIHsJMj7pnp1W3cJqrnVFhUieyX0Rb2buSuYYgCvRfSfFyuAKScU5jibm5gSy63Ggo3jsa-TzFIXeLpK3LQ8o2lWCxFUmb5_hLmWWXfnSgfw-rwo0wuyc5VWww4Puy-HXGHP8mMKptN-5WhU2HpDzz6Yy0N9LfZFdSuzIMEeh8w=s1999"></a></td></tr><tr><td>In the simulator, we find that model-free and other popular algorithms (<a href="https://arxiv.org/abs/1707.06347">PPO</a>, <a href="https://arxiv.org/abs/1312.5602v1">DQN</a>) perform approximately as well as our method.</td></tr></tbody></table>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjEImS_CNfS04TwBthPmxAhqzJ3IMOvNvL0HyCWs08Rr3C1P90J62x4aG0992fHPyVWDbof0hojwiKDCovnBdvhDUesZQnCNcsL9mnW43UfIJ2m36WX2U5_QXUNGjhFZ5It5ksp7sW7Q7W9zjZA-z28jRTXoQyqI6qTsQaRQkey56QgUsq1j_jR7aemTg=s1999"></a></td></tr><tr><td>However, these other methods take an order of magnitude more episodes to train to similar levels.</td></tr></tbody></table>
<h2>Conclusions and the Road Forward</h2><p>
 We have described a deep-learning approach to mechanical ventilation based on simulated dynamics learned from a physical test lung. However, this is only the beginning. To make an impact on real-world ventilators there are numerous other considerations and issues to take into account. Most important amongst them are non-invasive ventilators, which are significantly more challenging due to the difficulty of discerning pressure from lungs and mask pressure. Other directions are how to handle spontaneous breathing and coughing. To learn more and become involved in this important intersection of machine learning and health, see an <a href="https://sites.google.com/corp/view/nsc-tutorial/home">ICML tutorial</a> on control theory and learning, and consider participating in one of our <a href="https://www.kaggle.com/c/ventilator-pressure-prediction">kaggle competitions</a> for creating better ventilator simulators!</p>
<h2>Acknowledgements</h2><p>
<em>The primary work was based in the Google AI Princeton lab, in collaboration with Cohen lab at the Mechanical and Aerospace Engineering department at Princeton University. The research paper was authored by contributors from Google and Princeton University, including: Daniel Suo, Naman Agarwal, Wenhan Xia, Xinyi Chen, Udaya Ghai, Alexander Yu, Paula Gradu, Karan Singh, Cyril Zhang, Edgar Minasyan, Julienne LaChance, Tom Zajdel, Manuel Schottdorf, Daniel Cohen, and Elad Hazan.</em>
</p>
</div>||||I||||Skip to main content
 Machine Learning for Mechanical Ventilation Control
 Machine Learning for Mechanical Ventilation Control
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Machine Learning for Mechanical Ventilation Control

 Thursday, February 17, 2022

 Posted by Daniel Suo, Software Engineer and Elad Hazan, Research Scientist, Google Research, on behalf of the Google AI Princeton Team

 Mechanical ventilators provide critical support for patients who have difficulty breathing or are unable to breathe on their own. They see frequent use in scenarios ranging from routine anesthesia, to neonatal intensive care and life support during the COVID-19 pandemic. A typical ventilator consists of a compressed air source, valves to control the flow of air into and out of the lungs, and a "respiratory circuit" that connects the ventilator to the patient. In some cases, a sedated patient may be connected to the ventilator via a tube inserted through the trachea to their lungs, a process called invasive ventilation.

     
 A mechanical ventilator takes breaths for patients who are not fully capable of doing so on their own. In invasive ventilation, a controllable, compressed air source is connected to a sedated patient via tubing called a respiratory circuit.
 

 In both invasive and non-invasive ventilation, the ventilator follows a clinician-prescribed breathing waveform based on a respiratory measurement from the patient (e.g., airway pressure, tidal volume). In order to prevent harm, this demanding task requires both robustness to differences or changes in patients' lungs and adherence to the desired waveform. Consequently, ventilators require significant attention from highly-trained clinicians in order to ensure that their performance matches the patients’ needs and that they do not cause lung damage.

   
 Example of a clinician-prescribed breathing waveform (orange) in units of airway pressure and the actual pressure (blue), given some controller algorithm.
 

 In “Machine Learning for Mechanical Ventilation Control”, we present exploratory research into the design of a deep learning–based algorithm to improve medical ventilator control for invasive ventilation. Using signals from an artificial lung, we design a control algorithm that measures airway pressure and computes necessary adjustments to the airflow to better and more consistently match prescribed values. Compared to other approaches, we demonstrate improved robustness and better performance while requiring less manual intervention from clinicians, which suggests that this approach could reduce the likelihood of harm to a patient’s lungs.


 Current Methods

 Today, ventilators are controlled with methods belonging to the PID family (i.e., Proportional, Integral, Differential), which control a system based on the history of errors between the observed and desired states. A PID controller uses three characteristics for ventilator control: proportion (“P”) — a comparison of the measured and target pressure; integral (“I”) — the sum of previous measurements; and differential (“D”) — the difference between two previous measurements. Variants of PID have been used since the 17th century and today form the basis of many controllers in both industrial (e.g., controlling heat or fluids) and consumer (e.g., controlling espresso pressure) applications.

 PID control forms a solid baseline, relying on the sharp reactivity of P control to rapidly increase lung pressure when breathing in and the stability of I control to hold the breath in before exhaling. However, operators must tune the ventilator for specific patients, often repeatedly, to balance the “ringing” of overzealous P control against the ineffectually slow rise in lung pressure of dominant I control.

      
 Current PID methods are prone to over- and then under-shooting their target (ringing). Because patients differ in their physiology and may even change during treatment, highly-trained clinicians must constantly monitor and adjust existing methods to ensure such violent ringing as in the above example does not occur.
 

 To more effectively balance these characteristics, we propose a neural network–based controller to create a set of control signals that are more broad and adaptable than PID-generated controls.


 A Machine-Learned Ventilator Controller

 While one could tune the coefficients of a PID controller (either manually or via an exhaustive grid search) through a limited number of repeated trials, it is impossible to apply such a direct approach towards a deep controller, as deep neural networks (DNNs) are often parameter-rich and require significant training data. Similarly, popular model-free approaches, such as Q-Learning or Policy Gradient, are data-intensive and therefore unsuitable for the physical system at hand. Further, these approaches don't take into account the intrinsic differentiability of the ventilator dynamical system, which is deterministic, continuous and contact-free.

 We therefore adopt a model-based approach, where we first learn a DNN-based simulator of the ventilator-patient dynamical system. An advantage of learning such a simulator is that it provides a more accurate data-driven alternative to physics-based models, and can be more widely distributed for controller research.

 To train a faithful simulator, we built a dataset by exploring the space of controls and the resulting pressures, while balancing against physical safety, e.g., not over-inflating a test lung and causing damage. Though PID control can exhibit ringing behavior, it performs well enough to use as a baseline for generating training data. To safely explore and to faithfully capture the behavior of the system, we use PID controllers with varied control coefficients to generate the control-pressure trajectory data for simulator training. Further, we add random deviations to the PID controllers to capture the dynamics more robustly.

 We collect data for training by running mechanical ventilation tasks on a physical test lung using an open-source ventilator designed by Princeton University's People's Ventilator Project. We built a ventilator farm housing ten ventilator-lung systems on a server rack, which captures multiple airway resistance and compliance settings that span a spectrum of patient lung conditions, as required for practical applications of ventilator systems.

     
 We use a rack-based ventilator farm (10 ventilators / artificial lungs) to collect training data for a ventilator-lung simulator. Using this simulator, we train a DNN controller that we then validate on the physical ventilator farm.
 

 The true underlying state of the dynamical system is not available to the model directly, but rather only through observations of the airway pressure in the system. In the simulator we model the state of the system at any time as a collection of previous pressure observations and the control actions applied to the system (up to a limited lookback window). These inputs are fed into a DNN that predicts the subsequent pressure in the system. We train this simulator on the control-pressure trajectory data collected through interactions with the test lung.

 The performance of the simulator is measured via the sum of deviations of the simulator’s predictions (under self-simulation) from the ground truth.

      
 While it is infeasible to compare real dynamics with their simulated counterparts over all possible trajectories and control inputs, we measure the distance between simulation and the known safe trajectories. We introduce some random exploration around these safe trajectories for robustness.
 

 Having learned an accurate simulator, we then use it to train a DNN-based controller completely offline. This approach allows us to rapidly apply updates during controller training. Furthermore, the differentiable nature of the simulator allows for the stable use of the direct policy gradient, where we analytically compute the gradient of the loss with respect to the DNN parameters. We find this method to be significantly more efficient than model-free approaches.


 Results

 To establish a baseline, we run an exhaustive grid of PID controllers for multiple lung settings and select the best performing PID controller as measured by average absolute deviation between the desired pressure waveform and the actual pressure waveform. We compare these to our controllers and provide evidence that our DNN controllers are better performing and more robust.

 1. Breathing waveform tracking performance:

 We compare the best PID controller for a given lung setting against our controller trained on the learned simulator for the same setting. Our learned controller shows a 22% lower mean absolute error (MAE) between target and actual pressure waveforms.

 1.      
 Comparison of the MAE between target and actual pressure waveforms (lower is better) for the best PID controller (orange) for a given lung setting (shown for two settings, R=5 and R=20) against our controller (blue) trained on the learned simulator for the same setting. The learned controller performs up to 22% better.
 
 2. Robustness:

 Further, we compare the performance of the single best PID controller across the entire set of lung settings with our controller trained on a set of learned simulators over the same settings. Our controller performs up to 32% better in MAE between target and actual pressure waveforms, suggesting that it could require less manual intervention between patients or even as a patient's condition changes.

 2.    
 As above, but comparing the single best PID controller across the entire set of lung settings against our controller trained over the same settings. The learned controller performs up to 32% better, suggesting that it may require less manual intervention.
 

 Finally, we investigated the feasibility of using model-free and other popular RL algorithms (PPO, DQN), in comparison to a direct policy gradient trained on the simulator. We find that the simulator-trained direct policy gradient achieves slightly better scores and does so with a more stable training process that uses orders of magnitude fewer training samples and a significantly smaller hyperparameter search space.

   
 In the simulator, we find that model-free and other popular algorithms (PPO, DQN) perform approximately as well as our method.
 
   
 However, these other methods take an order of magnitude more episodes to train to similar levels.
 


 Conclusions and the Road Forward

 We have described a deep-learning approach to mechanical ventilation based on simulated dynamics learned from a physical test lung. However, this is only the beginning. To make an impact on real-world ventilators there are numerous other considerations and issues to take into account. Most important amongst them are non-invasive ventilators, which are significantly more challenging due to the difficulty of discerning pressure from lungs and mask pressure. Other directions are how to handle spontaneous breathing and coughing. To learn more and become involved in this important intersection of machine learning and health, see an ICML tutorial on control theory and learning, and consider participating in one of our kaggle competitions for creating better ventilator simulators!


 Acknowledgements

 The primary work was based in the Google AI Princeton lab, in collaboration with Cohen lab at the Mechanical and Aerospace Engineering department at Princeton University. The research paper was authored by contributors from Google and Princeton University, including: Daniel Suo, Naman Agarwal, Wenhan Xia, Xinyi Chen, Udaya Ghai, Alexander Yu, Paula Gradu, Karan Singh, Cyril Zhang, Edgar Minasyan, Julienne LaChance, Tom Zajdel, Manuel Schottdorf, Daniel Cohen, and Elad Hazan.

 Labels: Health Machine Learning Neural Networks
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Daniel Suo, Software Engineer and Elad Hazan, Research Scientist, Google Research, on behalf of the Google AI Princeton Team 
 Mechanical ventilators provide critical support for patients who have difficulty breathing or are unable to breathe on their own. They see frequent use in scenarios ranging from routine anesthesia, to neonatal intensive care and life support during the COVID-19 pandemic. A typical ventilator consists of a compressed air source, valves to control the flow of air into and out of the lungs, and a "respiratory circuit" that connects the ventilator to the patient. In some cases, a sedated patient may be connected to the ventilator via a tube inserted through the trachea to their lungs, a process called invasive ventilation.
          A mechanical ventilator takes breaths for patients who are not fully capable of doing so on their own. In invasive ventilation, a controllable, compressed air source is connected to a sedated patient via tubing called a respiratory circuit.   
 In both invasive and non-invasive ventilation, the ventilator follows a clinician-prescribed breathing waveform based on a respiratory measurement from the patient (e.g., airway pressure, tidal volume). In order to prevent harm, this demanding task requires both robustness to differences or changes in patients' lungs and adherence to the desired waveform. Consequently, ventilators require significant attention from highly-trained clinicians in order to ensure that their performance matches the patients’ needs and that they do not cause lung damage.
         Example of a clinician-prescribed breathing waveform (orange) in units of airway pressure and the actual pressure (blue), given some controller algorithm.   
In “Machine Learning for Mechanical Ventilation Control”, we present exploratory research into the design of a deep learning–based algorithm to improve medical ventilator control for invasive ventilation. Using signals from an artificial lung, we design a control algorithm that measures airway pressure and computes necessary adjustments to the airflow to better and more consistently match prescribed values. Compared to other approaches, we demonstrate improved robustness and better performance while requiring less manual intervention from clinicians, which suggests that this approach could reduce the likelihood of harm to a patient’s lungs.
Current Methods 
Today, ventilators are controlled with methods belonging to the PID family (i.e., Proportional, Integral, Differential), which control a system based on the history of errors between the observed and desired states. A PID controller uses three characteristics for ventilator control: proportion (“P”) — a comparison of the measured and target pressure; integral (“I”) — the sum of previous measurements; and differential (“D”) — the difference between two previous measurements. Variants of PID have been used since the 17th century and today form the basis of many controllers in both industrial (e.g., controlling heat or fluids) and consumer (e.g., controlling espresso pressure) applications. 
PID control forms a solid baseline, relying on the sharp reactivity of P control to rapidly increase lung pressure when breathing in and the stability of I control to hold the breath in before exhaling. However, operators must tune the ventilator for specific patients, often repeatedly, to balance the “ringing” of overzealous P control against the ineffectually slow rise in lung pressure of dominant I control.
         Current PID methods are prone to over- and then under-shooting their target (ringing). Because patients differ in their physiology and may even change during treatment, highly-trained clinicians must constantly monitor and adjust existing methods to ensure such violent ringing as in the above example does not occur.   
To more effectively balance these characteristics, we propose a neural network–based controller to create a set of control signals that are more broad and adaptable than PID-generated controls.
A Machine-Learned Ventilator Controller 
While one could tune the coefficients of a PID controller (either manually or via an exhaustive grid search) through a limited number of repeated trials, it is impossible to apply such a direct approach towards a deep controller, as deep neural networks (DNNs) are often parameter-rich and require significant training data. Similarly, popular model-free approaches, such as Q-Learning or Policy Gradient, are data-intensive and therefore unsuitable for the physical system at hand. Further, these approaches don't take into account the intrinsic differentiability of the ventilator dynamical system, which is deterministic, continuous and contact-free. 
We therefore adopt a model-based approach, where we first learn a DNN-based simulator of the ventilator-patient dynamical system. An advantage of learning such a simulator is that it provides a more accurate data-driven alternative to physics-based models, and can be more widely distributed for controller research. 
To train a faithful simulator, we built a dataset by exploring the space of controls and the resulting pressures, while balancing against physical safety, e.g., not over-inflating a test lung and causing damage. Though PID control can exhibit ringing behavior, it performs well enough to use as a baseline for generating training data. To safely explore and to faithfully capture the behavior of the system, we use PID controllers with varied control coefficients to generate the control-pressure trajectory data for simulator training. Further, we add random deviations to the PID controllers to capture the dynamics more robustly.
We collect data for training by running mechanical ventilation tasks on a physical test lung using an open-source ventilator designed by Princeton University's People's Ventilator Project. We built a ventilator farm housing ten ventilator-lung systems on a server rack, which captures multiple airway resistance and compliance settings that span a spectrum of patient lung conditions, as required for practical applications of ventilator systems. 
         We use a rack-based ventilator farm (10 ventilators / artificial lungs) to collect training data for a ventilator-lung simulator. Using this simulator, we train a DNN controller that we then validate on the physical ventilator farm.   
The true underlying state of the dynamical system is not available to the model directly, but rather only through observations of the airway pressure in the system. In the simulator we model the state of the system at any time as a collection of previous pressure observations and the control actions applied to the system (up to a limited lookback window). These inputs are fed into a DNN that predicts the subsequent pressure in the system. We train this simulator on the control-pressure trajectory data collected through interactions with the test lung.
The performance of the simulator is measured via the sum of deviations of the simulator’s predictions (under self-simulation) from the ground truth.
         While it is infeasible to compare real dynamics with their simulated counterparts over all possible trajectories and control inputs, we measure the distance between simulation and the known safe trajectories. We introduce some random exploration around these safe trajectories for robustness.   
Having learned an accurate simulator, we then use it to train a DNN-based controller completely offline. This approach allows us to rapidly apply updates during controller training. Furthermore, the differentiable nature of the simulator allows for the stable use of the direct policy gradient, where we analytically compute the gradient of the loss with respect to the DNN parameters.  We find this method to be significantly more efficient than model-free approaches.  
Results 
To establish a baseline, we run an exhaustive grid of PID controllers for multiple lung settings and select the best performing PID controller as measured by average absolute deviation between the desired pressure waveform and the actual pressure waveform. We compare these to our controllers and provide evidence that our DNN controllers are better performing and more robust.
 Breathing waveform tracking performance:
 We compare the best PID controller for a given lung setting against our controller trained on the learned simulator for the same setting. Our learned controller shows a 22% lower mean absolute error (MAE) between target and actual pressure waveforms. 
         Comparison of the MAE between target and actual pressure waveforms (lower is better) for the best PID controller (orange) for a given lung setting (shown for two settings, R=5 and R=20) against our controller (blue) trained on the learned simulator for the same setting. The learned controller performs up to 22% better.   
  Robustness: 
 Further, we compare the performance of the single best PID controller across the entire set of lung settings with our controller trained on a set of learned simulators over the same settings. Our controller performs up to 32% better in MAE between target and actual pressure waveforms, suggesting that it could require less manual intervention between patients or even as a patient's condition changes.
         As above, but comparing the single best PID controller across the entire set of lung settings against our controller trained over the same settings. The learned controller performs up to 32% better, suggesting that it may require less manual intervention.   
 
Finally, we investigated the feasibility of using model-free and other popular RL algorithms (PPO, DQN), in comparison to a direct policy gradient trained on the simulator. We find that the simulator-trained direct policy gradient achieves slightly better scores and does so with a more stable training process that uses orders of magnitude fewer training samples and a significantly smaller hyperparameter search space.
         In the simulator, we find that model-free and other popular algorithms (PPO, DQN) perform approximately as well as our method.   
         However, these other methods take an order of magnitude more episodes to train to similar levels.   
Conclusions and the Road Forward 
 We have described a deep-learning approach to mechanical ventilation based on simulated dynamics learned from a physical test lung. However, this is only the beginning. To make an impact on real-world ventilators there are numerous other considerations and issues to take into account. Most important amongst them are non-invasive ventilators, which are significantly more challenging due to the difficulty of discerning pressure from lungs and mask pressure. Other directions are how to handle spontaneous breathing and coughing. To learn more and become involved in this important intersection of machine learning and health, see an ICML tutorial on control theory and learning, and consider participating in one of our kaggle competitions for creating better ventilator simulators!
Acknowledgements 
The primary work was based in the Google AI Princeton lab, in collaboration with Cohen lab at the Mechanical and Aerospace Engineering department at Princeton University. The research paper was authored by contributors from Google and Princeton University, including: Daniel Suo, Naman Agarwal, Wenhan Xia, Xinyi Chen, Udaya Ghai, Alexander Yu, Paula Gradu, Karan Singh, Cyril Zhang, Edgar Minasyan, Julienne LaChance, Tom Zajdel, Manuel Schottdorf, Daniel Cohen, and Elad Hazan.
Machine Learning for Mechanical Ventilation Control
https://ai.googleblog.com/2022/02/machine-learning-for-mechanical.html
https://ai.googleblog.com/2022/02/the-balloon-learning-environment.html
The Balloon Learning Environment
2022-02-17
None
<div><div>
<p>Posted by Joshua Greaves, Software Engineer and Pablo Samuel Castro, Staff Software Engineer, Google Research, Brain Team</p><p>
Benchmark challenges have been a driving force in the advancement of machine learning (ML). In particular, difficult benchmark environments for reinforcement learning (RL) have been crucial for the rapid progress of the field by challenging researchers to overcome increasingly difficult tasks. The <a href="https://github.com/mgbellemare/Arcade-Learning-Environment">Arcade Learning Environment</a>, <a href="https://mujoco.org/">Mujoco</a>, and others have been used to push the envelope in <a href="https://deepmind.com/research/publications/2019/human-level-control-through-deep-reinforcement-learning">RL algorithms</a>, <a href="https://arxiv.org/abs/2106.08229">representation learning</a>, <a href="https://arxiv.org/abs/2109.11052">exploration</a>, and more. 
</p><p>
In “<a href="https://www.nature.com/articles/s41586-020-2939-8.epdf?sharing_token=JYZ0ZlvEivoTq9RkGfWPQtRgN0jAjWel9jnR3ZoTv0Mh-6OgaxBwChMnw6EOI9v07nMOMJGBruSSDc8BFPfwkG1QQ0R-p9CwTuKA6ZO41aQ8e-Y-ffoWrsFX1cztOZfL5cL1mwXL8qU58Plz4GAzu_SLyawhPWS5QV6GieUEDig%3D">Autonomous Navigation of Stratospheric Balloons Using Reinforcement Learning</a>”, published in <i><a href="https://www.nature.com/articles/s41586-020-2939-8">Nature</a></i> two years ago, we demonstrated how deep RL can be used to create a high-performing flight agent that can control <a href="https://en.wikipedia.org/wiki/Superpressure_balloon">stratospheric balloons</a> in the real world. This research confirmed that deep RL can be successfully applied outside of simulated environments, and contributed practical knowledge for integrating RL algorithms with complex dynamical systems.</p>
 
 <p>
 Today we are excited to announce the open-source release of the <a href="https://github.com/google/balloon-learning-environment">Balloon Learning Environment</a> (BLE), a new benchmark emulating the real-world problem of controlling stratospheric balloons. The BLE is a high-fidelity simulator, which we hope will provide researchers with a valuable resource for deep RL research.
</p>
<h2>Station-Keeping Stratospheric Balloons</h2>
<p>
Stratospheric balloons are filled with a buoyant gas that allows them to float for weeks or months at a time in the <a href="https://en.wikipedia.org/wiki/Stratosphere">stratosphere</a>, about twice as high as a passenger plane’s cruising altitude. Though there are many potential variations of stratospheric balloons, the kind emulated in the BLE are equipped with solar panels and batteries, which allow them to adjust their altitude by controlling the weight of air in their <a href="https://en.wikipedia.org/wiki/Ballast">ballast</a> using an electric pump. However, they have no means to propel themselves laterally, which means that they are subject to wind patterns in the air around them. 
 
</p><table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjEPKIemxd-WVimieQoVwHabCUHiBp88K2e_Joe5MXlEQf_RwAlJda1YIPoqTxCOxUvL5iiYV0udbcPAgdKmUvXU8egfF5c3J5DHzFhxz6gaUM9lXHNS4UHweX9L-APwXvK4_pux7ouCehvRHrS4ztZjyqlPJqsIoaBtfG4iQjGdp2JuGdvHU8fHYQn9g=s800"></a></td></tr><tr><td>By changing its altitude, a stratospheric balloon can surf winds moving in different directions.</td></tr></tbody></table>
<p>
The goal of an agent in the BLE is to <em>station-keep</em> — i.e., to control a balloon to stay within 50km of a fixed ground station — by changing its altitude to catch winds that it finds favorable. We measure how successful an agent is at station-keeping by measuring the fraction of time the balloon is within the specified radius, denoted TWR50 (i.e., the time within a radius of 50km).
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEirECHWtUIX1wIvK5caa5kT3g8dxWcveIMyfFY8NZmhh4gkweNLxLp6YaxrmCgvyIy1oUWjjaB0MeWyLpow5KNt6AMjMFnQOYtjsON_JqS2Dxa-HyQIJlywmLI7oWWXH3rc8Yh8xiFtTUQQmdMaaxxWHivg-DqtirD3JJSslLutjDstyLuDBcZvMLyZJQ=s720"></a></td></tr><tr><td>A station-seeking balloon must navigate a changing wind field to stay above a ground station. <b>Left</b>: Side elevation of a station-keeping balloon. <b>Right</b>: Birds-eye-view of the same balloon.</td></tr></tbody></table>
<h2>The Challenges of Station-Keeping</h2>
<p>
To create a realistic simulator (without including copious amounts of historical wind data), the BLE uses a <a href="https://arxiv.org/abs/1312.6114">variational autoencoder</a> (VAE) trained on historical data to generate wind forecasts that match the characteristics of real winds. A <a href="https://github.com/lmas/opensimplex">wind noise model</a> is then used to make the windfields more realistic to match what a balloon would encounter in real-world conditions. <br />
</p>
<p>
Navigating a stratospheric balloon through a wind field can be quite challenging. The winds at any given altitude rarely remain ideal for long, and a good balloon controller will need to move up and down through its wind column to discover more suitable winds. In RL parlance, the problem of station-keeping is <em>partially observable </em>because the agent only has access to forecasted wind data to make those decisions. An agent has access to wind forecasts at every altitude and the true wind at its current altitude. The BLE returns an observation which includes a notion of wind uncertainty.
 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi2zsmxItZRuuvwAl_Lnt6SU6-C7RAkcjSUoGq0dJbKJSUGG5pHlsLFbQKH_O22v-bmNzxNuuVKGtxcIfOb9v16fNgu67o6o65gpV8edAFv4i98ywkY3gBMAJiLmP5yqag1FFXsTwAtuD1Hrgl7uQNRMg3T6M4RjfckDqHZi_gF7e3LJkY3Z4oIlkPItg=s2880"></a></td></tr><tr><td>A stratospheric balloon must explore winds at different altitudes in order to find favorable winds. The observation returned by the BLE includes wind predictions and a measure of uncertainty, made by mixing a wind forecast and winds measured at the balloon’s altitude.</td></tr></tbody></table>
<p>In some situations, there may not be suitable winds anywhere in the balloon’s wind column. In this case, an expert agent is still able to fly towards the station by taking a more circuitous route through the wind field (a common example is when the balloon moves in a zig-zag fashion, akin to tacking on a sailboat). Below we demonstrate that even just remaining in range of the station usually requires significant acrobatics.</p>
 
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjcJQAYD86mtWMc-smoyGPg8BXgjuYG2Savohq51h4wrD68SOT1OpIz1UNPbZkUCQ-JoU6DzT65L7RJIpPNYFh8_xdk5EzW0tgnO7Nio3wFfSiGNPgjLpY0Pfw78SsQn6aTNXieFlvtkXyyDhSHRerLlUsrA__bBrwjB6WuCq8nuppQwCSG4xcLwCDiSA=s721"></a></td></tr><tr><td>An agent must handle long planning horizons to succeed in station-keeping. In this case, StationSeeker (an expert-designed controller) heads directly to the center of the station-keeping area and is pushed out, while Perciatelli44 (an RL agent) is able to plan ahead and stay in range longer by hugging the edge of the area.</td></tr></tbody></table>
<p>
Night-time adds a fresh element of difficulty to station-keeping in the BLE, which reflects the reality of night-time changes in physical conditions and power availability. While during the day the air pump is powered by solar panels, at night the balloon relies on its on-board batteries for energy. Using too much power early in the night typically results in limited maneuverability in the hours preceding dawn. This is where RL agents can discover quite creative solutions — such as reducing altitude in the afternoon in order to store potential energy.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjt1M2tRCDoT6TRneWZpxWwqgAJam5YrviImnGb5xWJ7NqjuSse_jzQ6bY3SjJKoT3i09lp0uWhAjlkXoQTCY5_9wRWSZdxRDiNXVL0BpuUR2L-mDtX-XYFWXgW2tq2xgyJsacHWKBPlTq8RYlMs1U9SKsCFSQqwyAMMA0e-Jprf1ax-uEtwIypqm7F4A=s733"></a></td></tr><tr><td>An agent needs to balance the station-keeping objective with a finite energy allowance at night.</td></tr></tbody></table>
<p>
Despite all these challenges, our research demonstrates that agents trained with reinforcement learning can learn to perform better than expert-designed controllers at station-keeping. Along with the BLE, we are releasing the main agents from our research: Perciatelli44 (an RL agent) and StationSeeker (an expert-designed controller). The BLE can be used with any reinforcement learning library, and to showcase this we include <a href="https://github.com/google/dopamine">Dopamine’s</a> DQN and QR-DQN agents, as well as <a href="https://github.com/deepmind/acme">Acme’s</a> QR-DQN agent (supporting both standalone and distributed training with <a href="https://github.com/deepmind/launchpad">Launchpad</a>).
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiaOsjgYMR2iYq9LYgfGwfpho9A40kWcHQgCRsmFbDTR9XgVhXG05v2uqAT1aCaemijgHTmGexRtdCQgB_P9t5GPU5gbslnwObw41_aLKy7WQklGYqqo3lNiO3yBEVVirOhaFdMxePj1nNnDAaX3uYFUfWLte5okyJb-lxfEsBHQvochsL50iHCr_gBrQ=s717"></a></td></tr><tr><td>Evaluation performance by the included benchmark agents on the BLE. “Finetuned” is a fine-tuned Perciatelli44 agent, and Acme is a QR-DQN agent trained with the Acme library.</td></tr></tbody></table>
<p>
The <a href="https://github.com/google/balloon-learning-environment">BLE source code</a> contains information on how to get started with the BLE, including training and evaluating agents, documentation on the various components of the simulator, and example code. It also includes <a href="https://www.tensorflow.org/datasets/catalog/ble_wind_field">the historical windfield data</a> (as a <a href="https://www.tensorflow.org/datasets/api_docs/python/tfds">TensorFlow DataSet</a>) used to train the VAE to allow researchers to experiment with their own models for windfield generation. We are excited to see the progress that the community will make on this benchmark.
</p>
<h2>Acknowledgements</h2>
<p>
<i>We would like to thank the Balloon Learning Environment team: Sal Candido, Marc G. Bellemare, Vincent Dumoulin, Ross Goroshin, and Sam Ponda. We’d also like to thank Tom Small for his excellent animation in this blog post and graphic design help, along with our colleagues, Bradley Rhodes, Daniel Eisenberg, Piotr Staczyk, Anton Raichuk, Nikola Momchev, Geoff Hinton, Hugo Larochelle, and the rest of the Brain team in Montreal.</i>
</p>
</div>||||I||||Skip to main content
 The Balloon Learning Environment
 The Balloon Learning Environment
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 The Balloon Learning Environment

 Thursday, February 17, 2022

 Posted by Joshua Greaves, Software Engineer and Pablo Samuel Castro, Staff Software Engineer, Google Research, Brain Team

 Benchmark challenges have been a driving force in the advancement of machine learning (ML). In particular, difficult benchmark environments for reinforcement learning (RL) have been crucial for the rapid progress of the field by challenging researchers to overcome increasingly difficult tasks. The Arcade Learning Environment, Mujoco, and others have been used to push the envelope in RL algorithms, representation learning, exploration, and more.

 In “Autonomous Navigation of Stratospheric Balloons Using Reinforcement Learning”, published in Nature two years ago, we demonstrated how deep RL can be used to create a high-performing flight agent that can control stratospheric balloons in the real world. This research confirmed that deep RL can be successfully applied outside of simulated environments, and contributed practical knowledge for integrating RL algorithms with complex dynamical systems.

 Today we are excited to announce the open-source release of the Balloon Learning Environment (BLE), a new benchmark emulating the real-world problem of controlling stratospheric balloons. The BLE is a high-fidelity simulator, which we hope will provide researchers with a valuable resource for deep RL research.


 Station-Keeping Stratospheric Balloons

 Stratospheric balloons are filled with a buoyant gas that allows them to float for weeks or months at a time in the stratosphere, about twice as high as a passenger plane’s cruising altitude. Though there are many potential variations of stratospheric balloons, the kind emulated in the BLE are equipped with solar panels and batteries, which allow them to adjust their altitude by controlling the weight of air in their ballast using an electric pump. However, they have no means to propel themselves laterally, which means that they are subject to wind patterns in the air around them.

  
 By changing its altitude, a stratospheric balloon can surf winds moving in different directions.
 

 The goal of an agent in the BLE is to station-keep — i.e., to control a balloon to stay within 50km of a fixed ground station — by changing its altitude to catch winds that it finds favorable. We measure how successful an agent is at station-keeping by measuring the fraction of time the balloon is within the specified radius, denoted TWR50 (i.e., the time within a radius of 50km).

    
 A station-seeking balloon must navigate a changing wind field to stay above a ground station. Left: Side elevation of a station-keeping balloon. Right: Birds-eye-view of the same balloon.
 


 The Challenges of Station-Keeping

 To create a realistic simulator (without including copious amounts of historical wind data), the BLE uses a variational autoencoder (VAE) trained on historical data to generate wind forecasts that match the characteristics of real winds. A wind noise model is then used to make the windfields more realistic to match what a balloon would encounter in real-world conditions.

 Navigating a stratospheric balloon through a wind field can be quite challenging. The winds at any given altitude rarely remain ideal for long, and a good balloon controller will need to move up and down through its wind column to discover more suitable winds. In RL parlance, the problem of station-keeping is partially observable because the agent only has access to forecasted wind data to make those decisions. An agent has access to wind forecasts at every altitude and the true wind at its current altitude. The BLE returns an observation which includes a notion of wind uncertainty.

     
 A stratospheric balloon must explore winds at different altitudes in order to find favorable winds. The observation returned by the BLE includes wind predictions and a measure of uncertainty, made by mixing a wind forecast and winds measured at the balloon’s altitude.
 

 In some situations, there may not be suitable winds anywhere in the balloon’s wind column. In this case, an expert agent is still able to fly towards the station by taking a more circuitous route through the wind field (a common example is when the balloon moves in a zig-zag fashion, akin to tacking on a sailboat). Below we demonstrate that even just remaining in range of the station usually requires significant acrobatics.

      
 An agent must handle long planning horizons to succeed in station-keeping. In this case, StationSeeker (an expert-designed controller) heads directly to the center of the station-keeping area and is pushed out, while Perciatelli44 (an RL agent) is able to plan ahead and stay in range longer by hugging the edge of the area.
 

 Night-time adds a fresh element of difficulty to station-keeping in the BLE, which reflects the reality of night-time changes in physical conditions and power availability. While during the day the air pump is powered by solar panels, at night the balloon relies on its on-board batteries for energy. Using too much power early in the night typically results in limited maneuverability in the hours preceding dawn. This is where RL agents can discover quite creative solutions — such as reducing altitude in the afternoon in order to store potential energy.

   
 An agent needs to balance the station-keeping objective with a finite energy allowance at night.
 

 Despite all these challenges, our research demonstrates that agents trained with reinforcement learning can learn to perform better than expert-designed controllers at station-keeping. Along with the BLE, we are releasing the main agents from our research: Perciatelli44 (an RL agent) and StationSeeker (an expert-designed controller). The BLE can be used with any reinforcement learning library, and to showcase this we include Dopamine’s DQN and QR-DQN agents, as well as Acme’s QR-DQN agent (supporting both standalone and distributed training with Launchpad).

    
 Evaluation performance by the included benchmark agents on the BLE. “Finetuned” is a fine-tuned Perciatelli44 agent, and Acme is a QR-DQN agent trained with the Acme library.
 

 The BLE source code contains information on how to get started with the BLE, including training and evaluating agents, documentation on the various components of the simulator, and example code. It also includes the historical windfield data (as a TensorFlow DataSet) used to train the VAE to allow researchers to experiment with their own models for windfield generation. We are excited to see the progress that the community will make on this benchmark.


 Acknowledgements

 We would like to thank the Balloon Learning Environment team: Sal Candido, Marc G. Bellemare, Vincent Dumoulin, Ross Goroshin, and Sam Ponda. We’d also like to thank Tom Small for his excellent animation in this blog post and graphic design help, along with our colleagues, Bradley Rhodes, Daniel Eisenberg, Piotr Staczyk, Anton Raichuk, Nikola Momchev, Geoff Hinton, Hugo Larochelle, and the rest of the Brain team in Montreal.

 Labels: open source Reinforcement Learning
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Joshua Greaves, Software Engineer and Pablo Samuel Castro, Staff Software Engineer, Google Research, Brain Team 
Benchmark challenges have been a driving force in the advancement of machine learning (ML). In particular, difficult benchmark environments for reinforcement learning (RL) have been crucial for the rapid progress of the field by challenging researchers to overcome increasingly difficult tasks. The Arcade Learning Environment, Mujoco, and others have been used to push the envelope in RL algorithms, representation learning, exploration, and more. 
 
In “Autonomous Navigation of Stratospheric Balloons Using Reinforcement Learning”, published in  Nature  two years ago, we demonstrated how deep RL can be used to create a high-performing flight agent that can control stratospheric balloons in the real world. This research confirmed that deep RL can be successfully applied outside of simulated environments, and contributed practical knowledge for integrating RL algorithms with complex dynamical systems.
 
 
 Today we are excited to announce the open-source release of the Balloon Learning Environment (BLE), a new benchmark emulating the real-world problem of controlling stratospheric balloons. The BLE is a high-fidelity simulator, which we hope will provide researchers with a valuable resource for deep RL research.
Station-Keeping Stratospheric Balloons
Stratospheric balloons are filled with a buoyant gas that allows them to float for weeks or months at a time in the stratosphere, about twice as high as a passenger plane’s cruising altitude. Though there are many potential variations of stratospheric balloons, the kind emulated in the BLE are equipped with solar panels and batteries, which allow them to adjust their altitude by controlling the weight of air in their ballast using an electric pump. However, they have no means to propel themselves laterally, which means that they are subject to wind patterns in the air around them. 
 
          By changing its altitude, a stratospheric balloon can surf winds moving in different directions.   
The goal of an agent in the BLE is to station-keep — i.e., to control a balloon to stay within 50km of a fixed ground station — by changing its altitude to catch winds that it finds favorable. We measure how successful an agent is at station-keeping by measuring the fraction of time the balloon is within the specified radius, denoted TWR50 (i.e., the time within a radius of 50km).
         A station-seeking balloon must navigate a changing wind field to stay above a ground station. Left: Side elevation of a station-keeping balloon. Right: Birds-eye-view of the same balloon.   
The Challenges of Station-Keeping
To create a realistic simulator (without including copious amounts of historical wind data), the BLE uses a variational autoencoder (VAE) trained on historical data to generate wind forecasts that match the characteristics of real winds. A wind noise model is then used to make the windfields more realistic to match what a balloon would encounter in real-world conditions. 
Navigating a stratospheric balloon through a wind field can be quite challenging. The winds at any given altitude rarely remain ideal for long, and a good balloon controller will need to move up and down through its wind column to discover more suitable winds. In RL parlance, the problem of station-keeping is partially observable because the agent only has access to forecasted wind data to make those decisions. An agent has access to wind forecasts at every altitude and the true wind at its current altitude. The BLE returns an observation which includes a notion of wind uncertainty.
 
         A stratospheric balloon must explore winds at different altitudes in order to find favorable winds. The observation returned by the BLE includes wind predictions and a measure of uncertainty, made by mixing a wind forecast and winds measured at the balloon’s altitude.   
In some situations, there may not be suitable winds anywhere in the balloon’s wind column. In this case, an expert agent is still able to fly towards the station by taking a more circuitous route through the wind field (a common example is when the balloon moves in a zig-zag fashion, akin to tacking on a sailboat). Below we demonstrate that even just remaining in range of the station usually requires significant acrobatics.
 
         An agent must handle long planning horizons to succeed in station-keeping. In this case, StationSeeker (an expert-designed controller) heads directly to the center of the station-keeping area and is pushed out, while Perciatelli44 (an RL agent) is able to plan ahead and stay in range longer by hugging the edge of the area.   
Night-time adds a fresh element of difficulty to station-keeping in the BLE, which reflects the reality of night-time changes in physical conditions and power availability. While during the day the air pump is powered by solar panels, at night the balloon relies on its on-board batteries for energy. Using too much power early in the night typically results in limited maneuverability in the hours preceding dawn. This is where RL agents can discover quite creative solutions — such as reducing altitude in the afternoon in order to store potential energy.
         An agent needs to balance the station-keeping objective with a finite energy allowance at night.   
Despite all these challenges, our research demonstrates that agents trained with reinforcement learning can learn to perform better than expert-designed controllers at station-keeping. Along with the BLE, we are releasing the main agents from our research: Perciatelli44 (an RL agent) and StationSeeker (an expert-designed controller). The BLE can be used with any reinforcement learning library, and to showcase this we include Dopamine’s DQN and QR-DQN agents, as well as Acme’s QR-DQN agent (supporting both standalone and distributed training with Launchpad).
         Evaluation performance by the included benchmark agents on the BLE. “Finetuned” is a fine-tuned Perciatelli44 agent, and Acme is a QR-DQN agent trained with the Acme library.   
The BLE source code contains information on how to get started with the BLE, including training and evaluating agents, documentation on the various components of the simulator, and example code. It also includes the historical windfield data (as a TensorFlow DataSet) used to train the VAE to allow researchers to experiment with their own models for windfield generation. We are excited to see the progress that the community will make on this benchmark.
Acknowledgements
We would like to thank the Balloon Learning Environment team: Sal Candido, Marc G. Bellemare, Vincent Dumoulin, Ross Goroshin, and Sam Ponda. We’d also like to thank Tom Small for his excellent animation in this blog post and graphic design help, along with our colleagues, Bradley Rhodes, Daniel Eisenberg, Piotr Staczyk, Anton Raichuk, Nikola Momchev, Geoff Hinton, Hugo Larochelle, and the rest of the Brain team in Montreal.
The Balloon Learning Environment
https://ai.googleblog.com/2022/02/the-balloon-learning-environment.html
https://ai.googleblog.com/2022/02/good-news-about-carbon-footprint-of.html
Good News About the Carbon Footprint of Machine Learning Training
2022-02-15
David Patterson; Distinguished Engineer; Google Research; Brain Team
<div><div>
<p>Posted by David Patterson, Distinguished Engineer, Google Research, Brain Team
</p><p>
Machine learning (ML) has become prominent in information technology, which has led some to raise concerns about the associated rise in the costs of computation, primarily the <a href="https://en.wikipedia.org/wiki/Carbon_footprint">carbon footprint</a>, i.e., total <a href="https://en.wikipedia.org/wiki/Greenhouse_gas_emissions">greenhouse gas</a> emissions. While these assertions rightfully elevated the discussion around carbon emissions in ML, they also highlight the need for accurate data to assess true carbon footprint, which can help identify strategies to mitigate carbon emission in ML.<br />
</p><p>
In “<a href="https://www.techrxiv.org/articles/preprint/The_Carbon_Footprint_of_Machine_Learning_Training_Will_Plateau_Then_Shrink/19139645">The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink</a>”, accepted for publication in <i><a href="https://www.computer.org/csdl/magazine/co">IEEE Computer</a></i>, we focus on <i>operational carbon emissions</i> — i.e., the energy cost of operating ML hardware, including data center overheads — from training of natural language processing (NLP) models and investigate best practices that could reduce the carbon footprint. We demonstrate four key practices that reduce the carbon (and energy) footprint of ML workloads by large margins, which we have employed to help keep ML under 15% of Google’s total energy use.
</p>
<h2>The 4Ms: Best Practices to Reduce Energy and Carbon Footprints</h2>
<p>
We identified four best practices that reduce energy and carbon emissions significantly — we call these the “4Ms” — all of which are being used at Google today and are available to anyone using Google Cloud services. 
</p>
<ul>
<li><i>Model. </i>Selecting efficient ML model architectures, such as <a href="https://ai.googleblog.com/2021/03/constructing-transformers-for-longer.html">sparse models</a>, can advance ML quality while reducing computation by 3x–10x.
</li><li><i>Machine. </i>Using processors and systems optimized for ML training<em>, </em>versus general-purpose processors, can improve performance and energy efficiency by 2x–5x.
</li><li><i>Mechanization. </i>Computing in the Cloud rather than on premise reduces energy usage and therefore emissions by 1.4x–2x. Cloud-based data centers are new, <a href="https://www.morganclaypool.com/doi/10.2200/S00874ED3V01Y201809CAC046">custom-designed</a> warehouses equipped for energy efficiency for 50,000 servers, resulting in very good <a href="https://en.wikipedia.org/wiki/Power_usage_effectiveness">power usage effectiveness</a> (PUE). On-premise data centers are often older and smaller and thus cannot amortize the cost of new energy-efficient cooling and power distribution systems. 
</li><li><i>Map Optimization. </i>Moreover, the cloud lets customers pick the location with the cleanest energy, further reducing the gross carbon footprint by 5x–10x. While one might worry that map optimization could lead to the greenest locations quickly reaching maximum capacity, user demand for efficient data centers will result in continued advancement in green data center design and deployment.
</li>
</ul>
<p>
These four practices together can reduce energy by 100x and emissions by 1000x. 
</p>
<p>
Note that Google matches 100% of its operational energy use with <a href="https://cloud.google.com/blog/topics/sustainability/google-achieves-four-consecutive-years-of-100-percent-renewable-energy">renewable energy sources</a>. Conventional <a href="https://en.wikipedia.org/wiki/Carbon_offset">carbon offsets</a> are usually retrospective up to a year after the carbon emissions and can be purchased anywhere on the same continent. Google has <a href="https://sustainability.google/progress/projects/24x7/">committed to decarbonizing</a> all energy consumption so that by <a href="https://sustainability.google/commitments/">2030</a>, it will operate on 100% carbon-free energy, 24 hours a day on the same grid where the energy is consumed. <a href="https://cloud.google.com/blog/topics/sustainability/google-releases-carbon-free-energy-percentage-for-2020">Some Google data centers already operate on 90% carbon-free energy</a>; the overall average was <a href="https://cloud.google.com/blog/topics/sustainability/the-year-in-carbon-free-energy-at-google">61% carbon-free energy in 2019 and 67% in 2020</a>.
</p>
<p>
Below, we illustrate the impact of improving the 4Ms in practice. Other <a href="https://arxiv.org/abs/1906.02243">studies</a> examined training the <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)">Transformer model</a> on an <a href="https://www.nvidia.com/en-us/data-center/tesla-p100/">Nvidia P100 GPU</a> in an average data center and energy mix consistent with the worldwide average. The recently introduced <a href="https://arxiv.org/abs/2109.08668">Primer model </a>reduces the computation needed to achieve the same accuracy by 4x. Using newer-generation ML hardware, like <a href="https://cloud.google.com/blog/products/ai-machine-learning/google-wins-mlperf-benchmarks-with-tpu-v4">TPUv4,</a> provides an additional 14x improvement over the P100, or 57x overall. Efficient cloud data centers gain 1.4x over the average data center, resulting in a total energy reduction of 83x. In addition, using a data center with a low-carbon energy source can reduce the carbon footprint another 9x, resulting in a 747x total reduction in carbon footprint over four years. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjoYEFFC00-8KnaNLqVEicJbvZJAoNNC4sSPEagMziqfVmqoFYwJkDrHYRro13r6-bVD_kqE2FcWlA_INRQdULmqVfqJMx0yNfzi19oaYJMGCnrDxExHBV80xOp53gr8BmAQAO1jC1FG7WwSuLSTQ6JTgX0hO2LEOElOVLtdgBBdMGVV7IWvOBFB--MaA=s1644"></a></td></tr><tr><td>Reduction in gross carbon dioxide equivalent emissions (CO<sub>2</sub>e) from applying the 4M best practices to the Transformer model trained on P100 GPUs in an average data center in 2017, <a href="https://arxiv.org/abs/1906.02243">as done in other studies</a>. Displayed values are the cumulative improvement successively addressing each of the 4Ms: updating the model to Primer; upgrading the ML accelerator to TPUv4; using a Google data center with better PUE than average; and training in a Google Oklahoma data center that uses very clean energy.</td></tr></tbody></table>
<h2>Overall Energy Consumption for ML</h2>
<p>
Google’s total energy usage increases annually, which is not surprising considering increased use of its services. ML workloads have grown rapidly, as has the computation per training run, but paying attention to the 4Ms — <a href="https://ai.googleblog.com/2022/02/unlocking-full-potential-of-datacenter.html">optimized models</a>, <a href="https://www.gwern.net/docs/ai/2021-jouppi.pdf">ML-specific hardware</a>, <a href="https://www.morganclaypool.com/doi/pdf/10.2200/S00874ED3V01Y201809CAC046">efficient data centers</a> — has largely compensated for this increased load. Our data shows that ML training and inference are only 10%–15% of Google’s total energy use for each of the last three years, each year split ⅗ for inference and ⅖ for training.
</p>
<h2>Prior Emission Estimates</h2>
<p>
Google uses <a href="https://arxiv.org/abs/1611.01578">neural architecture search</a> (NAS) to find <a href="https://arxiv.org/abs/1901.11117">better ML models</a>. NAS is typically performed once per problem domain/search space combination, and the resulting model can then be reused for thousands of applications — e.g., the <a href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/evolved_transformer.py">Evolved Transformer model found by NAS is open sourced</a> for all to use. As the optimized model found by NAS is often <a href="https://arxiv.org/abs/2109.08668">more efficient</a>, the one time cost of NAS is typically more than offset by emission reductions from subsequent use. 
</p>
<p>
A <a href="https://arxiv.org/abs/1906.02243">study from the University of Massachusetts</a> (UMass) estimated carbon emissions for the Evolved Transformer NAS. 
</p>
<ul>
<li>Without ready access to Google hardware or data centers, the study extrapolated from the available P100 GPUs instead of TPUv2s, and assumed US average data center efficiency instead of highly efficient hyperscale data centers. These assumptions increased the estimate by 5x over the energy used by the actual NAS computation that was performed in Google’s data center.
</li><li>In order to accurately estimate the emissions for NAS, it's important to understand the subtleties of how they work. NAS systems use a much smaller proxy task to search for the most efficient models to save time, and then scale up the found models to full size. The UMass study assumed that the search repeated full size model training thousands of times, resulting in emission estimates that are another 18.7x too high.
</li>
</ul>
<p>
The overshoot for the NAS was 88x: 5x for energy-efficient hardware in Google data centers and 18.7x for computation using proxies. The actual CO<sub>2</sub>e for the one-time search were 3,223 kg versus 284,019 kg, 88x less than the published estimate.
</p>
<p>
Unfortunately, some subsequent papers misinterpreted the NAS estimate as the training cost for the model it discovered, yet emissions for this particular NAS are ~1300x larger than for training the model. These papers estimated that training the Evolved Transformer model takes <a href="https://arxiv.org/abs/2007.05558">two million GPU hours</a>, costs millions of dollars, and that its carbon emissions are equivalent to <a href="https://www.sciencedirect.com/science/article/pii/S2666389921001884">five times the lifetime emissions of a car</a>. In reality, training the Evolved Transformer model on the task examined by the UMass researchers and following the 4M best practices takes 120 TPUv2 hours, costs $40, and emits only 2.4 kg (0.00004 car lifetimes), 120,000x less. This gap is nearly as large as if one overestimated the CO<sub>2</sub>e to <i>manufacture</i> a car by 100x and then used that number as the CO<sub>2</sub>e for <i>driving</i> a car. 
</p>
<h2>Outlook</h2>
<p>
Climate change is important, so we must get the numbers right to ensure that we focus on solving the biggest challenges. Within information technology, we believe these are much more likely the lifecycle costs — i.e., emission estimates that include the embedded carbon emitted from manufacturing all components involved, from chips to data center buildings — of manufacturing computing equipment of all types and sizes<a href="#1"><sup>1</sup></a> rather than the operational cost of ML training. 
</p>
<p>
Expect more good news if everyone improves the 4Ms. While these numbers may currently vary across companies, these simple measures can be followed across the industry:
</p>
<ul>
<li><i>Data center providers</i> should publish <a href="https://www.google.com/about/datacenters/efficiency/">data center efficiency</a> and the <a href="https://cloud.google.com/blog/topics/sustainability/google-releases-carbon-free-energy-percentage-for-2020">cleanliness of the energy supply</a> per location so that customers can understand and reduce their energy consumption and carbon footprint. 
</li><li><i>ML practitioners</i> should train models using the best processors in the greenest data centers, which today are often in the Cloud.
</li><li><i>ML researchers</i> should continue to <a href="https://arxiv.org/abs/2109.08668">develop more efficient ML models</a>, for example, by <a href="https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html">leveraging sparsity</a> or by <a href="https://arxiv.org/abs/2112.04426">integrating retrieval to shrink the model</a>. They should also publish energy consumption and carbon footprint, both to foster competition beyond model quality and to ensure accurate accounting of their work, which is difficult to do accurately post-hoc.
</li>
</ul>
<p>
If the 4Ms become widely recognized, we predict a virtuous circle that will bend the curve so that the global carbon footprint of ML training is actually shrinking, not increasing.
</p>
<h2>Acknowledgements</h2>
<p>
<i>Let me thank my co-authors who stayed with this long and winding investigation into a topic that was new to most of us: Jeff Dean, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, and Maud Texier. We also had a great deal of help from others along the way for an <a href="https://arxiv.org/abs/2104.10350">earlier study</a> that eventually led to this version of the paper. Emma Strubell made several suggestions for the prior paper, including the recommendation to examine the recent giant NLP models. Christopher Berner, Ilya Sutskever, OpenAI, and Microsoft shared information about GPT-3. Dmitry Lepikhin and Zongwei Zhou did a great deal of work to measure the performance and power of GPUs and TPUs in Google data centers. Hallie Cramer, Anna Escuer, Elke Michlmayr, Kelli Wright, and Nick Zakrasek helped with the data and policies for energy and CO<sub>2</sub>e emissions at Google. </i>
</p>
<hr /><p><br />
 <a><sup>1</sup></a>Worldwide IT manufacturing for 2021 included 1700M cell phones, 340M PCs, and 12M data center servers. <a href="#top1">  <sup>↩</sup></a><br />
</p></div>||||I||||Skip to main content
 Good News About the Carbon Footprint of Machine Learning Training
 Good News About the Carbon Footprint of Machine Learning Training
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Good News About the Carbon Footprint of Machine Learning Training

 Tuesday, February 15, 2022

 Posted by David Patterson, Distinguished Engineer, Google Research, Brain Team

 Machine learning (ML) has become prominent in information technology, which has led some to raise concerns about the associated rise in the costs of computation, primarily the carbon footprint, i.e., total greenhouse gas emissions. While these assertions rightfully elevated the discussion around carbon emissions in ML, they also highlight the need for accurate data to assess true carbon footprint, which can help identify strategies to mitigate carbon emission in ML.

 In “The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink”, accepted for publication in IEEE Computer, we focus on operational carbon emissions — i.e., the energy cost of operating ML hardware, including data center overheads — from training of natural language processing (NLP) models and investigate best practices that could reduce the carbon footprint. We demonstrate four key practices that reduce the carbon (and energy) footprint of ML workloads by large margins, which we have employed to help keep ML under 15% of Google’s total energy use.


 The 4Ms: Best Practices to Reduce Energy and Carbon Footprints

 We identified four best practices that reduce energy and carbon emissions significantly — we call these the “4Ms” — all of which are being used at Google today and are available to anyone using Google Cloud services.

 * Model. Selecting efficient ML model architectures, such as sparse models, can advance ML quality while reducing computation by 3x–10x.
 * Machine. Using processors and systems optimized for ML training, versus general-purpose processors, can improve performance and energy efficiency by 2x–5x.
 * Mechanization. Computing in the Cloud rather than on premise reduces energy usage and therefore emissions by 1.4x–2x. Cloud-based data centers are new, custom-designed warehouses equipped for energy efficiency for 50,000 servers, resulting in very good power usage effectiveness (PUE). On-premise data centers are often older and smaller and thus cannot amortize the cost of new energy-efficient cooling and power distribution systems.
 * Map Optimization. Moreover, the cloud lets customers pick the location with the cleanest energy, further reducing the gross carbon footprint by 5x–10x. While one might worry that map optimization could lead to the greenest locations quickly reaching maximum capacity, user demand for efficient data centers will result in continued advancement in green data center design and deployment.

 These four practices together can reduce energy by 100x and emissions by 1000x.

 Note that Google matches 100% of its operational energy use with renewable energy sources. Conventional carbon offsets are usually retrospective up to a year after the carbon emissions and can be purchased anywhere on the same continent. Google has committed to decarbonizing all energy consumption so that by 2030, it will operate on 100% carbon-free energy, 24 hours a day on the same grid where the energy is consumed. Some Google data centers already operate on 90% carbon-free energy; the overall average was 61% carbon-free energy in 2019 and 67% in 2020.

 Below, we illustrate the impact of improving the 4Ms in practice. Other studies examined training the Transformer model on an Nvidia P100 GPU in an average data center and energy mix consistent with the worldwide average. The recently introduced Primer model reduces the computation needed to achieve the same accuracy by 4x. Using newer-generation ML hardware, like TPUv4, provides an additional 14x improvement over the P100, or 57x overall. Efficient cloud data centers gain 1.4x over the average data center, resulting in a total energy reduction of 83x. In addition, using a data center with a low-carbon energy source can reduce the carbon footprint another 9x, resulting in a 747x total reduction in carbon footprint over four years.

         
 Reduction in gross carbon dioxide equivalent emissions (CO2e) from applying the 4M best practices to the Transformer model trained on P100 GPUs in an average data center in 2017, as done in other studies. Displayed values are the cumulative improvement successively addressing each of the 4Ms: updating the model to Primer; upgrading the ML accelerator to TPUv4; using a Google data center with better PUE than average; and training in a Google Oklahoma data center that uses very clean energy.
 


 Overall Energy Consumption for ML

 Google’s total energy usage increases annually, which is not surprising considering increased use of its services. ML workloads have grown rapidly, as has the computation per training run, but paying attention to the 4Ms — optimized models, ML-specific hardware, efficient data centers — has largely compensated for this increased load. Our data shows that ML training and inference are only 10%–15% of Google’s total energy use for each of the last three years, each year split ⅗ for inference and ⅖ for training.


 Prior Emission Estimates

 Google uses neural architecture search (NAS) to find better ML models. NAS is typically performed once per problem domain/search space combination, and the resulting model can then be reused for thousands of applications — e.g., the Evolved Transformer model found by NAS is open sourced for all to use. As the optimized model found by NAS is often more efficient, the one time cost of NAS is typically more than offset by emission reductions from subsequent use.

 A study from the University of Massachusetts (UMass) estimated carbon emissions for the Evolved Transformer NAS.

 * Without ready access to Google hardware or data centers, the study extrapolated from the available P100 GPUs instead of TPUv2s, and assumed US average data center efficiency instead of highly efficient hyperscale data centers. These assumptions increased the estimate by 5x over the energy used by the actual NAS computation that was performed in Google’s data center.
 * In order to accurately estimate the emissions for NAS, it's important to understand the subtleties of how they work. NAS systems use a much smaller proxy task to search for the most efficient models to save time, and then scale up the found models to full size. The UMass study assumed that the search repeated full size model training thousands of times, resulting in emission estimates that are another 18.7x too high.

 The overshoot for the NAS was 88x: 5x for energy-efficient hardware in Google data centers and 18.7x for computation using proxies. The actual CO2e for the one-time search were 3,223 kg versus 284,019 kg, 88x less than the published estimate.

 Unfortunately, some subsequent papers misinterpreted the NAS estimate as the training cost for the model it discovered, yet emissions for this particular NAS are ~1300x larger than for training the model. These papers estimated that training the Evolved Transformer model takes two million GPU hours, costs millions of dollars, and that its carbon emissions are equivalent to five times the lifetime emissions of a car. In reality, training the Evolved Transformer model on the task examined by the UMass researchers and following the 4M best practices takes 120 TPUv2 hours, costs $40, and emits only 2.4 kg (0.00004 car lifetimes), 120,000x less. This gap is nearly as large as if one overestimated the CO2e to manufacture a car by 100x and then used that number as the CO2e for driving a car.


 Outlook

 Climate change is important, so we must get the numbers right to ensure that we focus on solving the biggest challenges. Within information technology, we believe these are much more likely the lifecycle costs — i.e., emission estimates that include the embedded carbon emitted from manufacturing all components involved, from chips to data center buildings — of manufacturing computing equipment of all types and sizes1 rather than the operational cost of ML training.

 Expect more good news if everyone improves the 4Ms. While these numbers may currently vary across companies, these simple measures can be followed across the industry:

 * Data center providers should publish data center efficiency and the cleanliness of the energy supply per location so that customers can understand and reduce their energy consumption and carbon footprint.
 * ML practitioners should train models using the best processors in the greenest data centers, which today are often in the Cloud.
 * ML researchers should continue to develop more efficient ML models, for example, by leveraging sparsity or by integrating retrieval to shrink the model. They should also publish energy consumption and carbon footprint, both to foster competition beyond model quality and to ensure accurate accounting of their work, which is difficult to do accurately post-hoc.

 If the 4Ms become widely recognized, we predict a virtuous circle that will bend the curve so that the global carbon footprint of ML training is actually shrinking, not increasing.


 Acknowledgements

 Let me thank my co-authors who stayed with this long and winding investigation into a topic that was new to most of us: Jeff Dean, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, and Maud Texier. We also had a great deal of help from others along the way for an earlier study that eventually led to this version of the paper. Emma Strubell made several suggestions for the prior paper, including the recommendation to examine the recent giant NLP models. Christopher Berner, Ilya Sutskever, OpenAI, and Microsoft shared information about GPT-3. Dmitry Lepikhin and Zongwei Zhou did a great deal of work to measure the performance and power of GPUs and TPUs in Google data centers. Hallie Cramer, Anna Escuer, Elke Michlmayr, Kelli Wright, and Nick Zakrasek helped with the data and policies for energy and CO2e emissions at Google.


 1 Worldwide IT manufacturing for 2021 included 1700M cell phones, 340M PCs, and 12M data center servers. ↩
 Labels: Cloud Computing Machine Learning renewable energy TPU
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by David Patterson, Distinguished Engineer, Google Research, Brain Team
 
Machine learning (ML) has become prominent in information technology, which has led some to raise concerns about the associated rise in the costs of computation, primarily the carbon footprint, i.e., total greenhouse gas emissions. While these assertions rightfully elevated the discussion around carbon emissions in ML, they also highlight the need for accurate data to assess true carbon footprint, which can help identify strategies to mitigate carbon emission in ML.
 
In “The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink”, accepted for publication in  IEEE Computer , we focus on operational carbon emissions — i.e., the energy cost of operating ML hardware, including data center overheads — from training of natural language processing (NLP) models and investigate best practices that could reduce the carbon footprint. We demonstrate four key practices that reduce the carbon (and energy) footprint of ML workloads by large margins, which we have employed to help keep ML under 15% of Google’s total energy use.
The 4Ms: Best Practices to Reduce Energy and Carbon Footprints
We identified four best practices that reduce energy and carbon emissions significantly — we call these the “4Ms” — all of which are being used at Google today and are available to anyone using Google Cloud services. 
 Model. Selecting efficient ML model architectures, such as sparse models, can advance ML quality while reducing computation by 3x–10x.
  Machine. Using processors and systems optimized for ML training, versus general-purpose processors, can improve performance and energy efficiency by 2x–5x.
  Mechanization. Computing in the Cloud rather than on premise reduces energy usage and therefore emissions by 1.4x–2x. Cloud-based data centers are new, custom-designed warehouses equipped for energy efficiency for 50,000 servers, resulting in very good power usage effectiveness (PUE). On-premise data centers are often older and smaller and thus cannot amortize the cost of new energy-efficient cooling and power distribution systems. 
  Map Optimization. Moreover, the cloud lets customers pick the location with the cleanest energy, further reducing the gross carbon footprint by 5x–10x. While one might worry that map optimization could lead to the greenest locations quickly reaching maximum capacity, user demand for efficient data centers will result in continued advancement in green data center design and deployment.
These four practices together can reduce energy by 100x and emissions by 1000x. 
Note that Google matches 100% of its operational energy use with renewable energy sources. Conventional carbon offsets are usually retrospective up to a year after the carbon emissions and can be purchased anywhere on the same continent. Google has committed to decarbonizing all energy consumption so that by 2030, it will operate on 100% carbon-free energy, 24 hours a day on the same grid where the energy is consumed. Some Google data centers already operate on 90% carbon-free energy; the overall average was 61% carbon-free energy in 2019 and 67% in 2020.
Below, we illustrate the impact of improving the 4Ms in practice. Other studies examined training the Transformer model on an Nvidia P100 GPU in an average data center and energy mix consistent with the worldwide average. The recently introduced Primer model reduces the computation needed to achieve the same accuracy by 4x. Using newer-generation ML hardware, like TPUv4, provides an additional 14x improvement over the P100, or 57x overall. Efficient cloud data centers gain 1.4x over the average data center, resulting in a total energy reduction of 83x. In addition, using a data center with a low-carbon energy source can reduce the carbon footprint another 9x, resulting in a 747x total reduction in carbon footprint over four years. 
         Reduction in gross carbon dioxide equivalent emissions (CO2e) from applying the 4M best practices to the Transformer model trained on P100 GPUs in an average data center in 2017, as done in other studies. Displayed values are the cumulative improvement successively addressing each of the 4Ms: updating the model to Primer; upgrading the ML accelerator to TPUv4; using a Google data center with better PUE than average; and training in a Google Oklahoma data center that uses very clean energy.   
Overall Energy Consumption for ML
Google’s total energy usage increases annually, which is not surprising considering increased use of its services. ML workloads have grown rapidly, as has the computation per training run, but paying attention to the 4Ms — optimized models, ML-specific hardware, efficient data centers — has largely compensated for this increased load. Our data shows that ML training and inference are only 10%–15% of Google’s total energy use for each of the last three years, each year split ⅗ for inference and ⅖ for training.
Prior Emission Estimates
Google uses neural architecture search (NAS) to find better ML models. NAS is typically performed once per problem domain/search space combination, and the resulting model can then be reused for thousands of applications — e.g., the Evolved Transformer model found by NAS is open sourced for all to use. As the optimized model found by NAS is often more efficient, the one time cost of NAS is typically more than offset by emission reductions from subsequent use. 
A study from the University of Massachusetts (UMass) estimated carbon emissions for the Evolved Transformer NAS. 
Without ready access to Google hardware or data centers, the study extrapolated from the available P100 GPUs instead of TPUv2s, and assumed US average data center efficiency instead of highly efficient hyperscale data centers. These assumptions increased the estimate by 5x over the energy used by the actual NAS computation that was performed in Google’s data center.
 In order to accurately estimate the emissions for NAS, it's important to understand the subtleties of how they work. NAS systems use a much smaller proxy task to search for the most efficient models to save time, and then scale up the found models to full size. The UMass study assumed that the search repeated full size model training thousands of times, resulting in emission estimates that are another 18.7x too high.
The overshoot for the NAS was 88x: 5x for energy-efficient hardware in Google data centers and 18.7x for computation using proxies. The actual CO2e for the one-time search were 3,223 kg versus 284,019 kg, 88x less than the published estimate.
Unfortunately, some subsequent papers misinterpreted the NAS estimate as the training cost for the model it discovered, yet emissions for this particular NAS are ~1300x larger than for training the model. These papers estimated that training the Evolved Transformer model takes two million GPU hours, costs millions of dollars, and that its carbon emissions are equivalent to five times the lifetime emissions of a car. In reality, training the Evolved Transformer model on the task examined by the UMass researchers and following the 4M best practices takes 120 TPUv2 hours, costs $40, and emits only 2.4 kg (0.00004 car lifetimes), 120,000x less. This gap is nearly as large as if one overestimated the CO2e to manufacture a car by 100x and then used that number as the CO2e for driving a car. 
Outlook
Climate change is important, so we must get the numbers right to ensure that we focus on solving the biggest challenges. Within information technology, we believe these are much more likely the lifecycle costs — i.e., emission estimates that include the embedded carbon emitted from manufacturing all components involved, from chips to data center buildings — of manufacturing computing equipment of all types and sizes 1  rather than the operational cost of ML training. 
Expect more good news if everyone improves the 4Ms. While these numbers may currently vary across companies, these simple measures can be followed across the industry:
 Data center providers should publish data center efficiency and the cleanliness of the energy supply per location so that customers can understand and reduce their energy consumption and carbon footprint. 
  ML practitioners should train models using the best processors in the greenest data centers, which today are often in the Cloud.
  ML researchers should continue to develop more efficient ML models, for example, by leveraging sparsity or by integrating retrieval to shrink the model. They should also publish energy consumption and carbon footprint, both to foster competition beyond model quality and to ensure accurate accounting of their work, which is difficult to do accurately post-hoc.
If the 4Ms become widely recognized, we predict a virtuous circle that will bend the curve so that the global carbon footprint of ML training is actually shrinking, not increasing.
Acknowledgements
Let me thank my co-authors who stayed with this long and winding investigation into a topic that was new to most of us: Jeff Dean, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, and Maud Texier. We also had a great deal of help from others along the way for an earlier study that eventually led to this version of the paper. Emma Strubell made several suggestions for the prior paper, including the recommendation to examine the recent giant NLP models. Christopher Berner, Ilya Sutskever, OpenAI, and Microsoft shared information about GPT-3. Dmitry Lepikhin and Zongwei Zhou did a great deal of work to measure the performance and power of GPUs and TPUs in Google data centers. Hallie Cramer, Anna Escuer, Elke Michlmayr, Kelli Wright, and Nick Zakrasek helped with the data and policies for energy and CO2e emissions at Google. 
  
  1 Worldwide IT manufacturing for 2021 included 1700M cell phones, 340M PCs, and 12M data center servers.   ↩
Good News About the Carbon Footprint of Machine Learning Training
https://ai.googleblog.com/2022/02/good-news-about-carbon-footprint-of.html
https://ai.googleblog.com/2022/02/an-international-scientific-challenge.html
An International Scientific Challenge for the Diagnosis and Gleason Grading of Prostate Cancer
2022-02-11
Po-Hsuan Cameron Chen; Software Engineer; Google Health; Maggie Demkin; Program Manager; Kaggle
<div><div>
<p>Posted by Po-Hsuan Cameron Chen, Software Engineer, Google Health and Maggie Demkin, Program Manager, Kaggle</p><p>
In recent years, machine learning (ML) competitions in health have attracted ML scientists to work together to solve challenging clinical problems. These competitions provide access to relevant data and well-defined problems where experienced data scientists come to compete for solutions and learn new methods. However, a fundamental difficulty in organizing such challenges is obtaining and curating high quality datasets for model development and independent datasets for model evaluation. Importantly, to reduce the risk of bias and to ensure broad applicability of the algorithm, evaluation of the generalisability of resulting algorithms should ideally be performed on multiple independent evaluation datasets by an independent group of scientists. 
</p><p>
One clinical problem that has attracted <a href="https://blog.google/technology/health/promise-using-ai-help-prostate-cancer-care/">substantial ML research is prostate cancer</a>, a condition that 1 in 9 men develop in their lifetime. A prostate cancer diagnosis requires pathologists to examine biological tissue samples under a microscope to identify cancer and grade the cancer for signs of aggressive growth patterns in the cells. However, this cancer grading task (called <a href="https://en.wikipedia.org/wiki/Gleason_grading_system">Gleason grading</a>) is difficult and subjective due to the need for visual assessment of cell differentiation and Gleason pattern predominance. Building a large dataset of samples with expert annotations can help with the development of ML systems to aid in prostate cancer grading.
</p>
<p>
To help accelerate and enable more research in this area, Google Health, <a href="https://www.radboudumc.nl/en/research">Radboud University Medical Center</a> and <a href="https://ki.se/en">Karolinska Institutet</a> joined forces to organize a global competition, the <a href="https://www.kaggle.com/c/prostate-cancer-grade-assessment/overview">Prostate cANcer graDe Assessment (PANDA) Challenge</a>, on the open <a href="https://www.kaggle.com/c/prostate-cancer-grade-assessment">Kaggle platform</a>. In “<a href="https://www.nature.com/articles/s41591-021-01620-2">Artificial Intelligence for Diagnosis and Gleason Grading of Prostate Cancer: the PANDA challenge</a>”, published in <a href="https://www.nature.com/nm/">Nature Medicine</a>, we present the results of the challenge. The <a href="https://zenodo.org/record/3715938#.Yf11GPXML6U">study design</a> of the PANDA challenge provided the largest public whole-slide image dataset available and was open to participants from April 21st until July 23rd, 2020. The development datasets <a href="https://www.kaggle.com/c/prostate-cancer-grade-assessment/data">remain available</a> for further research. In this effort, we compiled and publicly released a European cohort of prostate cancer cases for algorithm development and pioneered a standardized evaluation setup for digital pathology that enabled independent, blinded external validation of the algorithms on data from both the United States and EU.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgjKeYkFZ3OYRfbIWiRc3Pyg7SutaZw_U-9FSwaur3vTGB61kI928zrUZqGRmFbiglhM5he1UnVvrb1P8mqBbRspHJYKrh58nlF39MVLY7GRfqjQNkdAFQWpoUg_g7FaXMiFZuhFZj8zv6Yz3EFiaIEYbVn19xCLOVMpYPclqLkiURUrhUEFimXtxzeZw=s4930"></a></td></tr><tr><td>The global competition attracted participants from 65 countries (the size of the circle for each country illustrates the number of participants).</td></tr></tbody></table>
<h2>Design of the Panda Challenge</h2><p>
The challenge had two phases: a development phase (i.e., the Kaggle competition) and a validation phase. During the competition, 1,290 developers from 65 countries competed in building the best performing Gleason grading algorithm, having full access to a development set for algorithm training. Throughout the competition teams submitted algorithms that were evaluated on a hidden tuning set.
</p>
<p>
In the validation phase, a selection of top performing algorithms were independently evaluated on internal and external validation datasets with <a href="https://www.thelancet.com/journals/landig/article/PIIS2589-7500(21)00216-8/fulltext">high quality reference grades from panels</a> of expert prostate pathologists. In addition, a group of general pathologists graded a subset of the same cases to put the difficulty of the task and dataset in context. The algorithms submitted by the teams were then compared to grades done by groups of international and US general pathologists on these subsets. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj0UN1OMlFYWkmDvocDSDKaUb3wFqJ0G4ocRjlB2M6e4GMxrpTGalzCqcmMw-VH_X9dj60VatUuUFQxFmd1cdh-1_PZZWIVyCwKFjA_tL3xcHun2yB8xaWeWcewWKDBVgxtEeaLTJMxbd-FveIJ6jkEzR4BkHeU6bRrfT29jMELTf90Ap3uLlGLNUpDaA=s4866"></a></td></tr><tr><td>Overview of the PANDA challenge’s phases for development and validation.</td></tr></tbody></table>
<h2>Research Velocity During the Challenge</h2><p>
We found that a group of Gleason grading ML algorithms developed during a global competition could achieve pathologist-level performance and generalize well to intercontinental and multinational cohorts. On all external validation sets, these algorithms achieved high agreement with urologic pathologists (prostate specialists) and high sensitivity for detecting tumor in biopsies. The Kaggle platform enabled the tracking of teams’ performance throughout the competition. Impressively, the first team achieving high agreement with the prostate pathologists at above 0.90 (quadratically weighted <a href="https://en.wikipedia.org/wiki/Cohen%27s_kappa">Cohen’s kappa</a>) on the internal validation set occurred within the <em>first 10 days</em> of the competition. By the 33rd day, the median performance of all teams exceeded a score of 0.85. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjDHD4kmt7SgLuWnp4kG-d9-DASL3fkqkLefpgamXrb0imhg0PJNYZRN_rRqEogGsytyGI6t-m4slo9rgAYxbVw9GDXKc31N54EjiF3hJmxXoyVdMZV6hgengVGTDHCKHvGQOJBAB6OU_Xszr4rDAWy10ZL9p_g4Z5CFKTO0UANqu2WwcHmL2HzgoD7lw=s1907"></a></td></tr><tr><td>Progression of algorithms’ performances throughout the competition, as shown by the highest score on the tuning and internal validation sets among all participating teams. During the competition teams could submit their algorithm for evaluation on the tuning set, after which they received their score. At the same time, algorithms were evaluated on the internal validation set, without disclosing these results to the participating teams. The development of the top score obtained by any team shows the rapid improvement of the algorithms.</td></tr></tbody></table>
<h2>Learning from the Challenge</h2><p>
By moderating the <a href="https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion">discussion forum</a> on the Kaggle platform, we learned that the teams’ openness in <a href="https://www.kaggle.com/c/prostate-cancer-grade-assessment/code">sharing code</a> via <a href="https://research.google.com/colaboratory/faq.html">colab notebooks</a> led to rapid improvement across the board, a promising sign for future public challenges, and a clear indication of the power of sharing knowledge on a common platform. 
</p>
<p>
Organizing a public challenge that evaluates algorithm generalization across independent cohorts using high quality reference standard panels presents substantial logistical difficulties. Assembling this size of a dataset across countries and organizations was a massive undertaking. This work benefited from an amazing collaboration between the three organizing institutions which have all contributed respective publications in this space, two in <a href="https://www.thelancet.com/journals/lanonc/article/PIIS1470-2045(19)30739-9/fulltext">Lancet</a> <a href="https://www.thelancet.com/journals/lanonc/article/PIIS1470-2045(19)30738-7/fulltext">Oncology</a> and one in <a href="https://jamanetwork.com/journals/jamaoncology/fullarticle/2768225">JAMA Oncology</a>. Combining these efforts provided a high quality foundation on which this competition could be based. With the publication, Radboud and Karolinska research groups are also open sourcing the <a href="https://www.kaggle.com/c/prostate-cancer-grade-assessment/data">PANDA challenge development datasets</a> to facilitate the further improvement of prostate Gleason grading algorithms. We look forward to seeing many more advancements in this field, and more challenges that can catalyze extensive international knowledge sharing and collaborative research. 
</p>
<h2>Acknowledgements</h2><p>
<em>Key contributors to this project at Google include Po-Hsuan Cameron Chen, Kunal Nagpal, Yuannan Cai, David F. Steiner, Maggie Demkin, Sohier Dane, Fraser Tan, Greg S. Corrado, Lily Peng, Craig H. Mermel. Collaborators on this project include Wouter Bulten, Kimmo Kartasalo, Peter Ström, Hans Pinckaers, Hester van Boven, Robert Vink, Christina Hulsbergen-van de Kaa, Jeroen van der Laak, Mahul B. Amin, Andrew J. Evans, Theodorus van der Kwast, Robert Allan, Peter A. Humphrey, Henrik Grönberg, Hemamali Samaratunga, Brett Delahunt, Toyonori Tsuzuki, Tomi Häkkinen, Lars Egevad, Masi Valkonen, Pekka Ruusuvuori, Geert Litjens, Martin Eklund and the PANDA Challenge consortium. We thank Ellery Wulczyn, Annisah Um'rani, Yun Liu, and Dale Webster for their feedback on the manuscript and guidance on the project. We thank our collaborators at NMCSD, particularly Niels Olson, for internal re-use of de-identified data which contributed to the US external validation set. Sincere appreciation also goes to Sami Lachgar, Ashley Zlatinov, and Lauren Winer for their feedback on the blogpost.</em>
</p>
</div>||||I||||Skip to main content
 An International Scientific Challenge for the Diagnosis and Gleason Grading of Prostate Cancer
 An International Scientific Challenge for the Diagnosis and Gleason Grading of Prostate Cancer
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 An International Scientific Challenge for the Diagnosis and Gleason Grading of Prostate Cancer

 Friday, February 11, 2022

 Posted by Po-Hsuan Cameron Chen, Software Engineer, Google Health and Maggie Demkin, Program Manager, Kaggle

 In recent years, machine learning (ML) competitions in health have attracted ML scientists to work together to solve challenging clinical problems. These competitions provide access to relevant data and well-defined problems where experienced data scientists come to compete for solutions and learn new methods. However, a fundamental difficulty in organizing such challenges is obtaining and curating high quality datasets for model development and independent datasets for model evaluation. Importantly, to reduce the risk of bias and to ensure broad applicability of the algorithm, evaluation of the generalisability of resulting algorithms should ideally be performed on multiple independent evaluation datasets by an independent group of scientists.

 One clinical problem that has attracted substantial ML research is prostate cancer, a condition that 1 in 9 men develop in their lifetime. A prostate cancer diagnosis requires pathologists to examine biological tissue samples under a microscope to identify cancer and grade the cancer for signs of aggressive growth patterns in the cells. However, this cancer grading task (called Gleason grading) is difficult and subjective due to the need for visual assessment of cell differentiation and Gleason pattern predominance. Building a large dataset of samples with expert annotations can help with the development of ML systems to aid in prostate cancer grading.

 To help accelerate and enable more research in this area, Google Health, Radboud University Medical Center and Karolinska Institutet joined forces to organize a global competition, the Prostate cANcer graDe Assessment (PANDA) Challenge, on the open Kaggle platform. In “Artificial Intelligence for Diagnosis and Gleason Grading of Prostate Cancer: the PANDA challenge”, published in Nature Medicine, we present the results of the challenge. The study design of the PANDA challenge provided the largest public whole-slide image dataset available and was open to participants from April 21st until July 23rd, 2020. The development datasets remain available for further research. In this effort, we compiled and publicly released a European cohort of prostate cancer cases for algorithm development and pioneered a standardized evaluation setup for digital pathology that enabled independent, blinded external validation of the algorithms on data from both the United States and EU.

   
 The global competition attracted participants from 65 countries (the size of the circle for each country illustrates the number of participants).
 


 Design of the Panda Challenge

 The challenge had two phases: a development phase (i.e., the Kaggle competition) and a validation phase. During the competition, 1,290 developers from 65 countries competed in building the best performing Gleason grading algorithm, having full access to a development set for algorithm training. Throughout the competition teams submitted algorithms that were evaluated on a hidden tuning set.

 In the validation phase, a selection of top performing algorithms were independently evaluated on internal and external validation datasets with high quality reference grades from panels of expert prostate pathologists. In addition, a group of general pathologists graded a subset of the same cases to put the difficulty of the task and dataset in context. The algorithms submitted by the teams were then compared to grades done by groups of international and US general pathologists on these subsets.

  
 Overview of the PANDA challenge’s phases for development and validation.
 


 Research Velocity During the Challenge

 We found that a group of Gleason grading ML algorithms developed during a global competition could achieve pathologist-level performance and generalize well to intercontinental and multinational cohorts. On all external validation sets, these algorithms achieved high agreement with urologic pathologists (prostate specialists) and high sensitivity for detecting tumor in biopsies. The Kaggle platform enabled the tracking of teams’ performance throughout the competition. Impressively, the first team achieving high agreement with the prostate pathologists at above 0.90 (quadratically weighted Cohen’s kappa) on the internal validation set occurred within the first 10 days of the competition. By the 33rd day, the median performance of all teams exceeded a score of 0.85.

         
 Progression of algorithms’ performances throughout the competition, as shown by the highest score on the tuning and internal validation sets among all participating teams. During the competition teams could submit their algorithm for evaluation on the tuning set, after which they received their score. At the same time, algorithms were evaluated on the internal validation set, without disclosing these results to the participating teams. The development of the top score obtained by any team shows the rapid improvement of the algorithms.
 


 Learning from the Challenge

 By moderating the discussion forum on the Kaggle platform, we learned that the teams’ openness in sharing code via colab notebooks led to rapid improvement across the board, a promising sign for future public challenges, and a clear indication of the power of sharing knowledge on a common platform.

 Organizing a public challenge that evaluates algorithm generalization across independent cohorts using high quality reference standard panels presents substantial logistical difficulties. Assembling this size of a dataset across countries and organizations was a massive undertaking. This work benefited from an amazing collaboration between the three organizing institutions which have all contributed respective publications in this space, two in Lancet Oncology and one in JAMA Oncology. Combining these efforts provided a high quality foundation on which this competition could be based. With the publication, Radboud and Karolinska research groups are also open sourcing the PANDA challenge development datasets to facilitate the further improvement of prostate Gleason grading algorithms. We look forward to seeing many more advancements in this field, and more challenges that can catalyze extensive international knowledge sharing and collaborative research.


 Acknowledgements

 Key contributors to this project at Google include Po-Hsuan Cameron Chen, Kunal Nagpal, Yuannan Cai, David F. Steiner, Maggie Demkin, Sohier Dane, Fraser Tan, Greg S. Corrado, Lily Peng, Craig H. Mermel. Collaborators on this project include Wouter Bulten, Kimmo Kartasalo, Peter Ström, Hans Pinckaers, Hester van Boven, Robert Vink, Christina Hulsbergen-van de Kaa, Jeroen van der Laak, Mahul B. Amin, Andrew J. Evans, Theodorus van der Kwast, Robert Allan, Peter A. Humphrey, Henrik Grönberg, Hemamali Samaratunga, Brett Delahunt, Toyonori Tsuzuki, Tomi Häkkinen, Lars Egevad, Masi Valkonen, Pekka Ruusuvuori, Geert Litjens, Martin Eklund and the PANDA Challenge consortium. We thank Ellery Wulczyn, Annisah Um'rani, Yun Liu, and Dale Webster for their feedback on the manuscript and guidance on the project. We thank our collaborators at NMCSD, particularly Niels Olson, for internal re-use of de-identified data which contributed to the US external validation set. Sincere appreciation also goes to Sami Lachgar, Ashley Zlatinov, and Lauren Winer for their feedback on the blogpost.

 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Po-Hsuan Cameron Chen, Software Engineer, Google Health and Maggie Demkin, Program Manager, Kaggle 
In recent years, machine learning (ML) competitions in health have attracted ML scientists to work together to solve challenging clinical problems. These competitions provide access to relevant data and well-defined problems where experienced data scientists come to compete for solutions and learn new methods. However, a fundamental difficulty in organizing such challenges is obtaining and curating high quality datasets for model development and independent datasets for model evaluation. Importantly, to reduce the risk of bias and to ensure broad applicability of the algorithm, evaluation of the generalisability of resulting algorithms should ideally be performed on multiple independent evaluation datasets by an independent group of scientists. 
 
One clinical problem that has attracted substantial ML research is prostate cancer, a condition that 1 in 9 men develop in their lifetime. A prostate cancer diagnosis requires pathologists to examine biological tissue samples under a microscope to identify cancer and grade the cancer for signs of aggressive growth patterns in the cells. However, this cancer grading task (called Gleason grading) is difficult and subjective due to the need for visual assessment of cell differentiation and Gleason pattern predominance. Building a large dataset of samples with expert annotations can help with the development of ML systems to aid in prostate cancer grading.
To help accelerate and enable more research in this area, Google Health, Radboud University Medical Center and Karolinska Institutet joined forces to organize a global competition, the Prostate cANcer graDe Assessment (PANDA) Challenge, on the open Kaggle platform. In “Artificial Intelligence for Diagnosis and Gleason Grading of Prostate Cancer: the PANDA challenge”, published in Nature Medicine, we present the results of the challenge. The study design of the PANDA challenge provided the largest public whole-slide image dataset available and was open to participants from April 21st until July 23rd, 2020. The development datasets remain available for further research. In this effort, we compiled and publicly released a European cohort of prostate cancer cases for algorithm development and pioneered a standardized evaluation setup for digital pathology that enabled independent, blinded external validation of the algorithms on data from both the United States and EU.
         The global competition attracted participants from 65 countries (the size of the circle for each country illustrates the number of participants).   
Design of the Panda Challenge 
The challenge had two phases: a development phase (i.e., the Kaggle competition) and a validation phase. During the competition, 1,290 developers from 65 countries competed in building the best performing Gleason grading algorithm, having full access to a development set for algorithm training. Throughout the competition teams submitted algorithms that were evaluated on a hidden tuning set.
In the validation phase, a selection of top performing algorithms were independently evaluated on internal and external validation datasets with high quality reference grades from panels of expert prostate pathologists. In addition, a group of general pathologists graded a subset of the same cases to put the difficulty of the task and dataset in context. The algorithms submitted by the teams were then compared to grades done by groups of international and US general pathologists on these subsets. 
         Overview of the PANDA challenge’s phases for development and validation.   
Research Velocity During the Challenge 
We found that a group of Gleason grading ML algorithms developed during a global competition could achieve pathologist-level performance and generalize well to intercontinental and multinational cohorts. On all external validation sets, these algorithms achieved high agreement with urologic pathologists (prostate specialists) and high sensitivity for detecting tumor in biopsies. The Kaggle platform enabled the tracking of teams’ performance throughout the competition. Impressively, the first team achieving high agreement with the prostate pathologists at above 0.90 (quadratically weighted Cohen’s kappa) on the internal validation set occurred within the first 10 days of the competition. By the 33rd day, the median performance of all teams exceeded a score of 0.85. 
         Progression of algorithms’ performances throughout the competition, as shown by the highest score on the tuning and internal validation sets among all participating teams. During the competition teams could submit their algorithm for evaluation on the tuning set, after which they received their score. At the same time, algorithms were evaluated on the internal validation set, without disclosing these results to the participating teams. The development of the top score obtained by any team shows the rapid improvement of the algorithms.   
Learning from the Challenge 
By moderating the discussion forum on the Kaggle platform, we learned that the teams’ openness in sharing code via colab notebooks led to rapid improvement across the board, a promising sign for future public challenges, and a clear indication of the power of sharing knowledge on a common platform. 
Organizing a public challenge that evaluates algorithm generalization across independent cohorts using high quality reference standard panels presents substantial logistical difficulties. Assembling this size of a dataset across countries and organizations was a massive undertaking. This work benefited from an amazing collaboration between the three organizing institutions which have all contributed respective publications in this space, two in Lancet Oncology and one in JAMA Oncology. Combining these efforts provided a high quality foundation on which this competition could be based. With the publication, Radboud and Karolinska research groups are also open sourcing the PANDA challenge development datasets to facilitate the further improvement of prostate Gleason grading algorithms. We look forward to seeing many more advancements in this field, and more challenges that can catalyze extensive international knowledge sharing and collaborative research. 
Acknowledgements 
Key contributors to this project at Google include Po-Hsuan Cameron Chen, Kunal Nagpal, Yuannan Cai, David F. Steiner, Maggie Demkin, Sohier Dane, Fraser Tan, Greg S. Corrado, Lily Peng, Craig H. Mermel. Collaborators on this project include Wouter Bulten, Kimmo Kartasalo, Peter Ström, Hans Pinckaers, Hester van Boven, Robert Vink, Christina Hulsbergen-van de Kaa, Jeroen van der Laak, Mahul B. Amin, Andrew J. Evans, Theodorus van der Kwast, Robert Allan, Peter A. Humphrey, Henrik Grönberg, Hemamali Samaratunga, Brett Delahunt, Toyonori Tsuzuki, Tomi Häkkinen, Lars Egevad, Masi Valkonen, Pekka Ruusuvuori, Geert Litjens, Martin Eklund and the PANDA Challenge consortium. We thank Ellery Wulczyn, Annisah Um'rani, Yun Liu, and Dale Webster for their feedback on the manuscript and guidance on the project. We thank our collaborators at NMCSD, particularly Niels Olson, for internal re-use of de-identified data which contributed to the US external validation set. Sincere appreciation also goes to Sami Lachgar, Ashley Zlatinov, and Lauren Winer for their feedback on the blogpost.
An International Scientific Challenge for the Diagnosis and Gleason Grading of Prostate Cancer
https://ai.googleblog.com/2022/02/an-international-scientific-challenge.html
https://ai.googleblog.com/2022/02/robot-see-robot-do.html
Robot See, Robot Do
2022-02-08
Kevin Zakka; Student Researcher; Andy Zeng; Research Scientist; Robotics
<div><div>
<p>Posted by Kevin Zakka, Student Researcher and Andy Zeng, Research Scientist, Robotics at Google</p><p>
People learn to do things by watching others — from mimicking new dance moves, to watching YouTube cooking videos. We’d like robots to do the same, i.e., to learn new skills<em> </em>by<em> </em>watching people do things during training. Today, however, the predominant paradigm for teaching robots is to remote control them using specialized hardware for <a href="https://youtu.be/QkNNlfYG7kg?t=83">teleoperation</a> and then train them to <a href="https://arxiv.org/abs/1811.06711">imitate</a> pre-recorded demonstrations. This limits both <em>who</em> can provide the demonstrations (programmers &amp; roboticists) and <em>where</em> they can be provided (lab settings). If robots could instead self-learn new tasks by watching humans, this capability could allow them to be deployed in more unstructured settings like the home, and make it dramatically easier for <em>anyone</em> to teach or communicate with them, expert or otherwise. Perhaps one day, they might even be able to use Youtube videos to grow their collection of skills over time.
</p><table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiMCy0jEmc6-jQw8phByZNi_p5w21glY-pU9yfwrPAdf2eby75Rh4j_kRdHvkSSVe5Dh0KcSEPsUtay_U030ZL6YPmURnBRLm2mIHIVAEi7lhZzvR8g2jZQNdCoQ9qrGJ-s_1WVPRmwp9BXMfzKUsbvUerbtK5Ciub8n3bv65RZ0stq2vCrT8xl1apv8A=s720"></a></td></tr><tr><td>Our motivation is to have robots watch people do tasks, naturally with their hands, and then use that data as demonstrations for learning. <a href="https://www.youtube.com/watch?v=kEtcBpaM_l8">Video</a> by Teh Aik Hui and Nathaniel Lim. License: CC-BY </td></tr></tbody></table>
<p>
However, an obvious but often overlooked problem is that a robot is physically different from a human, which means it often completes tasks differently than we do. For example, in the pen manipulation task below, the hand can grab all the pens together and quickly transfer them between containers, whereas the two-fingered gripper must transport one at a time. <a href="https://arxiv.org/abs/1703.02949">Prior</a> <a href="https://arxiv.org/abs/1704.06888v3">research</a> <a href="https://arxiv.org/abs/1912.04443">assumes</a> that humans and robots can do the same task similarly, which makes manually specifying one-to-one correspondences between human and robot actions easy. But with stark differences in physique, defining such correspondences for seemingly easy tasks can be surprisingly difficult and sometimes impossible.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi5mBx21KuWD83IfXrsArqNxVz32aGC2N5p85-cL-eHz7ji3IEFWpkQSprkpN8q7acWZ6aoztarknGWNm2UWnkgzSrQAcPm2cjsE-3oJObzP6NNv66S4b3FCCXNXW7jVTXuZLonCz4omHD9iEbmATdu6cB7XVjuDs27Ot4alr30DY2T51ostGyajjFeKA=s720"></a></td></tr><tr><td>Physically different <a href="https://en.wikipedia.org/wiki/Robot_end_effector">end-effectors</a> (i.e., “grippers”) (i.e., the part that interacts with the environment) induce different control strategies when solving the same task. <b>Left</b>: The hand grabs all pens and quickly transfers them between containers. <b>Right</b>: The two-fingered gripper transports one pen at a time. </td></tr></tbody></table>
<p>
In “<a href="https://arxiv.org/pdf/2106.03911.pdf">XIRL: Cross-Embodiment Inverse RL</a>”, presented as an <a href="https://x-irl.github.io/">oral paper</a> at <a href="https://www.robot-learning.org/">CoRL 2021</a>, we explore these challenges further and introduce a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL). Rather than focusing on how individual human actions should correspond to robot actions, XIRL learns the <em>high-level task objective</em> from videos, and summarizes that knowledge in the form of a reward function that is invariant to embodiment differences, such as shape, actions and <a href="https://en.wikipedia.org/wiki/Robot_end_effector">end-effector</a> dynamics. The learned rewards can then be used together with reinforcement learning to teach the task to agents with new physical embodiments through trial and error. Our approach is general and scales autonomously with data — the more embodiment diversity presented in the videos, the more invariant and robust the reward functions become. Experiments show that our learned reward functions lead to significantly more sample efficient (roughly 2 to 4 times) reinforcement learning on new embodiments compared to alternative methods. To extend and build on our work, we are releasing an accompanying <a href="https://github.com/google-research/google-research/tree/master/xirl">open-source implementation</a> of our method along with <a href="https://github.com/kevinzakka/x-magical">X-MAGICAL</a>, our new simulated benchmark for cross-embodiment imitation.
</p>
<h2>Cross-Embodiment Inverse Reinforcement Learning (XIRL)</h2>
<p>
The underlying observation in this work is that in spite of the many differences induced by different embodiments, there still exist visual cues that reflect progression towards a common task objective. For example, in the pen manipulation task above, the presence of pens in the cup but not the mug, or the absence of pens on the table, are key frames that are common to different embodiments and indirectly provide cues for how close to being complete a task is. The key idea behind XIRL is to automatically discover these key moments in videos of different length and cluster them meaningfully to encode task progression. This motivation shares many similarities with <a href="https://ai.googleblog.com/2019/08/video-understanding-using-temporal.html%E2%80%9D">unsupervised video alignment</a> research, from which we can leverage a method called <a href="https://sites.google.com/view/temporal-cycle-consistency/home">Temporal Cycle Consistency</a> (TCC), which aligns videos accurately while learning useful visual representations for fine-grained video understanding without requiring any ground-truth correspondences.
</p>
<p>
We leverage TCC to train an encoder to temporally align video demonstrations of different experts performing the same task. The TCC loss tries to maximize the number of cycle-consistent frames (or mutual nearest-neighbors) between pairs of sequences using a <a href="https://en.wikipedia.org/wiki/Differentiable_function">differentiable</a> formulation of <a href="https://en.wikipedia.org/wiki/Neighbourhood_components_analysis">soft nearest-neighbors</a>. Once the encoder is trained, we define our reward function as simply the negative <a href="https://en.wikipedia.org/wiki/Euclidean_distance">Euclidean distance</a> between the current observation and the goal observation in the learned embedding space. We can subsequently insert the reward into a standard <a href="https://en.wikipedia.org/wiki/Markov_decision_process">MDP</a> and use an RL algorithm to learn the demonstrated behavior. Surprisingly, we find that this simple reward formulation is effective for cross-embodiment imitation.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg2VsuqxQHP7aj31-gSMNdvB3zh6KSppLYCX3deV9rGLIJrJKC8PK9R8vYyKnaSOAdY2nA0gfwddzlREBtbZroLfqRf14Tju-a9WEQyG-WgummP-0S_tot7IDiGGTTBsLzGxcv2dBF15xd_ka7MG41OCMrbkHxS7UaqD1mPANYtW8E1yBo0zxxETMy_kg=s1500"></a></td></tr><tr><td>XIRL self-supervises reward functions from expert demonstrations using temporal cycle consistency (TCC), then uses them for downstream reinforcement learning to learn new skills from third-person demonstrations.</td></tr></tbody></table>
<h2>X-MAGICAL Benchmark</h2>
<p>
To evaluate the performance of XIRL and baseline alternatives (e.g., <a href="https://arxiv.org/abs/1704.06888">TCN</a>, <a href="https://arxiv.org/abs/1703.02949">LIFS</a>, <a href="https://arxiv.org/abs/1810.01531">Goal Classifier</a>) in a consistent environment, we created <a href="https://github.com/kevinzakka/x-magical">X-MAGICAL</a>, which is a simulated benchmark for cross-embodiment imitation. X-MAGICAL features a diverse set of agent embodiments, with differences in their shapes and end-effectors, designed to solve tasks in different ways. This leads to differences in execution speeds and state-action trajectories, which poses challenges for current imitation learning techniques, e.g., ones that use time as a heuristic for weak correspondences between two trajectories. The ability to generalize across embodiments is precisely what X-MAGICAL evaluates.
</p>
<p>
The <i>SweepToTop </i>task we considered for our experiments is a simplified 2D equivalent of a common household robotic sweeping task, where an agent has to push three objects into a goal zone in the environment. We chose this task specifically because its long-horizon nature highlights how different agent embodiments can generate entirely different trajectories (shown below). X-MAGICAL features a <a href="https://github.com/openai/gym">Gym</a> API and is designed to be easily extendable to new tasks and embodiments. You can try it out today with <i><a href="https://pypi.org/project/x-magical/">pip install x-magical</a></i>.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi3rV42swE9kiITAn3dFqTX7lbuG9-SHqtYp7BJ4Z_GTkHSIId-t_GRtL5rqm52COLu1tj9zQ2fkM1cH671ACDnVmqRa2mCG_BNBb_htSNnJt0Kl71PRrDIU_AFBg5_mCN8kAUdXJdEM7v667DrQdzKJHOvtNpGx4jm1a3TbKQ21ehqimUc-BpznTfHjA=s720"></a></td></tr><tr><td>Different agent shapes in the <em>SweepToTop</em> task in the X-MAGICAL benchmark need to use different strategies to reposition objects into the target area (pink), i.e., to “clear the debris”. For example, the long-stick can clear them all in one fell swoop, whereas the short-stick needs to do multiple consecutive back-and-forths.</td></tr></tbody></table>
<p><b>Left</b>: Heatmap of state visitation for each embodiment across all expert demonstrations. <b>Right</b>: Examples of expert trajectories for each embodiment.</p>
<h2>Highlights</h2>
<p>
In our first set of experiments, we checked whether our learned embodiment-invariant reward function can enable successful reinforcement learning, when the expert demonstrations are provided through the agent itself. We find that XIRL significantly outperforms alternative methods especially on the tougher agents (e.g., short-stick and gripper).
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiSFaweS6Q-U9By-51wlw0uAOeNvlXxbo02Q3KN1MCXS_4IBeADN0dT4_dC_qoH5oJFKrCkHOm3HjPx4gpTNMNZnyw_Y_idotfIRneDjF3UEIo1U53P1G_pAdiFaZLEycf02P6Z3kJP0GwpTShzbWF02zgUX5tQ6pM01BsEgerEV76My-pgIH8cKSJpdA=s1999"></a></td></tr><tr><td><b>Same-embodiment setting: </b>Comparison of XIRL with baseline reward functions, using <a href="https://arxiv.org/abs/1801.01290">SAC</a> for RL policy learning. XIRL is roughly 2 to 4 times more sample efficient than some of the baselines on the harder agents (short-stick and gripper).</td></tr></tbody></table>
<p>
We also find that our approach shows great potential for learning reward functions that generalize to novel embodiments. For instance, when reward learning is performed on embodiments that are different from the ones on which the policy is trained, we find that it results in significantly more sample efficient agents compared to the same baselines. Below, in the gripper subplot (bottom right) for example, the reward is first learned on demonstration videos from long-stick, medium-stick and short-stick, after which the reward function is used to train the gripper agent.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjyPfjufeRfNrQG-3DPWKvtVQPpSbJa6e-KWvj1fKc3hWfOqevGpekr6yvtf7UUQsBId8FIyev-SIMl6bA4DmRTmTsJ7rGNWBYfv4mPzY0sxwE-hvCydOBmbtcLqT9VQ6NaH5T_zkyRn0ZfjS2wFZ4yXuc_FRobGEIXge2_9E0l9TGkq2wz0XEFXWUsbQ=s1999"></a></td></tr><tr><td><b>Cross-embodiment setting: </b>XIRL performs favorably when compared with other baseline reward functions, trained on observation-only demonstrations from different embodiments. Each agent (long-stick, medium-stick, short-stick, gripper) had its reward trained using demonstrations from the other three embodiments.</td></tr></tbody></table>
<p>
We also find that we can train on real-world human demonstrations, and use the learned reward to train a <a href="https://www.google.com/url?q=https://www.youtube.com/watch?v%3DS4mULTknb2I&amp;sa=D&amp;source=docs&amp;ust=1644255265532577&amp;usg=AOvVaw1Hp9d3dfNJoczbo4PdSfqb">Sawyer arm</a> in simulation to push a puck to a designated target zone. In these experiments as well, our method outperforms baseline alternatives. For example, our XIRL variant trained only on the real-world demonstrations (purple in the plots below) reaches 80% of the total performance roughly 85% faster than the <a href="https://arxiv.org/abs/2011.06507v1">RLV</a> baseline (orange).
</p>
<p><a href="https://blogger.googleusercontent.com/img/a/AVvXsEibDWkZrbhZlfVjvaGTAHe1ASceh9Tcx8VhARVIqIUhOCXrhExGgwPGjlZSS9LyaXBC6JXtQUvIH163ZQ_omO15dk_eCCwupEMZRyMK1qp4zhaBN-TdqEE6wZY9ysQGOMCRxzpUr9dZqja71Tpd6ryoncdIrMtjgon5uVZMcARPmT58ZVpcog2Tqm5b4w=s720"></a></p>
<p><a href="https://blogger.googleusercontent.com/img/a/AVvXsEh1idFWgC_L_hlHUrlqiXllzIzhlJ9P37ipoHsu0c6ocLb7doF9XQwkTzetOMAPmfwKMtY7pDLTahuTQ-6odh5QjLwsNQyvpEGRJLGpmbOX8A_DILSW7JjwipZjHK02Mqiz0Awz7yOioco3OjvaqOQBZ_sabwaOg9-Jc59kPJply6p2sRdodiuKnV7reA=s1999"></a></p>
<h2>What Do The Learned Reward Functions Look Like?</h2>
<p>
To further explore the qualitative nature of our learned rewards in more challenging real-world scenarios, we collect a dataset of the pen transfer task using various household tools. 
</p>
<p><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhrKfU6Q075LErWi1101f44Q8yqwlVFlBcMqXR2nNSnwWjpz37GDkEshxlb8U3juslCN355MWvNHE4MLF8ohM4VSLUVQGqINdU-VddDrKU8ZpOseQ0qYOFmo3M_-lFf5dwpXOizhQ2mCBNihmyrzJM_yrR2rOh11WHrMhnooDHNz22OnAu-L29VFis3zA=s1999"></a></p>
<p>
Below, we show rewards extracted from a successful (top) and unsuccessful (bottom) demonstration. Both demonstrations follow a similar trajectory at the start of the task execution. The successful one nets a high reward for placing the pens consecutively into the mug then into the glass cup, while the unsuccessful one obtains a low reward because it drops the pens outside the glass cup towards the end of the execution (orange circle). These results are promising because they show that our learned encoder can represent fine-grained visual differences relevant to a task.
</p>
<p><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgOwqA3aMqLBmeYtZAvhWyZ4xJQksWTzQ3HaSKGefKPFhWeN4g7dJoPiLpzBG6OPkrB1VVNliI_KHVBdKihxXzWFsnGheARwvkepanImicyGBy__El-QmGEypERbJKlHpbpQ94s5Fc1N75aZI_3Ui4iy-RUACOsgVf6omMFlf3vnO-R7L37Ifb4hZHxZA=s1999"></a></p>
 
<h2>Conclusion</h2>
<p>
We highlighted XIRL, our approach to tackling the cross-embodiment imitation problem. XIRL learns an embodiment-invariant reward function that encodes task progress using a temporal cycle-consistency objective. Policies learned using our reward functions are significantly more sample-efficient than baseline alternatives. Furthermore, the reward functions do not require manually paired video frames between the demonstrator and the learner, giving them the ability to scale to an arbitrary number of embodiments or experts with varying skill levels. Overall, we are excited about this direction of work, and hope that our benchmark promotes further research in this area. For more details, please check out our <a href="https://arxiv.org/abs/2106.03911">paper</a> and download the code from our <a href="https://github.com/google-research/google-research/tree/master/xirl">GitHub repository</a>.
</p>
<h2>Acknowledgments</h2>
<p>
<i>Kevin and Andy summarized research performed together with Pete Florence, Jonathan Tompson, Jeannette Bohg (faculty at Stanford University) and Debidatta Dwibedi. All authors would additionally like to thank Alex Nichol, Nick Hynes, Sean Kirmani, Brent Yi, Jimmy Wu, Karl Schmeckpeper and Minttu Alakuijala for fruitful technical discussions, and Sam Toyer for invaluable help with setting up the simulated benchmark.</i>
</p>
</div>||||I||||Skip to main content
 Robot See, Robot Do
 Robot See, Robot Do
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Robot See, Robot Do

 Tuesday, February 08, 2022

 Posted by Kevin Zakka, Student Researcher and Andy Zeng, Research Scientist, Robotics at Google

 People learn to do things by watching others — from mimicking new dance moves, to watching YouTube cooking videos. We’d like robots to do the same, i.e., to learn new skills by watching people do things during training. Today, however, the predominant paradigm for teaching robots is to remote control them using specialized hardware for teleoperation and then train them to imitate pre-recorded demonstrations. This limits both who can provide the demonstrations (programmers & roboticists) and where they can be provided (lab settings). If robots could instead self-learn new tasks by watching humans, this capability could allow them to be deployed in more unstructured settings like the home, and make it dramatically easier for anyone to teach or communicate with them, expert or otherwise. Perhaps one day, they might even be able to use Youtube videos to grow their collection of skills over time.

    
 Our motivation is to have robots watch people do tasks, naturally with their hands, and then use that data as demonstrations for learning. Video by Teh Aik Hui and Nathaniel Lim. License: CC-BY
 

 However, an obvious but often overlooked problem is that a robot is physically different from a human, which means it often completes tasks differently than we do. For example, in the pen manipulation task below, the hand can grab all the pens together and quickly transfer them between containers, whereas the two-fingered gripper must transport one at a time. Prior research assumes that humans and robots can do the same task similarly, which makes manually specifying one-to-one correspondences between human and robot actions easy. But with stark differences in physique, defining such correspondences for seemingly easy tasks can be surprisingly difficult and sometimes impossible.

      
 Physically different end-effectors (i.e., “grippers”) (i.e., the part that interacts with the environment) induce different control strategies when solving the same task. Left: The hand grabs all pens and quickly transfers them between containers. Right: The two-fingered gripper transports one pen at a time.
 

 In “XIRL: Cross-Embodiment Inverse RL”, presented as an oral paper at CoRL 2021, we explore these challenges further and introduce a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL). Rather than focusing on how individual human actions should correspond to robot actions, XIRL learns the high-level task objective from videos, and summarizes that knowledge in the form of a reward function that is invariant to embodiment differences, such as shape, actions and end-effector dynamics. The learned rewards can then be used together with reinforcement learning to teach the task to agents with new physical embodiments through trial and error. Our approach is general and scales autonomously with data — the more embodiment diversity presented in the videos, the more invariant and robust the reward functions become. Experiments show that our learned reward functions lead to significantly more sample efficient (roughly 2 to 4 times) reinforcement learning on new embodiments compared to alternative methods. To extend and build on our work, we are releasing an accompanying open-source implementation of our method along with X-MAGICAL, our new simulated benchmark for cross-embodiment imitation.


 Cross-Embodiment Inverse Reinforcement Learning (XIRL)

 The underlying observation in this work is that in spite of the many differences induced by different embodiments, there still exist visual cues that reflect progression towards a common task objective. For example, in the pen manipulation task above, the presence of pens in the cup but not the mug, or the absence of pens on the table, are key frames that are common to different embodiments and indirectly provide cues for how close to being complete a task is. The key idea behind XIRL is to automatically discover these key moments in videos of different length and cluster them meaningfully to encode task progression. This motivation shares many similarities with unsupervised video alignment research, from which we can leverage a method called Temporal Cycle Consistency (TCC), which aligns videos accurately while learning useful visual representations for fine-grained video understanding without requiring any ground-truth correspondences.

 We leverage TCC to train an encoder to temporally align video demonstrations of different experts performing the same task. The TCC loss tries to maximize the number of cycle-consistent frames (or mutual nearest-neighbors) between pairs of sequences using a differentiable formulation of soft nearest-neighbors. Once the encoder is trained, we define our reward function as simply the negative Euclidean distance between the current observation and the goal observation in the learned embedding space. We can subsequently insert the reward into a standard MDP and use an RL algorithm to learn the demonstrated behavior. Surprisingly, we find that this simple reward formulation is effective for cross-embodiment imitation.

    
 XIRL self-supervises reward functions from expert demonstrations using temporal cycle consistency (TCC), then uses them for downstream reinforcement learning to learn new skills from third-person demonstrations.
 


 X-MAGICAL Benchmark

 To evaluate the performance of XIRL and baseline alternatives (e.g., TCN, LIFS, Goal Classifier) in a consistent environment, we created X-MAGICAL, which is a simulated benchmark for cross-embodiment imitation. X-MAGICAL features a diverse set of agent embodiments, with differences in their shapes and end-effectors, designed to solve tasks in different ways. This leads to differences in execution speeds and state-action trajectories, which poses challenges for current imitation learning techniques, e.g., ones that use time as a heuristic for weak correspondences between two trajectories. The ability to generalize across embodiments is precisely what X-MAGICAL evaluates.

 The SweepToTop task we considered for our experiments is a simplified 2D equivalent of a common household robotic sweeping task, where an agent has to push three objects into a goal zone in the environment. We chose this task specifically because its long-horizon nature highlights how different agent embodiments can generate entirely different trajectories (shown below). X-MAGICAL features a Gym API and is designed to be easily extendable to new tasks and embodiments. You can try it out today with pip install x-magical.

      
 Different agent shapes in the SweepToTop task in the X-MAGICAL benchmark need to use different strategies to reposition objects into the target area (pink), i.e., to “clear the debris”. For example, the long-stick can clear them all in one fell swoop, whereas the short-stick needs to do multiple consecutive back-and-forths.
 
 
 
 Left: Heatmap of state visitation for each embodiment across all expert demonstrations. Right: Examples of expert trajectories for each embodiment.
 


 Highlights

 In our first set of experiments, we checked whether our learned embodiment-invariant reward function can enable successful reinforcement learning, when the expert demonstrations are provided through the agent itself. We find that XIRL significantly outperforms alternative methods especially on the tougher agents (e.g., short-stick and gripper).

     
 Same-embodiment setting: Comparison of XIRL with baseline reward functions, using SAC for RL policy learning. XIRL is roughly 2 to 4 times more sample efficient than some of the baselines on the harder agents (short-stick and gripper).
 

 We also find that our approach shows great potential for learning reward functions that generalize to novel embodiments. For instance, when reward learning is performed on embodiments that are different from the ones on which the policy is trained, we find that it results in significantly more sample efficient agents compared to the same baselines. Below, in the gripper subplot (bottom right) for example, the reward is first learned on demonstration videos from long-stick, medium-stick and short-stick, after which the reward function is used to train the gripper agent.

      
 Cross-embodiment setting: XIRL performs favorably when compared with other baseline reward functions, trained on observation-only demonstrations from different embodiments. Each agent (long-stick, medium-stick, short-stick, gripper) had its reward trained using demonstrations from the other three embodiments.
 

 We also find that we can train on real-world human demonstrations, and use the learned reward to train a Sawyer arm in simulation to push a puck to a designated target zone. In these experiments as well, our method outperforms baseline alternatives. For example, our XIRL variant trained only on the real-world demonstrations (purple in the plots below) reaches 80% of the total performance roughly 85% faster than the RLV baseline (orange).

 
 
 
 


 What Do The Learned Reward Functions Look Like?

 To further explore the qualitative nature of our learned rewards in more challenging real-world scenarios, we collect a dataset of the pen transfer task using various household tools.

 
 

 Below, we show rewards extracted from a successful (top) and unsuccessful (bottom) demonstration. Both demonstrations follow a similar trajectory at the start of the task execution. The successful one nets a high reward for placing the pens consecutively into the mug then into the glass cup, while the unsuccessful one obtains a low reward because it drops the pens outside the glass cup towards the end of the execution (orange circle). These results are promising because they show that our learned encoder can represent fine-grained visual differences relevant to a task.

 
 


 Conclusion

 We highlighted XIRL, our approach to tackling the cross-embodiment imitation problem. XIRL learns an embodiment-invariant reward function that encodes task progress using a temporal cycle-consistency objective. Policies learned using our reward functions are significantly more sample-efficient than baseline alternatives. Furthermore, the reward functions do not require manually paired video frames between the demonstrator and the learner, giving them the ability to scale to an arbitrary number of embodiments or experts with varying skill levels. Overall, we are excited about this direction of work, and hope that our benchmark promotes further research in this area. For more details, please check out our paper and download the code from our GitHub repository.


 Acknowledgments

 Kevin and Andy summarized research performed together with Pete Florence, Jonathan Tompson, Jeannette Bohg (faculty at Stanford University) and Debidatta Dwibedi. All authors would additionally like to thank Alex Nichol, Nick Hynes, Sean Kirmani, Brent Yi, Jimmy Wu, Karl Schmeckpeper and Minttu Alakuijala for fruitful technical discussions, and Sam Toyer for invaluable help with setting up the simulated benchmark.

 Labels: Computer Vision Deep Learning Reinforcement Learning Robotics Self-Supervised Learning
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Kevin Zakka, Student Researcher and Andy Zeng, Research Scientist, Robotics at Google 
People learn to do things by watching others — from mimicking new dance moves, to watching YouTube cooking videos. We’d like robots to do the same, i.e., to learn new skills by watching people do things during training. Today, however, the predominant paradigm for teaching robots is to remote control them using specialized hardware for teleoperation and then train them to imitate pre-recorded demonstrations. This limits both who can provide the demonstrations (programmers &amp; roboticists) and where they can be provided (lab settings). If robots could instead self-learn new tasks by watching humans, this capability could allow them to be deployed in more unstructured settings like the home, and make it dramatically easier for anyone to teach or communicate with them, expert or otherwise. Perhaps one day, they might even be able to use Youtube videos to grow their collection of skills over time.
          Our motivation is to have robots watch people do tasks, naturally with their hands, and then use that data as demonstrations for learning. Video by Teh Aik Hui and Nathaniel Lim. License: CC-BY    
However, an obvious but often overlooked problem is that a robot is physically different from a human, which means it often completes tasks differently than we do. For example, in the pen manipulation task below, the hand can grab all the pens together and quickly transfer them between containers, whereas the two-fingered gripper must transport one at a time. Prior research assumes that humans and robots can do the same task similarly, which makes manually specifying one-to-one correspondences between human and robot actions easy. But with stark differences in physique, defining such correspondences for seemingly easy tasks can be surprisingly difficult and sometimes impossible.
         Physically different end-effectors (i.e., “grippers”) (i.e., the part that interacts with the environment) induce different control strategies when solving the same task. Left: The hand grabs all pens and quickly transfers them between containers. Right: The two-fingered gripper transports one pen at a time.    
In “XIRL: Cross-Embodiment Inverse RL”, presented as an oral paper at CoRL 2021, we explore these challenges further and introduce a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL). Rather than focusing on how individual human actions should correspond to robot actions, XIRL learns the high-level task objective from videos, and summarizes that knowledge in the form of a reward function that is invariant to embodiment differences, such as shape, actions and end-effector dynamics. The learned rewards can then be used together with reinforcement learning to teach the task to agents with new physical embodiments through trial and error. Our approach is general and scales autonomously with data — the more embodiment diversity presented in the videos, the more invariant and robust the reward functions become. Experiments show that our learned reward functions lead to significantly more sample efficient (roughly 2 to 4 times) reinforcement learning on new embodiments compared to alternative methods. To extend and build on our work, we are releasing an accompanying open-source implementation of our method along with X-MAGICAL, our new simulated benchmark for cross-embodiment imitation.
Cross-Embodiment Inverse Reinforcement Learning (XIRL)
The underlying observation in this work is that in spite of the many differences induced by different embodiments, there still exist visual cues that reflect progression towards a common task objective. For example, in the pen manipulation task above, the presence of pens in the cup but not the mug, or the absence of pens on the table, are key frames that are common to different embodiments and indirectly provide cues for how close to being complete a task is. The key idea behind XIRL is to automatically discover these key moments in videos of different length and cluster them meaningfully to encode task progression. This motivation shares many similarities with unsupervised video alignment research, from which we can leverage a method called Temporal Cycle Consistency (TCC), which aligns videos accurately while learning useful visual representations for fine-grained video understanding without requiring any ground-truth correspondences.
We leverage TCC to train an encoder to temporally align video demonstrations of different experts performing the same task. The TCC loss tries to maximize the number of cycle-consistent frames (or mutual nearest-neighbors) between pairs of sequences using a differentiable formulation of soft nearest-neighbors. Once the encoder is trained, we define our reward function as simply the negative Euclidean distance between the current observation and the goal observation in the learned embedding space. We can subsequently insert the reward into a standard MDP and use an RL algorithm to learn the demonstrated behavior. Surprisingly, we find that this simple reward formulation is effective for cross-embodiment imitation.
         XIRL self-supervises reward functions from expert demonstrations using temporal cycle consistency (TCC), then uses them for downstream reinforcement learning to learn new skills from third-person demonstrations.   
X-MAGICAL Benchmark
To evaluate the performance of XIRL and baseline alternatives (e.g., TCN, LIFS, Goal Classifier) in a consistent environment, we created X-MAGICAL, which is a simulated benchmark for cross-embodiment imitation. X-MAGICAL features a diverse set of agent embodiments, with differences in their shapes and end-effectors, designed to solve tasks in different ways. This leads to differences in execution speeds and state-action trajectories, which poses challenges for current imitation learning techniques, e.g., ones that use time as a heuristic for weak correspondences between two trajectories. The ability to generalize across embodiments is precisely what X-MAGICAL evaluates.
The SweepToTop task we considered for our experiments is a simplified 2D equivalent of a common household robotic sweeping task, where an agent has to push three objects into a goal zone in the environment. We chose this task specifically because its long-horizon nature highlights how different agent embodiments can generate entirely different trajectories (shown below). X-MAGICAL features a Gym API and is designed to be easily extendable to new tasks and embodiments. You can try it out today with  pip install x-magical .
         Different agent shapes in the SweepToTop task in the X-MAGICAL benchmark need to use different strategies to reposition objects into the target area (pink), i.e., to “clear the debris”. For example, the long-stick can clear them all in one fell swoop, whereas the short-stick needs to do multiple consecutive back-and-forths.   
 Left: Heatmap of state visitation for each embodiment across all expert demonstrations. Right: Examples of expert trajectories for each embodiment.
Highlights
In our first set of experiments, we checked whether our learned embodiment-invariant reward function can enable successful reinforcement learning, when the expert demonstrations are provided through the agent itself. We find that XIRL significantly outperforms alternative methods especially on the tougher agents (e.g., short-stick and gripper).
          Same-embodiment setting: Comparison of XIRL with baseline reward functions, using SAC for RL policy learning. XIRL is roughly 2 to 4 times more sample efficient than some of the baselines on the harder agents (short-stick and gripper).   
We also find that our approach shows great potential for learning reward functions that generalize to novel embodiments. For instance, when reward learning is performed on embodiments that are different from the ones on which the policy is trained, we find that it results in significantly more sample efficient agents compared to the same baselines. Below, in the gripper subplot (bottom right) for example, the reward is first learned on demonstration videos from long-stick, medium-stick and short-stick, after which the reward function is used to train the gripper agent.
          Cross-embodiment setting: XIRL performs favorably when compared with other baseline reward functions, trained on observation-only demonstrations from different embodiments. Each agent (long-stick, medium-stick, short-stick, gripper) had its reward trained using demonstrations from the other three embodiments.   
We also find that we can train on real-world human demonstrations, and use the learned reward to train a Sawyer arm in simulation to push a puck to a designated target zone. In these experiments as well, our method outperforms baseline alternatives. For example, our XIRL variant trained only on the real-world demonstrations (purple in the plots below) reaches 80% of the total performance roughly 85% faster than the RLV baseline (orange).
   
   
What Do The Learned Reward Functions Look Like?
To further explore the qualitative nature of our learned rewards in more challenging real-world scenarios, we collect a dataset of the pen transfer task using various household tools. 
   
Below, we show rewards extracted from a successful (top) and unsuccessful (bottom) demonstration. Both demonstrations follow a similar trajectory at the start of the task execution. The successful one nets a high reward for placing the pens consecutively into the mug then into the glass cup, while the unsuccessful one obtains a low reward because it drops the pens outside the glass cup towards the end of the execution (orange circle). These results are promising because they show that our learned encoder can represent fine-grained visual differences relevant to a task.
   
 
Conclusion
We highlighted XIRL, our approach to tackling the cross-embodiment imitation problem. XIRL learns an embodiment-invariant reward function that encodes task progress using a temporal cycle-consistency objective. Policies learned using our reward functions are significantly more sample-efficient than baseline alternatives. Furthermore, the reward functions do not require manually paired video frames between the demonstrator and the learner, giving them the ability to scale to an arbitrary number of embodiments or experts with varying skill levels. Overall, we are excited about this direction of work, and hope that our benchmark promotes further research in this area. For more details, please check out our paper and download the code from our GitHub repository.
Acknowledgments
Kevin and Andy summarized research performed together with Pete Florence, Jonathan Tompson, Jeannette Bohg (faculty at Stanford University) and Debidatta Dwibedi. All authors would additionally like to thank Alex Nichol, Nick Hynes, Sean Kirmani, Brent Yi, Jimmy Wu, Karl Schmeckpeper and Minttu Alakuijala for fruitful technical discussions, and Sam Toyer for invaluable help with setting up the simulated benchmark.
Robot See, Robot Do
https://ai.googleblog.com/2022/02/robot-see-robot-do.html
https://ai.googleblog.com/2022/02/unlocking-full-potential-of-datacenter.html
Unlocking the Full Potential of Datacenter ML Accelerators with Platform-Aware Neural Architecture Search
2022-02-08
Sheng Li; Staff Software Engineer; Norman P Jouppi; Google Fellow; Google Research
<div><div>
<p>Posted by Sheng Li, Staff Software Engineer and Norman P. Jouppi, Google Fellow, Google Research</p><p>
Continuing advances in the design and implementation of datacenter (DC) accelerators for machine learning (ML), such as <a href="https://cloud.google.com/blog/topics/tpus">TPUs</a> and <a href="https://cloud.google.com/gpu">GPUs</a>, have been critical for powering modern ML models and applications <a href="https://cloud.google.com/blog/topics/tpus/google-showcases-cloud-tpu-v4-pods-for-large-model-training">at scale</a>. These improved accelerators exhibit peak performance (e.g., <a href="https://en.wikipedia.org/wiki/FLOPS">FLOPs</a>) that is orders of magnitude better than traditional computing systems. However, there is a fast-widening gap between the available peak performance offered by state-of-the-art hardware and the actual achieved performance when ML models run on that hardware. 
</p><p>
One approach to address this gap is to design hardware-specific ML models that optimize both performance (e.g., throughput and latency) and model quality. Recent applications of <a href="https://arxiv.org/abs/1611.01578">neural architecture search</a> (NAS), an emerging paradigm to automate the design of ML model architectures, have employed a <a href="https://ai.googleblog.com/2018/08/mnasnet-towards-automating-design-of.html">platform-aware multi-objective approach</a> that includes a hardware performance objective. While this approach has yielded improved model performance in practice, the details of the underlying hardware architecture are opaque to the model. As a result, there is untapped potential to build full capability hardware-friendly ML model architectures, with hardware-specific optimizations, for powerful DC ML accelerators. 
</p>
<p>
In “<a href="https://openaccess.thecvf.com/content/CVPR2021/papers/Li_Searching_for_Fast_Model_Families_on_Datacenter_Accelerators_CVPR_2021_paper.pdf">Searching for Fast Model Families on Datacenter Accelerators</a>”, <a href="https://openaccess.thecvf.com/content/CVPR2021/html/Li_Searching_for_Fast_Model_Families_on_Datacenter_Accelerators_CVPR_2021_paper.html">published</a> at <a href="https://cvpr2021.thecvf.com/">CVPR 2021</a>, we advanced the state of the art of hardware-aware NAS by automatically adapting model architectures to the hardware on which they will be executed. The approach we propose finds optimized families of models for which additional hardware performance gains cannot be achieved without loss in model quality (called <a href="https://en.wikipedia.org/wiki/Pareto_efficiency">Pareto optimization</a>). To accomplish this, we infuse a deep understanding of hardware architecture into the design of the NAS search space for discovery of both <em>single</em> models and model <em>families</em>. We provide quantitative analysis of the performance gap between hardware and traditional model architectures and demonstrate the advantages of using true hardware performance (i.e., throughput and latency), instead of the performance proxy (FLOPs), as the <a href="https://ai.googleblog.com/2018/08/mnasnet-towards-automating-design-of.html">performance optimization objective</a>. Leveraging this advanced hardware-aware NAS and building upon the <a href="https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html">EfficientNet</a> architecture, we developed a family of models, called EfficientNetX, that demonstrate the effectiveness of this approach for Pareto-optimized ML models on TPUs and GPUs.
</p>
<h2>Platform-Aware NAS for DC ML Accelerators</h2>
<p>
To achieve high performance, ML models need to adapt to modern ML accelerators. Platform-aware NAS integrates knowledge of the hardware accelerator properties into all three pillars of NAS: (<i>i</i>) the search objectives; (<i>ii</i>) the search space; and (<i>iii</i>) the search algorithm (shown below). We focus on the new search space because it contains the building blocks needed to compose the models and is the key link between the ML model architectures and accelerator hardware architectures. 
</p>
<p>
We construct TPU/GPU specialized search spaces with TPU/GPU-friendly operations to infuse hardware awareness into NAS. For example, a key adaptation is maximizing <a href="https://en.wikipedia.org/wiki/Parallel_computing">parallelism</a> to ensure different hardware components inside the accelerators work together as efficiently as possible. This includes the <a href="https://cloud.google.com/tpu/docs/system-architecture-tpu-vm">matrix multiplication units</a> (MXUs) in TPUs and the <a href="https://www.nvidia.com/en-us/data-center/tensor-cores/">TensorCore</a> in GPUs for matrix/tensor computation, as well as the <a href="https://cloud.google.com/tpu/docs/system-architecture-tpu-vm">vector processing units</a> (VPUs) in TPUs and <a href="https://en.wikipedia.org/wiki/CUDA">CUDA</a> cores in GPUs for vector processing. Maximizing model <a href="https://en.wikipedia.org/wiki/Roofline_model#Arithmetic_intensity">arithmetic intensity</a> (i.e., optimizing the parallelism between computation and operations on the high bandwidth memory) is also critical to achieve top performance. To tap into the full potential of the hardware, it is crucial for ML models to achieve high parallelism inside and across these hardware components.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiHzTGiyJhcsv_KihRCxUv3eeyjZuQxIR35W_4yT5XY987-vK2rThmNDIU5fjV-M9qcQ1le_9SsTGxLb9AQJL8dRhTh6JRXMbzXnrVgNWa8UaQYZQrEgXf1wp4Z58NKWDOEjGxUIzDjxapLC6lpahdyW1xEx4s9I987JF7yzS3By2QIODbcT_XUgiItQw=s1999"></a></td></tr><tr><td>Overview of platform-aware NAS on TPUs/GPUs, highlighting the search space and search objectives.</td></tr></tbody></table>
<p>
Advanced platform-aware NAS has an optimized search space containing a set of complementary techniques to holistically improve parallelism for ML model execution on TPUs and GPUs: 
</p>
<ol>
<li>It uses specialized <a href="https://openaccess.thecvf.com/content/CVPR2021/html/Li_Searching_for_Fast_Model_Families_on_Datacenter_Accelerators_CVPR_2021_paper.html">tensor reshaping techniques</a> to maximize the parallelism in the MXUs / TensorCores. 
</li><li>It dynamically selects different activation functions depending on matrix operation types to ensure overlapping of vector and matrix/tensor processing. 
</li><li>It employs hybrid convolutions and a novel fusion strategy to strike a balance between total compute and <a href="https://en.wikipedia.org/wiki/Roofline_model#Arithmetic_intensity">arithmetic intensity</a> to ensure that computation and memory access happens in parallel and to reduce the contention on VPUs / CUDA cores.
</li><li>With <a href="https://openaccess.thecvf.com/content/CVPR2021/html/Li_Searching_for_Fast_Model_Families_on_Datacenter_Accelerators_CVPR_2021_paper.html">latency-aware compound scaling</a> (LACS), which uses hardware performance instead of FLOPs as the performance objective to search for model depth, width and resolutions, we ensure parallelism at all levels for the entire model family on the <a href="https://en.wikipedia.org/wiki/Pareto_front">Pareto-front</a>. 
</li>
</ol>
<h2>EfficientNet-X: Platform-Aware NAS-Optimized Computer Vision Models for TPUs and GPUs</h2>
<p>
Using this approach to platform-aware NAS, we have designed EfficientNet-X, an optimized computer vision model family for TPUs and GPUs. This family builds upon the EfficientNet architecture, which itself was originally designed by traditional multi-objective NAS without true hardware-awareness as the baseline. The resulting EfficientNet-X model family achieves an average speedup of ~1.5x–2x over EfficientNet on <a href="https://cloud.google.com/tpu">TPUv3</a> and <a href="https://cloud.google.com/compute/docs/gpus#introduction">GPUv100</a>, respectively, with comparable accuracy. 
</p>
<p>
In addition to the improved speeds, EfficientNet-X has shed light on the non-proportionality between FLOPs and true performance. Many think FLOPs are a good ML performance proxy (i.e., FLOPs and performance are proportional), but they are not. While FLOPs are a good performance proxy for simple hardware such as scalar machines, they can exhibit a margin of error of up to 400% on advanced matrix/tensor machines. For example, because of its hardware-friendly model architecture, EfficientNet-X requires ~2x more FLOPs than EfficientNet, but is ~2x faster on TPUs and GPUs. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhLM7_dTe2bN89tYSNWuxeRB1Si_o3lTj_glnUWObo9gMW9TK-RL6LQAkdaLNOo9TyucVIwX4RI3NebND0JjQDVMjGtG00n4Q9Kw-qhjll-zQhjErhxz5ERDlKBwDTF7dCR9tQQoKqQBFzq4VUipRPguTH6bbsWISgwu4UwSsFjmPdEhqLXpXzlqmo3Vw=s1936"></a></td></tr><tr><td>EfficientNet-X family achieves 1.5x–2x speedup on average over the state-of-the-art EfficientNet family, with comparable accuracy on TPUv3 and GPUv100.</td></tr></tbody></table>
<h2>Self-Driving ML Model Performance on New Accelerator Hardware Platforms</h2>
<p>
Platform-aware NAS exposes the inner workings of the hardware and leverages these properties when designing hardware-optimized ML models. In a sense, the “platform-awareness” of the model is a “gene” that preserves knowledge of how to optimize performance for a hardware family, even on new generations, without the need to redesign the models. For example, <a href="https://ieeexplore.ieee.org/document/9499913">TPUv4i</a> delivers up to 3x higher peak performance (FLOPS) than its predecessor <a href="https://cacm.acm.org/magazines/2020/7/245702-a-domain-specific-supercomputer-for-training-deep-neural-networks/fulltext">TPUv2</a>, but EfficientNet performance only improves by 30% when migrating from TPUv2 to TPUv4i. In comparison, EfficientNet-X retains its platform-aware properties even on new hardware and achieves a 2.6x speedup when migrating from TPUv2 to TPUv4i, utilizing almost all of the 3x peak performance gain expected when upgrading between the two generations. 
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjUq32G6UvNmvyz45jryFZ-Eg0exZfnBvOm2cjOjiOAwTmNDK4X_FrMNhsnt4ovch2v6XfjhgBMvNpUV1MIOsSRD52y2o3d7PrnF9Lh5pm_EkJfU598zEMSV3_a9Av6MfzFo504h1ft8FdR4S8Uv3AOkd5UTYCXeyrMwZhiHOFMSSU_qKrRJPZmn7z9ig=s1608"></a></td></tr><tr><td>Hardware peak performance ratio of TPUv2 to TPUv4i and the geometric mean speedup of EfficientNet-X and EfficientNet families, respectively, when migrating from TPUv2 to TPUv4i.</td></tr></tbody></table>
<h2>Conclusion and Future Work</h2>
<p>
We demonstrate how to improve the capabilities of platform-aware NAS for datacenter ML accelerators, especially TPUs and GPUs. Both platform-aware NAS and the EfficientNet-X model family have been deployed in production and materialize up to ~40% efficiency gains and significant quality improvements for various internal computer vision projects across Google. Additionally, because of its deep understanding of accelerator hardware architecture, platform-aware NAS was able to identify critical performance bottlenecks on TPUv2-v4i architectures and has enabled design enhancements to future TPUs with significant potential performance uplift. As next steps, we are working on expanding platform-aware NAS’s capabilities to the ML hardware and model design beyond computer vision.
</p>
<h2>Acknowledgements</h2>
<p>
<i>Special thanks to our co-authors: Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc Le. We also thank many collaborators including Jeff Dean, David Patterson, Shengqi Zhu, Yun Ni, Gang Wu, Tao Chen, Xin Li, Yuan Qi, Amit Sabne, Shahab Kamali, and many others from the broad Google research and engineering teams who helped on the research and the subsequent broad production deployment of platform-aware NAS.</i>
</p>
</div>||||I||||Skip to main content
 Unlocking the Full Potential of Datacenter ML Accelerators with Platform-Aware Neural Architecture Search
 Unlocking the Full Potential of Datacenter ML Accelerators with Platform-Aware Neural Architecture Search
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Unlocking the Full Potential of Datacenter ML Accelerators with Platform-Aware Neural Architecture Search

 Tuesday, February 08, 2022

 Posted by Sheng Li, Staff Software Engineer and Norman P. Jouppi, Google Fellow, Google Research

 Continuing advances in the design and implementation of datacenter (DC) accelerators for machine learning (ML), such as TPUs and GPUs, have been critical for powering modern ML models and applications at scale. These improved accelerators exhibit peak performance (e.g., FLOPs) that is orders of magnitude better than traditional computing systems. However, there is a fast-widening gap between the available peak performance offered by state-of-the-art hardware and the actual achieved performance when ML models run on that hardware.

 One approach to address this gap is to design hardware-specific ML models that optimize both performance (e.g., throughput and latency) and model quality. Recent applications of neural architecture search (NAS), an emerging paradigm to automate the design of ML model architectures, have employed a platform-aware multi-objective approach that includes a hardware performance objective. While this approach has yielded improved model performance in practice, the details of the underlying hardware architecture are opaque to the model. As a result, there is untapped potential to build full capability hardware-friendly ML model architectures, with hardware-specific optimizations, for powerful DC ML accelerators.

 In “Searching for Fast Model Families on Datacenter Accelerators”, published at CVPR 2021, we advanced the state of the art of hardware-aware NAS by automatically adapting model architectures to the hardware on which they will be executed. The approach we propose finds optimized families of models for which additional hardware performance gains cannot be achieved without loss in model quality (called Pareto optimization). To accomplish this, we infuse a deep understanding of hardware architecture into the design of the NAS search space for discovery of both single models and model families. We provide quantitative analysis of the performance gap between hardware and traditional model architectures and demonstrate the advantages of using true hardware performance (i.e., throughput and latency), instead of the performance proxy (FLOPs), as the performance optimization objective. Leveraging this advanced hardware-aware NAS and building upon the EfficientNet architecture, we developed a family of models, called EfficientNetX, that demonstrate the effectiveness of this approach for Pareto-optimized ML models on TPUs and GPUs.


 Platform-Aware NAS for DC ML Accelerators

 To achieve high performance, ML models need to adapt to modern ML accelerators. Platform-aware NAS integrates knowledge of the hardware accelerator properties into all three pillars of NAS: (i) the search objectives; (ii) the search space; and (iii) the search algorithm (shown below). We focus on the new search space because it contains the building blocks needed to compose the models and is the key link between the ML model architectures and accelerator hardware architectures.

 We construct TPU/GPU specialized search spaces with TPU/GPU-friendly operations to infuse hardware awareness into NAS. For example, a key adaptation is maximizing parallelism to ensure different hardware components inside the accelerators work together as efficiently as possible. This includes the matrix multiplication units (MXUs) in TPUs and the TensorCore in GPUs for matrix/tensor computation, as well as the vector processing units (VPUs) in TPUs and CUDA cores in GPUs for vector processing. Maximizing model arithmetic intensity (i.e., optimizing the parallelism between computation and operations on the high bandwidth memory) is also critical to achieve top performance. To tap into the full potential of the hardware, it is crucial for ML models to achieve high parallelism inside and across these hardware components.

  
 Overview of platform-aware NAS on TPUs/GPUs, highlighting the search space and search objectives.
 

 Advanced platform-aware NAS has an optimized search space containing a set of complementary techniques to holistically improve parallelism for ML model execution on TPUs and GPUs:

 1. It uses specialized tensor reshaping techniques to maximize the parallelism in the MXUs / TensorCores.
 2. It dynamically selects different activation functions depending on matrix operation types to ensure overlapping of vector and matrix/tensor processing.
 3. It employs hybrid convolutions and a novel fusion strategy to strike a balance between total compute and arithmetic intensity to ensure that computation and memory access happens in parallel and to reduce the contention on VPUs / CUDA cores.
 4. With latency-aware compound scaling (LACS), which uses hardware performance instead of FLOPs as the performance objective to search for model depth, width and resolutions, we ensure parallelism at all levels for the entire model family on the Pareto-front.


 EfficientNet-X: Platform-Aware NAS-Optimized Computer Vision Models for TPUs and GPUs

 Using this approach to platform-aware NAS, we have designed EfficientNet-X, an optimized computer vision model family for TPUs and GPUs. This family builds upon the EfficientNet architecture, which itself was originally designed by traditional multi-objective NAS without true hardware-awareness as the baseline. The resulting EfficientNet-X model family achieves an average speedup of ~1.5x–2x over EfficientNet on TPUv3 and GPUv100, respectively, with comparable accuracy.

 In addition to the improved speeds, EfficientNet-X has shed light on the non-proportionality between FLOPs and true performance. Many think FLOPs are a good ML performance proxy (i.e., FLOPs and performance are proportional), but they are not. While FLOPs are a good performance proxy for simple hardware such as scalar machines, they can exhibit a margin of error of up to 400% on advanced matrix/tensor machines. For example, because of its hardware-friendly model architecture, EfficientNet-X requires ~2x more FLOPs than EfficientNet, but is ~2x faster on TPUs and GPUs.

   
 EfficientNet-X family achieves 1.5x–2x speedup on average over the state-of-the-art EfficientNet family, with comparable accuracy on TPUv3 and GPUv100.
 


 Self-Driving ML Model Performance on New Accelerator Hardware Platforms

 Platform-aware NAS exposes the inner workings of the hardware and leverages these properties when designing hardware-optimized ML models. In a sense, the “platform-awareness” of the model is a “gene” that preserves knowledge of how to optimize performance for a hardware family, even on new generations, without the need to redesign the models. For example, TPUv4i delivers up to 3x higher peak performance (FLOPS) than its predecessor TPUv2, but EfficientNet performance only improves by 30% when migrating from TPUv2 to TPUv4i. In comparison, EfficientNet-X retains its platform-aware properties even on new hardware and achieves a 2.6x speedup when migrating from TPUv2 to TPUv4i, utilizing almost all of the 3x peak performance gain expected when upgrading between the two generations.

    
 Hardware peak performance ratio of TPUv2 to TPUv4i and the geometric mean speedup of EfficientNet-X and EfficientNet families, respectively, when migrating from TPUv2 to TPUv4i.
 


 Conclusion and Future Work

 We demonstrate how to improve the capabilities of platform-aware NAS for datacenter ML accelerators, especially TPUs and GPUs. Both platform-aware NAS and the EfficientNet-X model family have been deployed in production and materialize up to ~40% efficiency gains and significant quality improvements for various internal computer vision projects across Google. Additionally, because of its deep understanding of accelerator hardware architecture, platform-aware NAS was able to identify critical performance bottlenecks on TPUv2-v4i architectures and has enabled design enhancements to future TPUs with significant potential performance uplift. As next steps, we are working on expanding platform-aware NAS’s capabilities to the ML hardware and model design beyond computer vision.


 Acknowledgements

 Special thanks to our co-authors: Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc Le. We also thank many collaborators including Jeff Dean, David Patterson, Shengqi Zhu, Yun Ni, Gang Wu, Tao Chen, Xin Li, Yuan Qi, Amit Sabne, Shahab Kamali, and many others from the broad Google research and engineering teams who helped on the research and the subsequent broad production deployment of platform-aware NAS.

 Labels: Computer Vision TPU
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Sheng Li, Staff Software Engineer and Norman P. Jouppi, Google Fellow, Google Research 
Continuing advances in the design and implementation of datacenter (DC) accelerators for machine learning (ML), such as TPUs and GPUs, have been critical for powering modern ML models and applications at scale. These improved accelerators exhibit peak performance (e.g., FLOPs) that is orders of magnitude better than traditional computing systems. However, there is a fast-widening gap between the available peak performance offered by state-of-the-art hardware and the actual achieved performance when ML models run on that hardware. 
 
One approach to address this gap is to design hardware-specific ML models that optimize both performance (e.g., throughput and latency) and model quality. Recent applications of neural architecture search (NAS), an emerging paradigm to automate the design of ML model architectures, have employed a platform-aware multi-objective approach that includes a hardware performance objective. While this approach has yielded improved model performance in practice, the details of the underlying hardware architecture are opaque to the model. As a result, there is untapped potential to build full capability hardware-friendly ML model architectures, with hardware-specific optimizations, for powerful DC ML accelerators. 
In “Searching for Fast Model Families on Datacenter Accelerators”, published at CVPR 2021, we advanced the state of the art of hardware-aware NAS by automatically adapting model architectures to the hardware on which they will be executed. The approach we propose finds optimized families of models for which additional hardware performance gains cannot be achieved without loss in model quality (called Pareto optimization). To accomplish this, we infuse a deep understanding of hardware architecture into the design of the NAS search space for discovery of both single models and model families. We provide quantitative analysis of the performance gap between hardware and traditional model architectures and demonstrate the advantages of using true hardware performance (i.e., throughput and latency), instead of the performance proxy (FLOPs), as the performance optimization objective. Leveraging this advanced hardware-aware NAS and building upon the EfficientNet architecture, we developed a family of models, called EfficientNetX, that demonstrate the effectiveness of this approach for Pareto-optimized ML models on TPUs and GPUs.
Platform-Aware NAS for DC ML Accelerators
To achieve high performance, ML models need to adapt to modern ML accelerators. Platform-aware NAS integrates knowledge of the hardware accelerator properties into all three pillars of NAS: (i) the search objectives; (ii) the search space; and (iii) the search algorithm (shown below). We focus on the new search space because it contains the building blocks needed to compose the models and is the key link between the ML model architectures and accelerator hardware architectures. 
We construct TPU/GPU specialized search spaces with TPU/GPU-friendly operations to infuse hardware awareness into NAS. For example, a key adaptation is maximizing parallelism to ensure different hardware components inside the accelerators work together as efficiently as possible. This includes the matrix multiplication units (MXUs) in TPUs and the TensorCore in GPUs for matrix/tensor computation, as well as the vector processing units (VPUs) in TPUs and CUDA cores in GPUs for vector processing. Maximizing model arithmetic intensity (i.e., optimizing the parallelism between computation and operations on the high bandwidth memory) is also critical to achieve top performance. To tap into the full potential of the hardware, it is crucial for ML models to achieve high parallelism inside and across these hardware components.
         Overview of platform-aware NAS on TPUs/GPUs, highlighting the search space and search objectives.   
Advanced platform-aware NAS has an optimized search space containing a set of complementary techniques to holistically improve parallelism for ML model execution on TPUs and GPUs: 
It uses specialized tensor reshaping techniques to maximize the parallelism in the MXUs / TensorCores. 
 It dynamically selects different activation functions depending on matrix operation types to ensure overlapping of vector and matrix/tensor processing. 
 It employs hybrid convolutions and a novel fusion strategy to strike a balance between total compute and arithmetic intensity to ensure that computation and memory access happens in parallel and to reduce the contention on VPUs / CUDA cores.
 With latency-aware compound scaling (LACS), which uses hardware performance instead of FLOPs as the performance objective to search for model depth, width and resolutions, we ensure parallelism at all levels for the entire model family on the Pareto-front. 
EfficientNet-X: Platform-Aware NAS-Optimized Computer Vision Models for TPUs and GPUs
Using this approach to platform-aware NAS, we have designed EfficientNet-X, an optimized computer vision model family for TPUs and GPUs. This family builds upon the EfficientNet architecture, which itself was originally designed by traditional multi-objective NAS without true hardware-awareness as the baseline. The resulting EfficientNet-X model family achieves an average speedup of ~1.5x–2x over EfficientNet on TPUv3 and GPUv100, respectively, with comparable accuracy. 
In addition to the improved speeds, EfficientNet-X has shed light on the non-proportionality between FLOPs and true performance. Many think FLOPs are a good ML performance proxy (i.e., FLOPs and performance are proportional), but they are not. While FLOPs are a good performance proxy for simple hardware such as scalar machines, they can exhibit a margin of error of up to 400% on advanced matrix/tensor machines. For example, because of its hardware-friendly model architecture, EfficientNet-X requires ~2x more FLOPs than EfficientNet, but is ~2x faster on TPUs and GPUs. 
         EfficientNet-X family achieves 1.5x–2x speedup on average over the state-of-the-art EfficientNet family, with comparable accuracy on TPUv3 and GPUv100.   
Self-Driving ML Model Performance on New Accelerator Hardware Platforms
Platform-aware NAS exposes the inner workings of the hardware and leverages these properties when designing hardware-optimized ML models. In a sense, the “platform-awareness” of the model is a “gene” that preserves knowledge of how to optimize performance for a hardware family, even on new generations, without the need to redesign the models. For example, TPUv4i delivers up to 3x higher peak performance (FLOPS) than its predecessor TPUv2, but EfficientNet performance only improves by 30% when migrating from TPUv2 to TPUv4i. In comparison, EfficientNet-X retains its platform-aware properties even on new hardware and achieves a 2.6x speedup when migrating from TPUv2 to TPUv4i, utilizing almost all of the 3x peak performance gain expected when upgrading between the two generations. 
         Hardware peak performance ratio of TPUv2 to TPUv4i and the geometric mean speedup of EfficientNet-X and EfficientNet families, respectively, when migrating from TPUv2 to TPUv4i.   
Conclusion and Future Work
We demonstrate how to improve the capabilities of platform-aware NAS for datacenter ML accelerators, especially TPUs and GPUs. Both platform-aware NAS and the EfficientNet-X model family have been deployed in production and materialize up to ~40% efficiency gains and significant quality improvements for various internal computer vision projects across Google. Additionally, because of its deep understanding of accelerator hardware architecture, platform-aware NAS was able to identify critical performance bottlenecks on TPUv2-v4i architectures and has enabled design enhancements to future TPUs with significant potential performance uplift. As next steps, we are working on expanding platform-aware NAS’s capabilities to the ML hardware and model design beyond computer vision.
Acknowledgements
Special thanks to our co-authors: Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc Le. We also thank many collaborators including Jeff Dean, David Patterson, Shengqi Zhu, Yun Ni, Gang Wu, Tao Chen, Xin Li, Yuan Qi, Amit Sabne, Shahab Kamali, and many others from the broad Google research and engineering teams who helped on the research and the subsequent broad production deployment of platform-aware NAS.
Unlocking the Full Potential of Datacenter ML Accelerators with Platform-Aware Neural Architecture Search
https://ai.googleblog.com/2022/02/unlocking-full-potential-of-datacenter.html
https://ai.googleblog.com/2022/02/applying-differential-privacy-to-large.html
Applying Differential Privacy to Large Scale Image Classification
2022-02-01
Alexey Kurakin; Software Engineer; Roxana Geambasu; Visiting Faculty Researcher; Google Research
<div><div>
<p>Posted by Alexey Kurakin, Software Engineer and Roxana Geambasu, Visiting Faculty Researcher, Google Research</p><p>
Machine learning (ML) models are becoming increasingly valuable for improved performance across a variety of consumer products, from <a href="https://ai.googleblog.com/2021/04/flexible-scalable-differentiable.html">recommendations</a> to automatic <a href="https://ai.googleblog.com/2021/10/self-supervised-learning-advances.html">image classification</a>. However, despite aggregating large amounts of data, in theory it is possible for models to encode characteristics of individual entries from the training set. For example, experiments in controlled settings have shown that language models trained using email datasets <a href="https://arxiv.org/abs/1802.08232">may sometimes encode sensitive information</a> included in the training data and may have the potential to <a href="https://arxiv.org/abs/1610.05820">reveal the presence of a particular user’s data</a> in the training set. As such, it is important to prevent the encoding of such characteristics from individual training entries. To these ends, researchers are increasingly employing <a href="https://ai.googleblog.com/2021/11/predicting-text-selections-with.html">federated learning approaches</a>. 
</p><p>
<a href="https://link.springer.com/chapter/10.1007/11681878_14">Differential privacy</a> (DP) provides a rigorous mathematical framework that allows researchers to quantify and understand the privacy guarantees of a system or an algorithm. Within the DP framework, privacy guarantees of a system are usually characterized by a positive parameter ε, called the <em>privacy loss bound</em>, with smaller ε corresponding to better privacy. One usually trains a model with DP guarantees using <a href="https://arxiv.org/abs/1607.00133">DP-SGD</a>, a specialized training algorithm that provides DP guarantees for the trained model. 
</p>
<p>
However training with DP-SGD typically has two major drawbacks. First, most existing implementations of DP-SGD are inefficient and slow, which makes it hard to use on large datasets. Second, DP-SGD training often significantly impacts utility (such as model accuracy) to the point that models trained with DP-SGD may become unusable in practice. As a result most DP research papers evaluate DP algorithms on very small datasets (<a href="http://yann.lecun.com/exdb/mnist/">MNIST</a>, <a href="https://www.cs.toronto.edu/~kriz/cifar.html">CIFAR-10</a>, or <a href="http://archive.ics.uci.edu/ml/index.php">UCI</a>) and don’t even try to perform evaluation of larger datasets, such as <a href="https://image-net.org/index.php">ImageNet</a>.
</p>
<p>
In “<a href="https://arxiv.org/abs/2201.12328">Toward Training at ImageNet Scale with Differential Privacy</a>”, we share initial results from our ongoing effort to train a large image classification model on <a href="https://image-net.org/index.php">ImageNet</a> using DP while maintaining high accuracy and minimizing computational cost. We show that the combination of various training techniques, such as careful choice of the model and hyperparameters, large batch training, and <a href="https://en.wikipedia.org/wiki/Transfer_learning">transfer learning</a> from other datasets, can significantly boost accuracy of an ImageNet model trained with DP. To substantiate these discoveries and encourage follow-up research, we are also releasing the associated <a href="https://github.com/google-research/dp-imagenet">source code</a>. 
</p>
<h2>Testing Differential Privacy on ImageNet</h2><p>
We choose ImageNet classification as a demonstration of the practicality and efficacy of DP because: (1) it is an ambitious task for DP, for which no prior work shows sufficient progress; and (2) it is a public dataset on which other researchers can operate, so it represents an opportunity to collectively improve the utility of real-life DP training. Classification on ImageNet is challenging for DP because it requires large networks with many parameters. This translates into a significant amount of noise added into the computation, because the noise added scales with the size of the model.
</p>
<h2>Scaling Differential Privacy with JAX</h2><p>
Exploring multiple architectures and training configurations to research what works for DP can be debilitatingly slow. To streamline our efforts, we used JAX, a high-performance computational library based on <a href="https://www.tensorflow.org/xla">XLA</a> that can do efficient auto-vectorization and just-in-time compilation of the mathematical computations. Using these JAX features was <a href="https://arxiv.org/abs/2010.09063">previously recommended</a> as a good way to speed up DP-SGD in the context of <em>smaller</em> datasets such as CIFAR-10.
</p>
<p>
We created our own implementation of DP-SGD on JAX and benchmarked it against the large ImageNet dataset (the code is included in our <a href="https://github.com/google-research/dp-imagenet/blob/main/imagenet/imagenet_train.py">release</a>). The implementation in JAX was relatively simple and resulted in noticeable performance gains simply because of using the XLA compiler. Compared to other implementations of DP-SGD, such as that in <a href="https://github.com/tensorflow/privacy">Tensorflow Privacy</a>, the JAX implementation is consistently several times faster. It is typically even faster compared to the custom-built and optimized <a href="https://github.com/pytorch/opacus">PyTorch Opacus</a>. 
</p>
<p>
Each step of our DP-SGD implementation takes approximately two forward-backward passes through the network. While this is slower than non-private training, which requires only a single forward-backward pass, it is still the <a href="https://arxiv.org/abs/1510.01799">most efficient known approach</a> to train with the per-example gradients necessary for DP-SGD. The graph below shows training runtimes for two models on ImageNet with DP-SGD vs. non-private SGD, each on JAX. Overall, we find DP-SGD on JAX sufficiently fast to run large experiments just by slightly reducing the number of training runs used to find optimal hyperparameters compared to non-private training. This is significantly better than alternatives, such as Tensorflow Privacy, which we found to be ~5x–10x slower on our CIFAR10 and MNIST benchmarks.
</p>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhWDXCIFoq1WvNJBCCS1mJ6ri6GvRr7bTW5vqn8OoNEjq6I457xH2ncoFuuFJz385Tzc7f7USXxmVuSZt0jWA1qjtPx354eq4wUrommruVaNf9PbrSihCvOD618grNEUyOIEHw9Jr18MWloa5awU0kSpFk1XhU6FTBgjGgFRPdxAct_ghc7kPw3sghbcA=s768"></a></td></tr><tr><td>Time in seconds per training epoch on ImageNet using a <a href="https://arxiv.org/abs/1512.03385">Resnet18 or Resnet50</a> architecture with 8 V100 GPUs.</td></tr></tbody></table>
<h2>Combining Techniques for Improved Accuracy</h2><p>
It is possible that future training algorithms may improve DP’s privacy-utility tradeoff. However, with current algorithms, such as DP-SGD, our experience points to an engineering “bag-of-tricks” approach to make DP more practical on challenging tasks like ImageNet.
</p>
<p>
Because we can train models faster with JAX, we can iterate quickly and explore multiple configurations to find what works well for DP. We report the following combination of techniques as useful to achieve non-trivial accuracy and privacy on ImageNet:
</p>
<ul>
<li><em>Full-batch training</em> 
<p>Theoretically, it is known that larger <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">minibatch</a> sizes improve the utility of DP-SGD, with full-batch training (i.e., where a full dataset is one batch) giving the best utility [<a href="https://arxiv.org/abs/1411.5417">1</a>, <a href="https://proceedings.neurips.cc/paper/2020/file/2e2c4bf7ceaa4712a72dd5ee136dc9a8-Paper.pdf">2</a>], and <a href="https://arxiv.org/abs/2108.01624">empirical results</a> are emerging to support this theory. Indeed, our experiments demonstrate that increasing the batch size along with the number of training epochs leads to a decrease in ε while still maintaining accuracy. However, training with extremely large batches is non-trivial as the batch cannot fit into GPU/TPU memory. So, we employed virtual large-batch training by accumulating gradients for multiple steps before updating the weights instead of applying gradient updates on each training step.<br />
</p>
<table>
 <colgroup>
 <col></col>
 <col></col>
 <col></col>
 <col></col>
 <col></col>
 <col></col>
 <col></col>
 <col></col>
 <col></col>
 </colgroup><tbody>
 <tr><td><em>Batch size</em></td><td> </td><td>1024</td><td> </td><td>4 × 1024</td><td> </td><td>16 × 1024</td><td> </td><td>64 × 1024</td></tr>
 <tr><td><em>Number of epochs</em></td><td> </td><td>10</td><td> </td><td>40</td><td> </td><td>160</td><td> </td><td>640</td></tr>
 <tr><td><em>Accuracy</em></td><td> </td><td>56%</td><td> </td><td>57.5%</td><td> </td><td>57.9%</td><td> </td><td>57.2%</td></tr>
 <tr><td><em>Privacy loss bound</em> ε</td><td> </td><td>9.8 × 10<sup>8</sup></td><td> </td><td>6.1 × 10<sup>7</sup></td><td> </td><td>3.5 × 10<sup>6</sup></td><td> </td><td>6.7 × 10<sup>4</sup></td></tr>
 </tbody></table><br />
 </li>
 <li><em>Transfer learning from public data</em> 
<p>Pre-training on public data followed by DP fine-tuning on private data has previously been shown to improve accuracy on other benchmarks [<a href="https://arxiv.org/abs/2011.11660">3</a>, <a href="https://arxiv.org/abs/2110.06500">4</a>]. A question that remains is <em>what</em> public data to use for a given task to optimize transfer learning. In this work we simulate a private/public data split by using ImageNet as "private" data and using <a href="http://places2.csail.mit.edu/">Places365</a>, another image classification dataset, as a proxy for “public" data. We pre-trained our models on Places365 before fine-tuning them with DP-SGD on ImageNet. Places365 only has images of landscapes and buildings, <em>not</em> of animals as ImageNet, so it is quite different, making it a good candidate to demonstrate the ability of the model to transfer to a different but related domain. 
</p>
<p>We found that transfer learning from Places365 gave us 47.5% accuracy on ImageNet with a reasonable level of privacy (ε = 10). This is low compared to the 70% accuracy of a similar non-private model, but compared to naïve DP training on ImageNet, which yields either very low accuracy (2 - 5%) or no privacy (ε=10<sup>9</sup>), this is quite good.
</p>
</li>
</ul>
<table><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi6Hosk5carfsgrBucStwR5_6wdaBIjYVDxxp1pgNzTpiI6gqEf9UhPwAmJY1VfT1JHQ0OXVEsi-cv4qFD6dmJDAyMAZpymz3DI4lRHkBKEbCC7BGJkhP08raO0oDEAny2NrDYWEJOotWxbh2jDpAAI6FVZeo9FvSPYmtylUf4IKJe1oSOXOmHyVP7a6Q=s1200"></a></td></tr><tr><td>Privacy-accuracy tradeoff for Resnet-18 on ImageNet using large-batch training with transfer learning from <a href="http://places2.csail.mit.edu/">Places365</a>.</td></tr></tbody></table>
<h2>Next Steps</h2><p>
We hope these early results and source code provide an impetus for other researchers to work on improving DP for ambitious tasks such as ImageNet as a proxy for challenging production-scale tasks. With the much faster DP-SGD on JAX, we urge DP and ML researchers to explore diverse training regimes, model architectures, and algorithms to make DP more practical. To continue advancing the state of the field, we recommend researchers start with a baseline that incorporates full-batch training plus transfer learning.
</p>
<h2>Acknowledgments</h2><p>
<em>This work was carried out with the support of the <a href="https://research.google/outreach/visiting-researcher-program/">Google Visiting Researcher Program</a> while Prof. Geambasu, an Associate Professor with Columbia University, was on sabbatical with Google Research. This work received substantial contributions from Steve Chien, Shuang Song, Andreas Terzis and Abhradeep Guha Thakurta.</em>
</p>
</div>||||I||||Skip to main content
 Applying Differential Privacy to Large Scale Image Classification
 Applying Differential Privacy to Large Scale Image Classification
 Philosophy
 Research Areas
 Publications
 People
 Tools & Downloads
 Outreach
 Careers
 Blog
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
 Blog ›

 Applying Differential Privacy to Large Scale Image Classification

 Tuesday, February 01, 2022

 Posted by Alexey Kurakin, Software Engineer and Roxana Geambasu, Visiting Faculty Researcher, Google Research

 Machine learning (ML) models are becoming increasingly valuable for improved performance across a variety of consumer products, from recommendations to automatic image classification. However, despite aggregating large amounts of data, in theory it is possible for models to encode characteristics of individual entries from the training set. For example, experiments in controlled settings have shown that language models trained using email datasets may sometimes encode sensitive information included in the training data and may have the potential to reveal the presence of a particular user’s data in the training set. As such, it is important to prevent the encoding of such characteristics from individual training entries. To these ends, researchers are increasingly employing federated learning approaches.

 Differential privacy (DP) provides a rigorous mathematical framework that allows researchers to quantify and understand the privacy guarantees of a system or an algorithm. Within the DP framework, privacy guarantees of a system are usually characterized by a positive parameter ε, called the privacy loss bound, with smaller ε corresponding to better privacy. One usually trains a model with DP guarantees using DP-SGD, a specialized training algorithm that provides DP guarantees for the trained model.

 However training with DP-SGD typically has two major drawbacks. First, most existing implementations of DP-SGD are inefficient and slow, which makes it hard to use on large datasets. Second, DP-SGD training often significantly impacts utility (such as model accuracy) to the point that models trained with DP-SGD may become unusable in practice. As a result most DP research papers evaluate DP algorithms on very small datasets (MNIST, CIFAR-10, or UCI) and don’t even try to perform evaluation of larger datasets, such as ImageNet.

 In “Toward Training at ImageNet Scale with Differential Privacy”, we share initial results from our ongoing effort to train a large image classification model on ImageNet using DP while maintaining high accuracy and minimizing computational cost. We show that the combination of various training techniques, such as careful choice of the model and hyperparameters, large batch training, and transfer learning from other datasets, can significantly boost accuracy of an ImageNet model trained with DP. To substantiate these discoveries and encourage follow-up research, we are also releasing the associated source code.


 Testing Differential Privacy on ImageNet

 We choose ImageNet classification as a demonstration of the practicality and efficacy of DP because: (1) it is an ambitious task for DP, for which no prior work shows sufficient progress; and (2) it is a public dataset on which other researchers can operate, so it represents an opportunity to collectively improve the utility of real-life DP training. Classification on ImageNet is challenging for DP because it requires large networks with many parameters. This translates into a significant amount of noise added into the computation, because the noise added scales with the size of the model.


 Scaling Differential Privacy with JAX

 Exploring multiple architectures and training configurations to research what works for DP can be debilitatingly slow. To streamline our efforts, we used JAX, a high-performance computational library based on XLA that can do efficient auto-vectorization and just-in-time compilation of the mathematical computations. Using these JAX features was previously recommended as a good way to speed up DP-SGD in the context of smaller datasets such as CIFAR-10.

 We created our own implementation of DP-SGD on JAX and benchmarked it against the large ImageNet dataset (the code is included in our release). The implementation in JAX was relatively simple and resulted in noticeable performance gains simply because of using the XLA compiler. Compared to other implementations of DP-SGD, such as that in Tensorflow Privacy, the JAX implementation is consistently several times faster. It is typically even faster compared to the custom-built and optimized PyTorch Opacus.

 Each step of our DP-SGD implementation takes approximately two forward-backward passes through the network. While this is slower than non-private training, which requires only a single forward-backward pass, it is still the most efficient known approach to train with the per-example gradients necessary for DP-SGD. The graph below shows training runtimes for two models on ImageNet with DP-SGD vs. non-private SGD, each on JAX. Overall, we find DP-SGD on JAX sufficiently fast to run large experiments just by slightly reducing the number of training runs used to find optimal hyperparameters compared to non-private training. This is significantly better than alternatives, such as Tensorflow Privacy, which we found to be ~5x–10x slower on our CIFAR10 and MNIST benchmarks.

   
 Time in seconds per training epoch on ImageNet using a Resnet18 or Resnet50 architecture with 8 V100 GPUs.
 


 Combining Techniques for Improved Accuracy

 It is possible that future training algorithms may improve DP’s privacy-utility tradeoff. However, with current algorithms, such as DP-SGD, our experience points to an engineering “bag-of-tricks” approach to make DP more practical on challenging tasks like ImageNet.

 Because we can train models faster with JAX, we can iterate quickly and explore multiple configurations to find what works well for DP. We report the following combination of techniques as useful to achieve non-trivial accuracy and privacy on ImageNet:

 * Full-batch training

 Theoretically, it is known that larger minibatch sizes improve the utility of DP-SGD, with full-batch training (i.e., where a full dataset is one batch) giving the best utility [1, 2], and empirical results are emerging to support this theory. Indeed, our experiments demonstrate that increasing the batch size along with the number of training epochs leads to a decrease in ε while still maintaining accuracy. However, training with extremely large batches is non-trivial as the batch cannot fit into GPU/TPU memory. So, we employed virtual large-batch training by accumulating gradients for multiple steps before updating the weights instead of applying gradient updates on each training step.

 * Batch size 1024 4 × 1024 16 × 1024 64 × 1024
 Number of epochs 10 40 160 640 
 Accuracy 56% 57.5% 57.9% 57.2% 
 Privacy loss bound ε 9.8 × 108 6.1 × 107 3.5 × 106 6.7 × 104
 

 * Transfer learning from public data

 Pre-training on public data followed by DP fine-tuning on private data has previously been shown to improve accuracy on other benchmarks [3, 4]. A question that remains is what public data to use for a given task to optimize transfer learning. In this work we simulate a private/public data split by using ImageNet as "private" data and using Places365, another image classification dataset, as a proxy for “public" data. We pre-trained our models on Places365 before fine-tuning them with DP-SGD on ImageNet. Places365 only has images of landscapes and buildings, not of animals as ImageNet, so it is quite different, making it a good candidate to demonstrate the ability of the model to transfer to a different but related domain.

 We found that transfer learning from Places365 gave us 47.5% accuracy on ImageNet with a reasonable level of privacy (ε = 10). This is low compared to the 70% accuracy of a similar non-private model, but compared to naïve DP training on ImageNet, which yields either very low accuracy (2 - 5%) or no privacy (ε=109), this is quite good.

   
 Privacy-accuracy tradeoff for Resnet-18 on ImageNet using large-batch training with transfer learning from Places365.
 


 Next Steps

 We hope these early results and source code provide an impetus for other researchers to work on improving DP for ambitious tasks such as ImageNet as a proxy for challenging production-scale tasks. With the much faster DP-SGD on JAX, we urge DP and ML researchers to explore diverse training regimes, model architectures, and algorithms to make DP more practical. To continue advancing the state of the field, we recommend researchers start with a baseline that incorporates full-batch training plus transfer learning.


 Acknowledgments

 This work was carried out with the support of the Google Visiting Researcher Program while Prof. Geambasu, an Associate Professor with Columbia University, was on sabbatical with Google Research. This work received substantial contributions from Steve Chien, Shuang Song, Andreas Terzis and Abhradeep Guha Thakurta.

 Labels: Deep Learning Security and Privacy
 Archive
 Labels
 All A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

 Privacy Terms About Google Google Products
Posted by Alexey Kurakin, Software Engineer and Roxana Geambasu, Visiting Faculty Researcher, Google Research 
Machine learning (ML) models are becoming increasingly valuable for improved performance across a variety of consumer products, from recommendations to automatic image classification. However, despite aggregating large amounts of data, in theory it is possible for models to encode characteristics of individual entries from the training set. For example, experiments in controlled settings have shown that language models trained using email datasets may sometimes encode sensitive information included in the training data and may have the potential to reveal the presence of a particular user’s data in the training set. As such, it is important to prevent the encoding of such characteristics from individual training entries. To these ends, researchers are increasingly employing federated learning approaches. 
 
Differential privacy (DP) provides a rigorous mathematical framework that allows researchers to quantify and understand the privacy guarantees of a system or an algorithm. Within the DP framework, privacy guarantees of a system are usually characterized by a positive parameter ε, called the privacy loss bound, with smaller ε corresponding to better privacy. One usually trains a model with DP guarantees using DP-SGD, a specialized training algorithm that provides DP guarantees for the trained model. 
However training with DP-SGD typically has two major drawbacks. First, most existing implementations of DP-SGD are inefficient and slow, which makes it hard to use on large datasets. Second, DP-SGD training often significantly impacts utility (such as model accuracy) to the point that models trained with DP-SGD may become unusable in practice. As a result most DP research papers evaluate DP algorithms on very small datasets (MNIST, CIFAR-10, or UCI) and don’t even try to perform evaluation of larger datasets, such as ImageNet.
In “Toward Training at ImageNet Scale with Differential Privacy”, we share initial results from our ongoing effort to train a large image classification model on ImageNet using DP while maintaining high accuracy and minimizing computational cost. We show that the combination of various training techniques, such as careful choice of the model and hyperparameters, large batch training, and transfer learning from other datasets, can significantly boost accuracy of an ImageNet model trained with DP. To substantiate these discoveries and encourage follow-up research, we are also releasing the associated source code. 
Testing Differential Privacy on ImageNet 
We choose ImageNet classification as a demonstration of the practicality and efficacy of DP because: (1) it is an ambitious task for DP, for which no prior work shows sufficient progress; and (2) it is a public dataset on which other researchers can operate, so it represents an opportunity to collectively improve the utility of real-life DP training. Classification on ImageNet is challenging for DP because it requires large networks with many parameters. This translates into a significant amount of noise added into the computation, because the noise added scales with the size of the model.
Scaling Differential Privacy with JAX 
Exploring multiple architectures and training configurations to research what works for DP can be debilitatingly slow. To streamline our efforts, we used JAX, a high-performance computational library based on XLA that can do efficient auto-vectorization and just-in-time compilation of the mathematical computations. Using these JAX features was previously recommended as a good way to speed up DP-SGD in the context of smaller datasets such as CIFAR-10.
We created our own implementation of DP-SGD on JAX and benchmarked it against the large ImageNet dataset (the code is included in our release). The implementation in JAX was relatively simple and resulted in noticeable performance gains simply because of using the XLA compiler. Compared to other implementations of DP-SGD, such as that in Tensorflow Privacy, the JAX implementation is consistently several times faster. It is typically even faster compared to the custom-built and optimized PyTorch Opacus. 
Each step of our DP-SGD implementation takes approximately two forward-backward passes through the network. While this is slower than non-private training, which requires only a single forward-backward pass, it is still the most efficient known approach to train with the per-example gradients necessary for DP-SGD. The graph below shows training runtimes for two models on ImageNet with DP-SGD vs. non-private SGD, each on JAX. Overall, we find DP-SGD on JAX sufficiently fast to run large experiments just by slightly reducing the number of training runs used to find optimal hyperparameters compared to non-private training. This is significantly better than alternatives, such as Tensorflow Privacy, which we found to be ~5x–10x slower on our CIFAR10 and MNIST benchmarks.
         Time in seconds per training epoch on ImageNet using a Resnet18 or Resnet50 architecture with 8 V100 GPUs.   
Combining Techniques for Improved Accuracy 
It is possible that future training algorithms may improve DP’s privacy-utility tradeoff. However, with current algorithms, such as DP-SGD, our experience points to an engineering “bag-of-tricks” approach to make DP more practical on challenging tasks like ImageNet.
Because we can train models faster with JAX, we can iterate quickly and explore multiple configurations to find what works well for DP. We report the following combination of techniques as useful to achieve non-trivial accuracy and privacy on ImageNet:
 Full-batch training 
Theoretically, it is known that larger minibatch sizes improve the utility of DP-SGD, with full-batch training (i.e., where a full dataset is one batch) giving the best utility [1, 2], and empirical results are emerging to support this theory. Indeed, our experiments demonstrate that increasing the batch size along with the number of training epochs leads to a decrease in ε while still maintaining accuracy. However, training with extremely large batches is non-trivial as the batch cannot fit into GPU/TPU memory. So, we employed virtual large-batch training by accumulating gradients for multiple steps before updating the weights instead of applying gradient updates on each training step.
 
  
  
  
  
  
  
  
  
  
  
   Batch size    1024   4 × 1024   16 × 1024   64 × 1024 
   Number of epochs    10   40   160   640 
   Accuracy    56%   57.5%   57.9%   57.2% 
   Privacy loss bound ε   9.8 × 108    6.1 × 107    3.5 × 106    6.7 × 104  
   
 
  Transfer learning from public data 
Pre-training on public data followed by DP fine-tuning on private data has previously been shown to improve accuracy on other benchmarks [3, 4]. A question that remains is what public data to use for a given task to optimize transfer learning. In this work we simulate a private/public data split by using ImageNet as "private" data and using Places365, another image classification dataset, as a proxy for “public" data. We pre-trained our models on Places365 before fine-tuning them with DP-SGD on ImageNet. Places365 only has images of landscapes and buildings, not of animals as ImageNet, so it is quite different, making it a good candidate to demonstrate the ability of the model to transfer to a different but related domain. 
We found that transfer learning from Places365 gave us 47.5% accuracy on ImageNet with a reasonable level of privacy (ε = 10). This is low compared to the 70% accuracy of a similar non-private model, but compared to naïve DP training on ImageNet, which yields either very low accuracy (2 - 5%) or no privacy (ε=109), this is quite good.
         Privacy-accuracy tradeoff for Resnet-18 on ImageNet using large-batch training with transfer learning from Places365.   
Next Steps 
We hope these early results and source code provide an impetus for other researchers to work on improving DP for ambitious tasks such as ImageNet as a proxy for challenging production-scale tasks. With the much faster DP-SGD on JAX, we urge DP and ML researchers to explore diverse training regimes, model architectures, and algorithms to make DP more practical. To continue advancing the state of the field, we recommend researchers start with a baseline that incorporates full-batch training plus transfer learning.
Acknowledgments 
This work was carried out with the support of the Google Visiting Researcher Program while Prof. Geambasu, an Associate Professor with Columbia University, was on sabbatical with Google Research. This work received substantial contributions from Steve Chien, Shuang Song, Andreas Terzis and Abhradeep Guha Thakurta.
Applying Differential Privacy to Large Scale Image Classification
https://ai.googleblog.com/2022/02/applying-differential-privacy-to-large.html