AdvSecureNet: A Python Toolkit for Adversarial Machine Learning
Melih Catal Manuel Günther






My editor

AdvSecureNet: A Python Toolkit for Adversarial Machine Learning
    Melih Catal melihcatal@gmail.com 

       Software Evolution and Architecture Lab 

       University of Zurich, Switzerland
       Manuel Günther guenther@ifi.uzh.ch 

       Artificial Intelligence and Machine Learning Group

       University of Zurich, Switzerland

    September 9, 2024
=================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Machine learning models are vulnerable to adversarial attacks. Several tools have been developed to research these vulnerabilities, but they often lack comprehensive features and flexibility. We introduce AdvSecureNet, a PyTorch based toolkit for adversarial machine learning that is the first to natively support multi-GPU setups for attacks, defenses, and evaluation.  It is the first toolkit that supports both CLI and API interfaces and external YAML configuration files to enhance versatility and reproducibility.  The toolkit includes multiple attacks, defenses and evaluation metrics. Rigiorous software engineering practices are followed to ensure high code quality and maintainability. The project is available as an open-source project on GitHub at <https://github.com/melihcatal/advsecurenet> and installable via PyPI.




  Adversarial Machine Learning, Trustworthy AI, Research Toolkit, PyTorch




§ INTRODUCTION


Machine learning models are widely used in fields such as self-driving cars <cit.>, facial recognition <cit.>, and medical imaging <cit.>, as well as in natural language processing tasks like chatbots <cit.> and translation services <cit.>. However, these models are vulnerable to adversarial attacks – subtle input modifications that can deceive the models <cit.>, which can compromise their integrity, confidentiality, or availability <cit.>.

Several libraries, such as ART <cit.>, AdverTorch <cit.>, and CleverHans <cit.>, have been developed to research these vulnerabilities by providing tools for implementing attacks, defenses, and evaluation metrics. However, these libraries often lack key features necessary for comprehensive research and experimentation, such as native multi-GPU support, integrated CLI and API interfaces, and support for external configuration files.

To address these limitations, we introduce AdvSecureNet (Adversarial Secure Networks), a comprehensive and flexible Python toolkit that supports multiple adversarial attacks, defenses, and evaluation metrics, optimized for multi-GPU setups. It includes a command-line interface (CLI) and an application programming interface (API), providing users with versatile options for experimentation and research. This paper outlines the features, design, and contributions of AdvSecureNet to the adversarial machine learning community.









[1]SecML supports attacks from CleverHans <cit.> and FoolBox <cit.>.
[2]This feature is only available for adversarial training.




§ ADVSECURENET FEATURES



Adversarial Attacks and Defenses: AdvSecureNet supports a diverse range of evasion attacks on computer vision tasks, including gradient-based, decision-based, single-step, iterative, white-box, black-box, targeted, and untargeted attacks <cit.>. AdvSecureNet also includes defense mechanisms such as adversarial training <cit.>, which incorporates adversarial examples into the training process to enhance model resilience, and ensemble adversarial training <cit.>, which leverages multiple models or attacks to develop a more resilient defense strategy.

Evaluation Metrics: AdvSecureNet supports metrics like accuracy, robustness, transferability, and similarity. Accuracy measures performance on benign data, robustness assesses resistance to attacks, transferability evaluates how well adversarial examples deceive different models, and similarity quantifies perceptual differences using PSNR <cit.> and SSIM <cit.>.

Multi-GPU Support: AdvSecureNet is optimized for multi-GPU setup, enhancing the efficiency of training, evaluation, and adversarial attack generation, especially for large models and datasets. This parallel GPU utilization aims to reduce computational time, making the toolkit ideal for large-scale experiments.

Interfaces and Configuration: AdvSecureNet offers both CLI and API interfaces. The CLI allows for quick execution of attacks, defenses, and evaluations, while the API provides advanced integration and extension within user applications. The toolkit also supports YAML configuration files for easy parameter tuning and experimentation, enabling users to share experiments, reproduce results, and manage setups effectively.

Built-in Models, Datasets and Target Generation: AdvSecureNet supports all PyTorch vision library models and well-known datasets like CIFAR-10, CIFAR-100, MNIST, FashionMNIST, SVHN, and ImageNet, allowing users to start without additional setup. Additionally, it can automatically generate adversarial targets for targeted attacks to simplify the attack configuration process. Users can still provide target labels manually and use custom datasets and models if desired.




§ DESIGN AND IMPLEMENTATION




AdvSecureNet is a modular, extensible, and user-friendly toolkit built on PyTorch for efficient computation and GPU acceleration. It includes core modules for attacks, defenses, evaluation metrics, and utilities, each with well-defined interfaces. The toolkit follows best practices in software engineering, featuring comprehensive testing, documentation, and CI/CD pipelines. It adheres to PEP 8 guidelines and uses Black for code formatting, along with tools like Pylint <cit.> and MyPy <cit.> for static code analysis and type checking. SonarQube <cit.> and Radon <cit.> provide insights into code quality and complexity. The project is hosted on GitHub under MIT license. Documentation is available on GitHub Pages, which includes detailed guidance on installation, usage, and comprehensive API references. AdvSecureNet is also available as a pip package on PyPI for easy installation and use across various environments.






§ RELATED WORK AND COMPARISON WITH EXISTING TOOLKITS


The burgeoning field of machine learning security has led to the development of several libraries designed to aid researchers. Notable among these are ART <cit.>, AdverTorch <cit.>, SecML <cit.>, FoolBox <cit.>, Ares <cit.>, and CleverHans <cit.>. ART, developed by IBM, is recognized for its extensive range of attacks, defenses, and support for multiple frameworks. AdverTorch, created by Borealis AI, focuses on PyTorch and offers a wide array of attacks, though it lacks support for adversarial training. CleverHans, one of the earliest libraries in the field, was initially designed for testing adversarial attacks, and as a result, has limited defensive capabilities. SecML and Ares, while smaller in scale, provide unique features; Ares, for instance, supports distributed training and external configuration files. FoolBox is distinguished by its diverse attack portfolio and support for multiple frameworks, but it does not offer defensive methods. Unfortunately, many of these libraries are no longer maintained. Table <ref> provides a detailed comparison of the features offered by these libraries.

AdvSecureNet stands out among existing adversarial machine learning toolkits in both its features and performance. Regarding features, AdvSecureNet is one of the few toolkits that are actively maintained, which is crucial for ongoing support. While IBM ART offers the most extensive attacks and defenses, AdvSecureNet provides a balanced selection, including adversarial and ensemble adversarial training for defense and a diverse range of attacks for evasion. AdvSecureNet distinguishes itself by being the first toolkit to natively support multi-GPU setups for adversarial attacks, defenses, and evaluation, whereas ARES only supports distributed adversarial training. This makes AdvSecureNet ideal for large-scale experiments. It is also the first toolkit that fully supports both CLI and API usages and external YAML configuration files, aiding researchers in sharing and reproducing experiments.

AdvSecureNet shows its strength in performance, achieving faster execution times on multi-GPU setups compared to other toolkits. As shown in Table <ref>, AdvSecureNet’s multi-GPU PGD attack time (1.78 minutes) outperforms ARES’s best single GPU time (3.05 minutes). In adversarial training on CIFAR-10, AdvSecureNet reduces training time from 5.07 minutes on a single GPU to 2.77 minutes with 7 GPUs, a speedup of 1.83x. AdvSecureNet's performance is even more impressive on ImageNet, reducing training time from 240 minutes on a single GPU to 30 minutes with 7 GPUs, which is an 8x speedup. In comparison, ARES reduces training time from 627 minutes on a single GPU to 217 minutes with 7 GPUs, a less efficient speedup of 2.89x. IBM ART, which does not natively support multi-GPU setups, remains at 323 minutes on a single GPU. The results show that AdvSecureNet provides superior performance and scalability, making it an ideal choice for large-scale adversarial machine learning experiments.





§ FUTURE WORK AND CONCLUSION

The AdvSecureNet toolkit is an ongoing project, and we plan to continue improving and expanding its capabilities. Currently, the toolkit focuses on evasion attacks and defenses in computer vision tasks, but we aim to extend its functionality to other domains, such as natural language processing. Additionally, we plan to incorporate other aspects of the trustworthiness of machine learning models, including fairness and interpretability. 

In conclusion, AdvSecureNet is a comprehensive toolkit for adversarial machine learning research, offering a wide range of attacks, defenses, datasets, and evaluation metrics in addition to multi-GPU support, CLI and API interfaces, as well as external configuration files. By providing a flexible and efficient platform for experimentation, AdvSecureNet aims to advance the field of adversarial machine learning.



