I have a rust application and a python application. I'm looking for some easy to use mechanism to call the python code from within my rust application
https://pyo3.rs/v0.16.4/
PyO3 user guide
2020-08-02
None
<div>
 
 
 
 
 
 
 
 
 <nav>
 
 
 </nav>
 <div>
 <div>
 
 
 <div>
 
 <h2>PyO3 user guide</h2>
 
 </div>
 
 
 
 
 <div>
 <main>
 <h2><a href="#the-pyo3-user-guide">The PyO3 user guide</a></h2>
<p>Welcome to the PyO3 user guide! This book is a companion to <a href="https://docs.rs/pyo3">PyO3's API docs</a>. It contains examples and documentation to explain all of PyO3's use cases in detail.</p>
<p>Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README.</p>
<h2><a href="#pyo3">PyO3</a></h2>
<p><a href="https://github.com/PyO3/pyo3/actions"></a>
<a href="https://pyo3.rs/dev/bench/"></a>
<a href="https://codecov.io/gh/PyO3/pyo3"></a>
<a href="https://crates.io/crates/pyo3"></a>
<a href="https://rust-lang.github.io/rfcs/2495-min-rust-version.html"></a>
<a href="https://gitter.im/PyO3/Lobby"></a>
<a href="https://github.com/PyO3/pyo3/blob/main/Contributing.md"></a></p>
<p><a href="https://www.rust-lang.org/">Rust</a> bindings for <a href="https://www.python.org/">Python</a>, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.</p>
<ul>
<li>
<p>User Guide: <a href="https://pyo3.rs/">stable</a> | <a href="https://pyo3.rs/main">main</a></p>
</li>
<li>
<p>API Documentation: <a href="https://docs.rs/pyo3/">stable</a> | <a href="https://pyo3.rs/main/doc">main</a></p>
</li>
</ul>
<h2><a href="#usage">Usage</a></h2>
<p>PyO3 supports the following software versions:</p>
<ul>
<li>Python 3.7 and up (CPython and PyPy)</li>
<li>Rust 1.48 and up</li>
</ul>
<p>You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.</p>
<h3><a href="#using-rust-from-python">Using Rust from Python</a></h3>
<p>PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use <a href="https://github.com/PyO3/maturin"><code>maturin</code></a>. <code>maturin</code> is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install <code>maturin</code>, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.</p>
<p>First, follow the commands below to create a new directory containing a new Python <code>virtualenv</code>, and install <code>maturin</code> into the virtualenv using Python's package manager, <code>pip</code>:</p>
<pre><code># (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin
</code></pre>
<p>Still inside this <code>string_sum</code> directory, now run <code>maturin init</code>. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:</p>
<pre><code>$ maturin init
✔ 🤷 What kind of bindings to use? · pyo3
 ✨ Done! New project created string_sum
</code></pre>
<p>The most important files generated by this command are <code>Cargo.toml</code> and <code>lib.rs</code>, which will look roughly like the following:</p>
<p><strong><code>Cargo.toml</code></strong></p>
<pre><code>[package]
name = "string_sum"
version = "0.1.0"
edition = "2018"
[lib]
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.16.4", features = ["extension-module"] }
</code></pre>
<p><strong><code>src/lib.rs</code></strong></p>
<pre><pre><code>
#![allow(unused)]
fn main() {
use pyo3::prelude::*;
/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -&gt; PyResult&lt;String&gt; {
 Ok((a + b).to_string())
}
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn string_sum(_py: Python&lt;'_&gt;, m: &amp;PyModule) -&gt; PyResult&lt;()&gt; {
 m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
 Ok(())
}
}
</code></pre></pre>
<p>Finally, run <code>maturin develop</code>. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from <code>python</code>:</p>
<pre><code>$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
&gt;&gt;&gt; import string_sum
&gt;&gt;&gt; string_sum.sum_as_string(5, 20)
'25'
</code></pre>
<p>To make changes to the package, just edit the Rust source code and then re-run <code>maturin develop</code> to recompile.</p>
<p>To run this all as a single copy-and-paste, use the bash script below (replace <code>string_sum</code> in the first command with the desired package name):</p>
<pre><code>mkdir string_sum &amp;&amp; cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop
</code></pre>
<p>As well as with <code>maturin</code>, it is possible to build using <a href="https://github.com/PyO3/setuptools-rust"><code>setuptools-rust</code></a> or <a href="https://pyo3.rs/latest/building_and_distribution.html#manual-builds">manually</a>. Both offer more flexibility than <code>maturin</code> but require more configuration to get started.</p>
<h3><a href="#using-python-from-rust">Using Python from Rust</a></h3>
<p>To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.</p>
<p>To install the Python shared library on Ubuntu:</p>
<pre><code>sudo apt install python3-dev
</code></pre>
<p>Start a new project with <code>cargo new</code> and add <code>pyo3</code> to the <code>Cargo.toml</code> like this:</p>
<pre><code>[dependencies.pyo3]
version = "0.16.4"
features = ["auto-initialize"]
</code></pre>
<p>Example program displaying the value of <code>sys.version</code> and the current user name:</p>
<pre><pre><code>use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
fn main() -&gt; PyResult&lt;()&gt; {
 Python::with_gil(|py| {
 let sys = py.import("sys")?;
 let version: String = sys.getattr("version")?.extract()?;
 let locals = [("os", py.import("os")?)].into_py_dict(py);
 let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
 let user: String = py.eval(code, None, Some(&amp;locals))?.extract()?;
 println!("Hello {}, I'm Python {}", user, version);
 Ok(())
 })
}
</code></pre></pre>
<p>The guide has <a href="https://pyo3.rs/latest/python_from_rust.html">a section</a> with lots of examples
about this topic.</p>
<ul>
<li><a href="https://github.com/PyO3/maturin">maturin</a> <em>Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages</em></li>
<li><a href="https://github.com/PyO3/setuptools-rust">setuptools-rust</a> <em>Setuptools plugin for Rust support</em>.</li>
<li><a href="https://github.com/PyO3/pyo3-built">pyo3-built</a> <em>Simple macro to expose metadata obtained with the <a href="https://crates.io/crates/built"><code>built</code></a> crate as a <a href="https://docs.rs/pyo3/*/pyo3/types/struct.PyDict.html"><code>PyDict</code></a></em></li>
<li><a href="https://github.com/PyO3/rust-numpy">rust-numpy</a> <em>Rust binding of NumPy C-API</em></li>
<li><a href="https://github.com/gperinazzo/dict-derive">dict-derive</a> <em>Derive FromPyObject to automatically transform Python dicts into Rust structs</em></li>
<li><a href="https://github.com/vorner/pyo3-log">pyo3-log</a> <em>Bridge from Rust to Python logging</em></li>
<li><a href="https://github.com/davidhewitt/pythonize">pythonize</a> <em>Serde serializer for converting Rust objects to JSON-compatible Python objects</em></li>
<li><a href="https://github.com/awestlake87/pyo3-asyncio">pyo3-asyncio</a> <em>Utilities for working with Python's Asyncio library and async functions</em></li>
<li><a href="https://github.com/mityax/rustimport">rustimport</a> <em>Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically.</em></li>
</ul>
<h2><a href="#examples">Examples</a></h2>
<ul>
<li><a href="https://github.com/mre/hyperjson">hyperjson</a> <em>A hyper-fast Python module for reading/writing JSON data using Rust's serde-json</em></li>
<li><a href="https://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever">html-py-ever</a> <em>Using <a href="https://github.com/servo/html5ever">html5ever</a> through <a href="https://github.com/kuchiki-rs/kuchiki">kuchiki</a> to speed up html parsing and css-selecting.</em></li>
<li><a href="https://github.com/ManifoldFR/point-process-rust/tree/master/pylib">point-process</a> <em>High level API for pointprocesses as a Python library</em></li>
<li><a href="https://github.com/autopilot-rs/autopy">autopy</a> <em>A simple, cross-platform GUI automation library for Python and Rust.</em>
<ul>
<li>Contains an example of building wheels on TravisCI and appveyor using <a href="https://github.com/pypa/cibuildwheel">cibuildwheel</a></li>
</ul>
</li>
<li><a href="https://github.com/ijl/orjson">orjson</a> <em>Fast Python JSON library</em></li>
<li><a href="https://github.com/fusion-engineering/inline-python">inline-python</a> <em>Inline Python code directly in your Rust code</em></li>
<li><a href="https://github.com/kngwyu/rogue-gym">Rogue-Gym</a> <em>Customizable rogue-like game for AI experiments</em>
<ul>
<li>Contains an example of building wheels on Azure Pipelines</li>
</ul>
</li>
<li><a href="https://github.com/thedrow/fastuuid/">fastuuid</a> <em>Python bindings to Rust's UUID library</em></li>
<li><a href="https://github.com/wasmerio/wasmer-python">wasmer-python</a> <em>Python library to run WebAssembly binaries</em></li>
<li><a href="https://github.com/cds-astro/mocpy">mocpy</a> <em>Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere</em></li>
<li><a href="https://github.com/huggingface/tokenizers/tree/master/bindings/python">tokenizers</a> <em>Python bindings to the Hugging Face tokenizers (NLP) written in Rust</em></li>
<li><a href="https://github.com/Project-Dream-Weaver/pyre-http">pyre</a> <em>Fast Python HTTP server written in Rust</em></li>
<li><a href="https://github.com/Stranger6667/jsonschema-rs/tree/master/bindings/python">jsonschema-rs</a> <em>Fast JSON Schema validation library</em></li>
<li><a href="https://github.com/Stranger6667/css-inline/tree/master/bindings/python">css-inline</a> <em>CSS inlining for Python implemented in Rust</em></li>
<li><a href="https://github.com/pyca/cryptography/tree/main/src/rust">cryptography</a> <em>Python cryptography library with some functionality in Rust</em></li>
<li><a href="https://github.com/daggy1234/polaroid">polaroid</a> <em>Hyper Fast and safe image manipulation library for Python written in Rust</em></li>
<li><a href="https://github.com/aviramha/ormsgpack">ormsgpack</a> <em>Fast Python msgpack library</em></li>
<li><a href="https://github.com/fastlmm/bed-reader">bed-reader</a> <em>Read and write the PLINK BED format, simply and efficiently</em>
<ul>
<li>Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions</li>
</ul>
</li>
<li><a href="https://github.com/kevinheavey/pyheck">pyheck</a> <em>Fast case conversion library, built by wrapping <a href="https://github.com/withoutboats/heck">heck</a></em>
<ul>
<li>Quite easy to follow as there's not much code.</li>
</ul>
</li>
</ul>
<h2><a href="#articles-and-other-media">Articles and other media</a></h2>
<ul>
<li><a href="https://towardsdatascience.com/nine-rules-for-writing-python-extensions-in-rust-d35ea3a4ec29?sk=f8d808d5f414154fdb811e4137011437">Nine Rules for Writing Python Extensions in Rust</a> - Dec 31, 2021</li>
<li><a href="https://saidvandeklundert.net/learn/2021-11-18-calling-rust-from-python-using-pyo3/">Calling Rust from Python using PyO3</a> - Nov 18, 2021</li>
<li><a href="https://www.youtube.com/watch?v=-XyWG_klSAw&amp;t=320s">davidhewitt's 2021 talk at Rust Manchester meetup</a> - Aug 19, 2021</li>
<li><a href="https://blog.waleedkhan.name/port-python-to-rust/">Incrementally porting a small Python project to Rust</a> - Apr 29, 2021</li>
<li><a href="https://www.vortexa.com/insight/integrating-rust-into-python">Vortexa - Integrating Rust into Python</a> - Apr 12, 2021</li>
<li><a href="https://blog.yossarian.net/2020/08/02/Writing-and-publishing-a-python-module-in-rust">Writing and publishing a Python module in Rust</a> - Aug 2, 2020</li>
</ul>
<h2><a href="#contributing">Contributing</a></h2>
<p>Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:</p>
<ul>
<li>help PyO3 users with issues on GitHub and Gitter</li>
<li>improve documentation</li>
<li>write features and bugfixes</li>
<li>publish blogs and examples of how to use PyO3</li>
</ul>
<p>Our <a href="https://github.com/PyO3/pyo3/blob/main/Contributing.md">contributing notes</a> and <a href="https://github.com/PyO3/pyo3/blob/main/Architecture.md">architecture guide</a> have more resources if you wish to volunteer time for PyO3 and are searching where to start.</p>
<p>If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:</p>
<ul>
<li><a href="https://github.com/sponsors/davidhewitt">davidhewitt</a></li>
</ul>
<h2><a href="#license">License</a></h2>
<p>PyO3 is licensed under the <a href="https://opensource.org/licenses/APACHE-2.0">Apache-2.0 license</a>.
Python is licensed under the <a href="https://docs.python.org/3/license.html">Python License</a>.</p>
 </main>
 <nav>
 
 
 <a href="/v0.16.4/module">
 <i></i>
 </a>
 
 
 </nav>
 </div>
 </div>
 <nav>
 
 <a href="/v0.16.4/module">
 <i></i>
 </a>
 </nav>
 </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
||||I|||| 1. Introduction
 2. 
 3. 1. Python Modules
 4. 2. Python Functions
 5. 3. Python Classes
 6. 
 1. 3.1. Class customizations
 2. 
 1. 3.1.1. Basic object customization
 2. 3.1.2. Emulating numeric types
 3. 3.1.3. Emulating callable objects
 7. 4. Type Conversions
 8. 
 1. 4.1. Mapping of Rust types to Python types
 2. 4.2. Conversion traits
 9. 5. Python Exceptions
 10. 6. Calling Python from Rust
 11. 7. GIL, mutability and object types
 12. 8. Parallelism
 13. 9. Debugging
 14. 10. Features Reference
 15. 11. Memory Management
 16. 12. Advanced Topics
 17. 13. Building and Distribution
 18. 
 1. 13.1. Supporting multiple Python versions
 19. 14. Useful Crates
 20. 
 1. 14.1. Logging
 2. 14.2. Async / Await
 21. 15. FAQ & Troubleshooting
 22. 
 23. Appendix A: Migration Guide
 24. Appendix B: PyO3 and rust-cpython
 25. Appendix C: Trait bounds
 26. Appendix D: Python typing hints
 27. CHANGELOG
 * Light (default)
 * Rust
 * Coal
 * Navy
 * Ayu

 PyO3 user guide

 The PyO3 user guide

 Welcome to the PyO3 user guide! This book is a companion to PyO3's API docs. It contains examples and documentation to explain all of PyO3's use cases in detail.

 Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README.

 PyO3

 Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.

 * User Guide: stable | main

 * API Documentation: stable | main

 Usage

 PyO3 supports the following software versions:

 * Python 3.7 and up (CPython and PyPy)
 * Rust 1.48 and up

 You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.

 Using Rust from Python

 PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use maturin. maturin is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install maturin, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.

 First, follow the commands below to create a new directory containing a new Python virtualenv, and install maturin into the virtualenv using Python's package manager, pip:

 # (replace string_sum with the desired package name)
 $ mkdir string_sum
 $ cd string_sum
 $ python -m venv .env
 $ source .env/bin/activate
 $ pip install maturin
 

 Still inside this string_sum directory, now run maturin init. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:

 $ maturin init
 ✔ 🤷 What kind of bindings to use? · pyo3
 ✨ Done! New project created string_sum
 

 The most important files generated by this command are Cargo.toml and lib.rs, which will look roughly like the following:

 Cargo.toml

 [package]
 name = "string_sum"
 version = "0.1.0"
 edition = "2018"
 
 [lib]
 # The name of the native library. This is the name which will be used in Python to import the
 # library (i.e. `import string_sum`). If you change this, you must also change the name of the
 # `#[pymodule]` in `src/lib.rs`.
 name = "string_sum"
 # "cdylib" is necessary to produce a shared library for Python to import from.
 #
 # Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
 # to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
 # crate-type = ["cdylib", "rlib"]
 crate-type = ["cdylib"]
 
 [dependencies]
 pyo3 = { version = "0.16.4", features = ["extension-module"] }
 

 src/lib.rs

 
 #![allow(unused)]
 fn main() {
 use pyo3::prelude::*;
 
 /// Formats the sum of two numbers as string.
 #[pyfunction]
 fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
 Ok((a + b).to_string())
 }
 
 /// A Python module implemented in Rust. The name of this function must match
 /// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
 /// import the module.
 #[pymodule]
 fn string_sum(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
 m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
 Ok(())
 }
 }
 

 Finally, run maturin develop. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from python:

 $ maturin develop
 # lots of progress output as maturin runs the compilation...
 $ python
 >>> import string_sum
 >>> string_sum.sum_as_string(5, 20)
 '25'
 

 To make changes to the package, just edit the Rust source code and then re-run maturin develop to recompile.

 To run this all as a single copy-and-paste, use the bash script below (replace string_sum in the first command with the desired package name):

 mkdir string_sum && cd "$_"
 python -m venv .env
 source .env/bin/activate
 pip install maturin
 maturin init --bindings pyo3
 maturin develop
 

 As well as with maturin, it is possible to build using setuptools-rust or manually. Both offer more flexibility than maturin but require more configuration to get started.

 Using Python from Rust

 To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.

 To install the Python shared library on Ubuntu:

 sudo apt install python3-dev
 

 Start a new project with cargo new and add pyo3 to the Cargo.toml like this:

 [dependencies.pyo3]
 version = "0.16.4"
 features = ["auto-initialize"]
 

 Example program displaying the value of sys.version and the current user name:

 use pyo3::prelude::*;
 use pyo3::types::IntoPyDict;
 
 fn main() -> PyResult<()> {
 Python::with_gil(|py| {
 let sys = py.import("sys")?;
 let version: String = sys.getattr("version")?.extract()?;
 
 let locals = [("os", py.import("os")?)].into_py_dict(py);
 let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
 let user: String = py.eval(code, None, Some(&locals))?.extract()?;
 
 println!("Hello {}, I'm Python {}", user, version);
 Ok(())
 })
 }
 

 The guide has a section with lots of examples about this topic.

 Tools and libraries

 * maturin Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages
 * setuptools-rust Setuptools plugin for Rust support.
 * pyo3-built Simple macro to expose metadata obtained with the built crate as a PyDict
 * rust-numpy Rust binding of NumPy C-API
 * dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs
 * pyo3-log Bridge from Rust to Python logging
 * pythonize Serde serializer for converting Rust objects to JSON-compatible Python objects
 * pyo3-asyncio Utilities for working with Python's Asyncio library and async functions
 * rustimport Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically.

 Examples

 * hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json
 * html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting.
 * point-process High level API for pointprocesses as a Python library
 * autopy A simple, cross-platform GUI automation library for Python and Rust.
 + Contains an example of building wheels on TravisCI and appveyor using cibuildwheel
 * orjson Fast Python JSON library
 * inline-python Inline Python code directly in your Rust code
 * Rogue-Gym Customizable rogue-like game for AI experiments
 + Contains an example of building wheels on Azure Pipelines
 * fastuuid Python bindings to Rust's UUID library
 * wasmer-python Python library to run WebAssembly binaries
 * mocpy Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere
 * tokenizers Python bindings to the Hugging Face tokenizers (NLP) written in Rust
 * pyre Fast Python HTTP server written in Rust
 * jsonschema-rs Fast JSON Schema validation library
 * css-inline CSS inlining for Python implemented in Rust
 * cryptography Python cryptography library with some functionality in Rust
 * polaroid Hyper Fast and safe image manipulation library for Python written in Rust
 * ormsgpack Fast Python msgpack library
 * bed-reader Read and write the PLINK BED format, simply and efficiently
 + Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions
 * pyheck Fast case conversion library, built by wrapping heck
 + Quite easy to follow as there's not much code.

 Articles and other media

 * Nine Rules for Writing Python Extensions in Rust - Dec 31, 2021
 * Calling Rust from Python using PyO3 - Nov 18, 2021
 * davidhewitt's 2021 talk at Rust Manchester meetup - Aug 19, 2021
 * Incrementally porting a small Python project to Rust - Apr 29, 2021
 * Vortexa - Integrating Rust into Python - Apr 12, 2021
 * Writing and publishing a Python module in Rust - Aug 2, 2020

 Contributing

 Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:

 * help PyO3 users with issues on GitHub and Gitter
 * improve documentation
 * write features and bugfixes
 * publish blogs and examples of how to use PyO3

 Our contributing notes and architecture guide have more resources if you wish to volunteer time for PyO3 and are searching where to start.

 If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:

 * davidhewitt

 License

 PyO3 is licensed under the Apache-2.0 license. Python is licensed under the Python License.

PyO3 user guide
 
 
 
 
 
 
 
 
  The PyO3 user guide 
Welcome to the PyO3 user guide! This book is a companion to PyO3's API docs. It contains examples and documentation to explain all of PyO3's use cases in detail.
Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README.
 PyO3 
  
 
 
 
 
 
  
 Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.
User Guide: stable | main 
API Documentation: stable | main 
 Usage 
PyO3 supports the following software versions:
Python 3.7 and up (CPython and PyPy)
Rust 1.48 and up
You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.
 Using Rust from Python 
PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use  maturin . maturin is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install maturin, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.
First, follow the commands below to create a new directory containing a new Python virtualenv, and install maturin into the virtualenv using Python's package manager, pip:
 # (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin
 
Still inside this string_sum directory, now run maturin init. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:
 $ maturin init
✔ 🤷 What kind of bindings to use? · pyo3
 ✨ Done! New project created string_sum
 
The most important files generated by this command are Cargo.toml and lib.rs, which will look roughly like the following:
  Cargo.toml  
 [package]
name = "string_sum"
version = "0.1.0"
edition = "2018"
[lib]
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.16.4", features = ["extension-module"] }
 
  src/lib.rs  
  
#![allow(unused)]
fn main() {
use pyo3::prelude::*;
/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -&gt; PyResult&lt;String&gt; {
 Ok((a + b).to_string())
}
/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn string_sum(_py: Python&lt;'_&gt;, m: &amp;PyModule) -&gt; PyResult&lt;()&gt; {
 m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
 Ok(())
}
}
  
Finally, run maturin develop. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from python:
 $ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
&gt;&gt;&gt; import string_sum
&gt;&gt;&gt; string_sum.sum_as_string(5, 20)
'25'
 
To make changes to the package, just edit the Rust source code and then re-run maturin develop to recompile.
To run this all as a single copy-and-paste, use the bash script below (replace string_sum in the first command with the desired package name):
 mkdir string_sum &amp;&amp; cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop
 
As well as with maturin, it is possible to build using  setuptools-rust  or manually. Both offer more flexibility than maturin but require more configuration to get started.
 Using Python from Rust 
To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.
To install the Python shared library on Ubuntu:
 sudo apt install python3-dev
 
Start a new project with cargo new and add pyo3 to the Cargo.toml like this:
 [dependencies.pyo3]
version = "0.16.4"
features = ["auto-initialize"]
 
Example program displaying the value of sys.version and the current user name:
  use pyo3::prelude::*;
use pyo3::types::IntoPyDict;
fn main() -&gt; PyResult&lt;()&gt; {
 Python::with_gil(|py| {
 let sys = py.import("sys")?;
 let version: String = sys.getattr("version")?.extract()?;
 let locals = [("os", py.import("os")?)].into_py_dict(py);
 let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
 let user: String = py.eval(code, None, Some(&amp;locals))?.extract()?;
 println!("Hello {}, I'm Python {}", user, version);
 Ok(())
 })
}
  
The guide has a section with lots of examples
about this topic.
 maturin Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages 
 setuptools-rust Setuptools plugin for Rust support.
 pyo3-built Simple macro to expose metadata obtained with the  built  crate as a  PyDict   
 rust-numpy Rust binding of NumPy C-API 
 dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs 
 pyo3-log Bridge from Rust to Python logging 
 pythonize Serde serializer for converting Rust objects to JSON-compatible Python objects 
 pyo3-asyncio Utilities for working with Python's Asyncio library and async functions 
 rustimport Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically. 
 Examples 
 hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json 
 html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting. 
 point-process High level API for pointprocesses as a Python library 
 autopy A simple, cross-platform GUI automation library for Python and Rust.
Contains an example of building wheels on TravisCI and appveyor using cibuildwheel 
 orjson Fast Python JSON library 
 inline-python Inline Python code directly in your Rust code 
 Rogue-Gym Customizable rogue-like game for AI experiments
Contains an example of building wheels on Azure Pipelines
 fastuuid Python bindings to Rust's UUID library 
 wasmer-python Python library to run WebAssembly binaries 
 mocpy Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere 
 tokenizers Python bindings to the Hugging Face tokenizers (NLP) written in Rust 
 pyre Fast Python HTTP server written in Rust 
 jsonschema-rs Fast JSON Schema validation library 
 css-inline CSS inlining for Python implemented in Rust 
 cryptography Python cryptography library with some functionality in Rust 
 polaroid Hyper Fast and safe image manipulation library for Python written in Rust 
 ormsgpack Fast Python msgpack library 
 bed-reader Read and write the PLINK BED format, simply and efficiently
Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions
 pyheck Fast case conversion library, built by wrapping heck 
Quite easy to follow as there's not much code.
 Articles and other media 
 Nine Rules for Writing Python Extensions in Rust - Dec 31, 2021
 Calling Rust from Python using PyO3 - Nov 18, 2021
 davidhewitt's 2021 talk at Rust Manchester meetup - Aug 19, 2021
 Incrementally porting a small Python project to Rust - Apr 29, 2021
 Vortexa - Integrating Rust into Python - Apr 12, 2021
 Writing and publishing a Python module in Rust - Aug 2, 2020
 Contributing 
Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:
help PyO3 users with issues on GitHub and Gitter
improve documentation
write features and bugfixes
publish blogs and examples of how to use PyO3
Our contributing notes and architecture guide have more resources if you wish to volunteer time for PyO3 and are searching where to start.
If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:
 davidhewitt 
 License 
PyO3 is licensed under the Apache-2.0 license.
Python is licensed under the Python License.
PyO3 user guide
https://pyo3.rs/v0.16.4/
https://blog.m-ou.se/writing-python-inside-rust-2/
Writing Python inside your Rust code — Part 2
2020-04-25
Mara Bos
<div><div>
 <p>In this part, we’ll extend our <code>python!{}</code>-macro to be able to seamlessly
use Rust variables in the Python code within.
We explore a few options, and implement two alternatives.</p>
<h2>Defining globals for Python</h2>
<p>In <a href="../writing-python-inside-rust-1">Part 1</a>,
we wrote a <code>run_python</code> function that executes Python code using PyO3’s <code>Python::run</code> function:</p>
<div><pre><code>fn run_python(code: &amp;str) {
 let py = pyo3::Python::acquire_gil();
 if let Err(e) = py.python().run(code, None, None) {
 e.print(py.python());
 panic!("Python code failed");
 }
}
</code></pre></div><p>Let’s take a look again at <a href="https://docs.rs/pyo3/0.9.2/pyo3/struct.Python.html#method.run">the signature of <code>pyo3::Python::run</code></a>:</p>
<div><pre><code> pub fn run(
 self,
 code: &amp;str,
 globals: Option&lt;&amp;PyDict&gt;,
 locals: Option&lt;&amp;PyDict&gt;
 ) -&gt; PyResult&lt;()&gt;
</code></pre></div><p>Looks like we can give a <code>PyDict</code> of global variables where we give <code>None</code> now.</p>
<p>Setting variables inside a <code>PyDict</code> looks easy enough: <a href="https://docs.rs/pyo3/0.9.2/pyo3/types/struct.PyDict.html#method.set_item"><code>PyDict::set_item</code></a>
only requires the key and value to implement <a href="https://docs.rs/pyo3/0.9.2/pyo3/conversion/trait.ToPyObject.html"><code>ToPyObject</code></a>,
which is already implemented for a lot of types, including strings, integers, vectors, and many more.</p>
<p>Let’s see how it works, by defining <code>five = 5</code> from Rust:</p>
<div><pre><code>fn run_python(code: &amp;str) {
 let py = pyo3::Python::acquire_gil();
 let globals = pyo3::types::PyDict::new(py.python());
 // "five" and 5 are automatically converted to a PyObject by ToPyObject.
 globals.set_item("five", 5).unwrap();
 if let Err(e) = py.python().run(code, Some(globals), None) {
 e.print(py.python());
 panic!("Python code failed");
 }
}
</code></pre></div><div><pre><code>fn main() {
 python! {
 print(five + 1)
 }
}
</code></pre></div><div><pre><code>$ cargo r
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.34s
 Running `target/debug/scratchpad`
6
</code></pre></div><p>Hey, that works!</p>
<h2>Transparent syntax</h2>
<p>Now that we know converting objects from Rust to Python is not going to be a problem (thanks to PyO3’s <code>ToPyObject</code>),
we can move on to the problem of how the user will indicate which of their variables need to be converted and injected in the <code>globals</code> dictionary.</p>
<p>The most ergonomic way would probably be something that’s completely transparent:</p>
<div><pre><code>let a = 5;
python! {
 b = 10
 print(a + b)
}
</code></pre></div><p>If we could make this work, users could just refer to any variable in scope as if they didn’t even switch languages.
Sounds perfect.</p>
<p>However, our procedural macro does not have access to the surrounding code,
so would not know that <code>a</code> even exists in the Rust code.
In order to know <code>a</code> comes from Rust, but <code>b</code> does not,
it’d have to understand the Python code and see that <code>a</code> is used without being initialized,
but <code>b</code> is initialized inside the Python code.
This goes much further than parsing <code>b = 10</code> and understanding that defines <code>b</code>, meaning it should not be captured.
For example, <code>print</code> is not explicitly defined anywhere, but it does not refer to anything from Rust.</p>
<p>If we think a bit more about this, it only gets worse:</p>
<div><pre><code>let a = 5;
python! {
 from somelibrary import *
 globals()['c'] = 10
 print(a + b + c)
}
</code></pre></div><p>Without fully parsing <em>and running</em> all of the Python code, we can’t possibly do this.</p>
<p>So, there needs to be <em>some</em> way the user can tell the macro which variables need to be captured.</p>
<h2>Capture list</h2>
<p>Capturing variables..
That sounds like something <em>closures</em> (sometimes referred to as <em>lambdas</em>) do.
Maybe we can draw some inspiration from there.</p>
<p>In Rust, a closure is defined using <code>||</code>:</p>
<div><pre><code>let a = 5;
let f = |b| a + b;
assert_eq!(f(10), 15);
</code></pre></div><p>Here, <code>f</code> behaves like a function. It takes one argument (<code>b</code>), and returns <code>a + b</code>.
<code>a</code> was implicitly captured from the environment.
This implicit capturing is exactly what we discussed before,
which is not feasible for <code>python!{}</code>.</p>
<p>In Rust, closures always implicitly capture whatever they need from their environment,
as <a href="https://doc.rust-lang.org/book/ch13-01-closures.html">described in the Rust Book</a>.
However, if we look at closures in other languages, we see that C++ is one of the few where
we can <a href="https://en.cppreference.com/w/cpp/language/lambda#Lambda_capture">manually specify what should be captured</a>,
using <code>[]</code>-syntax:</p>
<div><pre><code>int main() {
 int a = 5;
 auto x = [a] (int b) { return a + b; };
 // └┬┘ └──┬──┘
 // │ └── Parameter list
 // └── Capture list
 assert(x(10) == 15);
}
</code></pre></div><p>Looks like C++ already solved our problem, using a <em>capture list</em>.
Let’s just steal the idea:</p>
<div><pre><code>let a = 3;
let b = 20;
python! {
 [a b] // List of variables to capture
 c = 100
 print(a + b + c)
}
</code></pre></div><p>I’ve omitted commas or any other syntax apart from the <code>[]</code> and the names,
to keep parsing as simple as possible.
If it turns out to work well, we can always improve the syntax later.</p>
<h3>Implementation</h3>
<p>Alright, let’s see if we can implement this.</p>
<p>We’ll have to parse the capture list
and generate code that calls <code>globals.set_item("var", var)</code> for each variable.
This code should end up in <code>run_python</code> and executed after making the <code>globals</code> dictionary,
but before executing <code>Python::run</code>.</p>
<p>Passing arbitrary code to a function is easy in Rust:
using <code>impl Fn</code>, <code>run_python</code> can accept a closure containing the code with all the <code>set_item</code> calls.
We’ll have to give the closure access to the <code>PyDict</code>:</p>
<div><pre><code>fn run_python(
 code: &amp;str,
 f: impl FnOnce(&amp;pyo3::types::PyDict), // new
) {
 let py = pyo3::Python::acquire_gil();
 let globals = pyo3::types::PyDict::new(py.python());
 f(globals); // new
 if let Err(e) = py.python().run(code, Some(globals), None) {
 e.print(py.python());
 panic!("Python code failed");
 }
}
</code></pre></div><p>We use <code>FnOnce</code> instead of <code>Fn</code>, because we only call the function once.
This means it can also accept closures that move out of their own captures,
and therefore cannot be called more than once.</p>
<p>Now on to the hard part, the macro implementation.</p>
<p>First, a function that checks if there’s a capture list, and extracts the names from it:</p>
<div><pre><code>fn get_captures(input: TokenStream) -&gt; Option&lt;(Vec&lt;String&gt;, TokenStream)&gt; {
 let mut input = input.into_iter();
 let captures = match input.next() {
 Some(TokenTree::Group(g)) if g.delimiter() == Delimiter::Bracket =&gt; g.stream(),
 _ =&gt; return None,
 };
 let captures = captures
 .into_iter()
 .map(|token| {
 if let TokenTree::Ident(ident) = token {
 ident.to_string()
 } else {
 panic!("Invalid token in capture list");
 }
 })
 .collect();
 Some((captures, TokenStream::from_iter(input)))
}
</code></pre></div><p>Let’s go through this step by step:</p>
<ul>
<li>It returns an <code>Option</code>, because there might not be a capture list, in which case we’ll just return <code>None</code>.</li>
<li>It turns the <code>TokenStream</code> into an iterator, so we can extract the first token from it, while tracking our position in the stream.</li>
<li>It checks if the first token is a <code>Group</code> delimited by <code>[..]</code>, and extracts the tokens inside this group.</li>
<li>It loops over these tokens, checking if they are actually identifiers, and converts them to strings.</li>
<li>It returns the vector of captured variable names, together with the rest of the token stream.</li>
</ul>
<p>Let’s test it by adding it to our procedural macro:</p>
<div><pre><code>#[proc_macro]
pub fn python(input: TokenStream) -&gt; TokenStream {
 let (captures, input) = get_captures(input.clone()).unwrap_or((vec![], input));
 dbg!(captures);
 ...
</code></pre></div><p>We clone the <code>input</code> before giving it to <code>get_captures</code>, so we still have the original one around in case there was no capture list.</p>
<p>We still have to generate the <code>set_item</code>-code, but for now let’s see if parsing the capture list worked:</p>
<div><pre><code>fn main() {
 python! {
 [a b]
 ...
 }
}
</code></pre></div><div><pre><code>$ cargo b
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
[python-macro/src/lib.rs:37] captures = [
 "a",
 "b",
]
error[E0061]: this function takes 2 arguments but 1 argument was supplied
</code></pre></div><p>That part works.
It shows the names of the variables that should be captured,
and then errors out because we’re missing the second argument to <code>run_python</code>.</p>
<p>On to generating that part.</p>
<p>For a capture list like <code>[a b]</code>, our macro should generate something like:</p>
<div><pre><code>|globals| {
 globals.set_item("a", a).unwrap();
 globals.set_item("b", b).unwrap();
}
</code></pre></div><p>We can do this directly inside our existing <code>quote!()</code>:</p>
<div><pre><code> quote!(
 run_python(
 #source,
 |globals| {
 #(
 globals
 .set_item(stringify!(#captures), #captures)
 .expect("Conversion failed");
 )*
 }
 );
 ).into()
</code></pre></div><p>That <code>#(..)*</code>-syntax is <a href="https://docs.rs/quote/1.0.3/quote/macro.quote.html#interpolation">a <code>quote!</code>-feature</a>
which will repeat its contents as many times as needed. In this case, for each element of <code>captures</code>.
We again use <code>stringify!()</code> to turn the variable name into a string.</p>
<p>Let’s try!</p>
<div><pre><code>fn main() {
 let a = 3;
 let b = 20;
 python! {
 [a b]
 c = 100
 print(a + b + c)
 }
}
</code></pre></div><div><pre><code>$ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
warning: unused variable: `a`
 --&gt; src/main.rs:5:9
 |
5 | let a = 3;
 | ^ help: if this is intentional, prefix it with an underscore: `_a`
 |
 = note: `#[warn(unused_variables)]` on by default
warning: unused variable: `b`
 --&gt; src/main.rs:6:9
 |
6 | let b = 20;
 | ^ help: if this is intentional, prefix it with an underscore: `_b`
warning: 2 warnings emitted
 Finished dev [unoptimized + debuginfo] target(s) in 0.57s
 Running `target/debug/scratchpad`
Traceback (most recent call last):
 File "&lt;string&gt;", line 10, in &lt;module&gt;
NameError: name 'a' is not defined
</code></pre></div><p>Hold up. <code>a</code> unused? <code>b</code> unused? <code>name 'a'</code> not defined? What is going on?
Warnings aside, the code compiled fine. So the macro did generate valid code.</p>
<p>But it’s wrong, somehow.</p>
<p>Now how do we debug our procedural macro?</p>
<p>There is an unstable <code>rustc</code> option called <code>--pretty=expanded</code>, which will show us the code <em>after</em> all macro expansions.
Unlike what the name suggests, the output is usually not very pretty, so it’s a good idea to pass the output through <code>rustfmt</code> for readability.
The <code>cargo rustc</code> command allows us to pass custom options to <code>rustc</code>:</p>
<div><pre><code>$ cargo rustc -- -Z unstable-options --pretty=expanded | rustfmt
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.11s
&lt;snip&gt;
fn main() {
 let a = 3;
 let b = 20;
 run_python("\n\n\n\n\n\n\n\nc = 100\nprint(a + b + c)", |globals| {
 globals.set_item("\"a\"", "a").expect("Conversion failed");
 globals.set_item("\"b\"", "b").expect("Conversion failed");
 });
}
&lt;snip&gt;
</code></pre></div><p>Oh, those <code>set_item</code> calls look off.</p>
<div><pre><code>// What it generated:
globals.set_item("\"a\"", "a").expect("Conversion failed");
// What we wanted:
globals.set_item("a", a).expect("Conversion failed");
</code></pre></div><p>Instead of just the identifier <code>a</code>, <code>quote!()</code> produced the string literal <code>"a"</code>.
So instead of <code>(stringify!(a), a)</code>, we got <code>(stringify!("a"), "a")</code>, which expands to <code>("\"a\"", "a")</code>.</p>
<p>We can’t really blame <code>quote!()</code> here, because we did ask it to insert a <code>String</code> (from <code>captures</code>, which is a <code>Vec&lt;String&gt;</code>).
It makes sense that it’ll turn <code>String</code>s into string literals.</p>
<p>To fix this, we should instead give it <code>Ident</code>s, which represent identifiers.</p>
<p>Looking at the <a href="https://doc.rust-lang.org/stable/proc_macro/struct.Ident.html#method.new">documentation of <code>Ident</code></a>,
we see that <code>Ident</code>s are made from a string (the name) and a <code>Span</code>.
This span is used not only for compiler errors to display their location, but also for <em>macro hygiene</em>.
It makes sure that names defined in macros do not get mixed up with names on the outside.</p>
<p>In this case, we do want to refer to variables on the outside, and it’d be nice
if errors (e.g. about a variable not existing) would point to the place where
the user named it in the capture list.</p>
<p>This means we should not make our own <code>Ident</code>s, but simply use the original ones we got out of the <code>TokenStream</code>.
This is a pretty simple change to <code>get_captures</code>. All we have to do is change the return type:</p>
<div><pre><code>fn get_captures(input: TokenStream) -&gt; Option&lt;(Vec&lt;Ident&gt;, TokenStream)&gt;
// ^^^^^
</code></pre></div><p>And replace <code>ident.to_string()</code> by <code>ident</code>.</p>
<p>Easy!</p>
<div><pre><code>$ cargo r
 Compiling python-macro v0.1.0
error[E0277]: the trait bound `proc_macro::Ident: quote::to_tokens::ToTokens` is not satisfied
 --&gt; python-macro/src/lib.rs:47:5
 |
47 | / quote!(
48 | | run_python(
49 | | #source,
50 | | |globals| {
... |
57 | | );
58 | | ).into()
 | |_____^ the trait `quote::to_tokens::ToTokens` is not implemented for `proc_macro::Ident`
 |
 = note: required because of the requirements on the impl of `quote::to_tokens::ToTokens` for `&amp;proc_macro::Ident`
 = note: required because of the requirements on the impl of `quote::to_tokens::ToTokens` for `quote::__private::RepInterp&lt;&amp;proc_macro::Ident&gt;`
 = note: required by `quote::to_tokens::ToTokens::to_tokens`
 = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
</code></pre></div><p>Uh. Looks like <code>quote!()</code> can’t handle <code>Ident</code>? <code>quote::ToTokens</code> isn’t implemented for it?
But <a href="https://docs.rs/quote/1.0.3/quote/trait.ToTokens.html#impl-ToTokens-for-Ident">the documentation</a> said—</p>
<p>…</p>
<p>Oh. Wait a minute.</p>
<p>That <code>Ident</code> there links to <a href="https://docs.rs/proc-macro2/1.0.9/proc_macro2/struct.Ident.html"><code>proc_macro2::Ident</code></a>,
not <a href="https://doc.rust-lang.org/stable/proc_macro/struct.Ident.html"><code>proc_macro::Ident</code></a>.
What’s up with that? A second <code>proc_macro</code> crate?</p>
<p>The <a href="https://docs.rs/proc-macro2/1.0.9/proc_macro2/">documentation of <code>proc-macro2</code></a>
explains it is simply a wrapper around <code>proc_macro</code>,
but one which also allows using it outside procedural macros, where the compiler-provided <code>proc_macro</code> crate doesn’t exist.
Useful for unit testing and more.</p>
<p>For procedural macros, it suggests converting <code>proc_macro::TokenStream</code>s
directly to <code>proc_macro2::TokenStream</code>s before doing anything else,
since crates like <code>quote</code> and <code>syn</code> use <code>proc_macro2</code> for everything.</p>
<p>Okay, let’s see.</p>
<p>First we add it as a new dependency in <code>python-macro/Cargo.toml</code>:</p>
<div><pre><code>[dependencies]
quote = "1.0"
proc-macro2 = { version = "1.0", features = ["span-locations"] }
</code></pre></div><p>We enable the <code>span-locations</code> feature to enable line and column information, which is still unstable.</p>
<p>Then, in <code>python-macro/lib.rs</code>, we only have to make a few changes:</p>
<div><pre><code>use proc_macro2::{Delimiter, Ident, LineColumn, Spacing, TokenStream, TokenTree};
// ^ Only added the 2 here.
#[proc_macro]
pub fn python(input: proc_macro::TokenStream) -&gt; proc_macro::TokenStream {
 // ^^^^^^^^^^^^ ^^^^^^^^^^^^
 let input = TokenStream::from(input); // Convert it to proc_macro2's TokenStream.
 ...
</code></pre></div><p>Okay, fingers crossed…</p>
<div><pre><code>$ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.76s
 Running `target/debug/scratchpad`
123
</code></pre></div><p>Whoa. It worked!</p>
<p>Thanks to <code>quote</code>, generating the right code was quite easy.
Quite ergonomic how it allows placeholders like <code>#a</code> to substitute variables.</p>
<p>Uh. Wait. Placeholders. That gives me an idea.</p>
<h2>Placeholder syntax</h2>
<p>What if we use <code>quote</code>’s solution, instead of the one we stole from C++’s closures?</p>
<div><pre><code>let a = 3;
let b = 20;
python! {
 c = 100
 print(#a + #b + c)
}
</code></pre></div><p>I like it.</p>
<p>Now, is <code>#a</code> the best option? Or should we use <code>@a</code>, <code>$a</code>, <code>^a</code>, <code>rust:a</code>, <code>«a»</code>, or something else?</p>
<p>First of all, it needs to be something that the Rust tokeniser allows.
As we’ve seen in <a href="../writing-python-inside-rust-1a/#syntax-issues">part 1A</a>, there’s no way around that.</p>
<p>Then, it’s important to pick something that doesn’t already have a meaning in Python.
<code>#</code> is used for comments, <code>@</code> is used for <a href="https://www.python.org/dev/peps/pep-3107/">annotations</a>, etc.</p>
<p>Another consideration is syntax highlighting.
Users of our macro will probably be writing their code in an editor that knows how to syntax-highlight Rust code,
but knows nothing about our <code>python!{}</code> macro.
It’d be nice if our syntax doesn’t completely break syntax highlighting,
or better: if our placeholders would show up in some recognizable way.</p>
<p>To do that, we need to pick some existing Rust syntax, which has no meaning in Python.</p>
<p>The first thing that comes to mind is lifetime syntax: <code>'a</code>.
Most Rust editors will understand the <code>'</code> and the name afterwards belong together,
and although single quoted strings are already a thing in Python,
those are already unusable in our macro anyway.
(See <a href="../writing-python-inside-rust-1a/#single-quoted-strings">part 1A</a> for details.)</p>
<p>Sounds like this can work!</p>
<p>Here’s how that would look:</p>
<div><pre><code>let a = 3;
let b = 20;
python! {
 c = 100
 print('a + 'b + c)
}
</code></pre></div><p>Nice. Let’s do this.</p>
<h3>Implementation</h3>
<p>What do we need to do to implement this? Let’s see…</p>
<ul>
<li>Throw out the <code>get_captures</code> function and call.</li>
<li>Modify our reconstruction function to detect <code>'a</code>-syntax, replace it by a variable, and remember the <code>Ident</code> for later.</li>
<li>Modify the <code>quote!()</code> to use our new list of <code>Ident</code>s.</li>
<li>Done?</li>
</ul>
<p>Users might refer to the same variable multiple times, but we should capture them only once.
So it’s probably a good idea to use a set or map to store all the placeholders we find:</p>
<div><pre><code>struct Source {
 // &lt;&lt;snip&gt;
 captures: BTreeMap&lt;String, Ident&gt;,
}
</code></pre></div><p>In our <code>Source::reconstruct_from</code> function,
we now need to look for <code>'</code>-tokens and extract the token after it.</p>
<p>It currently looks like this:</p>
<div><pre><code> fn reconstruct_from(&amp;mut self, input: TokenStream) {
 for t in input {
 // &lt;snip&gt;
 }
 }
</code></pre></div><p>We now no longer want to process exactly one token per iteration,
but sometimes consume the next one as well (to see what’s after a <code>'</code>).
This means we can’t easily use a for-loop anymore,
as it doesn’t give us access to the underlying iterator.
Using a <code>while let</code> lets us define the iterator ourselves:</p>
<div><pre><code> let mut input = input.into_iter();
 while let Some(t) = input.next() {
 // Now we can use input.next() to consume more tokens.
 }
</code></pre></div><p>The body of the loop looked like this:</p>
<div><pre><code> if let TokenTree::Group(g) = t {
 // &lt;snip&gt;
 } else {
 self.add_whitespace(t.span().start());
 self.add_str(&amp;t.to_string());
 }
</code></pre></div><p>There’s one case for handling group tokens and to recurse into them,
and another case for all other tokens, including <code>'</code>-tokens.</p>
<p>Right after the <code>add_whitespace</code> in the <code>else</code> case, we add:</p>
<div><pre><code> if let TokenTree::Punct(t) = &amp;t {
 if t.as_char() == '\'' {
 if let Some(TokenTree::Ident(var)) = input.next() {
 let varname = format!("_rust_{}", var);
 self.col += var.to_string().len() + 1;
 self.source += &amp;varname;
 self.captures.entry(varname).or_insert(var);
 continue;
 } else {
 unreachable!();
 }
 }
 }
</code></pre></div><p>If the token we see is a <code>'</code>-token, we consume the next token
and assume that’s an identifier.
(If it wasn’t, the Rust tokeniser would’ve already rejected the code.)</p>
<p>Instead of <code>'a</code>, we add <code>_rust_a</code> to the Python code (since we can’t name variables <code>'a</code>),
and remember both this name and the <code>Ident</code> in our <code>capatures</code> map if it wasn’t seen before.</p>
<p>Finally, we update our <code>quote!()</code>, which doesn’t change much:</p>
<div><pre><code> let name = s.captures.keys();
 let var = s.captures.values();
 quote!(
 run_python(
 #source,
 |globals| {
 #(
 globals
 .set_item(#name, #var)
 .expect("Conversion failed");
 )*
 }
 );
 ).into()
</code></pre></div><p>Done?</p>
<p>Let’s try!</p>
<div><pre><code>fn main() {
 let a = 3;
 let b = 20;
 python! {
 c = 100
 print('a + 'b + c)
 }
}
</code></pre></div><div><pre><code>$ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.72s
 Running `target/debug/scratchpad`
123
</code></pre></div><p>Success!</p>
<p>🎉</p>
<h2>Identifier spans</h2>
<p>Since we used the original <code>Ident</code>s everywhere, we should get nice error messages
when trying to use a variable that doesn’t exist:</p>
<div><pre><code>let a = 3;
let b = 20;
python! {
 print('a + 'b + 'c)
}
</code></pre></div><div><pre><code>$ cargo r
 Compiling scratchpad v0.1.0
error[E0425]: cannot find value `c` in this scope
 --&gt; src/main.rs:8:25
 |
8 | print('a + 'b + 'c)
 | ^^ help: a local variable with a similar name exists: `a`
</code></pre></div><p>Awesome!</p>
<p>Note how it points to both the <code>'</code> and <code>c</code>.
<code>rustc</code> knows these two tokens belong together,
because we used a syntax it is already familiar with.</p>
<h2>What’s next</h2>
<p>Now we have a way to very easily get data into our <code>python!{}</code> blocks.
It works for everything that implements <code>ToPyObject</code>, including strings, numbers,
and all kind of collections.</p>
<p>What’s still missing, is a nice way to get data out.
We’ll look at that in a later post,
but first we’re going to look at a very different topic.</p>
<p>In <a href="../writing-python-inside-rust-3">part 3</a>,
we’re going to <a href="https://docs.python.org/3/library/py_compile.html">‘compile’</a>
the Python code into <a href="https://towardsdatascience.com/understanding-python-bytecode-e7edaae8734d">Python byte-code</a>
<em>at compile time</em>, to catch errors like invalid Python syntax before even running it.
This also speeds up execution times,
by moving part of the work into the compilation step.</p>
 </div>||||I|||| Mara's Blog

 Writing Python inside your Rust code — Part 2

 2020-04-25 2456 words 12 mins read

 Contents

 * Defining globals for Python
 * Transparent syntax
 * Capture list
 + Implementation
 * Placeholder syntax
 + Implementation
 * Identifier spans
 * What’s next

 In this part, we’ll extend our python!{}-macro to be able to seamlessly use Rust variables in the Python code within. We explore a few options, and implement two alternatives.

 Previously: Part 1A

 Next: Part 3

 Defining globals for Python

 In Part 1, we wrote a run_python function that executes Python code using PyO3’s Python::run function:

 fn run_python(code: &str) {
 let py = pyo3::Python::acquire_gil();
 if let Err(e) = py.python().run(code, None, None) {
 e.print(py.python());
 panic!("Python code failed");
 }
 }
 

 Let’s take a look again at the signature of pyo3::Python::run:

 pub fn run(
 self,
 code: &str,
 globals: Option<&PyDict>,
 locals: Option<&PyDict>
 ) -> PyResult<()>
 

 Looks like we can give a PyDict of global variables where we give None now.

 Setting variables inside a PyDict looks easy enough: PyDict::set_item only requires the key and value to implement ToPyObject, which is already implemented for a lot of types, including strings, integers, vectors, and many more.

 Let’s see how it works, by defining five = 5 from Rust:

 fn run_python(code: &str) {
 let py = pyo3::Python::acquire_gil();
 let globals = pyo3::types::PyDict::new(py.python());
 
 // "five" and 5 are automatically converted to a PyObject by ToPyObject.
 globals.set_item("five", 5).unwrap();
 
 if let Err(e) = py.python().run(code, Some(globals), None) {
 e.print(py.python());
 panic!("Python code failed");
 }
 }
 
 fn main() {
 python! {
 print(five + 1)
 }
 }
 
 $ cargo r
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.34s
 Running `target/debug/scratchpad`
 6
 

 Hey, that works!

 Transparent syntax

 Now that we know converting objects from Rust to Python is not going to be a problem (thanks to PyO3’s ToPyObject), we can move on to the problem of how the user will indicate which of their variables need to be converted and injected in the globals dictionary.

 The most ergonomic way would probably be something that’s completely transparent:

 let a = 5;
 python! {
 b = 10
 print(a + b)
 }
 

 If we could make this work, users could just refer to any variable in scope as if they didn’t even switch languages. Sounds perfect.

 However, our procedural macro does not have access to the surrounding code, so would not know that a even exists in the Rust code. In order to know a comes from Rust, but b does not, it’d have to understand the Python code and see that a is used without being initialized, but b is initialized inside the Python code. This goes much further than parsing b = 10 and understanding that defines b, meaning it should not be captured. For example, print is not explicitly defined anywhere, but it does not refer to anything from Rust.

 If we think a bit more about this, it only gets worse:

 let a = 5;
 python! {
 from somelibrary import *
 globals()['c'] = 10
 print(a + b + c)
 }
 

 Without fully parsing and running all of the Python code, we can’t possibly do this.

 So, there needs to be some way the user can tell the macro which variables need to be captured.

 Capture list

 Capturing variables.. That sounds like something closures (sometimes referred to as lambdas) do. Maybe we can draw some inspiration from there.

 In Rust, a closure is defined using ||:

 let a = 5;
 let f = |b| a + b;
 assert_eq!(f(10), 15);
 

 Here, f behaves like a function. It takes one argument (b), and returns a + b. a was implicitly captured from the environment. This implicit capturing is exactly what we discussed before, which is not feasible for python!{}.

 In Rust, closures always implicitly capture whatever they need from their environment, as described in the Rust Book. However, if we look at closures in other languages, we see that C++ is one of the few where we can manually specify what should be captured, using []-syntax:

 int main() {
 int a = 5;
 
 auto x = [a] (int b) { return a + b; };
 // └┬┘ └──┬──┘
 // │ └── Parameter list
 // └── Capture list
 
 assert(x(10) == 15);
 }
 

 Looks like C++ already solved our problem, using a capture list. Let’s just steal the idea:

 let a = 3;
 let b = 20;
 python! {
 [a b] // List of variables to capture
 c = 100
 print(a + b + c)
 }
 

 I’ve omitted commas or any other syntax apart from the [] and the names, to keep parsing as simple as possible. If it turns out to work well, we can always improve the syntax later.

 Implementation

 Alright, let’s see if we can implement this.

 We’ll have to parse the capture list and generate code that calls globals.set_item("var", var) for each variable. This code should end up in run_python and executed after making the globals dictionary, but before executing Python::run.

 Passing arbitrary code to a function is easy in Rust: using impl Fn, run_python can accept a closure containing the code with all the set_item calls. We’ll have to give the closure access to the PyDict:

 fn run_python(
 code: &str,
 f: impl FnOnce(&pyo3::types::PyDict), // new
 ) {
 let py = pyo3::Python::acquire_gil();
 let globals = pyo3::types::PyDict::new(py.python());
 
 f(globals); // new
 
 if let Err(e) = py.python().run(code, Some(globals), None) {
 e.print(py.python());
 panic!("Python code failed");
 }
 }
 

 We use FnOnce instead of Fn, because we only call the function once. This means it can also accept closures that move out of their own captures, and therefore cannot be called more than once.

 Now on to the hard part, the macro implementation.

 First, a function that checks if there’s a capture list, and extracts the names from it:

 fn get_captures(input: TokenStream) -> Option<(Vec<String>, TokenStream)> {
 let mut input = input.into_iter();
 
 let captures = match input.next() {
 Some(TokenTree::Group(g)) if g.delimiter() == Delimiter::Bracket => g.stream(),
 _ => return None,
 };
 
 let captures = captures
 .into_iter()
 .map(|token| {
 if let TokenTree::Ident(ident) = token {
 ident.to_string()
 } else {
 panic!("Invalid token in capture list");
 }
 })
 .collect();
 
 Some((captures, TokenStream::from_iter(input)))
 }
 

 Let’s go through this step by step:

 * It returns an Option, because there might not be a capture list, in which case we’ll just return None.
 * It turns the TokenStream into an iterator, so we can extract the first token from it, while tracking our position in the stream.
 * It checks if the first token is a Group delimited by [..], and extracts the tokens inside this group.
 * It loops over these tokens, checking if they are actually identifiers, and converts them to strings.
 * It returns the vector of captured variable names, together with the rest of the token stream.

 Let’s test it by adding it to our procedural macro:

 #[proc_macro]
 pub fn python(input: TokenStream) -> TokenStream {
 let (captures, input) = get_captures(input.clone()).unwrap_or((vec![], input));
 
 dbg!(captures);
 
 ...
 

 We clone the input before giving it to get_captures, so we still have the original one around in case there was no capture list.

 We still have to generate the set_item-code, but for now let’s see if parsing the capture list worked:

 fn main() {
 python! {
 [a b]
 ...
 }
 }
 
 $ cargo b
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 [python-macro/src/lib.rs:37] captures = [
 "a",
 "b",
 ]
 error[E0061]: this function takes 2 arguments but 1 argument was supplied
 

 That part works. It shows the names of the variables that should be captured, and then errors out because we’re missing the second argument to run_python.

 On to generating that part.

 For a capture list like [a b], our macro should generate something like:

 |globals| {
 globals.set_item("a", a).unwrap();
 globals.set_item("b", b).unwrap();
 }
 

 We can do this directly inside our existing quote!():

 quote!(
 run_python(
 #source,
 |globals| {
 #(
 globals
 .set_item(stringify!(#captures), #captures)
 .expect("Conversion failed");
 )*
 }
 );
 ).into()
 

 That #(..)*-syntax is a quote!-feature which will repeat its contents as many times as needed. In this case, for each element of captures. We again use stringify!() to turn the variable name into a string.

 Let’s try!

 fn main() {
 let a = 3;
 let b = 20;
 python! {
 [a b]
 c = 100
 print(a + b + c)
 }
 }
 
 $ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 warning: unused variable: `a`
 --> src/main.rs:5:9
 |
 5 | let a = 3;
 | ^ help: if this is intentional, prefix it with an underscore: `_a`
 |
 = note: `#[warn(unused_variables)]` on by default
 
 warning: unused variable: `b`
 --> src/main.rs:6:9
 |
 6 | let b = 20;
 | ^ help: if this is intentional, prefix it with an underscore: `_b`
 
 warning: 2 warnings emitted
 
 Finished dev [unoptimized + debuginfo] target(s) in 0.57s
 Running `target/debug/scratchpad`
 Traceback (most recent call last):
 File "<string>", line 10, in <module>
 NameError: name 'a' is not defined
 

 Hold up. a unused? b unused? name 'a' not defined? What is going on? Warnings aside, the code compiled fine. So the macro did generate valid code.

 But it’s wrong, somehow.

 Now how do we debug our procedural macro?

 There is an unstable rustc option called --pretty=expanded, which will show us the code after all macro expansions. Unlike what the name suggests, the output is usually not very pretty, so it’s a good idea to pass the output through rustfmt for readability. The cargo rustc command allows us to pass custom options to rustc:

 $ cargo rustc -- -Z unstable-options --pretty=expanded | rustfmt
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.11s
 <snip>
 
 fn main() {
 let a = 3;
 let b = 20;
 run_python("\n\n\n\n\n\n\n\nc = 100\nprint(a + b + c)", |globals| {
 globals.set_item("\"a\"", "a").expect("Conversion failed");
 globals.set_item("\"b\"", "b").expect("Conversion failed");
 });
 }
 
 <snip>
 

 Oh, those set_item calls look off.

 // What it generated:
 globals.set_item("\"a\"", "a").expect("Conversion failed");
 
 // What we wanted:
 globals.set_item("a", a).expect("Conversion failed");
 

 Instead of just the identifier a, quote!() produced the string literal "a". So instead of (stringify!(a), a), we got (stringify!("a"), "a"), which expands to ("\"a\"", "a").

 We can’t really blame quote!() here, because we did ask it to insert a String (from captures, which is a Vec<String>). It makes sense that it’ll turn Strings into string literals.

 To fix this, we should instead give it Idents, which represent identifiers.

 Looking at the documentation of Ident, we see that Idents are made from a string (the name) and a Span. This span is used not only for compiler errors to display their location, but also for macro hygiene. It makes sure that names defined in macros do not get mixed up with names on the outside.

 In this case, we do want to refer to variables on the outside, and it’d be nice if errors (e.g. about a variable not existing) would point to the place where the user named it in the capture list.

 This means we should not make our own Idents, but simply use the original ones we got out of the TokenStream. This is a pretty simple change to get_captures. All we have to do is change the return type:

 fn get_captures(input: TokenStream) -> Option<(Vec<Ident>, TokenStream)>
 // ^^^^^
 

 And replace ident.to_string() by ident.

 Easy!

 $ cargo r
 Compiling python-macro v0.1.0
 error[E0277]: the trait bound `proc_macro::Ident: quote::to_tokens::ToTokens` is not satisfied
 --> python-macro/src/lib.rs:47:5
 |
 47 | / quote!(
 48 | | run_python(
 49 | | #source,
 50 | | |globals| {
 ... |
 57 | | );
 58 | | ).into()
 | |_____^ the trait `quote::to_tokens::ToTokens` is not implemented for `proc_macro::Ident`
 |
 = note: required because of the requirements on the impl of `quote::to_tokens::ToTokens` for `&proc_macro::Ident`
 = note: required because of the requirements on the impl of `quote::to_tokens::ToTokens` for `quote::__private::RepInterp<&proc_macro::Ident>`
 = note: required by `quote::to_tokens::ToTokens::to_tokens`
 = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
 

 Uh. Looks like quote!() can’t handle Ident? quote::ToTokens isn’t implemented for it? But the documentation said—

 impl ToTokens for Ident
 

 …

 Oh. Wait a minute.

 That Ident there links to proc_macro2::Ident, not proc_macro::Ident. What’s up with that? A second proc_macro crate?

 The documentation of proc-macro2 explains it is simply a wrapper around proc_macro, but one which also allows using it outside procedural macros, where the compiler-provided proc_macro crate doesn’t exist. Useful for unit testing and more.

 For procedural macros, it suggests converting proc_macro::TokenStreams directly to proc_macro2::TokenStreams before doing anything else, since crates like quote and syn use proc_macro2 for everything.

 Okay, let’s see.

 First we add it as a new dependency in python-macro/Cargo.toml:

 [dependencies]
 quote = "1.0"
 proc-macro2 = { version = "1.0", features = ["span-locations"] }
 

 We enable the span-locations feature to enable line and column information, which is still unstable.

 Then, in python-macro/lib.rs, we only have to make a few changes:

 use proc_macro2::{Delimiter, Ident, LineColumn, Spacing, TokenStream, TokenTree};
 // ^ Only added the 2 here.
 
 #[proc_macro]
 pub fn python(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
 // ^^^^^^^^^^^^ ^^^^^^^^^^^^
 
 let input = TokenStream::from(input); // Convert it to proc_macro2's TokenStream.
 
 ...
 

 Okay, fingers crossed…

 $ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.76s
 Running `target/debug/scratchpad`
 123
 

 Whoa. It worked!

 Thanks to quote, generating the right code was quite easy. Quite ergonomic how it allows placeholders like #a to substitute variables.

 Uh. Wait. Placeholders. That gives me an idea.

 Placeholder syntax

 What if we use quote’s solution, instead of the one we stole from C++’s closures?

 let a = 3;
 let b = 20;
 python! {
 c = 100
 print(#a + #b + c)
 }
 

 I like it.

 Now, is #a the best option? Or should we use @a, $a, ^a, rust:a, «a», or something else?

 First of all, it needs to be something that the Rust tokeniser allows. As we’ve seen in part 1A, there’s no way around that.

 Then, it’s important to pick something that doesn’t already have a meaning in Python. # is used for comments, @ is used for annotations, etc.

 Another consideration is syntax highlighting. Users of our macro will probably be writing their code in an editor that knows how to syntax-highlight Rust code, but knows nothing about our python!{} macro. It’d be nice if our syntax doesn’t completely break syntax highlighting, or better: if our placeholders would show up in some recognizable way.

 To do that, we need to pick some existing Rust syntax, which has no meaning in Python.

 The first thing that comes to mind is lifetime syntax: 'a. Most Rust editors will understand the ' and the name afterwards belong together, and although single quoted strings are already a thing in Python, those are already unusable in our macro anyway. (See part 1A for details.)

 Sounds like this can work!

 Here’s how that would look:

 let a = 3;
 let b = 20;
 python! {
 c = 100
 print('a + 'b + c)
 }
 

 Nice. Let’s do this.

 Implementation

 What do we need to do to implement this? Let’s see…

 * Throw out the get_captures function and call.
 * Modify our reconstruction function to detect 'a-syntax, replace it by a variable, and remember the Ident for later.
 * Modify the quote!() to use our new list of Idents.
 * Done?

 Users might refer to the same variable multiple times, but we should capture them only once. So it’s probably a good idea to use a set or map to store all the placeholders we find:

 struct Source {
 // <<snip>
 captures: BTreeMap<String, Ident>,
 }
 

 In our Source::reconstruct_from function, we now need to look for '-tokens and extract the token after it.

 It currently looks like this:

 fn reconstruct_from(&mut self, input: TokenStream) {
 for t in input {
 // <snip>
 }
 }
 

 We now no longer want to process exactly one token per iteration, but sometimes consume the next one as well (to see what’s after a '). This means we can’t easily use a for-loop anymore, as it doesn’t give us access to the underlying iterator. Using a while let lets us define the iterator ourselves:

 let mut input = input.into_iter();
 while let Some(t) = input.next() {
 // Now we can use input.next() to consume more tokens.
 }
 

 The body of the loop looked like this:

 if let TokenTree::Group(g) = t {
 // <snip>
 } else {
 self.add_whitespace(t.span().start());
 self.add_str(&t.to_string());
 }
 

 There’s one case for handling group tokens and to recurse into them, and another case for all other tokens, including '-tokens.

 Right after the add_whitespace in the else case, we add:

 if let TokenTree::Punct(t) = &t {
 if t.as_char() == '\'' {
 if let Some(TokenTree::Ident(var)) = input.next() {
 let varname = format!("_rust_{}", var);
 self.col += var.to_string().len() + 1;
 self.source += &varname;
 self.captures.entry(varname).or_insert(var);
 continue;
 } else {
 unreachable!();
 }
 }
 }
 

 If the token we see is a '-token, we consume the next token and assume that’s an identifier. (If it wasn’t, the Rust tokeniser would’ve already rejected the code.)

 Instead of 'a, we add _rust_a to the Python code (since we can’t name variables 'a), and remember both this name and the Ident in our capatures map if it wasn’t seen before.

 Finally, we update our quote!(), which doesn’t change much:

 let name = s.captures.keys();
 let var = s.captures.values();
 
 quote!(
 run_python(
 #source,
 |globals| {
 #(
 globals
 .set_item(#name, #var)
 .expect("Conversion failed");
 )*
 }
 );
 ).into()
 

 Done?

 Let’s try!

 fn main() {
 let a = 3;
 let b = 20;
 python! {
 c = 100
 print('a + 'b + c)
 }
 }
 
 $ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.72s
 Running `target/debug/scratchpad`
 123
 

 Success!

 🎉

 Identifier spans

 Since we used the original Idents everywhere, we should get nice error messages when trying to use a variable that doesn’t exist:

 let a = 3;
 let b = 20;
 python! {
 print('a + 'b + 'c)
 }
 
 $ cargo r
 Compiling scratchpad v0.1.0
 error[E0425]: cannot find value `c` in this scope
 --> src/main.rs:8:25
 |
 8 | print('a + 'b + 'c)
 | ^^ help: a local variable with a similar name exists: `a`
 

 Awesome!

 Note how it points to both the ' and c. rustc knows these two tokens belong together, because we used a syntax it is already familiar with.

 What’s next

 Now we have a way to very easily get data into our python!{} blocks. It works for everything that implements ToPyObject, including strings, numbers, and all kind of collections.

 What’s still missing, is a nice way to get data out. We’ll look at that in a later post, but first we’re going to look at a very different topic.

 In part 3, we’re going to ‘compile’ the Python code into Python byte-code at compile time, to catch errors like invalid Python syntax before even running it. This also speeds up execution times, by moving part of the work into the compilation step.

 Rust Python inline-python
 Please enable JavaScript to view the comments.

 © 2013–2022 ♥ Mara Bos

 Powered by Hugo | Theme derived from Even
In this part, we’ll extend our python!{}-macro to be able to seamlessly
use Rust variables in the Python code within.
We explore a few options, and implement two alternatives.
Defining globals for Python
In Part 1,
we wrote a run_python function that executes Python code using PyO3’s Python::run function:
  fn run_python(code: &amp;str) {
 let py = pyo3::Python::acquire_gil();
 if let Err(e) = py.python().run(code, None, None) {
 e.print(py.python());
 panic!("Python code failed");
 }
}
   Let’s take a look again at the signature of pyo3::Python::run :
   pub fn run(
 self,
 code: &amp;str,
 globals: Option&lt;&amp;PyDict&gt;,
 locals: Option&lt;&amp;PyDict&gt;
 ) -&gt; PyResult&lt;()&gt;
   Looks like we can give a PyDict of global variables where we give None now.
Setting variables inside a PyDict looks easy enough:  PyDict::set_item 
only requires the key and value to implement  ToPyObject ,
which is already implemented for a lot of types, including strings, integers, vectors, and many more.
Let’s see how it works, by defining five = 5 from Rust:
  fn run_python(code: &amp;str) {
 let py = pyo3::Python::acquire_gil();
 let globals = pyo3::types::PyDict::new(py.python());
 // "five" and 5 are automatically converted to a PyObject by ToPyObject.
 globals.set_item("five", 5).unwrap();
 if let Err(e) = py.python().run(code, Some(globals), None) {
 e.print(py.python());
 panic!("Python code failed");
 }
}
     fn main() {
 python! {
 print(five + 1)
 }
}
     $ cargo r
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.34s
 Running `target/debug/scratchpad`
6
   Hey, that works!
Transparent syntax
Now that we know converting objects from Rust to Python is not going to be a problem (thanks to PyO3’s ToPyObject),
we can move on to the problem of how the user will indicate which of their variables need to be converted and injected in the globals dictionary.
The most ergonomic way would probably be something that’s completely transparent:
  let a = 5;
python! {
 b = 10
 print(a + b)
}
   If we could make this work, users could just refer to any variable in scope as if they didn’t even switch languages.
Sounds perfect.
However, our procedural macro does not have access to the surrounding code,
so would not know that a even exists in the Rust code.
In order to know a comes from Rust, but b does not,
it’d have to understand the Python code and see that a is used without being initialized,
but b is initialized inside the Python code.
This goes much further than parsing b = 10 and understanding that defines b, meaning it should not be captured.
For example, print is not explicitly defined anywhere, but it does not refer to anything from Rust.
If we think a bit more about this, it only gets worse:
  let a = 5;
python! {
 from somelibrary import *
 globals()['c'] = 10
 print(a + b + c)
}
   Without fully parsing and running all of the Python code, we can’t possibly do this.
So, there needs to be some way the user can tell the macro which variables need to be captured.
Capture list
Capturing variables..
That sounds like something closures (sometimes referred to as lambdas) do.
Maybe we can draw some inspiration from there.
In Rust, a closure is defined using ||:
  let a = 5;
let f = |b| a + b;
assert_eq!(f(10), 15);
   Here, f behaves like a function. It takes one argument (b), and returns a + b.
a was implicitly captured from the environment.
This implicit capturing is exactly what we discussed before,
which is not feasible for python!{}.
In Rust, closures always implicitly capture whatever they need from their environment,
as described in the Rust Book.
However, if we look at closures in other languages, we see that C++ is one of the few where
we can manually specify what should be captured,
using []-syntax:
  int main() {
 int a = 5;
 auto x = [a] (int b) { return a + b; };
 // └┬┘ └──┬──┘
 // │ └── Parameter list
 // └── Capture list
 assert(x(10) == 15);
}
   Looks like C++ already solved our problem, using a capture list.
Let’s just steal the idea:
  let a = 3;
let b = 20;
python! {
 [a b] // List of variables to capture
 c = 100
 print(a + b + c)
}
   I’ve omitted commas or any other syntax apart from the [] and the names,
to keep parsing as simple as possible.
If it turns out to work well, we can always improve the syntax later.
Implementation
Alright, let’s see if we can implement this.
We’ll have to parse the capture list
and generate code that calls globals.set_item("var", var) for each variable.
This code should end up in run_python and executed after making the globals dictionary,
but before executing Python::run.
Passing arbitrary code to a function is easy in Rust:
using impl Fn, run_python can accept a closure containing the code with all the set_item calls.
We’ll have to give the closure access to the PyDict:
  fn run_python(
 code: &amp;str,
 f: impl FnOnce(&amp;pyo3::types::PyDict), // new
) {
 let py = pyo3::Python::acquire_gil();
 let globals = pyo3::types::PyDict::new(py.python());
 f(globals); // new
 if let Err(e) = py.python().run(code, Some(globals), None) {
 e.print(py.python());
 panic!("Python code failed");
 }
}
   We use FnOnce instead of Fn, because we only call the function once.
This means it can also accept closures that move out of their own captures,
and therefore cannot be called more than once.
Now on to the hard part, the macro implementation.
First, a function that checks if there’s a capture list, and extracts the names from it:
  fn get_captures(input: TokenStream) -&gt; Option&lt;(Vec&lt;String&gt;, TokenStream)&gt; {
 let mut input = input.into_iter();
 let captures = match input.next() {
 Some(TokenTree::Group(g)) if g.delimiter() == Delimiter::Bracket =&gt; g.stream(),
 _ =&gt; return None,
 };
 let captures = captures
 .into_iter()
 .map(|token| {
 if let TokenTree::Ident(ident) = token {
 ident.to_string()
 } else {
 panic!("Invalid token in capture list");
 }
 })
 .collect();
 Some((captures, TokenStream::from_iter(input)))
}
   Let’s go through this step by step:
It returns an Option, because there might not be a capture list, in which case we’ll just return None.
It turns the TokenStream into an iterator, so we can extract the first token from it, while tracking our position in the stream.
It checks if the first token is a Group delimited by [..], and extracts the tokens inside this group.
It loops over these tokens, checking if they are actually identifiers, and converts them to strings.
It returns the vector of captured variable names, together with the rest of the token stream.
Let’s test it by adding it to our procedural macro:
  #[proc_macro]
pub fn python(input: TokenStream) -&gt; TokenStream {
 let (captures, input) = get_captures(input.clone()).unwrap_or((vec![], input));
 dbg!(captures);
 ...
   We clone the input before giving it to get_captures, so we still have the original one around in case there was no capture list.
We still have to generate the set_item-code, but for now let’s see if parsing the capture list worked:
  fn main() {
 python! {
 [a b]
 ...
 }
}
     $ cargo b
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
[python-macro/src/lib.rs:37] captures = [
 "a",
 "b",
]
error[E0061]: this function takes 2 arguments but 1 argument was supplied
   That part works.
It shows the names of the variables that should be captured,
and then errors out because we’re missing the second argument to run_python.
On to generating that part.
For a capture list like [a b], our macro should generate something like:
  |globals| {
 globals.set_item("a", a).unwrap();
 globals.set_item("b", b).unwrap();
}
   We can do this directly inside our existing quote!():
   quote!(
 run_python(
 #source,
 |globals| {
 #(
 globals
 .set_item(stringify!(#captures), #captures)
 .expect("Conversion failed");
 )*
 }
 );
 ).into()
   That #(..)*-syntax is a quote!-feature
which will repeat its contents as many times as needed. In this case, for each element of captures.
We again use stringify!() to turn the variable name into a string.
Let’s try!
  fn main() {
 let a = 3;
 let b = 20;
 python! {
 [a b]
 c = 100
 print(a + b + c)
 }
}
     $ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
warning: unused variable: `a`
 --&gt; src/main.rs:5:9
 |
5 | let a = 3;
 | ^ help: if this is intentional, prefix it with an underscore: `_a`
 |
 = note: `#[warn(unused_variables)]` on by default
warning: unused variable: `b`
 --&gt; src/main.rs:6:9
 |
6 | let b = 20;
 | ^ help: if this is intentional, prefix it with an underscore: `_b`
warning: 2 warnings emitted
 Finished dev [unoptimized + debuginfo] target(s) in 0.57s
 Running `target/debug/scratchpad`
Traceback (most recent call last):
 File "&lt;string&gt;", line 10, in &lt;module&gt;
NameError: name 'a' is not defined
   Hold up. a unused? b unused? name 'a' not defined? What is going on?
Warnings aside, the code compiled fine. So the macro did generate valid code.
But it’s wrong, somehow.
Now how do we debug our procedural macro?
There is an unstable rustc option called --pretty=expanded, which will show us the code after all macro expansions.
Unlike what the name suggests, the output is usually not very pretty, so it’s a good idea to pass the output through rustfmt for readability.
The cargo rustc command allows us to pass custom options to rustc:
  $ cargo rustc -- -Z unstable-options --pretty=expanded | rustfmt
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.11s
&lt;snip&gt;
fn main() {
 let a = 3;
 let b = 20;
 run_python("\n\n\n\n\n\n\n\nc = 100\nprint(a + b + c)", |globals| {
 globals.set_item("\"a\"", "a").expect("Conversion failed");
 globals.set_item("\"b\"", "b").expect("Conversion failed");
 });
}
&lt;snip&gt;
   Oh, those set_item calls look off.
  // What it generated:
globals.set_item("\"a\"", "a").expect("Conversion failed");
// What we wanted:
globals.set_item("a", a).expect("Conversion failed");
   Instead of just the identifier a, quote!() produced the string literal "a".
So instead of (stringify!(a), a), we got (stringify!("a"), "a"), which expands to ("\"a\"", "a").
We can’t really blame quote!() here, because we did ask it to insert a String (from captures, which is a Vec&lt;String&gt;).
It makes sense that it’ll turn Strings into string literals.
To fix this, we should instead give it Idents, which represent identifiers.
Looking at the documentation of Ident ,
we see that Idents are made from a string (the name) and a Span.
This span is used not only for compiler errors to display their location, but also for macro hygiene.
It makes sure that names defined in macros do not get mixed up with names on the outside.
In this case, we do want to refer to variables on the outside, and it’d be nice
if errors (e.g. about a variable not existing) would point to the place where
the user named it in the capture list.
This means we should not make our own Idents, but simply use the original ones we got out of the TokenStream.
This is a pretty simple change to get_captures. All we have to do is change the return type:
  fn get_captures(input: TokenStream) -&gt; Option&lt;(Vec&lt;Ident&gt;, TokenStream)&gt;
// ^^^^^
   And replace ident.to_string() by ident.
Easy!
  $ cargo r
 Compiling python-macro v0.1.0
error[E0277]: the trait bound `proc_macro::Ident: quote::to_tokens::ToTokens` is not satisfied
 --&gt; python-macro/src/lib.rs:47:5
 |
47 | / quote!(
48 | | run_python(
49 | | #source,
50 | | |globals| {
... |
57 | | );
58 | | ).into()
 | |_____^ the trait `quote::to_tokens::ToTokens` is not implemented for `proc_macro::Ident`
 |
 = note: required because of the requirements on the impl of `quote::to_tokens::ToTokens` for `&amp;proc_macro::Ident`
 = note: required because of the requirements on the impl of `quote::to_tokens::ToTokens` for `quote::__private::RepInterp&lt;&amp;proc_macro::Ident&gt;`
 = note: required by `quote::to_tokens::ToTokens::to_tokens`
 = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
   Uh. Looks like quote!() can’t handle Ident? quote::ToTokens isn’t implemented for it?
But the documentation said—
…
Oh. Wait a minute.
That Ident there links to  proc_macro2::Ident ,
not  proc_macro::Ident .
What’s up with that? A second proc_macro crate?
The documentation of proc-macro2 
explains it is simply a wrapper around proc_macro,
but one which also allows using it outside procedural macros, where the compiler-provided proc_macro crate doesn’t exist.
Useful for unit testing and more.
For procedural macros, it suggests converting proc_macro::TokenStreams
directly to proc_macro2::TokenStreams before doing anything else,
since crates like quote and syn use proc_macro2 for everything.
Okay, let’s see.
First we add it as a new dependency in python-macro/Cargo.toml:
  [dependencies]
quote = "1.0"
proc-macro2 = { version = "1.0", features = ["span-locations"] }
   We enable the span-locations feature to enable line and column information, which is still unstable.
Then, in python-macro/lib.rs, we only have to make a few changes:
  use proc_macro2::{Delimiter, Ident, LineColumn, Spacing, TokenStream, TokenTree};
// ^ Only added the 2 here.
#[proc_macro]
pub fn python(input: proc_macro::TokenStream) -&gt; proc_macro::TokenStream {
 // ^^^^^^^^^^^^ ^^^^^^^^^^^^
 let input = TokenStream::from(input); // Convert it to proc_macro2's TokenStream.
 ...
   Okay, fingers crossed…
  $ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.76s
 Running `target/debug/scratchpad`
123
   Whoa. It worked!
Thanks to quote, generating the right code was quite easy.
Quite ergonomic how it allows placeholders like #a to substitute variables.
Uh. Wait. Placeholders. That gives me an idea.
Placeholder syntax
What if we use quote’s solution, instead of the one we stole from C++’s closures?
  let a = 3;
let b = 20;
python! {
 c = 100
 print(#a + #b + c)
}
   I like it.
Now, is #a the best option? Or should we use @a, $a, ^a, rust:a, «a», or something else?
First of all, it needs to be something that the Rust tokeniser allows.
As we’ve seen in part 1A, there’s no way around that.
Then, it’s important to pick something that doesn’t already have a meaning in Python.
# is used for comments, @ is used for annotations, etc.
Another consideration is syntax highlighting.
Users of our macro will probably be writing their code in an editor that knows how to syntax-highlight Rust code,
but knows nothing about our python!{} macro.
It’d be nice if our syntax doesn’t completely break syntax highlighting,
or better: if our placeholders would show up in some recognizable way.
To do that, we need to pick some existing Rust syntax, which has no meaning in Python.
The first thing that comes to mind is lifetime syntax: 'a.
Most Rust editors will understand the ' and the name afterwards belong together,
and although single quoted strings are already a thing in Python,
those are already unusable in our macro anyway.
(See part 1A for details.)
Sounds like this can work!
Here’s how that would look:
  let a = 3;
let b = 20;
python! {
 c = 100
 print('a + 'b + c)
}
   Nice. Let’s do this.
Implementation
What do we need to do to implement this? Let’s see…
Throw out the get_captures function and call.
Modify our reconstruction function to detect 'a-syntax, replace it by a variable, and remember the Ident for later.
Modify the quote!() to use our new list of Idents.
Done?
Users might refer to the same variable multiple times, but we should capture them only once.
So it’s probably a good idea to use a set or map to store all the placeholders we find:
  struct Source {
 // &lt;&lt;snip&gt;
 captures: BTreeMap&lt;String, Ident&gt;,
}
   In our Source::reconstruct_from function,
we now need to look for '-tokens and extract the token after it.
It currently looks like this:
   fn reconstruct_from(&amp;mut self, input: TokenStream) {
 for t in input {
 // &lt;snip&gt;
 }
 }
   We now no longer want to process exactly one token per iteration,
but sometimes consume the next one as well (to see what’s after a ').
This means we can’t easily use a for-loop anymore,
as it doesn’t give us access to the underlying iterator.
Using a while let lets us define the iterator ourselves:
   let mut input = input.into_iter();
 while let Some(t) = input.next() {
 // Now we can use input.next() to consume more tokens.
 }
   The body of the loop looked like this:
   if let TokenTree::Group(g) = t {
 // &lt;snip&gt;
 } else {
 self.add_whitespace(t.span().start());
 self.add_str(&amp;t.to_string());
 }
   There’s one case for handling group tokens and to recurse into them,
and another case for all other tokens, including '-tokens.
Right after the add_whitespace in the else case, we add:
   if let TokenTree::Punct(t) = &amp;t {
 if t.as_char() == '\'' {
 if let Some(TokenTree::Ident(var)) = input.next() {
 let varname = format!("_rust_{}", var);
 self.col += var.to_string().len() + 1;
 self.source += &amp;varname;
 self.captures.entry(varname).or_insert(var);
 continue;
 } else {
 unreachable!();
 }
 }
 }
   If the token we see is a '-token, we consume the next token
and assume that’s an identifier.
(If it wasn’t, the Rust tokeniser would’ve already rejected the code.)
Instead of 'a, we add _rust_a to the Python code (since we can’t name variables 'a),
and remember both this name and the Ident in our capatures map if it wasn’t seen before.
Finally, we update our quote!(), which doesn’t change much:
   let name = s.captures.keys();
 let var = s.captures.values();
 quote!(
 run_python(
 #source,
 |globals| {
 #(
 globals
 .set_item(#name, #var)
 .expect("Conversion failed");
 )*
 }
 );
 ).into()
   Done?
Let’s try!
  fn main() {
 let a = 3;
 let b = 20;
 python! {
 c = 100
 print('a + 'b + c)
 }
}
     $ cargo r
 Compiling python-macro v0.1.0
 Compiling scratchpad v0.1.0
 Finished dev [unoptimized + debuginfo] target(s) in 0.72s
 Running `target/debug/scratchpad`
123
   Success!
🎉
Identifier spans
Since we used the original Idents everywhere, we should get nice error messages
when trying to use a variable that doesn’t exist:
  let a = 3;
let b = 20;
python! {
 print('a + 'b + 'c)
}
     $ cargo r
 Compiling scratchpad v0.1.0
error[E0425]: cannot find value `c` in this scope
 --&gt; src/main.rs:8:25
 |
8 | print('a + 'b + 'c)
 | ^^ help: a local variable with a similar name exists: `a`
   Awesome!
Note how it points to both the ' and c.
rustc knows these two tokens belong together,
because we used a syntax it is already familiar with.
What’s next
Now we have a way to very easily get data into our python!{} blocks.
It works for everything that implements ToPyObject, including strings, numbers,
and all kind of collections.
What’s still missing, is a nice way to get data out.
We’ll look at that in a later post,
but first we’re going to look at a very different topic.
In part 3,
we’re going to ‘compile’
the Python code into Python byte-code
at compile time, to catch errors like invalid Python syntax before even running it.
This also speeds up execution times,
by moving part of the work into the compilation step.
Writing Python inside your Rust code — Part 2
https://blog.m-ou.se/writing-python-inside-rust-2/
https://www.infinyon.com/blog/2021/03/python-client/
How we built our Python Client that's mostly Rust
2021-03-24
Sebastian Imlay
<div><div>
 <p>This week, we’re happy to announce the addition of a <a href="https://github.com/infinyon/fluvio-client-python">Python client library for Fluvio</a>.
Using the Python client is just as easy as using our other clients. Check out the
<a href="https://www.fluvio.io/api/official/python/examples/">hello world in Python tutorial</a> or <a href="https://infinyon.github.io/fluvio-client-python/fluvio.html">documentation</a> for usage.</p>
<p>In this post, we’ll talk about how we were able to leverage some of the great
Rust tools to build a Python client without writing much Python itself.</p>
<div><h2><a href="#overview"><i></i></a>Overview</h2></div>
<p>In short, we will:</p>
<ul>
<li>use <a href="https://github.com/Dushistov/flapigen-rs">flapigen</a> to define how our Rust structs will go across the FFI.</li>
<li>use the <a href="https://github.com/dgrunwald/rust-cpython">rust-cpython</a> extension in our Python project and call it.</li>
</ul>
<div><h2><a href="#setup"><i></i></a>Setup</h2></div>
<p>To get started, we’ll create a new project folder that’s set up for both
Rust and Python, using <code>cargo</code> and <code>venv</code>.</p>
<div><pre><code>cargo new --lib my-python-lib
cd my-python-lib
python -m venv venv
source venv/bin/activate
</code></pre></div><p>The above creates a Rust crate named <code>my-python-lib</code>, then sets-up a Python
<a href="https://docs.python.org/3/tutorial/venv.html">virtual environment</a>.</p>
<div><h2><a href="#rust-glue"><i></i></a>Rust glue</h2></div>
<p>We’ll need to add this to your <code>Cargo.toml</code>:</p>
<div><pre><code>[lib]
crate-type = ["cdylib"]
[dependencies]
cpython = { version = "0.5", features = ["extension-module"] }
[build-dependencies]
flapigen = "0.6.0-pre7"
</code></pre></div><p>The <code>crate-type = ["cdylib"]</code> tells Rust to build our crate as a C-compatible
<a href="https://en.wikipedia.org/wiki/Dynamic_linker">dynamic library</a> rather than a typical crate. This crate type will allow our Python code
to interact with our library as if it were compiled C code rather than Rust.</p>
<p>Now we’ll create a <a href="https://doc.rust-lang.org/cargo/reference/build-scripts.html">build script</a> in <code>build.rs</code> by adding the following:</p>
<div><pre><code>use flapigen::{LanguageConfig, PythonConfig};
use std::{env, path::Path};
fn main() {
 let in_src = Path::new("src").join("glue.rs.in");
 let out_dir = env::var("OUT_DIR").unwrap();
 let out_src = Path::new(&amp;out_dir).join("glue.rs");
 let python_cfg = PythonConfig::new("my_python_lib".to_owned());
 let flap_gen =
 flapigen::Generator::new(LanguageConfig::PythonConfig(python_cfg)).rustfmt_bindings(true);
 flap_gen.expand("python bindings", &amp;in_src, &amp;out_src);
 println!("cargo:rerun-if-changed={}", in_src.display());
}
</code></pre></div><p>The code sets up <code>flapigen</code> to run on our project. At build time,
it will read the “glue code” we write in <code>src/glue.rs.in</code>, and generate
Rust code to interact with Python and place it in <code>${OUT_DIR}/glue.rs</code>.</p>
<p>Now we’ll add a <code>src/glue.rs.in</code> file with something like the following:</p>
<div><pre><code>pub struct Foo {
 val: i32
}
impl Foo {
 pub fn new(val: i32) -&gt; Self {
 Self {
 val
 }
 }
 pub fn set_field(&amp;mut self, new_val: i32) {
 self.val = new_val;
 }
 pub fn val(&amp;self) -&gt; i32 {
 self.val
 }
}
foreign_class!(class Foo {
 self_type Foo;
 constructor Foo::new(_: i32) -&gt; Foo;
 fn Foo::set_field(&amp;mut self, _: i32);
 fn Foo::val(&amp;self) -&gt; i32;
});
</code></pre></div><p>This simple example was published in the <a href="https://dushistov.github.io/flapigen-rs/foreign-class.html">flapigen book</a> and we can copy and paste it here.</p>
<p>The <code>src/lib.rs</code> should currently have some basic tests. We’ll change it to the following:</p>
<div><pre><code>#![allow(non_snake_case, unused)]
include!(concat!(env!("OUT_DIR"), "/glue.rs"));
</code></pre></div><p>This is a typical Rust pattern when using build scripts. The code takes the
file in <code>${OUT_DIR}/glue.rs</code> and includes the contents into <code>src/lib.rs</code> in the
build directory. The result will be as if we hand-wrote the generated code in
our <code>lib.rs</code> file.</p>
<p>This section uses flapigen to expand the <a href="https://dushistov.github.io/flapigen-rs/foreign-class.html"><code>foreign_class</code></a> macro into many
<a href="https://github.com/dgrunwald/rust-cpython">cpython</a> functions as an <a href="https://docs.python.org/3/extending/extending.html">extension module</a>, and cargo compiles it as a
<a href="https://doc.rust-lang.org/cargo/reference/cargo-targets.html#library"><code>cdylib</code></a>. If you want to see what that looks like, install <a href="https://crates.io/crates/cargo-expand"><code>cargo-expand</code></a>
and run <code>cargo expand</code>. You’ll get a lot of generated rust code.</p>
<div><h2><a href="#python-glue"><i></i></a>Python Glue</h2></div>
<p>In the <a href="#setup"><code>setup</code></a>, we created a virtual environment, and now we’ll need to
install some Python tools via:</p>
<div><pre><code>$ source venv/bin/activate &amp;&amp; pip install setuptools-rust
</code></pre></div><p>Now to create a python package, you create a file called <code>setup.py</code> with:</p>
<div><pre><code>from setuptools import setup
from setuptools_rust import Binding, RustExtension
setup(
 name="my-python-lib",
 version="1.0",
 rust_extensions=[RustExtension("my_python_lib", binding=Binding.RustCPython)],
 # rust extensions are not zip safe, just like C-extensions.
 zip_safe=False,
)
</code></pre></div><p>This is the most basic <a href="https://github.com/PyO3/setuptools-rust#setuppy">setuptools-rust</a> setup, except for using
<a href="https://setuptools-rust.readthedocs.io/en/latest/reference.html#setuptools_rust.Binding"><code>RustCPython</code></a> as the binding.</p>
<p>To build the Rust and the Python packages just run <code>python setup.py develop</code>.
Python calls <code>cargo</code> and moves <code>cdylib</code> into your local directory.</p>
<div><h2><a href="#testing-it-all-out"><i></i></a>Testing it all out</h2></div>
<p>Create a <code>simple.py</code> script with the following in it:</p>
<div><pre><code>from my_python_lib import Foo
foo = Foo(1)
print(foo.val())
foo.set_field(11)
print(foo.val())
</code></pre></div><p>Running the script via <code>python simple.py</code> should result in:</p>
<p>And there you go, you’ve called Rust from Python!</p>
<div><h2><a href="#conclusion"><i></i></a>Conclusion</h2></div>
<p>You can get the source for this post in our <a href="https://github.com/infinyon/fluvio-demo-apps-rust/tree/master/my-python-lib-blog-post">fluvio-demo-apps-rust</a> repository.</p>
<p>These are just the basics for setting up a Python wrapper. In the Fluvio Python Client, the Rust crate is <a href="https://github.com/infinyon/fluvio-client-python/blob/c6d82d63a001376325d8583c75319c41fd5bfcd5/build.rs#L10"><code>_fluvio_python</code></a>; a private python module that wraps the rust structs with python classes, giving us nice <a href="https://infinyon.github.io/fluvio-client-python/fluvio.html">documentation generation</a>.</p>
<p>Packaging, testing, and publishing on <code>pypi</code> is beyond the scope of this blog. Checkout the <a href="https://github.com/infinyon/fluvio-client-python/blob/main/Makefile"><code>Makefile</code></a> and the <a href="https://github.com/infinyon/fluvio-client-python/blob/main/.github/workflows/publish.yml">github publishing workflow</a> for additional information.</p>
</div>||||I|||| Workshop: Best practices for building real-time data pipelines with transformations: Register Now
 * Products
 + InfinyOn Cloud
 + Fluvio OSS
 * Blog
 * Docs
 + Cloud Docs
 + APIs
 + CLI
 + Fluvio Docs
 * Learn
 + Webinars
 + Use Cases
 * Company
 + About Us
 + Careers
 + Press Releases
 + Resources
 * Try Cloud

 How we built our Python Client that's mostly Rust

 by Sebastian Imlay (@simlay)
 | March 24, 2021
 | Reading time : 3 minutes

 This week, we’re happy to announce the addition of a Python client library for Fluvio. Using the Python client is just as easy as using our other clients. Check out the hello world in Python tutorial or documentation for usage.

 In this post, we’ll talk about how we were able to leverage some of the great Rust tools to build a Python client without writing much Python itself.

 Overview

 In short, we will:

 * use flapigen to define how our Rust structs will go across the FFI.
 * use the rust-cpython extension in our Python project and call it.

 Setup

 To get started, we’ll create a new project folder that’s set up for both Rust and Python, using cargo and venv.

 cargo new --lib my-python-lib
 cd my-python-lib
 python -m venv venv
 source venv/bin/activate
 

 The above creates a Rust crate named my-python-lib, then sets-up a Python virtual environment.

 Note: You’ll need to have the rust toolchain and python 3.6 or above installed.

 Rust glue

 We’ll need to add this to your Cargo.toml:

 [lib]
 crate-type = ["cdylib"]
 
 [dependencies]
 cpython = { version = "0.5", features = ["extension-module"] }
 
 [build-dependencies]
 flapigen = "0.6.0-pre7"
 

 The crate-type = ["cdylib"] tells Rust to build our crate as a C-compatible dynamic library rather than a typical crate. This crate type will allow our Python code to interact with our library as if it were compiled C code rather than Rust.

 Now we’ll create a build script in build.rs by adding the following:

 use flapigen::{LanguageConfig, PythonConfig};
 use std::{env, path::Path};
 
 fn main() {
 let in_src = Path::new("src").join("glue.rs.in");
 let out_dir = env::var("OUT_DIR").unwrap();
 let out_src = Path::new(&out_dir).join("glue.rs");
 
 let python_cfg = PythonConfig::new("my_python_lib".to_owned());
 let flap_gen =
 flapigen::Generator::new(LanguageConfig::PythonConfig(python_cfg)).rustfmt_bindings(true);
 flap_gen.expand("python bindings", &in_src, &out_src);
 println!("cargo:rerun-if-changed={}", in_src.display());
 }
 

 The code sets up flapigen to run on our project. At build time, it will read the “glue code” we write in src/glue.rs.in, and generate Rust code to interact with Python and place it in ${OUT_DIR}/glue.rs.

 Now we’ll add a src/glue.rs.in file with something like the following:

 pub struct Foo {
 val: i32
 }
 impl Foo {
 pub fn new(val: i32) -> Self {
 Self {
 val
 }
 }
 pub fn set_field(&mut self, new_val: i32) {
 self.val = new_val;
 }
 pub fn val(&self) -> i32 {
 self.val
 }
 }
 foreign_class!(class Foo {
 self_type Foo;
 constructor Foo::new(_: i32) -> Foo;
 fn Foo::set_field(&mut self, _: i32);
 fn Foo::val(&self) -> i32;
 });
 

 This simple example was published in the flapigen book and we can copy and paste it here.

 The src/lib.rs should currently have some basic tests. We’ll change it to the following:

 #![allow(non_snake_case, unused)]
 
 include!(concat!(env!("OUT_DIR"), "/glue.rs"));
 

 This is a typical Rust pattern when using build scripts. The code takes the file in ${OUT_DIR}/glue.rs and includes the contents into src/lib.rs in the build directory. The result will be as if we hand-wrote the generated code in our lib.rs file.

 This section uses flapigen to expand the foreign_class macro into many cpython functions as an extension module, and cargo compiles it as a cdylib. If you want to see what that looks like, install cargo-expand and run cargo expand. You’ll get a lot of generated rust code.

 Python Glue

 In the setup, we created a virtual environment, and now we’ll need to install some Python tools via:

 $ source venv/bin/activate && pip install setuptools-rust
 

 Now to create a python package, you create a file called setup.py with:

 from setuptools import setup
 from setuptools_rust import Binding, RustExtension
 
 setup(
 name="my-python-lib",
 version="1.0",
 rust_extensions=[RustExtension("my_python_lib", binding=Binding.RustCPython)],
 # rust extensions are not zip safe, just like C-extensions.
 zip_safe=False,
 )
 

 This is the most basic setuptools-rust setup, except for using RustCPython as the binding.

 To build the Rust and the Python packages just run python setup.py develop. Python calls cargo and moves cdylib into your local directory.

 Testing it all out

 Create a simple.py script with the following in it:

 from my_python_lib import Foo
 foo = Foo(1)
 print(foo.val())
 foo.set_field(11)
 print(foo.val())
 

 Running the script via python simple.py should result in:

 $ python simple.py
 1
 11
 

 And there you go, you’ve called Rust from Python!

 Conclusion

 You can get the source for this post in our fluvio-demo-apps-rust repository.

 These are just the basics for setting up a Python wrapper. In the Fluvio Python Client, the Rust crate is _fluvio_python; a private python module that wraps the rust structs with python classes, giving us nice documentation generation.

 Packaging, testing, and publishing on pypi is beyond the scope of this blog. Checkout the Makefile and the github publishing workflow for additional information.

 Stream Key/Value records in Fluvio
 Build Your Own Custom Robot Assistant
 Share On
 Y
 Author
 Sebastian Imlay
 Fluvio Open Source
 Try Infinyon Cloud
 Subscribe to RSS feed
 * 
 * 
 * 
 * 
 * Terms of Use
 * |
 * Security
 * |
 * Privacy Policy
This week, we’re happy to announce the addition of a Python client library for Fluvio.
Using the Python client is just as easy as using our other clients. Check out the
hello world in Python tutorial or documentation for usage.
In this post, we’ll talk about how we were able to leverage some of the great
Rust tools to build a Python client without writing much Python itself.
     Overview 
In short, we will:
use flapigen to define how our Rust structs will go across the FFI.
use the rust-cpython extension in our Python project and call it.
     Setup 
To get started, we’ll create a new project folder that’s set up for both
Rust and Python, using cargo and venv.
  cargo new --lib my-python-lib
cd my-python-lib
python -m venv venv
source venv/bin/activate
   The above creates a Rust crate named my-python-lib, then sets-up a Python
virtual environment.
     Rust glue 
We’ll need to add this to your Cargo.toml:
  [lib]
crate-type = ["cdylib"]
[dependencies]
cpython = { version = "0.5", features = ["extension-module"] }
[build-dependencies]
flapigen = "0.6.0-pre7"
   The crate-type = ["cdylib"] tells Rust to build our crate as a C-compatible
dynamic library rather than a typical crate. This crate type will allow our Python code
to interact with our library as if it were compiled C code rather than Rust.
Now we’ll create a build script in build.rs by adding the following:
  use flapigen::{LanguageConfig, PythonConfig};
use std::{env, path::Path};
fn main() {
 let in_src = Path::new("src").join("glue.rs.in");
 let out_dir = env::var("OUT_DIR").unwrap();
 let out_src = Path::new(&amp;out_dir).join("glue.rs");
 let python_cfg = PythonConfig::new("my_python_lib".to_owned());
 let flap_gen =
 flapigen::Generator::new(LanguageConfig::PythonConfig(python_cfg)).rustfmt_bindings(true);
 flap_gen.expand("python bindings", &amp;in_src, &amp;out_src);
 println!("cargo:rerun-if-changed={}", in_src.display());
}
   The code sets up flapigen to run on our project. At build time,
it will read the “glue code” we write in src/glue.rs.in, and generate
Rust code to interact with Python and place it in ${OUT_DIR}/glue.rs.
Now we’ll add a src/glue.rs.in file with something like the following:
  pub struct Foo {
 val: i32
}
impl Foo {
 pub fn new(val: i32) -&gt; Self {
 Self {
 val
 }
 }
 pub fn set_field(&amp;mut self, new_val: i32) {
 self.val = new_val;
 }
 pub fn val(&amp;self) -&gt; i32 {
 self.val
 }
}
foreign_class!(class Foo {
 self_type Foo;
 constructor Foo::new(_: i32) -&gt; Foo;
 fn Foo::set_field(&amp;mut self, _: i32);
 fn Foo::val(&amp;self) -&gt; i32;
});
   This simple example was published in the flapigen book and we can copy and paste it here.
The src/lib.rs should currently have some basic tests. We’ll change it to the following:
  #![allow(non_snake_case, unused)]
include!(concat!(env!("OUT_DIR"), "/glue.rs"));
   This is a typical Rust pattern when using build scripts. The code takes the
file in ${OUT_DIR}/glue.rs and includes the contents into src/lib.rs in the
build directory. The result will be as if we hand-wrote the generated code in
our lib.rs file.
This section uses flapigen to expand the  foreign_class  macro into many
cpython functions as an extension module, and cargo compiles it as a
 cdylib . If you want to see what that looks like, install  cargo-expand 
and run cargo expand. You’ll get a lot of generated rust code.
     Python Glue 
In the  setup , we created a virtual environment, and now we’ll need to
install some Python tools via:
  $ source venv/bin/activate &amp;&amp; pip install setuptools-rust
   Now to create a python package, you create a file called setup.py with:
  from setuptools import setup
from setuptools_rust import Binding, RustExtension
setup(
 name="my-python-lib",
 version="1.0",
 rust_extensions=[RustExtension("my_python_lib", binding=Binding.RustCPython)],
 # rust extensions are not zip safe, just like C-extensions.
 zip_safe=False,
)
   This is the most basic setuptools-rust setup, except for using
 RustCPython  as the binding.
To build the Rust and the Python packages just run python setup.py develop.
Python calls cargo and moves cdylib into your local directory.
     Testing it all out 
Create a simple.py script with the following in it:
  from my_python_lib import Foo
foo = Foo(1)
print(foo.val())
foo.set_field(11)
print(foo.val())
   Running the script via python simple.py should result in:
And there you go, you’ve called Rust from Python!
     Conclusion 
You can get the source for this post in our fluvio-demo-apps-rust repository.
These are just the basics for setting up a Python wrapper. In the Fluvio Python Client, the Rust crate is  _fluvio_python ; a private python module that wraps the rust structs with python classes, giving us nice documentation generation.
Packaging, testing, and publishing on pypi is beyond the scope of this blog. Checkout the  Makefile  and the github publishing workflow for additional information.
How we built our Python Client that's mostly Rust
https://www.infinyon.com/blog/2021/03/python-client/
<div><div><div><h2>Practical Lessons from Upgrading Bed-Reader, a Python Bioinformatics Package</h2><div><a href="https://medium.com/@carlmkadie?source=post_page-----d35ea3a4ec29--------------------------------"><div><p></p></div></a><a href="https://towardsdatascience.com/?source=post_page-----d35ea3a4ec29--------------------------------"><div><p></p></div></a></div></div><figure><figcaption>Photo by <a href="https://unsplash.com/@davidclode?utm_source=medium&amp;utm_medium=referral">David Clode</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><p><em>Updated on 4/25/2023. For video version, see this </em><a href="https://www.youtube.com/watch?v=B6E0Jb6yj34&amp;t=87s"><em>YouTube from the PyData 2023 Conference</em></a><em> in Seattle.</em></p></blockquote><p>One year ago, I got fed up with our package’s C++ extension. I rewrote it in Rust. The resulting extension was as fast a C/C++ but with better compatibility and safety. Along the way, I learned nine rules that can help you create better extension code:</p><ol><li>Create a single repository containing both Rust and Python projects</li><li>Use maturin &amp; PyO3 to create Python-callable translator functions in Rust</li><li>Have the Rust translator functions call “nice” Rust functions</li><li>Preallocate memory in Python</li><li>Translate nice Rust error handling into nice Python error handling</li><li>Multithread with Rayon and ndarray::parallel, returning any errors</li><li>Allow users to control the number of parallel threads</li><li>Translate nice dynamically-type Python functions into nice Rust generic functions</li><li>Create both Rust and Python tests</li></ol><p>The word “nice” here means created with best practices and native types. In other words, the general strategy is this: At the top, write good Python code. In the middle, write a thin layer of translator code in Rust. At the bottom, write good Rust code.</p><figure><figcaption>Three Layers</figcaption></figure><p>This strategy may seem obvious but following it can be tricky. This article gives practical advice and examples on how to follow each rule.</p></div><div><p><a href="https://pypi.org/project/bed-reader/">Bed-Reader</a> is a Python package for reading and writing PLINK Bed Files, a binary format used in bioinformatics to store DNA data. Files in Bed format can be as large as a terabyte. Bed-Reader gives users fast, random access to large subsets of the data. It returns a NumPy array in the user’s choice of int8, float32, or float64.</p><p>I wanted the Bed-Reader extension code to be:</p><ul><li>Faster than Python</li><li>Compatible with NumPy</li><li>Fully data-parallel multithreading</li><li>Compatible with all other packages doing data-parallel multithreading</li><li>Safe</li></ul><p>Our original C++ extension gave me speed, NumPy compatibility, and — with <a href="https://en.wikipedia.org/wiki/OpenMP">OpenMP</a> — data-parallel multithreading. Sadly, OpenMP requires a runtime library and different Python packages can depend on different, incompatible, versions of that runtime library.</p><p>Rust gave me everything offered by C++. Beyond that, it solved the runtime-compatibility problem by providing data-parallel multithreading without a runtime library. Moreover, the Rust compiler <em>guarantees</em> thread safety. (It even uncovered a race condition in the original algorithm. In Rust, “<a href="https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html">Thread safety isn’t just documentation; it’s law</a>”.)</p><p>Creating a Python extension in Rust requires many design decisions. Based on my experience with Bed-Reader, here are the decisions I recommend. To avoid wishy-washiness, I’ll express these recommendations as rules.</p><h2>Rule 1: Create a single repository containing both Rust and Python projects</h2><p>The table below shows how to layout files.</p><figure></figure><p>Create files <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/Cargo.toml">Cargo.toml</a></code> and <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code> using Rust’s <a href="https://doc.rust-lang.org/cargo/guide/creating-a-new-project.html">usual ‘cargo new</a>’ command. There is no <code>setup.py</code> file for Python. Instead, <code><a href="https://github.com/fastlmm/bed-reader/blob/master/pyproject.toml">pyproject.toml</a></code> contains PyPi package information, such as the name of the package, its version number, the location of its README file, etc. To work without a <code>setup.py</code>, <code><a href="https://github.com/fastlmm/bed-reader/blob/rust/pyproject.toml">pyproject.toml</a></code> must contain:</p><pre><span>[build-system]<br/>requires = ["maturin&gt;=0.14,&lt;0.15"]<br/>build-backend = "maturin"</span></pre><p>We’ll talk about “maturin” and file <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/python_module.rs">src/python_module.rs</a></code> in Rule #2. We’ll talk about testing (<code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/tests.rs">src/tests.rs</a></code> and <code><a href="https://github.com/fastlmm/bed-reader/tree/mar2023/bed_reader/tests">bed_reader/tests</a></code>) in Rule #9.</p><p>Python settings go in <code><a href="https://github.com/fastlmm/bed-reader/blob/master/pyproject.toml">pyproject.toml</a></code> when possible (and in files such as a <code>pytest.ini</code> when not). Python code goes in its own subfolder, here <code><a href="https://github.com/fastlmm/bed-reader/tree/mar2023/bed_reader">bed_reader</a></code>.</p><p>Finally, we use GitHub actions to build, test, and ready deployment. That script lives in <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/.github/workflows/ci.yml">.github/workflows/ci.yml</a></code>.</p><h2>Rule 2: Use maturin &amp; PyO3 to create Python-callable translator functions in Rust</h2><p><a href="https://pypi.org/project/maturin/">Maturin</a> is a PyPi package for (among other things) building and publishing Python extensions via PyO3. <a href="https://pyo3.rs/master/doc/pyo3/index.html">PyO3</a> is a Rust crate for (among other things) writing Python extensions in Rust.</p><p>In <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/Cargo.toml">Cargo.toml</a></code>, include these Rust dependencies:</p><pre><span>[dependencies]<br/>thiserror = "1.0.30"<br/>ndarray-npy = { version = "0.8.1", default-features = false }<br/>rayon = "1.5.2"<br/>numpy = "0.18.0"<br/>ndarray = { version = "0.15.4", features = ["approx", "rayon"] }<br/>pyo3 = { version = "0.18.2", features = ["extension-module"], optional = true }<p>[build-dependencies]<br/>pyo3-build-config = "0.18.2"</p></span></pre><p>At the top of <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code>, include these two lines:</p><pre><span>mod python_module;<br/>mod tests;</span></pre><p>Create a <code>src/build.rs</code> containing:</p><pre><span>// https://pyo3.rs/v0.14.4/building_and_distribution.html#macos<br/>fn main() {<br/>    pyo3_build_config::add_extension_module_link_args();<br/>}</span></pre><h2>Rule 3: Have the Rust translator functions call “nice” Rust functions</h2><p>In <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code> define “nice” Rust functions. These functions will do the core work of your package. They will input and output standard Rust types and try to follow Rust best practices. For example, for the Bed-Reader package, <code>read_and_fill</code> is the nice Rust method for reading and returning values from a PLINK Bed file.</p><p>Python, however, can’t directly call these functions. So, in file <code><a href="https://github.com/fastlmm/bed-reader/blob/rust/src/python_module.rs">src/python_module.rs</a></code> define Rust translator functions that Python can call. Here is an example translator function:</p><pre><span>    #[pyfn(m)]<br/>    #[pyo3(name = "read_f64")]<br/>    #[allow(clippy::too_many_arguments)]<br/>    fn read_f64(<br/>        filename: &amp;str,<br/>        iid_count: usize,<br/>        sid_count: usize,<br/>        is_a1_counted: bool,<br/>        iid_index: &amp;PyArray1&lt;isize&gt;,<br/>        sid_index: &amp;PyArray1&lt;isize&gt;,<br/>        val: &amp;PyArray2&lt;f64&gt;,<br/>        num_threads: usize,<br/>    ) -&gt; Result&lt;(), PyErr&gt; {<br/>        let iid_index = iid_index.readonly();<br/>        let sid_index = sid_index.readonly();<br/>        let ii = &amp;iid_index.as_slice()?;<br/>        let si = &amp;sid_index.as_slice()?;<p>        // let mut val = unsafe { val.as_array_mut() };<br/>        // update 12/23/2023<br/>        let mut val = val.readwrite();<br/>        let mut val = val.as_array_mut();</p><p>        let mut bed = Bed::builder(filename)<br/>            .iid_count(iid_count)<br/>            .sid_count(sid_count)<br/>            .build()?;</p><p>        ReadOptions::builder()<br/>            .iid_index(*ii)<br/>            .sid_index(*si)<br/>            .is_a1_counted(is_a1_counted)<br/>            .num_threads(num_threads)<br/>            .read_and_fill(&amp;mut bed, &amp;mut val.view_mut())?;</p><p>        Ok(())<br/>    }</p></span></pre><p>This function takes as input a file name, some integers related to the size of the file, and two 1-D NumPy arrays that tell which subset of the data to read. The function reads values from the file and fills in <code>val</code>, a preallocated 2-D NumPy array.</p><p>Notice that this function:</p><ul><li>translates Python NumPy 1-D arrays into Rust slices, the standard Rust 1-D data structure, via:<br/><code>let iid_index = iid_index.readonly();<br/> let ii = &amp;iid_index.as_slice()?;</code></li><li>translates Python NumPy 2-D arrays into 2-D Rust ndarray objects, via:<br/><code>// let mut val = unsafe { val.as_array_mut() }<br/> // update 12/23/2023<br/> let mut val = val.readwrite();<br/> let mut val = val.as_array_mut();</code></li><li>calls <code>read_and_fill</code>, a nice Rust method in <code><a href="https://github.com/fastlmm/bed-reader/blob/rust/src/lib.rs">src/lib.rs</a></code> that does the core work.</li><li>(Later rules will cover preallocation, <code>f64</code>, <code>PyErr,</code> and <code>num_threads</code>)</li></ul><p>Doing the core work in a Rust function defined on standard Rust types lets us use Rust best practices for testing, generics, errors, etc. It also gives us a pathway to offer a Rust version of our package.</p><h2><strong>Rule 4: </strong>Preallocate memory in Python</h2><p>Preallocating the memory for our results in Python simplifies the Rust code. On the Python side, in <a href="https://github.com/fastlmm/bed-reader/blob/mar2023/bed_reader/_open_bed.py">bed_reader/_open_bed.py</a>, we import the Rust translator function:</p><pre><span>from .bed_reader import [...] read_f64 [...]</span></pre><p>Then, we define a nice Python function that allocates memory, calls the Rust translator function, and returns the result.</p><pre><span>def read([...]) -&gt; np.ndarray:<br/>  [...]<br/>  val = np.zeros((len(iid_index), len(sid_index)), order=order, dtype=dtype)<br/>  [...]<br/>  reader = read_f64<br/>  [...]<br/>  reader(<br/>      str(self.filepath),<br/>      iid_count=self.iid_count,<br/>      sid_count=self.sid_count,<br/>      is_a1_counted=self.count_A1,<br/>      iid_index=iid_index,<br/>      sid_index=sid_index,<br/>      val=val,<br/>      num_threads=num_threads,<br/>  )<br/>  [...]<br/>  return val</span></pre><p>(Later rules will explain <code>reader = read_f64</code> and <code>num_threads=num_threads</code>.)</p><h2>Rule 5: Translate nice Rust error handling into nice Python error handling</h2><p>To see how to handle errors, let’s trace two possible errors in <code>open_and_check</code> (called by <code>read_and_fill</code>, our nice Rust method in <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code>).</p><p><strong>Example Error 1: An error from a standard function </strong>— What if Rust’s standard <code>File::open</code> function can’t find the file or can’t open it? In that case, the question mark in this line:</p><pre><span>let mut buf_reader = BufReader::new(File::open(&amp;path)?);</span></pre><p>will cause the function to return with some <code>std::io::Error</code> value. To define a function that can return these values, we give the function a return type of <code>Result&lt;(), BedErrorPlus&gt;</code>. We define <code>BedErrorPlus</code> to include all of <code>std::io::Error</code> like so:</p><pre><span>use thiserror::Error;<br/>[...]<br/>/// All possible errors returned by this library and the libraries it depends on.<br/>// Based on `&lt;https://nick.groenen.me/posts/rust-error-handling/#the-library-error-type&gt;`<br/>#[derive(Error, Debug)]<br/>pub enum BedErrorPlus {<br/>    #[allow(missing_docs)]<br/>    #[error(transparent)]<br/>    BedError(#[from] BedError),<p>    #[allow(missing_docs)]<br/>    #[error(transparent)]<br/>    IOError(#[from] std::io::Error),</p><p>    #[allow(missing_docs)]<br/>    #[error(transparent)]<br/>    ThreadPoolError(#[from] ThreadPoolBuildError),<br/>[...]<br/>}</p></span></pre><p>This is nice Rust error handling, but Python doesn’t understand it. So, in <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/python_module.rs">src/python_module.rs</a></code>, we translate. First, we define our translator function <code>read_f64</code> to return <code>PyErr</code>. Second, we implement a converter from <code>BedErrorPlus</code> to <code>PyErr</code>. The converter creates the right class of Python error (<code>IOError</code>, <code>ValueError</code>, or <code>IndexError</code>) with the right error message. It looks like:</p><pre><span>    impl std::convert::From&lt;BedErrorPlus&gt; for PyErr {<br/>        fn from(err: BedErrorPlus) -&gt; PyErr {<br/>            match err {<br/>                BedErrorPlus::BedError(BedError::IidIndexTooBig(_))<br/>                | BedErrorPlus::BedError(BedError::SidIndexTooBig(_))<br/>                | BedErrorPlus::BedError(BedError::IndexMismatch(_, _, _, _))<br/>                | BedErrorPlus::BedError(BedError::IndexesTooBigForFiles(_, _))<br/>                | BedErrorPlus::BedError(BedError::SubsetMismatch(_, _, _, _)) =&gt; {<br/>                    PyIndexError::new_err(err.to_string())<br/>                }<p>                BedErrorPlus::IOError(_) =&gt; PyIOError::new_err(err.to_string()),</p><p>                _ =&gt; PyValueError::new_err(err.to_string()),<br/>            }<br/>        }<br/>    }</p></span></pre><p><strong>Example Error 2: An error specific to our function </strong>— What if our nice function <code>read_no_alloc</code> can open the file but then realizes the file’s format is wrong? It should raise a custom error like so:</p><pre><span>    if (BED_FILE_MAGIC1 != bytes_vector[0]) || (BED_FILE_MAGIC2 != bytes_vector[1]) {<br/>        return Err(BedError::IllFormed(path_ref_to_string(&amp;path)).into());<br/>    }</span></pre><p>The custom error of type <code>BedError::IllFormed</code> is defined in<code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a>:</code></p><pre><span>/// All errors specific to this library.<br/>#[derive(Error, Debug, Clone)]<br/>pub enum BedError {<br/>    #[allow(missing_docs)]<br/>    #[error("Ill-formed BED file. BED file header is incorrect or length is wrong. '{0}'")]<br/>    IllFormed(String),<br/>[...]<br/>}</span></pre><p>The rest of the error handling is the same as in Example Error #1.</p><p>In the end, for both Rust and Python, for both standard errors and custom errors, the result is a specific error type with an informative error message.</p><h2>Rule 6: Multithread with Rayon and ndarray::parallel, returning any errors</h2><p>The Rust Rayon crate provides easy and lightweight data-parallel multithreading. The ndarray::parallel module applies Rayon to arrays. The usual pattern is to parallelize across the columns (or rows) of one or more 2-D arrays. One challenge is to return any error message from the parallel threads. I’ll highlight two approaches to parallelizing array operations with error handling. Both examples appear in Bed-Reader’s <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code> file.</p><p><strong>Approach 1: </strong><code><strong>par_bridge().try_for_each</strong></code></p><p>Rayon’s <code><a href="https://doc.servo.org/rayon/iter/par_bridge/struct.IterBridge.html"><strong>par_bridge</strong></a></code> turns a sequential iterator into a parallel iterator. Its <code><a href="https://doc.servo.org/rayon/iter/trait.ParallelIterator.html#method.try_for_each"><strong>try_for_each</strong></a></code> method will stop all processing as quickly as it can if an error is hit.</p><p>In this example, we iterate through two things zipped together:</p><ul><li>a DNA location’s binary data and</li><li>the columns of our output array.</li></ul><p>We read the binary data sequentially but process each column’s piece of that data in parallel. We stop on any errors.</p><pre><span>[... not shown, read bytes for DNA location's data ...]<br/>// Zip in the column of the output array<br/>.zip(out_val.axis_iter_mut(nd::Axis(1)))<br/>// In parallel, decompress the iid info and put it in its column<br/>.par_bridge() // This seems faster that parallel zip<br/>.try_for_each(|(bytes_vector_result, mut col)| match bytes_vector_result {<br/>    Err(e) =&gt; Err(e),<br/>    Ok(bytes_vector) =&gt; {<br/>        for out_iid_i in 0..iid_index.len() {<br/>            let i_div_4 = i_div_4_array[out_iid_i];<br/>            let i_mod_4_times_2 = i_mod_4_times_2_array[out_iid_i];<br/>            let genotype_byte: u8 = (bytes_vector[i_div_4] &gt;&gt; i_mod_4_times_2) &amp; 0x03;<br/>            col[out_iid_i] = from_two_bits_to_value[genotype_byte as usize];<br/>        }<br/>        Ok(())<br/>    }<br/>})?;</span></pre><p><strong>Approach 2: </strong><code>par_azip!</code></p><p>The ndarray packages’s <code><a href="https://docs.rs/ndarray/latest/ndarray/macro.par_azip.html">par_azip!</a></code> macro lets one march through, in parallel, one or more zipped-together arrays (or array pieces). It is, in my opinion, very readable. It doesn’t, however, directly support error handling. We can add error handling by saving any error to a results list.</p><p>Here is an example from a utility function. The full utility function computes statistics (mean and variance) from three arrays of counts and sums. It does its work in parallel. If it finds an error in the data, it records that error in a result list. After all processing, it checks the result list for errors.</p><pre><span>[...]<br/>let mut result_list: Vec&lt;Result&lt;(), BedError&gt;&gt; = vec![Ok(()); sid_count];<br/>nd::par_azip!((mut stats_row in stats.axis_iter_mut(nd::Axis(0)),<br/>     &amp;n_observed in &amp;n_observed_array,<br/>     &amp;sum_s in &amp;sum_s_array,<br/>     &amp;sum2_s in &amp;sum2_s_array,<br/>     result_ptr in &amp;mut result_list)<br/>{<br/>  [...some code not shown...]<br/>});<br/>// Check the result list for errors<br/>result_list.par_iter().try_for_each(|x| (*x).clone())?;<br/>[...]</span></pre><p>Rayon and ndarray::parallel offer many other nice approaches to data-parallel processing. Feel free to use them, just be sure to gather and return any errors. (Do not just use Rust’s “panic”.)</p><h2>Rule 7: Allow users to control the number of parallel threads</h2><p>To play nicely with a user’s other code, the user must be able to control the number of parallel threads each function will use.</p><p>In the nice Python <code>read</code> function, we give the user an optional <code>num_threads</code>argument. If they don’t set it, Python sets it via this function:</p><pre><span>def get_num_threads(num_threads=None):<br/>    if num_threads is not None:<br/>        return num_threads<br/>    if "PST_NUM_THREADS" in os.environ:<br/>        return int(os.environ["PST_NUM_THREADS"])<br/>    if "NUM_THREADS" in os.environ:<br/>        return int(os.environ["NUM_THREADS"])<br/>    if "MKL_NUM_THREADS" in os.environ:<br/>        return int(os.environ["MKL_NUM_THREADS"])<br/>    return multiprocessing.cpu_count()</span></pre><p>Next, on the Rust side, we define <code>create_pool</code>. This helper function constructs a Rayon ThreadPool object from <code>num_threads</code>.</p><pre><span>pub fn create_pool(num_threads: usize) -&gt; Result&lt;rayon::ThreadPool, BedErrorPlus&gt; {<br/>   match rayon::ThreadPoolBuilder::new()<br/>      .num_threads(num_threads)<br/>      .build()<br/>   {<br/>      Err(e) =&gt; Err(e.into()),<br/>      Ok(pool) =&gt; Ok(pool),<br/>   }<br/>}</span></pre><p>Finally, in the Rust translator function<code>read_f64</code> eventually calls <code>internal_read_no_alloc</code>. We put that call inside a <code>create_pool(num_threads)?.install(...)</code>. This limits all Rayon functions to the <code>num_threads</code> we set.</p><pre><span>[...]<br/>    create_pool(num_threads)?.install(|| {<br/>        [...]<br/>        internal_read_no_alloc(<br/>            buf_reader,<br/>            [...]<br/>        )<br/>     })?;<br/>[...]</span></pre><h2>Rule 8: Translate nice dynamically-type Python functions into nice Rust generic functions</h2><p>Users of the nice Python <code>read</code> function can specify the dtype of the returned NumPy array (int8, float32 or float64). From this choice, the function looks up the appropriate Rust translator function (<code>read_i8</code>, <code>read_f32</code>, or <code>read_f64</code>), which is then called.</p><pre><span>def read(<br/>    [...]<br/>    dtype: Optional[Union[type, str]] = "float32",<br/>    [...]<br/>    )<br/>    [...]<br/>    if dtype == np.int8:<br/>        reader = read_i8<br/>    elif dtype == np.float64:<br/>        reader = read_f64<br/>    elif dtype == np.float32:<br/>        reader = read_f32<br/>    else:<br/>        raise ValueError(<br/>          f"dtype '{val.dtype}' not known, only "<br/>          + "'int8', 'float32', and 'float64' are allowed."<br/>        )</span><span>     reader(<br/>       str(self.filepath),<br/>       [...]<br/>     )</span></pre><p>The three Rust translator functions (in<code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/python_module.rs">src/python_module.rs</a></code>) call the same nice Rust function, <code>read_and_fill</code>, defined in <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code>. Here are the relevant parts of translator function <code>read_64</code>:</p><pre><span>#[pyfn(m)]<br/>#[pyo3(name = "read_f64")]<br/>#[allow(clippy::too_many_arguments)]<br/>fn read_f64(<br/>    [...]<br/>    val: &amp;PyArray2&lt;f64&gt;,<br/>    num_threads: usize,<br/>) -&gt; Result&lt;(), PyErr&gt; {<br/>    [...]<br/>    // let mut val = unsafe { val.as_array_mut() };<br/>    // update 12/23/2023<br/>    let mut val = val.readwrite();<br/>    let mut val = val.as_array_mut();<p>    let mut bed = Bed::builder(filename)<br/>        .iid_count(iid_count)<br/>        .sid_count(sid_count)<br/>        .build()?;</p><p>    ReadOptions::builder()<br/>        .iid_index(*ii)<br/>        .sid_index(*si)<br/>        .is_a1_counted(is_a1_counted)<br/>        .num_threads(num_threads)<br/>        .read_and_fill(&amp;mut bed, &amp;mut val.view_mut())?;</p><p>    Ok(())<br/>}</p></span></pre><p>We define the nice<code>read_and_fill</code> method <em>generically</em> in <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/src/lib.rs">src/lib.rs</a></code>. That is, it will work on any type <code>TVal</code> with the right traits. The relevant parts of its code are here:</p><pre><span>/// A trait alias, used internally, for the values of a .bed file, namely i8, f32, f64.<br/>pub trait BedVal: Copy + Default + From&lt;i8&gt; + Debug + Sync + Send + Missing + PartialEq {}<br/>impl&lt;T&gt; BedVal for T where T: Copy + Default + From&lt;i8&gt; + Debug + Sync + Send + Missing + PartialEq {<br/>[...]<br/>impl&lt;TVal: BedVal&gt; ReadOptionsBuilder&lt;TVal&gt; {<br/>[...]<br/>    pub fn read_and_fill(<br/>        &amp;self,<br/>        bed: &amp;mut Bed,<br/>        val: &amp;mut nd::ArrayViewMut2&lt;'_, TVal&gt;, //mutable slices additionally allow to modify elements. But slices cannot grow - they are just a view into some vector.<br/>    ) -&gt; Result&lt;(), BedErrorPlus&gt; {<br/>        let read_options = self.build()?;<br/>        bed.read_and_fill_with_options(val, &amp;read_options)<br/>    }<br/>[...]<br/>}</span></pre><p>Organizing the code in these three levels (nice Python, translator Rust, nice Rust) lets us offer code with dynamic types to our Python users while still writing nice, generic code in Rust.</p><h2>Rule 9: Create both Rust and Python tests</h2><p>You might be tempted to write only Python tests that will call Rust. You should, however, also write Rust tests. The addition of Rust tests lets you run tests interactively and debug interactively. Rust tests also give you a path to later offering a Rust version of your package. In the example project, both sets of tests read test files from <code><a href="https://github.com/fastlmm/bed-reader/tree/rust/bed_reader/tests/data">bed_reader/tests/data</a></code>.</p><p>Where practical, I also recommend writing pure Python versions of your functions. You can then use these slow Python function to test the results of your fast Rust functions.</p><p>Here are my notes on how I run the Rust and Python tests:</p><pre><span># test Rust<br/>cargo test<p># install packages, compile Rust for Python, and test Python<br/>pip install -r requirements.txt<br/>pip install -r requirements-dev.txt<br/>maturin develop<br/>pytest bed_reader</p></span></pre><p>Finally, your CI script, for example, <code><a href="https://github.com/fastlmm/bed-reader/blob/mar2023/.github/workflows/ci.yml">bed-reader/ci.yml</a></code>, should run both your Rust and Python tests.</p></div><div><p>So, there you have it: nine rules for writing Python extensions in Rust. Rust is a great language for writing Python extensions. It offers speed, safe data-parallel multithreading, and compatibility with NumPy arrays and other packages. Follow these nine rules to create great code in both Python and Rust.</p><blockquote><p>Aside: If you’re interested in future articles, please <a href="https://medium.com/@carlmkadie">follow me on Medium</a>. I write on scientific programming in Rust and Python, machine learning, and statistics. I tend to write about one article per month.</p></blockquote></div></div>||||I||||          Open in app

            Sign up

              Sign in

                  Write

              Sign up

                Sign in

                        Nine Rules for Writing Python Extensions in Rust

                        Practical Lessons from Upgrading Bed-Reader, a Python Bioinformatics Package

                                                Carl M. Kadie

                                          ·

                                          Follow

                                          Published in

                                              Towards Data Science

                                          ·
                                        11 min read
                                          ·
                                        Dec 31, 2021

                                        --

                                      4

                                                    Listen

                                          Share

                      Photo by David Clode on Unsplash

                      Updated on 4/25/2023. For video version, see this YouTube from the PyData 2023 Conference in Seattle.

                      One year ago, I got fed up with our package’s C++ extension. I rewrote it in Rust. The resulting extension was as fast a C/C++ but with better compatibility and safety. Along the way, I learned nine rules that can help you create better extension code:

                       1. Create a single repository containing both Rust and Python projects
                       2. Use maturin & PyO3 to create Python-callable translator functions in Rust
                       3. Have the Rust translator functions call “nice” Rust functions
                       4. Preallocate memory in Python
                       5. Translate nice Rust error handling into nice Python error handling
                       6. Multithread with Rayon and ndarray::parallel, returning any errors
                       7. Allow users to control the number of parallel threads
                       8. Translate nice dynamically-type Python functions into nice Rust generic functions
                       9. Create both Rust and Python tests

                      The word “nice” here means created with best practices and native types. In other words, the general strategy is this: At the top, write good Python code. In the middle, write a thin layer of translator code in Rust. At the bottom, write good Rust code.

                      Three Layers

                      This strategy may seem obvious but following it can be tricky. This article gives practical advice and examples on how to follow each rule.

                      Bed-Reader is a Python package for reading and writing PLINK Bed Files, a binary format used in bioinformatics to store DNA data. Files in Bed format can be as large as a terabyte. Bed-Reader gives users fast, random access to large subsets of the data. It returns a NumPy array in the user’s choice of int8, float32, or float64.

                      I wanted the Bed-Reader extension code to be:

                        * Faster than Python
                        * Compatible with NumPy
                        * Fully data-parallel multithreading
                        * Compatible with all other packages doing data-parallel multithreading
                        * Safe

                      Our original C++ extension gave me speed, NumPy compatibility, and — with OpenMP — data-parallel multithreading. Sadly, OpenMP requires a runtime library and different Python packages can depend on different, incompatible, versions of that runtime library.

                      Rust gave me everything offered by C++. Beyond that, it solved the runtime-compatibility problem by providing data-parallel multithreading without a runtime library. Moreover, the Rust compiler guarantees thread safety. (It even uncovered a race condition in the original algorithm. In Rust, “Thread safety isn’t just documentation; it’s law”.)

                      Creating a Python extension in Rust requires many design decisions. Based on my experience with Bed-Reader, here are the decisions I recommend. To avoid wishy-washiness, I’ll express these recommendations as rules.

                      Rule 1: Create a single repository containing both Rust and Python projects

                      The table below shows how to layout files.

                      Create files Cargo.toml and src/lib.rs using Rust’s usual ‘cargo new’ command. There is no setup.py file for Python. Instead, pyproject.toml contains PyPi package information, such as the name of the package, its version number, the location of its README file, etc. To work without a setup.py, pyproject.toml must contain:

                      [build-system]
                      requires = ["maturin>=0.14,<0.15"]
                      build-backend = "maturin"

                      We’ll talk about “maturin” and file src/python_module.rs in Rule #2. We’ll talk about testing (src/tests.rs and bed_reader/tests) in Rule #9.

                      Python settings go in pyproject.toml when possible (and in files such as a pytest.ini when not). Python code goes in its own subfolder, here bed_reader.

                      Finally, we use GitHub actions to build, test, and ready deployment. That script lives in .github/workflows/ci.yml.

                      Rule 2: Use maturin & PyO3 to create Python-callable translator functions in Rust

                      Maturin is a PyPi package for (among other things) building and publishing Python extensions via PyO3. PyO3 is a Rust crate for (among other things) writing Python extensions in Rust.

                      In Cargo.toml, include these Rust dependencies:

                      [dependencies]
                      thiserror = "1.0.30"
                      ndarray-npy = { version = "0.8.1", default-features = false }
                      rayon = "1.5.2"
                      numpy = "0.18.0"
                      ndarray = { version = "0.15.4", features = ["approx", "rayon"] }
                      pyo3 = { version = "0.18.2", features = ["extension-module"], optional = true }

                      [build-dependencies]
                      pyo3-build-config = "0.18.2"

                      At the top of src/lib.rs, include these two lines:

                      mod python_module;
                      mod tests;

                      Create a src/build.rs containing:

                      // https://pyo3.rs/v0.14.4/building_and_distribution.html#macos
                      fn main() {
                          pyo3_build_config::add_extension_module_link_args();
                      }

                      Rule 3: Have the Rust translator functions call “nice” Rust functions

                      In src/lib.rs define “nice” Rust functions. These functions will do the core work of your package. They will input and output standard Rust types and try to follow Rust best practices. For example, for the Bed-Reader package, read_and_fill is the nice Rust method for reading and returning values from a PLINK Bed file.

                      Python, however, can’t directly call these functions. So, in file src/python_module.rs define Rust translator functions that Python can call. Here is an example translator function:

                          #[pyfn(m)]
                          #[pyo3(name = "read_f64")]
                          #[allow(clippy::too_many_arguments)]
                          fn read_f64(
                              filename: &str,
                              iid_count: usize,
                              sid_count: usize,
                              is_a1_counted: bool,
                              iid_index: &PyArray1<isize>,
                              sid_index: &PyArray1<isize>,
                              val: &PyArray2<f64>,
                              num_threads: usize,
                          ) -> Result<(), PyErr> {
                              let iid_index = iid_index.readonly();
                              let sid_index = sid_index.readonly();
                              let ii = &iid_index.as_slice()?;
                              let si = &sid_index.as_slice()?;

                              // let mut val = unsafe { val.as_array_mut() };
                              // update 12/23/2023
                              let mut val = val.readwrite();
                              let mut val = val.as_array_mut();

                              let mut bed = Bed::builder(filename)
                                  .iid_count(iid_count)
                                  .sid_count(sid_count)
                                  .build()?;

                              ReadOptions::builder()
                                  .iid_index(*ii)
                                  .sid_index(*si)
                                  .is_a1_counted(is_a1_counted)
                                  .num_threads(num_threads)
                                  .read_and_fill(&mut bed, &mut val.view_mut())?;

                              Ok(())
                          }

                      This function takes as input a file name, some integers related to the size of the file, and two 1-D NumPy arrays that tell which subset of the data to read. The function reads values from the file and fills in val, a preallocated 2-D NumPy array.

                      Notice that this function:

                        * translates Python NumPy 1-D arrays into Rust slices, the standard Rust 1-D data structure, via:
                          let iid_index = iid_index.readonly();
                          let ii = &iid_index.as_slice()?;
                        * translates Python NumPy 2-D arrays into 2-D Rust ndarray objects, via:
                          // let mut val = unsafe { val.as_array_mut() }
                          // update 12/23/2023
                          let mut val = val.readwrite();
                          let mut val = val.as_array_mut();
                        * calls read_and_fill, a nice Rust method in src/lib.rs that does the core work.
                        * (Later rules will cover preallocation, f64, PyErr, and num_threads)

                      Doing the core work in a Rust function defined on standard Rust types lets us use Rust best practices for testing, generics, errors, etc. It also gives us a pathway to offer a Rust version of our package.

                      Rule 4: Preallocate memory in Python

                      Preallocating the memory for our results in Python simplifies the Rust code. On the Python side, in bed_reader/_open_bed.py, we import the Rust translator function:

                      from .bed_reader import [...] read_f64 [...]

                      Then, we define a nice Python function that allocates memory, calls the Rust translator function, and returns the result.

                      def read([...]) -> np.ndarray:
                        [...]
                        val = np.zeros((len(iid_index), len(sid_index)), order=order, dtype=dtype)
                        [...]
                        reader = read_f64
                        [...]
                        reader(
                            str(self.filepath),
                            iid_count=self.iid_count,
                            sid_count=self.sid_count,
                            is_a1_counted=self.count_A1,
                            iid_index=iid_index,
                            sid_index=sid_index,
                            val=val,
                            num_threads=num_threads,
                        )
                        [...]
                        return val

                      (Later rules will explain reader = read_f64 and num_threads=num_threads.)

                      Rule 5: Translate nice Rust error handling into nice Python error handling

                      To see how to handle errors, let’s trace two possible errors in open_and_check (called by read_and_fill, our nice Rust method in src/lib.rs).

                      Example Error 1: An error from a standard function — What if Rust’s standard File::open function can’t find the file or can’t open it? In that case, the question mark in this line:

                      let mut buf_reader = BufReader::new(File::open(&path)?);

                      will cause the function to return with some std::io::Error value. To define a function that can return these values, we give the function a return type of Result<(), BedErrorPlus>. We define BedErrorPlus to include all of std::io::Error like so:

                      use thiserror::Error;
                      [...]
                      /// All possible errors returned by this library and the libraries it depends on.
                      // Based on `<https://nick.groenen.me/posts/rust-error-handling/#the-library-error-type>`
                      #[derive(Error, Debug)]
                      pub enum BedErrorPlus {
                          #[allow(missing_docs)]
                          #[error(transparent)]
                          BedError(#[from] BedError),

                          #[allow(missing_docs)]
                          #[error(transparent)]
                          IOError(#[from] std::io::Error),

                          #[allow(missing_docs)]
                          #[error(transparent)]
                          ThreadPoolError(#[from] ThreadPoolBuildError),
                      [...]
                      }

                      This is nice Rust error handling, but Python doesn’t understand it. So, in src/python_module.rs, we translate. First, we define our translator function read_f64 to return PyErr. Second, we implement a converter from BedErrorPlus to PyErr. The converter creates the right class of Python error (IOError, ValueError, or IndexError) with the right error message. It looks like:

                          impl std::convert::From<BedErrorPlus> for PyErr {
                              fn from(err: BedErrorPlus) -> PyErr {
                                  match err {
                                      BedErrorPlus::BedError(BedError::IidIndexTooBig(_))
                                      | BedErrorPlus::BedError(BedError::SidIndexTooBig(_))
                                      | BedErrorPlus::BedError(BedError::IndexMismatch(_, _, _, _))
                                      | BedErrorPlus::BedError(BedError::IndexesTooBigForFiles(_, _))
                                      | BedErrorPlus::BedError(BedError::SubsetMismatch(_, _, _, _)) => {
                                          PyIndexError::new_err(err.to_string())
                                      }

                                      BedErrorPlus::IOError(_) => PyIOError::new_err(err.to_string()),

                                      _ => PyValueError::new_err(err.to_string()),
                                  }
                              }
                          }

                      Example Error 2: An error specific to our function — What if our nice function read_no_alloc can open the file but then realizes the file’s format is wrong? It should raise a custom error like so:

                          if (BED_FILE_MAGIC1 != bytes_vector[0]) || (BED_FILE_MAGIC2 != bytes_vector[1]) {
                              return Err(BedError::IllFormed(path_ref_to_string(&path)).into());
                          }

                      The custom error of type BedError::IllFormed is defined insrc/lib.rs:

                      /// All errors specific to this library.
                      #[derive(Error, Debug, Clone)]
                      pub enum BedError {
                          #[allow(missing_docs)]
                          #[error("Ill-formed BED file. BED file header is incorrect or length is wrong. '{0}'")]
                          IllFormed(String),
                      [...]
                      }

                      The rest of the error handling is the same as in Example Error #1.

                      In the end, for both Rust and Python, for both standard errors and custom errors, the result is a specific error type with an informative error message.

                      Rule 6: Multithread with Rayon and ndarray::parallel, returning any errors

                      The Rust Rayon crate provides easy and lightweight data-parallel multithreading. The ndarray::parallel module applies Rayon to arrays. The usual pattern is to parallelize across the columns (or rows) of one or more 2-D arrays. One challenge is to return any error message from the parallel threads. I’ll highlight two approaches to parallelizing array operations with error handling. Both examples appear in Bed-Reader’s src/lib.rs file.

                      Approach 1: par_bridge().try_for_each

                      Rayon’s par_bridge turns a sequential iterator into a parallel iterator. Its try_for_each method will stop all processing as quickly as it can if an error is hit.

                      In this example, we iterate through two things zipped together:

                        * a DNA location’s binary data and
                        * the columns of our output array.

                      We read the binary data sequentially but process each column’s piece of that data in parallel. We stop on any errors.

                      [... not shown, read bytes for DNA location's data ...]
                      // Zip in the column of the output array
                      .zip(out_val.axis_iter_mut(nd::Axis(1)))
                      // In parallel, decompress the iid info and put it in its column
                      .par_bridge() // This seems faster that parallel zip
                      .try_for_each(|(bytes_vector_result, mut col)| match bytes_vector_result {
                          Err(e) => Err(e),
                          Ok(bytes_vector) => {
                              for out_iid_i in 0..iid_index.len() {
                                  let i_div_4 = i_div_4_array[out_iid_i];
                                  let i_mod_4_times_2 = i_mod_4_times_2_array[out_iid_i];
                                  let genotype_byte: u8 = (bytes_vector[i_div_4] >> i_mod_4_times_2) & 0x03;
                                  col[out_iid_i] = from_two_bits_to_value[genotype_byte as usize];
                              }
                              Ok(())
                          }
                      })?;

                      Approach 2: par_azip!

                      The ndarray packages’s par_azip! macro lets one march through, in parallel, one or more zipped-together arrays (or array pieces). It is, in my opinion, very readable. It doesn’t, however, directly support error handling. We can add error handling by saving any error to a results list.

                      Here is an example from a utility function. The full utility function computes statistics (mean and variance) from three arrays of counts and sums. It does its work in parallel. If it finds an error in the data, it records that error in a result list. After all processing, it checks the result list for errors.

                      [...]
                      let mut result_list: Vec<Result<(), BedError>> = vec![Ok(()); sid_count];
                      nd::par_azip!((mut stats_row in stats.axis_iter_mut(nd::Axis(0)),
                           &n_observed in &n_observed_array,
                           &sum_s in &sum_s_array,
                           &sum2_s in &sum2_s_array,
                           result_ptr in &mut result_list)
                      {
                        [...some code not shown...]
                      });
                      // Check the result list for errors
                      result_list.par_iter().try_for_each(|x| (*x).clone())?;
                      [...]

                      Rayon and ndarray::parallel offer many other nice approaches to data-parallel processing. Feel free to use them, just be sure to gather and return any errors. (Do not just use Rust’s “panic”.)

                      Rule 7: Allow users to control the number of parallel threads

                      To play nicely with a user’s other code, the user must be able to control the number of parallel threads each function will use.

                      In the nice Python read function, we give the user an optional num_threadsargument. If they don’t set it, Python sets it via this function:

                      def get_num_threads(num_threads=None):
                          if num_threads is not None:
                              return num_threads
                          if "PST_NUM_THREADS" in os.environ:
                              return int(os.environ["PST_NUM_THREADS"])
                          if "NUM_THREADS" in os.environ:
                              return int(os.environ["NUM_THREADS"])
                          if "MKL_NUM_THREADS" in os.environ:
                              return int(os.environ["MKL_NUM_THREADS"])
                          return multiprocessing.cpu_count()

                      Next, on the Rust side, we define create_pool. This helper function constructs a Rayon ThreadPool object from num_threads.

                      pub fn create_pool(num_threads: usize) -> Result<rayon::ThreadPool, BedErrorPlus> {
                         match rayon::ThreadPoolBuilder::new()
                            .num_threads(num_threads)
                            .build()
                         {
                            Err(e) => Err(e.into()),
                            Ok(pool) => Ok(pool),
                         }
                      }

                      Finally, in the Rust translator functionread_f64 eventually calls internal_read_no_alloc. We put that call inside a create_pool(num_threads)?.install(...). This limits all Rayon functions to the num_threads we set.

                      [...]
                          create_pool(num_threads)?.install(|| {
                              [...]
                              internal_read_no_alloc(
                                  buf_reader,
                                  [...]
                              )
                           })?;
                      [...]

                      Rule 8: Translate nice dynamically-type Python functions into nice Rust generic functions

                      Users of the nice Python read function can specify the dtype of the returned NumPy array (int8, float32 or float64). From this choice, the function looks up the appropriate Rust translator function (read_i8, read_f32, or read_f64), which is then called.

                      def read(
                          [...]
                          dtype: Optional[Union[type, str]] = "float32",
                          [...]
                          )
                          [...]
                          if dtype == np.int8:
                              reader = read_i8
                          elif dtype == np.float64:
                              reader = read_f64
                          elif dtype == np.float32:
                              reader = read_f32
                          else:
                              raise ValueError(
                                f"dtype '{val.dtype}' not known, only "
                                + "'int8', 'float32', and 'float64' are allowed."
                              )     reader(
                             str(self.filepath),
                             [...]
                           )

                      The three Rust translator functions (insrc/python_module.rs) call the same nice Rust function, read_and_fill, defined in src/lib.rs. Here are the relevant parts of translator function read_64:

                      #[pyfn(m)]
                      #[pyo3(name = "read_f64")]
                      #[allow(clippy::too_many_arguments)]
                      fn read_f64(
                          [...]
                          val: &PyArray2<f64>,
                          num_threads: usize,
                      ) -> Result<(), PyErr> {
                          [...]
                          // let mut val = unsafe { val.as_array_mut() };
                          // update 12/23/2023
                          let mut val = val.readwrite();
                          let mut val = val.as_array_mut();

                          let mut bed = Bed::builder(filename)
                              .iid_count(iid_count)
                              .sid_count(sid_count)
                              .build()?;

                          ReadOptions::builder()
                              .iid_index(*ii)
                              .sid_index(*si)
                              .is_a1_counted(is_a1_counted)
                              .num_threads(num_threads)
                              .read_and_fill(&mut bed, &mut val.view_mut())?;

                          Ok(())
                      }

                      We define the niceread_and_fill method generically in src/lib.rs. That is, it will work on any type TVal with the right traits. The relevant parts of its code are here:

                      /// A trait alias, used internally, for the values of a .bed file, namely i8, f32, f64.
                      pub trait BedVal: Copy + Default + From<i8> + Debug + Sync + Send + Missing + PartialEq {}
                      impl<T> BedVal for T where T: Copy + Default + From<i8> + Debug + Sync + Send + Missing + PartialEq {
                      [...]
                      impl<TVal: BedVal> ReadOptionsBuilder<TVal> {
                      [...]
                          pub fn read_and_fill(
                              &self,
                              bed: &mut Bed,
                              val: &mut nd::ArrayViewMut2<'_, TVal>, //mutable slices additionally allow to modify elements. But slices cannot grow - they are just a view into some vector.
                          ) -> Result<(), BedErrorPlus> {
                              let read_options = self.build()?;
                              bed.read_and_fill_with_options(val, &read_options)
                          }
                      [...]
                      }

                      Organizing the code in these three levels (nice Python, translator Rust, nice Rust) lets us offer code with dynamic types to our Python users while still writing nice, generic code in Rust.

                      Rule 9: Create both Rust and Python tests

                      You might be tempted to write only Python tests that will call Rust. You should, however, also write Rust tests. The addition of Rust tests lets you run tests interactively and debug interactively. Rust tests also give you a path to later offering a Rust version of your package. In the example project, both sets of tests read test files from bed_reader/tests/data.

                      Where practical, I also recommend writing pure Python versions of your functions. You can then use these slow Python function to test the results of your fast Rust functions.

                      Here are my notes on how I run the Rust and Python tests:

                      # test Rust
                      cargo test

                      # install packages, compile Rust for Python, and test Python
                      pip install -r requirements.txt
                      pip install -r requirements-dev.txt
                      maturin develop
                      pytest bed_reader

                      Finally, your CI script, for example, bed-reader/ci.yml, should run both your Rust and Python tests.

                      So, there you have it: nine rules for writing Python extensions in Rust. Rust is a great language for writing Python extensions. It offers speed, safe data-parallel multithreading, and compatibility with NumPy arrays and other packages. Follow these nine rules to create great code in both Python and Rust.

                      Aside: If you’re interested in future articles, please follow me on Medium. I write on scientific programming in Rust and Python, machine learning, and statistics. I tend to write about one article per month.

                  Python
                  Rust
                  Bioinformatics
                  Parallel Computing
                  Programming

                        --

                        --

                        4

                    Follow

                    Written by Carl M. Kadie

                      535 Followers
                      · Writer for

                          Towards Data Science

                    Ph.D. in CS and Machine Learning. Retired Microsoft & Microsoft Research. Volunteer, open-source projects related to ML, Genomics, Rust, and Python.

                    Follow

                    Help

                    Status

                    About

                    Careers

                    Press

                    Blog

                    Privacy

                    Terms

                    Text to speech

                    Teams

Open in app
            Sign up
              Sign in
                  Write
              Sign up
                Sign in
                        Nine Rules for Writing Python Extensions in Rust
                        Practical Lessons from Upgrading Bed-Reader, a Python Bioinformatics Package
                                                Carl M. Kadie
                                          ·
                                          Follow
                                          Published in
                                              Towards Data Science
                                          ·
                                        11 min read
                                          ·
                                        Dec 31, 2021
                                        --
                                      4
                                                    Listen
                                          Share
                      Photo by David Clode on Unsplash
                      Updated on 4/25/2023. For video version, see this YouTube from the PyData 2023 Conference in Seattle.
                      One year ago, I got fed up with our package’s C++ extension. I rewrote it in Rust. The resulting extension was as fast a C/C++ but with better compatibility and safety. Along the way, I learned nine rules that can help you create better extension code:
                       1. Create a single repository containing both Rust and Python projects
                       2. Use maturin & PyO3 to create Python-callable translator functions in Rust
                       3. Have the Rust translator functions call “nice” Rust functions
                       4. Preallocate memory in Python
                       5. Translate nice Rust error handling into nice Python error handling
                       6. Multithread with Rayon and ndarray::parallel, returning any errors
                       7. Allow users to control the number of parallel threads
                       8. Translate nice dynamically-type Python functions into nice Rust generic functions
                       9. Create both Rust and Python tests
                      The word “nice” here means created with best practices and native types. In other words, the general strategy is this: At the top, write good Python code. In the middle, write a thin layer of translator code in Rust. At the bottom, write good Rust code.
                      Three Layers
                      This strategy may seem obvious but following it can be tricky. This article gives practical advice and examples on how to follow each rule.
                      Bed-Reader is a Python package for reading and writing PLINK Bed Files, a binary format used in bioinformatics to store DNA data. Files in Bed format can be as large as a terabyte. Bed-Reader gives users fast, random access to large subsets of the data. It returns a NumPy array in the user’s choice of int8, float32, or float64.
                      I wanted the Bed-Reader extension code to be:
                        * Faster than Python
                        * Compatible with NumPy
                        * Fully data-parallel multithreading
                        * Compatible with all other packages doing data-parallel multithreading
                        * Safe
                      Our original C++ extension gave me speed, NumPy compatibility, and — with OpenMP — data-parallel multithreading. Sadly, OpenMP requires a runtime library and different Python packages can depend on different, incompatible, versions of that runtime library.
                      Rust gave me everything offered by C++. Beyond that, it solved the runtime-compatibility problem by providing data-parallel multithreading without a runtime library. Moreover, the Rust compiler guarantees thread safety. (It even uncovered a race condition in the original algorithm. In Rust, “Thread safety isn’t just documentation; it’s law”.)
                      Creating a Python extension in Rust requires many design decisions. Based on my experience with Bed-Reader, here are the decisions I recommend. To avoid wishy-washiness, I’ll express these recommendations as rules.
                      Rule 1: Create a single repository containing both Rust and Python projects
                      The table below shows how to layout files.
                      Create files Cargo.toml and src/lib.rs using Rust’s usual ‘cargo new’ command. There is no setup.py file for Python. Instead, pyproject.toml contains PyPi package information, such as the name of the package, its version number, the location of its README file, etc. To work without a setup.py, pyproject.toml must contain:
                      [build-system]
                      requires = ["maturin>=0.14,,
                              sid_index: &PyArray1,
                              val: &PyArray2,
                              num_threads: usize,
                          ) -> Result {
                              let iid_index = iid_index.readonly();
                              let sid_index = sid_index.readonly();
                              let ii = &iid_index.as_slice()?;
                              let si = &sid_index.as_slice()?;
                              // let mut val = unsafe { val.as_array_mut() };
                              // update 12/23/2023
                              let mut val = val.readwrite();
                              let mut val = val.as_array_mut();
                              let mut bed = Bed::builder(filename)
                                  .iid_count(iid_count)
                                  .sid_count(sid_count)
                                  .build()?;
                              ReadOptions::builder()
                                  .iid_index(*ii)
                                  .sid_index(*si)
                                  .is_a1_counted(is_a1_counted)
                                  .num_threads(num_threads)
                                  .read_and_fill(&mut bed, &mut val.view_mut())?;
                              Ok(())
                          }
                      This function takes as input a file name, some integers related to the size of the file, and two 1-D NumPy arrays that tell which subset of the data to read. The function reads values from the file and fills in val, a preallocated 2-D NumPy array.
                      Notice that this function:
                        * translates Python NumPy 1-D arrays into Rust slices, the standard Rust 1-D data structure, via:
                          let iid_index = iid_index.readonly();
                          let ii = &iid_index.as_slice()?;
                        * translates Python NumPy 2-D arrays into 2-D Rust ndarray objects, via:
                          // let mut val = unsafe { val.as_array_mut() }
                          // update 12/23/2023
                          let mut val = val.readwrite();
                          let mut val = val.as_array_mut();
                        * calls read_and_fill, a nice Rust method in src/lib.rs that does the core work.
                        * (Later rules will cover preallocation, f64, PyErr, and num_threads)
                      Doing the core work in a Rust function defined on standard Rust types lets us use Rust best practices for testing, generics, errors, etc. It also gives us a pathway to offer a Rust version of our package.
                      Rule 4: Preallocate memory in Python
                      Preallocating the memory for our results in Python simplifies the Rust code. On the Python side, in bed_reader/_open_bed.py, we import the Rust translator function:
                      from .bed_reader import [...] read_f64 [...]
                      Then, we define a nice Python function that allocates memory, calls the Rust translator function, and returns the result.
                      def read([...]) -> np.ndarray:
                        [...]
                        val = np.zeros((len(iid_index), len(sid_index)), order=order, dtype=dtype)
                        [...]
                        reader = read_f64
                        [...]
                        reader(
                            str(self.filepath),
                            iid_count=self.iid_count,
                            sid_count=self.sid_count,
                            is_a1_counted=self.count_A1,
                            iid_index=iid_index,
                            sid_index=sid_index,
                            val=val,
                            num_threads=num_threads,
                        )
                        [...]
                        return val
                      (Later rules will explain reader = read_f64 and num_threads=num_threads.)
                      Rule 5: Translate nice Rust error handling into nice Python error handling
                      To see how to handle errors, let’s trace two possible errors in open_and_check (called by read_and_fill, our nice Rust method in src/lib.rs).
                      Example Error 1: An error from a standard function — What if Rust’s standard File::open function can’t find the file or can’t open it? In that case, the question mark in this line:
                      let mut buf_reader = BufReader::new(File::open(&path)?);
                      will cause the function to return with some std::io::Error value. To define a function that can return these values, we give the function a return type of Result. We define BedErrorPlus to include all of std::io::Error like so:
                      use thiserror::Error;
                      [...]
                      /// All possible errors returned by this library and the libraries it depends on.
                      // Based on ``
                      #[derive(Error, Debug)]
                      pub enum BedErrorPlus {
                          #[allow(missing_docs)]
                          #[error(transparent)]
                          BedError(#[from] BedError),
                          #[allow(missing_docs)]
                          #[error(transparent)]
                          IOError(#[from] std::io::Error),
                          #[allow(missing_docs)]
                          #[error(transparent)]
                          ThreadPoolError(#[from] ThreadPoolBuildError),
                      [...]
                      }
                      This is nice Rust error handling, but Python doesn’t understand it. So, in src/python_module.rs, we translate. First, we define our translator function read_f64 to return PyErr. Second, we implement a converter from BedErrorPlus to PyErr. The converter creates the right class of Python error (IOError, ValueError, or IndexError) with the right error message. It looks like:
                          impl std::convert::From for PyErr {
                              fn from(err: BedErrorPlus) -> PyErr {
                                  match err {
                                      BedErrorPlus::BedError(BedError::IidIndexTooBig(_))
                                      | BedErrorPlus::BedError(BedError::SidIndexTooBig(_))
                                      | BedErrorPlus::BedError(BedError::IndexMismatch(_, _, _, _))
                                      | BedErrorPlus::BedError(BedError::IndexesTooBigForFiles(_, _))
                                      | BedErrorPlus::BedError(BedError::SubsetMismatch(_, _, _, _)) => {
                                          PyIndexError::new_err(err.to_string())
                                      }
                                      BedErrorPlus::IOError(_) => PyIOError::new_err(err.to_string()),
                                      _ => PyValueError::new_err(err.to_string()),
                                  }
                              }
                          }
                      Example Error 2: An error specific to our function — What if our nice function read_no_alloc can open the file but then realizes the file’s format is wrong? It should raise a custom error like so:
                          if (BED_FILE_MAGIC1 != bytes_vector[0]) || (BED_FILE_MAGIC2 != bytes_vector[1]) {
                              return Err(BedError::IllFormed(path_ref_to_string(&path)).into());
                          }
                      The custom error of type BedError::IllFormed is defined insrc/lib.rs:
                      /// All errors specific to this library.
                      #[derive(Error, Debug, Clone)]
                      pub enum BedError {
                          #[allow(missing_docs)]
                          #[error("Ill-formed BED file. BED file header is incorrect or length is wrong. '{0}'")]
                          IllFormed(String),
                      [...]
                      }
                      The rest of the error handling is the same as in Example Error #1.
                      In the end, for both Rust and Python, for both standard errors and custom errors, the result is a specific error type with an informative error message.
                      Rule 6: Multithread with Rayon and ndarray::parallel, returning any errors
                      The Rust Rayon crate provides easy and lightweight data-parallel multithreading. The ndarray::parallel module applies Rayon to arrays. The usual pattern is to parallelize across the columns (or rows) of one or more 2-D arrays. One challenge is to return any error message from the parallel threads. I’ll highlight two approaches to parallelizing array operations with error handling. Both examples appear in Bed-Reader’s src/lib.rs file.
                      Approach 1: par_bridge().try_for_each
                      Rayon’s par_bridge turns a sequential iterator into a parallel iterator. Its try_for_each method will stop all processing as quickly as it can if an error is hit.
                      In this example, we iterate through two things zipped together:
                        * a DNA location’s binary data and
                        * the columns of our output array.
                      We read the binary data sequentially but process each column’s piece of that data in parallel. We stop on any errors.
                      [... not shown, read bytes for DNA location's data ...]
                      // Zip in the column of the output array
                      .zip(out_val.axis_iter_mut(nd::Axis(1)))
                      // In parallel, decompress the iid info and put it in its column
                      .par_bridge() // This seems faster that parallel zip
                      .try_for_each(|(bytes_vector_result, mut col)| match bytes_vector_result {
                          Err(e) => Err(e),
                          Ok(bytes_vector) => {
                              for out_iid_i in 0..iid_index.len() {
                                  let i_div_4 = i_div_4_array[out_iid_i];
                                  let i_mod_4_times_2 = i_mod_4_times_2_array[out_iid_i];
                                  let genotype_byte: u8 = (bytes_vector[i_div_4] >> i_mod_4_times_2) & 0x03;
                                  col[out_iid_i] = from_two_bits_to_value[genotype_byte as usize];
                              }
                              Ok(())
                          }
                      })?;
                      Approach 2: par_azip!
                      The ndarray packages’s par_azip! macro lets one march through, in parallel, one or more zipped-together arrays (or array pieces). It is, in my opinion, very readable. It doesn’t, however, directly support error handling. We can add error handling by saving any error to a results list.
                      Here is an example from a utility function. The full utility function computes statistics (mean and variance) from three arrays of counts and sums. It does its work in parallel. If it finds an error in the data, it records that error in a result list. After all processing, it checks the result list for errors.
                      [...]
                      let mut result_list: Vec> = vec![Ok(()); sid_count];
                      nd::par_azip!((mut stats_row in stats.axis_iter_mut(nd::Axis(0)),
                           &n_observed in &n_observed_array,
                           &sum_s in &sum_s_array,
                           &sum2_s in &sum2_s_array,
                           result_ptr in &mut result_list)
                      {
                        [...some code not shown...]
                      });
                      // Check the result list for errors
                      result_list.par_iter().try_for_each(|x| (*x).clone())?;
                      [...]
                      Rayon and ndarray::parallel offer many other nice approaches to data-parallel processing. Feel free to use them, just be sure to gather and return any errors. (Do not just use Rust’s “panic”.)
                      Rule 7: Allow users to control the number of parallel threads
                      To play nicely with a user’s other code, the user must be able to control the number of parallel threads each function will use.
                      In the nice Python read function, we give the user an optional num_threadsargument. If they don’t set it, Python sets it via this function:
                      def get_num_threads(num_threads=None):
                          if num_threads is not None:
                              return num_threads
                          if "PST_NUM_THREADS" in os.environ:
                              return int(os.environ["PST_NUM_THREADS"])
                          if "NUM_THREADS" in os.environ:
                              return int(os.environ["NUM_THREADS"])
                          if "MKL_NUM_THREADS" in os.environ:
                              return int(os.environ["MKL_NUM_THREADS"])
                          return multiprocessing.cpu_count()
                      Next, on the Rust side, we define create_pool. This helper function constructs a Rayon ThreadPool object from num_threads.
                      pub fn create_pool(num_threads: usize) -> Result {
                         match rayon::ThreadPoolBuilder::new()
                            .num_threads(num_threads)
                            .build()
                         {
                            Err(e) => Err(e.into()),
                            Ok(pool) => Ok(pool),
                         }
                      }
                      Finally, in the Rust translator functionread_f64 eventually calls internal_read_no_alloc. We put that call inside a create_pool(num_threads)?.install(...). This limits all Rayon functions to the num_threads we set.
                      [...]
                          create_pool(num_threads)?.install(|| {
                              [...]
                              internal_read_no_alloc(
                                  buf_reader,
                                  [...]
                              )
                           })?;
                      [...]
                      Rule 8: Translate nice dynamically-type Python functions into nice Rust generic functions
                      Users of the nice Python read function can specify the dtype of the returned NumPy array (int8, float32 or float64). From this choice, the function looks up the appropriate Rust translator function (read_i8, read_f32, or read_f64), which is then called.
                      def read(
                          [...]
                          dtype: Optional[Union[type, str]] = "float32",
                          [...]
                          )
                          [...]
                          if dtype == np.int8:
                              reader = read_i8
                          elif dtype == np.float64:
                              reader = read_f64
                          elif dtype == np.float32:
                              reader = read_f32
                          else:
                              raise ValueError(
                                f"dtype '{val.dtype}' not known, only "
                                + "'int8', 'float32', and 'float64' are allowed."
                              )     reader(
                             str(self.filepath),
                             [...]
                           )
                      The three Rust translator functions (insrc/python_module.rs) call the same nice Rust function, read_and_fill, defined in src/lib.rs. Here are the relevant parts of translator function read_64:
                      #[pyfn(m)]
                      #[pyo3(name = "read_f64")]
                      #[allow(clippy::too_many_arguments)]
                      fn read_f64(
                          [...]
                          val: &PyArray2,
                          num_threads: usize,
                      ) -> Result {
                          [...]
                          // let mut val = unsafe { val.as_array_mut() };
                          // update 12/23/2023
                          let mut val = val.readwrite();
                          let mut val = val.as_array_mut();
                          let mut bed = Bed::builder(filename)
                              .iid_count(iid_count)
                              .sid_count(sid_count)
                              .build()?;
                          ReadOptions::builder()
                              .iid_index(*ii)
                              .sid_index(*si)
                              .is_a1_counted(is_a1_counted)
                              .num_threads(num_threads)
                              .read_and_fill(&mut bed, &mut val.view_mut())?;
                          Ok(())
                      }
                      We define the niceread_and_fill method generically in src/lib.rs. That is, it will work on any type TVal with the right traits. The relevant parts of its code are here:
                      /// A trait alias, used internally, for the values of a .bed file, namely i8, f32, f64.
                      pub trait BedVal: Copy + Default + From + Debug + Sync + Send + Missing + PartialEq {}
                      impl BedVal for T where T: Copy + Default + From + Debug + Sync + Send + Missing + PartialEq {
                      [...]
                      impl ReadOptionsBuilder {
                      [...]
                          pub fn read_and_fill(
                              &self,
                              bed: &mut Bed,
                              val: &mut nd::ArrayViewMut2, //mutable slices additionally allow to modify elements. But slices cannot grow - they are just a view into some vector.
                          ) -> Result {
                              let read_options = self.build()?;
                              bed.read_and_fill_with_options(val, &read_options)
                          }
                      [...]
                      }
                      Organizing the code in these three levels (nice Python, translator Rust, nice Rust) lets us offer code with dynamic types to our Python users while still writing nice, generic code in Rust.
                      Rule 9: Create both Rust and Python tests
                      You might be tempted to write only Python tests that will call Rust. You should, however, also write Rust tests. The addition of Rust tests lets you run tests interactively and debug interactively. Rust tests also give you a path to later offering a Rust version of your package. In the example project, both sets of tests read test files from bed_reader/tests/data.
                      Where practical, I also recommend writing pure Python versions of your functions. You can then use these slow Python function to test the results of your fast Rust functions.
                      Here are my notes on how I run the Rust and Python tests:
                      # test Rust
                      cargo test
                      # install packages, compile Rust for Python, and test Python
                      pip install -r requirements.txt
                      pip install -r requirements-dev.txt
                      maturin develop
                      pytest bed_reader
                      Finally, your CI script, for example, bed-reader/ci.yml, should run both your Rust and Python tests.
                      So, there you have it: nine rules for writing Python extensions in Rust. Rust is a great language for writing Python extensions. It offers speed, safe data-parallel multithreading, and compatibility with NumPy arrays and other packages. Follow these nine rules to create great code in both Python and Rust.
                      Aside: If you’re interested in future articles, please follow me on Medium. I write on scientific programming in Rust and Python, machine learning, and statistics. I tend to write about one article per month.
                  Python
                  Rust
                  Bioinformatics
                  Parallel Computing
                  Programming
                        --
                        --
                        4
                    Follow
                    Written by Carl M. Kadie
                      535 Followers
                      · Writer for
                          Towards Data Science
                    Ph.D. in CS and Machine Learning. Retired Microsoft & Microsoft Research. Volunteer, open-source projects related to ML, Genomics, Rust, and Python.
                    Follow
                    Help
                    Status
                    About
                    Careers
                    Press
                    Blog
                    Privacy
                    Terms
                    Text to speech
                    Teams
Nine Rules for Writing Python Extensions in Rust
https://towardsdatascience.com/nine-rules-for-writing-python-extensions-in-rust-d35ea3a4ec29
https://www.bmc.com/blogs/python-vs-rust/
Python vs Rust: Choosing Your Language
2022-01-14
January
<div><div> <main> <article> <p>
</p> <div> <p>Python is one of the most ubiquitous and <a href="https://www.bmc.com/blogs/programming-languages/">popular programming languages</a> nowadays. It can power anything from simple scripts and web applications to data analytics. On the other hand, Rust is an up-and-coming programming language quickly gaining popularity in the tech community.</p> <p>Both these programming languages offer their own distinct advantages and disadvantages. In this article, let’s compare how each language stacks up against the other and find the preferable language for specific development needs.</p> <h2>What is Python?</h2> <p>Python was first introduced in 1991 by Guido van Rossum. It is a multiparadigm programming language designed to be easily extensible and help users work efficiently.</p> <p>Python eliminates staples of other programming languages like semicolons and curly brackets while providing a simple programming experience with a simple syntax that increases code readability. Python is considered a more beginner-friendly language due to its simplicity.</p> <p>Python’s extensibility and versatility allow using it across many domains, from <a href="https://www.bmc.com/blogs/system-administrator-vs-security-administrator-whats-the-difference/">system administration</a> and <a href="https://www.bmc.com/blogs/sdlc-software-development-lifecycle/">application development</a> to <a href="https://www.bmc.com/blogs/data-analytics-vs-data-analysis">data analytics</a>, machine learning, and artificial intelligence development.</p> <p><em>(See why <a href="https://www.bmc.com/blogs/python-big-data-analytics/">Python is perfect</a> for big data.)</em></p> <h3>Advantages of Python</h3> <ul> <li><strong>Python has a relatively smaller learning curve compared to other languages.</strong> It can provide a simpler development experience without compromising functionality. The <a href="https://www.bmc.com/blogs/asynchronous-programming/">asynchronous coding style</a> allows developers to easily handle complex coding requirements.</li> <li><strong>A massive collection of libraries and frameworks is available.</strong> Python has gained an impressive number of libraries and frameworks due to its maturity and popularity. As a developer, there is a high chance that you can find a library or framework for any kind of functionality.</li> <li><strong>Python integrates with a wide variety of software,</strong> including enterprise applications and databases. It can be easily integrated with other languages like PHP and .NET.</li> </ul> <h3>Disadvantages of Python</h3> <ul> <li>Python is slower compared to compiled options such as C++ and Java since it is an interpreted language.</li> <li>While Python is easy to debug, some errors won’t be shown until runtime.</li> </ul> <p><em>(Read our comparisons of Python to <a href="https://www.bmc.com/blogs/python-vs-java/">Java</a> &amp; <a href="https://www.bmc.com/blogs/go-vs-python/">Go</a>.)</em></p> <h2>What is Rust?</h2> <p>
Rust is a multiparadigm general-purpose programming language introduced by Graydon Hoare from Mozilla Research. Rust is focused on safety, stability, and performance. It is a statically typed programming language with a memory-efficient architecture and is C/C++ compliant.</p> <p>Even though Rust is a newer language compared to Python, it has quickly gained popularity within the developer community and is the most loved technology, according to the <a href="https://insights.stackoverflow.com/survey/2021#technology-most-loved-dreaded-and-wanted">2021 StackOverflow developer survey</a>. Rust can also be used in many different domains such as:</p> <ul> <li>System developments</li> <li>Web applications</li> <li>Embedded systems</li> <li>Blockchain</li> <li>Game engines</li> </ul> <h3>Advantages of Rust</h3> <ul> <li>Rust is performance-oriented compared to other languages with its fast and memory-efficient architecture with no runtime or garbage collection.</li> <li>Enforces strict safe memory allocations and secure coding practices.</li> <li>Direct safe control over low-level resources. (Comparable to C/C++)</li> </ul> <h3>Disadvantages of Rust</h3> <ul> <li>Relatively higher learning curve compared to languages like Python. A higher degree of coding knowledge is required to use Rust efficiently.</li> <li>Low level of <a href="https://dev.to/napoleon039/monkey-patching-what-is-it-and-should-you-be-using-it-50db">monkey patching</a> support.</li> <li>The compiler can be slow compared to other languages.</li> </ul> <h2>Comparing Python vs Rust</h2> <p>Since we have a basic overview of Python and Rust now, let’s compare them to understand how they stack up against each other.</p> <h3>Ease of coding</h3> <p>Python is inherently designed to provide a simpler development experience. Its highly readable code structure and simple syntax provide developers with a better coding experience. Additionally, users can easily adapt Python to any need and start development quickly as it can be used for many use cases.</p> <p>Meanwhile, Rust is geared more towards system programming and better suited for specific use-cases. Rust will have a steeper learning curve to utilize its features properly.</p> <h3>Performance</h3> <p>As an interpreted language, Python is slower, even with options like CPython geared towards speed. Rust is faster and can be more than twice as fast as Python. Since Rust is compiled directly into machine code, there is no interpreter or virtual machine between the code and the hardware. Another factor that improves the performance of Rust is its memory management.</p> <p>Even without a garbage collector like in Python, Rust ensures proper memory management from the get-go by enforcing checks for memory leaks and irregular memory behaviors. Overall, Rust has comparable performance to languages such as C and C++ without overheads.</p> <h3>Garbage collection</h3> <p>Rust provides developers the option to store data on the stack or the heap. This feature can be used during the program compilation to determine when the memory is no longer required and should be cleaned.</p> <p>Moreover, this option clears out data without the need for the program to decide on allocating and cleaning memory. Rust can be easily integrated with other languages without adversely affecting them as it eliminates the need to run a garbage collector constantly.</p> <p>Python utilizes a garbage collector to check for memory that is not in use and cleans up that unnecessary memory while the program is running.</p> <h3>Documentation</h3> <p>Both these programming languages have excellent documentation by official sources as well as community-provided sources. Python documentation is more beginner-friendly, and even community contributions are clearly defined in an easily understandable manner.</p> <p>Rust also has simple-to-understand documentation. Yet it’s a bit geared towards more technically experienced users and can be a little complex compared to Python.</p> <h3>Extensibility</h3> <p>Python offers a clear advantage in terms of extensibility due to the sheer number of libraries, frameworks, software, and services that are available for Python or support Python.</p> <p>While Rust is a relatively new language, it has a rapidly growing ecosystem due to its popularity. However, it is not comparable to the options available with Python.</p> <h3>Error handling</h3> <p>How Python and Rust handle errors is entirely different. Python will throw an exception when an error is encountered. Rust will return a value when an error is found, while Python will simply throw an error without providing any suggestions on how to fix it. Meanwhile, Rust will provide some recommendations to easily pinpoint and fix the issues.</p> <p>Rust will provide an improved development experience and a better and easier debugging experience than other compiled languages due to its features like:</p> <ul> <li>Guaranteed memory safety</li> <li>Reliability and consistency</li> <li>Comparative user-friendliness</li> </ul> <p>Unlike Python, Rust prevents the need for users to wait until runtime to determine some errors.</p> <h3>Security</h3> <p>Rust emphasizes security, and the guaranteed memory is what separates Rust from other similar languages like C or C++. Rust is completely memory safe unless explicitly specified by the developer. According to <a href="https://anssi-fr.github.io/rust-guide/">Secure Rust Guidelines</a>, the compiler tracks how many variables refer to given data and enforce a set of rules to manage and secure memory at any point for a Rust program.</p> <p>In contrast, Python requires developers to configure the memory management and prevent any memory leaks by themselves.</p> <h3>Community</h3> <p>As both languages are open-source projects, the community is directly involved with the development and improvement of them. Python has a considerably larger community as the more mature platform. You can easily find resources for any kind of Python development need, and you can find guides or answers to most problems with a quick google search.</p> <p>Rust also has a small yet friendly and highly active community. However, you might need to spend a bit more time finding resources for your exact needs.</p> <h2>Choosing between Python &amp; Rust</h2> <p>Both these languages have their unique approaches to development, with each language excelling in one aspect or another. What to choose depends on the <a href="https://www.bmc.com/blogs/application-developer-roles-responsibilities/">specific use case of the developer</a>.</p> <p>In general, Python:</p> <ul> <li>Will be the easiest language to learn, if you are starting up</li> <li>Also provides a simpler development experience</li> </ul> <p>Meanwhile, Rust:</p> <ul> <li>Can have a higher barrier to entry with its complex feature set that will be daunting to new developers.</li> <li>May be an ideal candidate if you want to expand your skillset and learn a second language.</li> </ul> <p>The versatility and extendability of Python are still unmatched. The ability to use Python across many disciplines from web and backend developments, DevOps (Scripting), scientific computing, enterprise applications to machine learning has contributed to the immense popularity of the language. When this versatility is coupled with the user-friendliness of the language, Python is one of the most sought-after languages.</p> <p><em>(Learn about <a href="https://www.bmc.com/blogs/python-tooling/">Python tools for beginners</a>.)</em></p> <p>However, Rust will be the more attractive option if speed and safety are your primary considerations. Its memory-safe nature and speed make Rust the ideal language for tasks like system development, embedded integrations, game engine development, file systems, and VR. Rust provides a programming language with modern sensibilities while facilitating <a href="https://www.bairesdev.com/blog/when-speed-matters-comparing-rust-and-c/">comparable speeds to languages like C/C++</a>.</p> <p>Rust is quickly gaining popularity between the wider technological community as well as organizations like Mozilla, Dropbox, and Cloudflare. AWS has even provided its SDK for Rust.</p> <h2>Both support powerful programming</h2> <p>Both these languages are powerful programming tools. However, as mentioned previously, the best language depends on the specific requirement of the user. Python will continue to be a popular language with its maturity, widespread adoption, and ease of use. Yet, Rust is also quickly gaining inroads to the developer community with its superior speed and secure architecture.</p> <h2>Related reading</h2> <ul> <li><a href="https://www.bmc.com/blogs/categories/devops/">BMC DevOps Blog</a></li> <li><a href="https://www.bmc.com/blogs/lua-programming-language/">The Lua Programming Language Beginner’s Guide</a></li> <li><a href="https://www.bmc.com/blogs/spring-framework/">The Spring Framework Beginner’s Guide: Features, Architecture &amp; Getting Started</a></li> <li><a href="https://www.bmc.com/blogs/chaos-monkey/">Chaos Monkey: A Beginner’s Guide</a></li> <li><a href="https://www.bmc.com/blogs/api-developer-portals/">API/Developer Portals: How To Create Great API Portals</a></li> <li><a href="https://www.bmc.com/blogs/software-development-programming-conferences/">Top Conferences for Programming &amp; Software Development</a></li> </ul> </div>
<div> <h3>Learn ML with our free downloadable guide</h3> <div> <div> <p>This e-book teaches machine learning in the simplest way possible. This book is for managers, programmers, directors – and anyone else who wants to learn machine learning. We start with very basic stats and algebra and build upon that. </p><br/> </div> <div> <p><a href="https://www.bmc.com/forms/machine-learning-ebook.html">
</a></p> </div> </div></div> <div> <p>These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.</p> <p>See an error or have a suggestion? Please let us know by emailing <a href="mailto:blogs@bmc.com">blogs@bmc.com</a>.</p> </div> </article> </main> <section> <h3>Business, Faster than Humanly Possible</h3> <p>BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an Autonomous Digital Enterprise that conquers the opportunities ahead.<br/> <a href="https://www.bmc.com/corporate/about-bmc-software.html">Learn more about BMC ›</a></p> </section> <div> <h3>You may also like</h3> </div> <section> <h3>About the author</h3> <div> <p>
</p> <div> <h4>Shanika Wickramasinghe</h4> <p>Shanika Wickramasinghe is a software engineer by profession and a graduate in Information Technology. Her specialties are Web and Mobile Development. Shanika considers writing the best medium to learn and share her knowledge. She is passionate about everything she does, loves to travel, and enjoys nature whenever she takes a break from her busy work schedule. You can connect with her on LinkedIn.</p> </div> </div> </section> </div></div>||||I|||| BMC
 * 
 * 
 * Solutions
 + 
 o Main Menu
 o Multi-Cloud Management
 o Accelerate Multi-Cloud Adoption
 o Gain Asset Visibility
 o Manage Performance in the Cloud
 o Automate Across Clouds
 o Manage Services Across Clouds
 o Automation & DevOps
 o Automate Application Workflows
 o Automate File Transfers
 o Automate Application Deployment
 o Automate Your Data Center
 o Automate Database Changes
 o Mainframe Enterprise DevOps
 o Modern Developer Experience
 + 
 o Security & Compliance
 o Build a SecOps Strategy
 o Remediate Vulnerabilities
 o Secure Your Mainframe
 o Observability and AIOps
 o Provide Continuous Service Assurance
 o Deliver End-to-End ITOM
 o Optimize Your Mainframe Costs
 o Achieve Peak Database Performance
 o Service Management Excellence
 o 360o Customer Service Management
 o Transform HR Service Management
 o Deliver Modern, Intelligent ITSM
 o See All Your IT Assets
 o Enable a Digital Workplace
 + 
 o AI & Machine Learning
 o Lead ServiceOps Transformation
 o Apply Artificial Intelligence to IT (AIOps)
 o Gain Big Data Insights
 o Accelerate With a Self-Managing Mainframe
 o Solutions by Industry
 o Communication Service Providers
 o Financial Services
 o Healthcare
 o Manufacturing
 o Public Sector
 o Retail
 + 
 o 
 * Products
 o Main Menu
 Featured Products
 + 
 o BMC Helix SaaS Solutions
 o BMC Helix Service Management
 o BMC Helix ITSM
 o BMC Helix Business Workflows
 o BMC Helix Digital Workplace
 o BMC Helix Virtual Agent
 o BMC Helix Operations Management
 o BMC Helix Platform
 o BMC Helix Discovery
 o BMC Helix Continuous Optimization
 o BMC Helix Control-M
 + 
 o Control-M Application Workflow Orchestration
 o Jobs-as-Code (DevOps)
 o Control-M and Managed File Transfer
 o Control-M and Workload Automation
 + 
 o Automated Mainframe Intelligence (BMC AMI)
 o BMC AMI Data
 o BMC AMI Ops
 o BMC AMI Security
 o BMC AMI DevX
 View All BMC Products
 * Partners
 + 
 o Partners
 o Locate a Partner
 o Become a BMC Partner
 o Partner Resources
 o Global Partnerships
 + 
 o Developers
 o Developer Community
 o BMC Helix Platform
 o Jobs-as-Code with Control-M
 + 
 o Integrations & Connectors
 o BMC Helix
 o Control-M
 o BMC Helix Control-M
 o Mainframe
 * Support & Services
 + 
 o Main Menu
 o Education & Certification
 o BMC Academy Login (iLearn)
 o Search for Courses
 o Learning Paths
 o IT Certifications
 + 
 o Services & Consulting
 o Success Subscriptions
 o Strategic Services
 o Mainframe Services
 o Deployment Services
 o Managed Services
 + 
 o Support
 o Customer Support Offerings
 o Downloads
 o Documentation
 o Support by Product Name
 o Knowledge Base
 o Case and Defect Management
 o Ask the Community
 o Supported Products (A-Z)
 Support Central
 * About BMC
 + 
 o Main Menu
 o BMC’s Approach to IT
 o Autonomous Digital Enterprise
 o Transcendent Customer Experience
 o Automation Everywhere
 o Enterprise DevOps
 o Data-Driven Business
 o Adaptive Cybersecurity
 + 
 o Why Choose BMC ?
 o About BMC
 o Leadership Team
 o Customer Stories
 o Corporate Social Responsibility
 o Customer Engagement Program
 o Voice of the Customer Program
 o BMC Innovation Labs
 + 
 o Corporate
 o Contact BMC
 o Careers
 o BMC Community
 o BMC Blogs
 o Newsroom
 * Free Trials & Demos |
 * Get Pricing |
 * Contact BMC
 Search
 Search Result

 View all results

 Popular Searches
 Helix Remedy Control-M TrueSight CMDB Client Management Discovery ITIL Track-It! FootPrints Careers
 * BMC Software
 * AIOps
 * BMC Beat
 * Cloud
 * DevOps
 * Innovation
 * ITSM
 * Mainframe
 * Workload Automation
 * More
 + Big Data
 + The Business of IT
 + IT Operations
 + Security
 + Industry Topics
 + BMC Bloggers List
 + BMC Guides
 + Blogs Sitemap
 * 
 + 
 DevOps Blog

 Python vs Rust: Choosing Your Language

 January 14, 2022
 6 minute read
 Shanika Wickramasinghe

 Python is one of the most ubiquitous and popular programming languages nowadays. It can power anything from simple scripts and web applications to data analytics. On the other hand, Rust is an up-and-coming programming language quickly gaining popularity in the tech community.

 Both these programming languages offer their own distinct advantages and disadvantages. In this article, let’s compare how each language stacks up against the other and find the preferable language for specific development needs.

 What is Python?

 Python was first introduced in 1991 by Guido van Rossum. It is a multiparadigm programming language designed to be easily extensible and help users work efficiently.

 Python eliminates staples of other programming languages like semicolons and curly brackets while providing a simple programming experience with a simple syntax that increases code readability. Python is considered a more beginner-friendly language due to its simplicity.

 Python’s extensibility and versatility allow using it across many domains, from system administration and application development to data analytics, machine learning, and artificial intelligence development.

 (See why Python is perfect for big data.)

 Advantages of Python

 * Python has a relatively smaller learning curve compared to other languages. It can provide a simpler development experience without compromising functionality. The asynchronous coding style allows developers to easily handle complex coding requirements.
 * A massive collection of libraries and frameworks is available. Python has gained an impressive number of libraries and frameworks due to its maturity and popularity. As a developer, there is a high chance that you can find a library or framework for any kind of functionality.
 * Python integrates with a wide variety of software, including enterprise applications and databases. It can be easily integrated with other languages like PHP and .NET.

 Disadvantages of Python

 * Python is slower compared to compiled options such as C++ and Java since it is an interpreted language.
 * While Python is easy to debug, some errors won’t be shown until runtime.

 (Read our comparisons of Python to Java & Go.)

 What is Rust?

 Rust is a multiparadigm general-purpose programming language introduced by Graydon Hoare from Mozilla Research. Rust is focused on safety, stability, and performance. It is a statically typed programming language with a memory-efficient architecture and is C/C++ compliant.

 Even though Rust is a newer language compared to Python, it has quickly gained popularity within the developer community and is the most loved technology, according to the 2021 StackOverflow developer survey. Rust can also be used in many different domains such as:

 * System developments
 * Web applications
 * Embedded systems
 * Blockchain
 * Game engines

 Advantages of Rust

 * Rust is performance-oriented compared to other languages with its fast and memory-efficient architecture with no runtime or garbage collection.
 * Enforces strict safe memory allocations and secure coding practices.
 * Direct safe control over low-level resources. (Comparable to C/C++)

 Disadvantages of Rust

 * Relatively higher learning curve compared to languages like Python. A higher degree of coding knowledge is required to use Rust efficiently.
 * Low level of monkey patching support.
 * The compiler can be slow compared to other languages.

 Comparing Python vs Rust

 Since we have a basic overview of Python and Rust now, let’s compare them to understand how they stack up against each other.

 Ease of coding

 Python is inherently designed to provide a simpler development experience. Its highly readable code structure and simple syntax provide developers with a better coding experience. Additionally, users can easily adapt Python to any need and start development quickly as it can be used for many use cases.

 Meanwhile, Rust is geared more towards system programming and better suited for specific use-cases. Rust will have a steeper learning curve to utilize its features properly.

 Performance

 As an interpreted language, Python is slower, even with options like CPython geared towards speed. Rust is faster and can be more than twice as fast as Python. Since Rust is compiled directly into machine code, there is no interpreter or virtual machine between the code and the hardware. Another factor that improves the performance of Rust is its memory management.

 Even without a garbage collector like in Python, Rust ensures proper memory management from the get-go by enforcing checks for memory leaks and irregular memory behaviors. Overall, Rust has comparable performance to languages such as C and C++ without overheads.

 Garbage collection

 Rust provides developers the option to store data on the stack or the heap. This feature can be used during the program compilation to determine when the memory is no longer required and should be cleaned.

 Moreover, this option clears out data without the need for the program to decide on allocating and cleaning memory. Rust can be easily integrated with other languages without adversely affecting them as it eliminates the need to run a garbage collector constantly.

 Python utilizes a garbage collector to check for memory that is not in use and cleans up that unnecessary memory while the program is running.

 Documentation

 Both these programming languages have excellent documentation by official sources as well as community-provided sources. Python documentation is more beginner-friendly, and even community contributions are clearly defined in an easily understandable manner.

 Rust also has simple-to-understand documentation. Yet it’s a bit geared towards more technically experienced users and can be a little complex compared to Python.

 Extensibility

 Python offers a clear advantage in terms of extensibility due to the sheer number of libraries, frameworks, software, and services that are available for Python or support Python.

 While Rust is a relatively new language, it has a rapidly growing ecosystem due to its popularity. However, it is not comparable to the options available with Python.

 Error handling

 How Python and Rust handle errors is entirely different. Python will throw an exception when an error is encountered. Rust will return a value when an error is found, while Python will simply throw an error without providing any suggestions on how to fix it. Meanwhile, Rust will provide some recommendations to easily pinpoint and fix the issues.

 Rust will provide an improved development experience and a better and easier debugging experience than other compiled languages due to its features like:

 * Guaranteed memory safety
 * Reliability and consistency
 * Comparative user-friendliness

 Unlike Python, Rust prevents the need for users to wait until runtime to determine some errors.

 Security

 Rust emphasizes security, and the guaranteed memory is what separates Rust from other similar languages like C or C++. Rust is completely memory safe unless explicitly specified by the developer. According to Secure Rust Guidelines, the compiler tracks how many variables refer to given data and enforce a set of rules to manage and secure memory at any point for a Rust program.

 In contrast, Python requires developers to configure the memory management and prevent any memory leaks by themselves.

 Community

 As both languages are open-source projects, the community is directly involved with the development and improvement of them. Python has a considerably larger community as the more mature platform. You can easily find resources for any kind of Python development need, and you can find guides or answers to most problems with a quick google search.

 Rust also has a small yet friendly and highly active community. However, you might need to spend a bit more time finding resources for your exact needs.

 Choosing between Python & Rust

 Both these languages have their unique approaches to development, with each language excelling in one aspect or another. What to choose depends on the specific use case of the developer.

 In general, Python:

 * Will be the easiest language to learn, if you are starting up
 * Also provides a simpler development experience

 Meanwhile, Rust:

 * Can have a higher barrier to entry with its complex feature set that will be daunting to new developers.
 * May be an ideal candidate if you want to expand your skillset and learn a second language.

 The versatility and extendability of Python are still unmatched. The ability to use Python across many disciplines from web and backend developments, DevOps (Scripting), scientific computing, enterprise applications to machine learning has contributed to the immense popularity of the language. When this versatility is coupled with the user-friendliness of the language, Python is one of the most sought-after languages.

 (Learn about Python tools for beginners.)

 However, Rust will be the more attractive option if speed and safety are your primary considerations. Its memory-safe nature and speed make Rust the ideal language for tasks like system development, embedded integrations, game engine development, file systems, and VR. Rust provides a programming language with modern sensibilities while facilitating comparable speeds to languages like C/C++.

 Rust is quickly gaining popularity between the wider technological community as well as organizations like Mozilla, Dropbox, and Cloudflare. AWS has even provided its SDK for Rust.

 Both support powerful programming

 Both these languages are powerful programming tools. However, as mentioned previously, the best language depends on the specific requirement of the user. Python will continue to be a popular language with its maturity, widespread adoption, and ease of use. Yet, Rust is also quickly gaining inroads to the developer community with its superior speed and secure architecture.

 Related reading

 * BMC DevOps Blog
 * The Lua Programming Language Beginner’s Guide
 * The Spring Framework Beginner’s Guide: Features, Architecture & Getting Started
 * Chaos Monkey: A Beginner’s Guide
 * API/Developer Portals: How To Create Great API Portals
 * Top Conferences for Programming & Software Development

 Learn ML with our free downloadable guide

 This e-book teaches machine learning in the simplest way possible. This book is for managers, programmers, directors – and anyone else who wants to learn machine learning. We start with very basic stats and algebra and build upon that.

 Download e-book ›
 Download e-book ›

 These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

 See an error or have a suggestion? Please let us know by emailing blogs@bmc.com.

 * 
 * 
 * 

 Business, Faster than Humanly Possible

 BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an Autonomous Digital Enterprise that conquers the opportunities ahead.
 Learn more about BMC ›

 You may also like

 What Is Google Cloud Run?

 GitHub, GitLab, Bitbucket & Azure DevOps: What’s The Difference?

 Getting Started with Cloud Native Applications

 How Containers & Kubernetes Work Together

 Introduction To Jaeger

 Top DevOps Trends in 2022

 About the author

 Shanika Wickramasinghe

 Shanika Wickramasinghe is a software engineer by profession and a graduate in Information Technology. Her specialties are Web and Mobile Development. Shanika considers writing the best medium to learn and share her knowledge. She is passionate about everything she does, loves to travel, and enjoys nature whenever she takes a break from her busy work schedule. You can connect with her on LinkedIn.

 View all posts

 Learn ML with our free downloadable guide

 Download e-book ›

 * Contact
 * Free Trials
 * Legal
 * Privacy Policy
 * Update my preferences
 * Trust Center
 ©Copyright 2005-2024 BMC Software, Inc. Use of this site signifies your acceptance of BMC’s Terms of Use. BMC, the BMC logo, and other BMC marks are assets of BMC Software, Inc. These trademarks are registered and may be registered in the U.S. and in other countries.
Python is one of the most ubiquitous and popular programming languages nowadays. It can power anything from simple scripts and web applications to data analytics. On the other hand, Rust is an up-and-coming programming language quickly gaining popularity in the tech community. Both these programming languages offer their own distinct advantages and disadvantages. In this article, let’s compare how each language stacks up against the other and find the preferable language for specific development needs. What is Python? Python was first introduced in 1991 by Guido van Rossum. It is a multiparadigm programming language designed to be easily extensible and help users work efficiently. Python eliminates staples of other programming languages like semicolons and curly brackets while providing a simple programming experience with a simple syntax that increases code readability. Python is considered a more beginner-friendly language due to its simplicity. Python’s extensibility and versatility allow using it across many domains, from system administration and application development to data analytics, machine learning, and artificial intelligence development.  (See why Python is perfect for big data.)  Advantages of Python   Python has a relatively smaller learning curve compared to other languages. It can provide a simpler development experience without compromising functionality. The asynchronous coding style allows developers to easily handle complex coding requirements.  A massive collection of libraries and frameworks is available. Python has gained an impressive number of libraries and frameworks due to its maturity and popularity. As a developer, there is a high chance that you can find a library or framework for any kind of functionality.  Python integrates with a wide variety of software, including enterprise applications and databases. It can be easily integrated with other languages like PHP and .NET.  Disadvantages of Python  Python is slower compared to compiled options such as C++ and Java since it is an interpreted language. While Python is easy to debug, some errors won’t be shown until runtime.   (Read our comparisons of Python to Java &amp; Go.)  What is Rust? 
Rust is a multiparadigm general-purpose programming language introduced by Graydon Hoare from Mozilla Research. Rust is focused on safety, stability, and performance. It is a statically typed programming language with a memory-efficient architecture and is C/C++ compliant. Even though Rust is a newer language compared to Python, it has quickly gained popularity within the developer community and is the most loved technology, according to the 2021 StackOverflow developer survey. Rust can also be used in many different domains such as:  System developments Web applications Embedded systems Blockchain Game engines  Advantages of Rust  Rust is performance-oriented compared to other languages with its fast and memory-efficient architecture with no runtime or garbage collection. Enforces strict safe memory allocations and secure coding practices. Direct safe control over low-level resources. (Comparable to C/C++)  Disadvantages of Rust  Relatively higher learning curve compared to languages like Python. A higher degree of coding knowledge is required to use Rust efficiently. Low level of monkey patching support. The compiler can be slow compared to other languages.  Comparing Python vs Rust Since we have a basic overview of Python and Rust now, let’s compare them to understand how they stack up against each other. Ease of coding Python is inherently designed to provide a simpler development experience. Its highly readable code structure and simple syntax provide developers with a better coding experience. Additionally, users can easily adapt Python to any need and start development quickly as it can be used for many use cases. Meanwhile, Rust is geared more towards system programming and better suited for specific use-cases. Rust will have a steeper learning curve to utilize its features properly. Performance As an interpreted language, Python is slower, even with options like CPython geared towards speed. Rust is faster and can be more than twice as fast as Python. Since Rust is compiled directly into machine code, there is no interpreter or virtual machine between the code and the hardware. Another factor that improves the performance of Rust is its memory management. Even without a garbage collector like in Python, Rust ensures proper memory management from the get-go by enforcing checks for memory leaks and irregular memory behaviors. Overall, Rust has comparable performance to languages such as C and C++ without overheads. Garbage collection Rust provides developers the option to store data on the stack or the heap. This feature can be used during the program compilation to determine when the memory is no longer required and should be cleaned. Moreover, this option clears out data without the need for the program to decide on allocating and cleaning memory. Rust can be easily integrated with other languages without adversely affecting them as it eliminates the need to run a garbage collector constantly. Python utilizes a garbage collector to check for memory that is not in use and cleans up that unnecessary memory while the program is running. Documentation Both these programming languages have excellent documentation by official sources as well as community-provided sources. Python documentation is more beginner-friendly, and even community contributions are clearly defined in an easily understandable manner. Rust also has simple-to-understand documentation. Yet it’s a bit geared towards more technically experienced users and can be a little complex compared to Python. Extensibility Python offers a clear advantage in terms of extensibility due to the sheer number of libraries, frameworks, software, and services that are available for Python or support Python. While Rust is a relatively new language, it has a rapidly growing ecosystem due to its popularity. However, it is not comparable to the options available with Python. Error handling How Python and Rust handle errors is entirely different. Python will throw an exception when an error is encountered. Rust will return a value when an error is found, while Python will simply throw an error without providing any suggestions on how to fix it. Meanwhile, Rust will provide some recommendations to easily pinpoint and fix the issues. Rust will provide an improved development experience and a better and easier debugging experience than other compiled languages due to its features like:  Guaranteed memory safety Reliability and consistency Comparative user-friendliness  Unlike Python, Rust prevents the need for users to wait until runtime to determine some errors. Security Rust emphasizes security, and the guaranteed memory is what separates Rust from other similar languages like C or C++. Rust is completely memory safe unless explicitly specified by the developer. According to Secure Rust Guidelines, the compiler tracks how many variables refer to given data and enforce a set of rules to manage and secure memory at any point for a Rust program. In contrast, Python requires developers to configure the memory management and prevent any memory leaks by themselves. Community As both languages are open-source projects, the community is directly involved with the development and improvement of them. Python has a considerably larger community as the more mature platform. You can easily find resources for any kind of Python development need, and you can find guides or answers to most problems with a quick google search. Rust also has a small yet friendly and highly active community. However, you might need to spend a bit more time finding resources for your exact needs. Choosing between Python &amp; Rust Both these languages have their unique approaches to development, with each language excelling in one aspect or another. What to choose depends on the specific use case of the developer. In general, Python:  Will be the easiest language to learn, if you are starting up Also provides a simpler development experience  Meanwhile, Rust:  Can have a higher barrier to entry with its complex feature set that will be daunting to new developers. May be an ideal candidate if you want to expand your skillset and learn a second language.  The versatility and extendability of Python are still unmatched. The ability to use Python across many disciplines from web and backend developments, DevOps (Scripting), scientific computing, enterprise applications to machine learning has contributed to the immense popularity of the language. When this versatility is coupled with the user-friendliness of the language, Python is one of the most sought-after languages.  (Learn about Python tools for beginners.)  However, Rust will be the more attractive option if speed and safety are your primary considerations. Its memory-safe nature and speed make Rust the ideal language for tasks like system development, embedded integrations, game engine development, file systems, and VR. Rust provides a programming language with modern sensibilities while facilitating comparable speeds to languages like C/C++. Rust is quickly gaining popularity between the wider technological community as well as organizations like Mozilla, Dropbox, and Cloudflare. AWS has even provided its SDK for Rust. Both support powerful programming Both these languages are powerful programming tools. However, as mentioned previously, the best language depends on the specific requirement of the user. Python will continue to be a popular language with its maturity, widespread adoption, and ease of use. Yet, Rust is also quickly gaining inroads to the developer community with its superior speed and secure architecture. Related reading   BMC DevOps Blog   The Lua Programming Language Beginner’s Guide   The Spring Framework Beginner’s Guide: Features, Architecture &amp; Getting Started   Chaos Monkey: A Beginner’s Guide   API/Developer Portals: How To Create Great API Portals   Top Conferences for Programming &amp; Software Development   
 Learn ML with our free downloadable guide   This e-book teaches machine learning in the simplest way possible. This book is for managers, programmers, directors – and anyone else who wants to learn machine learning. We start with very basic stats and algebra and build upon that.      
      These postings are my own and do not necessarily represent BMC's position, strategies, or opinion. See an error or have a suggestion? Please let us know by emailing blogs@bmc.com.     Business, Faster than Humanly Possible BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to become an Autonomous Digital Enterprise that conquers the opportunities ahead. Learn more about BMC ›    You may also like   About the author  
  Shanika Wickramasinghe Shanika Wickramasinghe is a software engineer by profession and a graduate in Information Technology. Her specialties are Web and Mobile Development. Shanika considers writing the best medium to learn and share her knowledge. She is passionate about everything she does, loves to travel, and enjoys nature whenever she takes a break from her busy work schedule. You can connect with her on LinkedIn.
Python vs Rust: Choosing Your Language
https://www.bmc.com/blogs/python-vs-rust/
https://thenextweb.com/news/thought-you-loved-python-meet-rust-programming-syndication
Thought you loved Python? Wait until you meet Rust
2022-04-10
Ari Joury
<div><div>
 <p>“Technology from the past comes to save the future from itself.” That’s how Graydon Hoare, the creator of Rust, <a href="https://steveklabnik.com/writing/rust-is-more-than-safety">describes what he wants to achieve</a>.</p>
<p>That’s one of the key hallmarks of Rust: using technologies that are well-known to academia but rarely implemented in contemporary programming languages. Technologies that are old, reliable, and sometimes forgotten. But which, above all, work extremely well.</p>
<p>These technologies are used mainly for one thing: safety.</p>
<p>Sounds boring? It is not, if you ask the community. A <a href="https://insights.stackoverflow.com/survey/2021#most-loved-dreaded-and-wanted-language-love-dread">whopping 87 percent of Rust developers love this language</a> above all others, making it the most-loved language since 2016 according to this year’s StackOverflow developer survey.</p>
<p>You’d think that software developers are among the most innovative humans on this planet. Rust, however, is the exact opposite of the “move fast and break things” mantra. Nevertheless, Rust developers are almost guaranteed to learn concepts they’ve never heard about before.</p>
<p>From the novelty of systems programming for some developers over algebraic data types to Rust’s very own approach to memory safety: Every developer can find something new and incredibly useful to learn. And <a href="https://stackoverflow.blog/2020/01/20/what-is-rust-and-why-is-it-so-popular/">there are more reasons</a> to fall in love with Rust.</p>
<h2>More memory safety without garbage collection</h2>
<p>One challenge of every programming language is managing a computer’s memory in a safe and efficient way. Python, for example, has a garbage collector that constantly looks out for no longer used memory and cleans it up while the program runs.</p>
<p>In other languages, like C and C++, the programmer must explicitly allocate and free memory as they go. Since all memory-related problems are therefore cleared before the program runs, this approach is much better for optimizing performance.</p>
<p>On the other hand, memory is yet another thing that developers need to think about all the time. This is one of the reasons why it takes much longer to write a program in C than in Python, even if it does the same thing at the end of the day.</p>
<p>Rust goes an alternative way: Memory is allocated through a system of ownership at compile time. It’s a neat hack to ensure that unused data is being cleaned without forcing the programmer to think about allocating and freeing memory all the time.</p>
<p>Basically, ownership is a collection of three <a href="https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html">rules</a>:</p>
<ol>
<li>Each value in Rust has a variable called owner.</li>
<li>There can only be one owner at a time.</li>
<li>When the owner goes out of scope, the value will be dropped, thus freeing memory.</li>
</ol>
<p>A simple example is assigning a vector in Rust:</p>
<figure></figure>
<p>In the second line, the vector [1, 2, 3] with the owner a is created. After that, b becomes the owner of the vector. Since the correct owner is called in the print statement, this program compiles and returns the expected result when executed:</p>
<figure></figure>
<p>On the other hand, you could try to call the vector with its previous owner, A, like so:</p>
<figure></figure>
<p>In this case, the compiler throws an error because a has already been dropped in the third line. There is <a href="https://depth-first.com/articles/2020/01/27/rust-ownership-by-example/">a lot more depth</a> to the subject, but this is the basic idea.</p>
<p>In comparison, Python would run through in the second case. Its garbage collector would drop A only after the last time that it is called, which is nice for the developer but not so nice in terms of memory space.</p>
<p>In C, things would be a bit more complicated: you would have to allocate memory space for a, then point it to the vector, then allocate more memory space for B, point B to A, and finally free the space taken up by A and B when you’re done.</p>
<p>In this sense, the approach of Rust to memory is a compromise between developing speed and performance. While it is not as dead-easy to write as Python, it’s not as clunky as C once you’ve understood the concept of ownership.</p>
<p>On the other hand, the efficiency is quite astounding: The developer team at Tilde, for example, managed to r<a href="https://www.rust-lang.org/static/pdfs/Rust-Tilde-Whitepaper.pdf">educe their memory usage by 90 percent</a> after rewriting some JavaHTTP pieces in Rust.</p>
<h2>Static typing without getting ugly</h2>
<p>It’s almost a religious war between fans of dynamic typing versus static typing. While it’s much easier to produce software in languages with dynamic typing, the code can become unmaintainable pretty quickly. This is one of the reasons why Python code can be quite hard to maintain, compared to C for example.</p>
<p>On the other hand, having to declare the type of each variable C-style can get rather annoying. If you’ve ever tried to use a [double] in a function that returns a [float] type in C, you know what I mean.</p>
<p>Rust goes a middle way: It’s a static type system, but it only requires the programmer to specify top-level types like function arguments and constants. Inside function bodies, Python-style type inference is allowed.</p>
<p><a href="https://learning-rust.github.io/docs/e3.option_and_result.html">One particularly useful feature</a> of Rust is that it has a None type as well. This allows you to handle exceptions at compile-time so that the program is guaranteed to run smoothly at the end-user. Consider this example, where we can get the full name of a person regardless of whether he has a middle name or not:</p>
<figure></figure>
<p>While versions of the None workaround exist in other languages, too, it showcases the ambition of Rust in a neat way: not making things too hard for writing while keeping the code as durable and maintainable as possible.</p>
<p>While Python is a general-purpose programming language, Rust is, like C, decidedly for systems programming. While Rust isn’t the ideal language if you’re making an application for your end-user, it is perfect for building pieces of software that provide services to other software.</p>
<h2>A nifty approach to systems programming</h2>
<p>As such, efficiency is at the core Rust. That’s best demonstrated by zero-cost abstractions, which interpret code while keeping memory usage to a minimum. As Bjarne Stroustrup, the inventor of C++, <a href="https://www.stroustrup.com/ETAPS-corrected-draft.pdf">puts it</a>: “What you don’t use, you don’t pay for. And further: What you do use, you couldn’t hand code any better.”</p>
<p>For example, consider adding all integer numbers up to 1,000 in Python:</p>
<figure></figure>
<p>This does 1,000 iterations and additions every time the code runs — and you can imagine how much that can slow the code down. In contrast, consider the same thing in Rust:</p>
<figure></figure>
<p>This compiles down to the constant 499500. Effectively, the memory usage has just been decreased by a factor of 1,000.</p>
<p>While these abstractions exist in C as well, Rust makes heavy use of them — in fact, one goal is to add as many zero-cost abstractions to the language as possible. In this sense, Rust is a bit like next-level C.</p>
<p>C has been around for more than 40 years, and Rust aims to do the same. Rust places such a high emphasis on backwards-compatibility that you can still run code in Rust 1.0 today. Likewise, if you write Rust code today, you should still be able to run it in twenty years. Rust won’t get rusty!</p>
<h2>A small but incredible community</h2>
<p>With its emphasis on safety and sustainability and all its nifty details speaking for it, it’s no wonder that <a href="https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine">Dropbox has rewritten a lot of its core structure</a> in Rust. Mozilla, Rust’s first big sponsor, has <a href="https://medium.com/mozilla-tech/a-quantum-leap-for-the-web-a3b7174b3c12">written vital parts of Firefox in it</a>. Microsoft deems C and C++ <a href="https://thenewstack.io/microsoft-rust-is-the-industrys-best-chance-at-safe-systems-programming/#">no longer safe for mission-critical software</a> and is investing more and more in Rust.</p>
<p>And it’s not only big corporations — the love for Rust translates down to the individual programmer. Even though only five percent of StackOverflow’s survey respondents use Rust so far, these developers are extremely enthusiastic about the language.</p>
<p>And there are reasons for that. Not only are the language specification and the compiler very well thought through. There is rustup to install and manage toolchains. There’s Cargo, a command line tool that comes with each Rust installation and helps manage dependencies, run tests and generate documentation.</p>
<p>There’s <a href="https://crates.io/">crates.io</a> where users can share and discover libraries, and docs.rs where they are documented. There are<a href="https://github.com/rust-lang/rust-clippy"> compiler lints from Clippy</a> and <a href="https://github.com/rust-lang/rustfmt">automatic formatting from rustfmt</a>.</p>
<p>Beyond that, there are official and unofficial chats, subreddits, user’s forums, StackOverflow questions, and conferences all over the world. With a c<a href="https://steveklabnik.com/writing/rust-is-more-than-safety">ommunity that puts friendliness above everything</a>, is there anything more to ask for?</p>
<h2>The downside: the need to run before you can walk</h2>
<p>The one thing that is discouraging about Rust is the high startup cost. While you would need one or two days to get productive in most languages, it’s more like one or two weeks in Rust.</p>
<p>This is due to the many new concepts that other languages don’t use, and the fact that there are typically a lot of errors at compile time. You need to handle all exceptions on day one, and can’t just write a scratch code that runs and add exceptions later, like you’d do in Python.</p>
<p>In addition, since Rust is still pretty new, not all libraries that you might want are there yet. Apart from the official documentation and various questions on StackOverflow, there are also not that many tutorials.</p>
<p>The good news is that once you’ve learned the concepts and got your program compiled, it runs through like a charm. Plus, it should still work in twenty years given the backwards-compatibility.</p>
<p>Given the sustainability of your code and the fact that Rust is backed by a number of large corporations, one or two weeks of up-front learning might well be worth it, despite the downsides.</p>
<h2>The bottom line: hack without fear</h2>
<p><a href="https://steveklabnik.com/writing/rust-is-more-than-safety">Rust is more than safety</a>. But it’s hard to deny that many of its core concepts aim at eliminating memory leaks and other safety issues. And in an age where software is everything, safety is a must.</p>
<p>There is probably a space for every upcoming language: <a href="https://towardsdatascience.com/one-in-two-pythonistas-should-learn-golang-now-ba8dacaf06e8">Go</a> is populating the space of Python and Java more and more, <a href="https://thenextweb.com/news/watch-out-python-julia-programming-coding-language-coming-for-crown-syndication">Julia is going after Python in Data Science</a>, and Rust is growing in the realm of Python and C++. What makes Rust special is its incredible community, its innovative features, and the fact that it’s designed to work for decades to come.</p>
<p><a href="https://graydon2.dreamwidth.org/253769.html">There’s a lot more work to do</a>, and only a fraction of that can and will be done in Rust. The new languages of today have a strong chance of sticking around for a while, even though other languages will emerge in the next few years, too. But if I had to place my chips on one language, Rust would be a safe bet.</p>
<p><strong>Further reading:</strong></p>
<p><a href="https://thenextweb.com/news/python-progamming-language-energy-analysis"><em>Python sucks in terms of energy efficiency — literally</em></a></p>
<p><a href="https://thenextweb.com/news/move-over-python-rust-highest-paid-programming-language-2021">Move over Python — Rust is the highest paid programming language of 2021</a></p>
<p><em>This article was written by Ari Joury and was originally published on <a href="https://towardsdatascience.com/">Towards Data Science</a>. You can <a href="https://towardsdatascience.com/thought-you-loved-python-wait-until-you-meet-rust-64a06d976ce">read it here</a>. </em></p>
 </div>||||I|||| News

 * News
 * Events
 + TNW Conference 2023 June 16 & 17, 2022
 + TNW València 2023
 + All events
 * Spaces
 * Programs
 * Newsletters
 * Partner with us
 * 
 * 
 * 
 * 
 * 
 * 
 * Jobs
 * Contact
 News news news news
 * Latest
 * Data and security
 * Deeptech
 * Ecosystems
 * Fintech and ecommerce
 * Future of work
 * Sustainability
 * More
 + Startups and technology

 The heart of tech is coming to the heart of the Mediterranean. Join TNW in València this March 🇪🇸 →

 This article was published on April 10, 2022

 * Programmer

 Thought you loved Python? Wait until you meet Rust

 How a niche phenomenon became StackOverflow’s most beloved language


 April 10, 2022 - 11:00 am

 Story by

 Ari Joury

 Story by

 Ari Joury

 “Technology from the past comes to save the future from itself.” That’s how Graydon Hoare, the creator of Rust, describes what he wants to achieve.

 That’s one of the key hallmarks of Rust: using technologies that are well-known to academia but rarely implemented in contemporary programming languages. Technologies that are old, reliable, and sometimes forgotten. But which, above all, work extremely well.

 These technologies are used mainly for one thing: safety.

 Discover the future of tech!

 Join us at TNW Conference June 15 & 16 in Amsterdam

 Join now

 Sounds boring? It is not, if you ask the community. A whopping 87 percent of Rust developers love this language above all others, making it the most-loved language since 2016 according to this year’s StackOverflow developer survey.

 You’d think that software developers are among the most innovative humans on this planet. Rust, however, is the exact opposite of the “move fast and break things” mantra. Nevertheless, Rust developers are almost guaranteed to learn concepts they’ve never heard about before.

 From the novelty of systems programming for some developers over algebraic data types to Rust’s very own approach to memory safety: Every developer can find something new and incredibly useful to learn. And there are more reasons to fall in love with Rust.

 More memory safety without garbage collection

 One challenge of every programming language is managing a computer’s memory in a safe and efficient way. Python, for example, has a garbage collector that constantly looks out for no longer used memory and cleans it up while the program runs.

 In other languages, like C and C++, the programmer must explicitly allocate and free memory as they go. Since all memory-related problems are therefore cleared before the program runs, this approach is much better for optimizing performance.

 On the other hand, memory is yet another thing that developers need to think about all the time. This is one of the reasons why it takes much longer to write a program in C than in Python, even if it does the same thing at the end of the day.

 Rust goes an alternative way: Memory is allocated through a system of ownership at compile time. It’s a neat hack to ensure that unused data is being cleaned without forcing the programmer to think about allocating and freeing memory all the time.

 Basically, ownership is a collection of three rules :

 1. Each value in Rust has a variable called owner.
 2. There can only be one owner at a time.
 3. When the owner goes out of scope, the value will be dropped, thus freeing memory.

 A simple example is assigning a vector in Rust:

 In the second line, the vector [1, 2, 3] with the owner a is created. After that, b becomes the owner of the vector. Since the correct owner is called in the print statement, this program compiles and returns the expected result when executed:

 On the other hand, you could try to call the vector with its previous owner, A, like so:

 In this case, the compiler throws an error because a has already been dropped in the third line. There is a lot more depth to the subject, but this is the basic idea.

 In comparison, Python would run through in the second case. Its garbage collector would drop A only after the last time that it is called, which is nice for the developer but not so nice in terms of memory space.

 In C, things would be a bit more complicated: you would have to allocate memory space for a, then point it to the vector, then allocate more memory space for B, point B to A, and finally free the space taken up by A and B when you’re done.

 In this sense, the approach of Rust to memory is a compromise between developing speed and performance. While it is not as dead-easy to write as Python, it’s not as clunky as C once you’ve understood the concept of ownership.

 On the other hand, the efficiency is quite astounding: The developer team at Tilde, for example, managed to reduce their memory usage by 90 percent after rewriting some JavaHTTP pieces in Rust.

 Static typing without getting ugly

 It’s almost a religious war between fans of dynamic typing versus static typing. While it’s much easier to produce software in languages with dynamic typing, the code can become unmaintainable pretty quickly. This is one of the reasons why Python code can be quite hard to maintain, compared to C for example.

 On the other hand, having to declare the type of each variable C-style can get rather annoying. If you’ve ever tried to use a [double] in a function that returns a [float] type in C, you know what I mean.

 Rust goes a middle way: It’s a static type system, but it only requires the programmer to specify top-level types like function arguments and constants. Inside function bodies, Python-style type inference is allowed.

 One particularly useful feature of Rust is that it has a None type as well. This allows you to handle exceptions at compile-time so that the program is guaranteed to run smoothly at the end-user. Consider this example, where we can get the full name of a person regardless of whether he has a middle name or not:

 While versions of the None workaround exist in other languages, too, it showcases the ambition of Rust in a neat way: not making things too hard for writing while keeping the code as durable and maintainable as possible.

 While Python is a general-purpose programming language, Rust is, like C, decidedly for systems programming. While Rust isn’t the ideal language if you’re making an application for your end-user, it is perfect for building pieces of software that provide services to other software.

 A nifty approach to systems programming

 As such, efficiency is at the core Rust. That’s best demonstrated by zero-cost abstractions, which interpret code while keeping memory usage to a minimum. As Bjarne Stroustrup, the inventor of C++, puts it: “What you don’t use, you don’t pay for. And further: What you do use, you couldn’t hand code any better.”

 For example, consider adding all integer numbers up to 1,000 in Python:

 This does 1,000 iterations and additions every time the code runs — and you can imagine how much that can slow the code down. In contrast, consider the same thing in Rust:

 This compiles down to the constant 499500. Effectively, the memory usage has just been decreased by a factor of 1,000.

 While these abstractions exist in C as well, Rust makes heavy use of them — in fact, one goal is to add as many zero-cost abstractions to the language as possible. In this sense, Rust is a bit like next-level C.

 C has been around for more than 40 years, and Rust aims to do the same. Rust places such a high emphasis on backwards-compatibility that you can still run code in Rust 1.0 today. Likewise, if you write Rust code today, you should still be able to run it in twenty years. Rust won’t get rusty!

 A small but incredible community

 With its emphasis on safety and sustainability and all its nifty details speaking for it, it’s no wonder that Dropbox has rewritten a lot of its core structure in Rust. Mozilla, Rust’s first big sponsor, has written vital parts of Firefox in it. Microsoft deems C and C++ no longer safe for mission-critical software and is investing more and more in Rust.

 And it’s not only big corporations — the love for Rust translates down to the individual programmer. Even though only five percent of StackOverflow’s survey respondents use Rust so far, these developers are extremely enthusiastic about the language.

 And there are reasons for that. Not only are the language specification and the compiler very well thought through. There is rustup to install and manage toolchains. There’s Cargo, a command line tool that comes with each Rust installation and helps manage dependencies, run tests and generate documentation.

 There’s crates.io where users can share and discover libraries, and docs.rs where they are documented. There are compiler lints from Clippy and automatic formatting from rustfmt.

 Beyond that, there are official and unofficial chats, subreddits, user’s forums, StackOverflow questions, and conferences all over the world. With a community that puts friendliness above everything, is there anything more to ask for?

 The downside: the need to run before you can walk

 The one thing that is discouraging about Rust is the high startup cost. While you would need one or two days to get productive in most languages, it’s more like one or two weeks in Rust.

 This is due to the many new concepts that other languages don’t use, and the fact that there are typically a lot of errors at compile time. You need to handle all exceptions on day one, and can’t just write a scratch code that runs and add exceptions later, like you’d do in Python.

 In addition, since Rust is still pretty new, not all libraries that you might want are there yet. Apart from the official documentation and various questions on StackOverflow, there are also not that many tutorials.

 The good news is that once you’ve learned the concepts and got your program compiled, it runs through like a charm. Plus, it should still work in twenty years given the backwards-compatibility.

 Given the sustainability of your code and the fact that Rust is backed by a number of large corporations, one or two weeks of up-front learning might well be worth it, despite the downsides.

 The bottom line: hack without fear

 Rust is more than safety. But it’s hard to deny that many of its core concepts aim at eliminating memory leaks and other safety issues. And in an age where software is everything, safety is a must.

 There is probably a space for every upcoming language: Go is populating the space of Python and Java more and more, Julia is going after Python in Data Science, and Rust is growing in the realm of Python and C++. What makes Rust special is its incredible community, its innovative features, and the fact that it’s designed to work for decades to come.

 There’s a lot more work to do, and only a fraction of that can and will be done in Rust. The new languages of today have a strong chance of sticking around for a while, even though other languages will emerge in the next few years, too. But if I had to place my chips on one language, Rust would be a safe bet.

 Further reading:

 Python sucks in terms of energy efficiency — literally

 Move over Python — Rust is the highest paid programming language of 2021

 This article was written by Ari Joury and was originally published on Towards Data Science. You can read it here.

 Also tagged with


 * Rust (programming language)
 * programming language
 * Python
 Published April 10, 2022 - 11:00 am UTC
 Back to top


 Popular on TNW

 1. 1

 What Greek myths can teach us about the dangers of AI

 2. 2

 6 ways EU startups can cut spending during the recession

 3. 3

 How to live off the grid as a techie

 4. 4

 How to market… without a marketing department

 5. 5

 Why TensorFlow for Python is dying a slow death

 Related Articles

 house of talent

 Following programming ‘best practices’ will slow you down

 science

 Stop using ‘rainbow’ maps — it doesn’t do your data justice

 Join the fiesta in València!

 TNW is coming to València in March, join us now →

 Discover more

 code word

 The guide to find variance using Python

 readme

 The most important skill for enterprise architects may surprise you: Communication

 The heart of tech

 * 
 * 
 * 
 * 
 * 
 * 
 * 


 More TNW

 * Media
 * Events
 * Programs
 * Spaces
 * Newsletters
 * Deals

 About TNW

 * Partner with us
 * Jobs
 * Terms & Conditions
 * Cookie Statement
 * Privacy Statement
 * Editorial Policy
 * Masthead

 Copyright © 2006—2023, The Next Web B.V. Made with <3 in Amsterdam.

“Technology from the past comes to save the future from itself.” That’s how Graydon Hoare, the creator of Rust, describes what he wants to achieve.
That’s one of the key hallmarks of Rust: using technologies that are well-known to academia but rarely implemented in contemporary programming languages. Technologies that are old, reliable, and sometimes forgotten. But which, above all, work extremely well.
These technologies are used mainly for one thing: safety.
Sounds boring? It is not, if you ask the community. A whopping 87 percent of Rust developers love this language above all others, making it the most-loved language since 2016 according to this year’s StackOverflow developer survey.
You’d think that software developers are among the most innovative humans on this planet. Rust, however, is the exact opposite of the “move fast and break things” mantra. Nevertheless, Rust developers are almost guaranteed to learn concepts they’ve never heard about before.
From the novelty of systems programming for some developers over algebraic data types to Rust’s very own approach to memory safety: Every developer can find something new and incredibly useful to learn. And there are more reasons to fall in love with Rust.
More memory safety without garbage collection
One challenge of every programming language is managing a computer’s memory in a safe and efficient way. Python, for example, has a garbage collector that constantly looks out for no longer used memory and cleans it up while the program runs.
In other languages, like C and C++, the programmer must explicitly allocate and free memory as they go. Since all memory-related problems are therefore cleared before the program runs, this approach is much better for optimizing performance.
On the other hand, memory is yet another thing that developers need to think about all the time. This is one of the reasons why it takes much longer to write a program in C than in Python, even if it does the same thing at the end of the day.
Rust goes an alternative way: Memory is allocated through a system of ownership at compile time. It’s a neat hack to ensure that unused data is being cleaned without forcing the programmer to think about allocating and freeing memory all the time.
Basically, ownership is a collection of three rules:
Each value in Rust has a variable called owner.
There can only be one owner at a time.
When the owner goes out of scope, the value will be dropped, thus freeing memory.
A simple example is assigning a vector in Rust:
 
In the second line, the vector [1, 2, 3] with the owner a is created. After that, b becomes the owner of the vector. Since the correct owner is called in the print statement, this program compiles and returns the expected result when executed:
 
On the other hand, you could try to call the vector with its previous owner, A, like so:
 
In this case, the compiler throws an error because a has already been dropped in the third line. There is a lot more depth to the subject, but this is the basic idea.
In comparison, Python would run through in the second case. Its garbage collector would drop A only after the last time that it is called, which is nice for the developer but not so nice in terms of memory space.
In C, things would be a bit more complicated: you would have to allocate memory space for a, then point it to the vector, then allocate more memory space for B, point B to A, and finally free the space taken up by A and B when you’re done.
In this sense, the approach of Rust to memory is a compromise between developing speed and performance. While it is not as dead-easy to write as Python, it’s not as clunky as C once you’ve understood the concept of ownership.
On the other hand, the efficiency is quite astounding: The developer team at Tilde, for example, managed to reduce their memory usage by 90 percent after rewriting some JavaHTTP pieces in Rust.
Static typing without getting ugly
It’s almost a religious war between fans of dynamic typing versus static typing. While it’s much easier to produce software in languages with dynamic typing, the code can become unmaintainable pretty quickly. This is one of the reasons why Python code can be quite hard to maintain, compared to C for example.
On the other hand, having to declare the type of each variable C-style can get rather annoying. If you’ve ever tried to use a [double] in a function that returns a [float] type in C, you know what I mean.
Rust goes a middle way: It’s a static type system, but it only requires the programmer to specify top-level types like function arguments and constants. Inside function bodies, Python-style type inference is allowed.
 One particularly useful feature of Rust is that it has a None type as well. This allows you to handle exceptions at compile-time so that the program is guaranteed to run smoothly at the end-user. Consider this example, where we can get the full name of a person regardless of whether he has a middle name or not:
 
While versions of the None workaround exist in other languages, too, it showcases the ambition of Rust in a neat way: not making things too hard for writing while keeping the code as durable and maintainable as possible.
While Python is a general-purpose programming language, Rust is, like C, decidedly for systems programming. While Rust isn’t the ideal language if you’re making an application for your end-user, it is perfect for building pieces of software that provide services to other software.
A nifty approach to systems programming
As such, efficiency is at the core Rust. That’s best demonstrated by zero-cost abstractions, which interpret code while keeping memory usage to a minimum. As Bjarne Stroustrup, the inventor of C++, puts it: “What you don’t use, you don’t pay for. And further: What you do use, you couldn’t hand code any better.”
For example, consider adding all integer numbers up to 1,000 in Python:
 
This does 1,000 iterations and additions every time the code runs — and you can imagine how much that can slow the code down. In contrast, consider the same thing in Rust:
 
This compiles down to the constant 499500. Effectively, the memory usage has just been decreased by a factor of 1,000.
While these abstractions exist in C as well, Rust makes heavy use of them — in fact, one goal is to add as many zero-cost abstractions to the language as possible. In this sense, Rust is a bit like next-level C.
C has been around for more than 40 years, and Rust aims to do the same. Rust places such a high emphasis on backwards-compatibility that you can still run code in Rust 1.0 today. Likewise, if you write Rust code today, you should still be able to run it in twenty years. Rust won’t get rusty!
A small but incredible community
With its emphasis on safety and sustainability and all its nifty details speaking for it, it’s no wonder that Dropbox has rewritten a lot of its core structure in Rust. Mozilla, Rust’s first big sponsor, has written vital parts of Firefox in it. Microsoft deems C and C++ no longer safe for mission-critical software and is investing more and more in Rust.
And it’s not only big corporations — the love for Rust translates down to the individual programmer. Even though only five percent of StackOverflow’s survey respondents use Rust so far, these developers are extremely enthusiastic about the language.
And there are reasons for that. Not only are the language specification and the compiler very well thought through. There is rustup to install and manage toolchains. There’s Cargo, a command line tool that comes with each Rust installation and helps manage dependencies, run tests and generate documentation.
There’s crates.io where users can share and discover libraries, and docs.rs where they are documented. There are compiler lints from Clippy and automatic formatting from rustfmt.
Beyond that, there are official and unofficial chats, subreddits, user’s forums, StackOverflow questions, and conferences all over the world. With a community that puts friendliness above everything, is there anything more to ask for?
The downside: the need to run before you can walk
The one thing that is discouraging about Rust is the high startup cost. While you would need one or two days to get productive in most languages, it’s more like one or two weeks in Rust.
This is due to the many new concepts that other languages don’t use, and the fact that there are typically a lot of errors at compile time. You need to handle all exceptions on day one, and can’t just write a scratch code that runs and add exceptions later, like you’d do in Python.
In addition, since Rust is still pretty new, not all libraries that you might want are there yet. Apart from the official documentation and various questions on StackOverflow, there are also not that many tutorials.
The good news is that once you’ve learned the concepts and got your program compiled, it runs through like a charm. Plus, it should still work in twenty years given the backwards-compatibility.
Given the sustainability of your code and the fact that Rust is backed by a number of large corporations, one or two weeks of up-front learning might well be worth it, despite the downsides.
The bottom line: hack without fear
 Rust is more than safety. But it’s hard to deny that many of its core concepts aim at eliminating memory leaks and other safety issues. And in an age where software is everything, safety is a must.
There is probably a space for every upcoming language: Go is populating the space of Python and Java more and more, Julia is going after Python in Data Science, and Rust is growing in the realm of Python and C++. What makes Rust special is its incredible community, its innovative features, and the fact that it’s designed to work for decades to come.
 There’s a lot more work to do, and only a fraction of that can and will be done in Rust. The new languages of today have a strong chance of sticking around for a while, even though other languages will emerge in the next few years, too. But if I had to place my chips on one language, Rust would be a safe bet.
 Further reading: 
  Python sucks in terms of energy efficiency — literally  
 Move over Python — Rust is the highest paid programming language of 2021 
 This article was written by Ari Joury and was originally published on Towards Data Science. You can read it here.
Thought you loved Python? Wait until you meet Rust
https://thenextweb.com/news/thought-you-loved-python-meet-rust-programming-syndication
https://softwaresim.com/blog/rust-programming-for-python-developers/
Rust programming for Python developers
2021-10-09
None
<div><div><p><strong>Which programming language should you choose for your next business project?</strong></p><p>In many situations, the right answer would be Python. Python is a hyper-popular programming language that can be used to build anything from a calculator to a robust website backend. With that said, Python can be lackluster for some applications.</p><p>Here’s where Rust comes in. Rust is obviously not the only alternative to Python, but it possesses certain features that make it highly preferable in certain environments.</p><p>Rust is a relatively young programming language. Rust has won over developers, researchers, and hobbyists with its emphasis on performance, efficiency, and security. Rust’s adoption is still in its infancy, but the language promises to become the prime tool for speed- and safety-oriented projects.</p><p>So, in the context of the team’s needs, would Python be able to satisfy the requirements? Or should you instead consider incorporating Rust in the team’s IT workflows?</p><p>To answer these questions, we need to better understand what makes these programming languages unique!</p><p>If you happen to be looking for a programming language for a data science project, <a href="https://softwaresim.com/blog/the-best-programming-languages-for-statistics/">read our guide detailing the most popular statistical languages</a>. This guide will help you come up with a structured approach to choosing a programming language, and it will also introduce you to the pros and cons of several languages, including Python, R, and MATLAB.</p><h2>What is Python?</h2><p></p><p>Python in a nutshell:</p><ul><li>Possesses hundreds of thousands of libraries.</li><li>Retains a simple and easily readable syntax.</li><li>Demonstrates a very beginner-friendly environment.</li><li>Exhibits flexibility and can be used in virtually any situation.</li><li>Reveals its popularity in machine learning, deep learning, data science, and scientific computing.</li><li>Displays slow performance (mitigated in packages that are implemented in fast programming languages).</li></ul><p>Python is a general-purpose programming language that emphasizes simplicity and clarity in coding. According to the <a href="https://insights.stackoverflow.com/survey/2021#technology-most-popular-technologies">Stack Overflow 2021 Developer Survey</a>, Python is the third most popular programming language out there, with 48.24% of respondents using it extensively in their projects.</p><p>The straightforward syntax of Python has made it the go-to language for beginner programmers. What’s more, Python’s ecosystem contains hundreds of thousands of libraries (packages) for virtually any need. These libraries include modules for web development, game development, GUI (graphical user interface) building, web scraping, scientific computing, and artificial intelligence.</p><p>The Python programming language is an especially strong player in the industries of machine learning and deep learning. Existing off-the-shelf packages hide away the math-intensive implementation details of AI algorithms, allowing anyone to build and use complex models to enhance their business operations.</p><h2>What is Rust?</h2><p></p><p>Rust in a nutshell:</p><ul><li>Demonstrates excellent performance (on par with C and C++).</li><li>Heavily emphasizes memory safety.</li><li>Establishes strict rules to make code secure, safe, and bug-free.</li><li>Has tens of thousands of libraries.</li><li>Relatively difficult to learn due to the C++-like syntax and unique coding rules.</li></ul><p>Released in 2010, Rust is still in the early stages of adoption and is no match to Python when it comes to popularity. Only about 7.03% of developers use Rust, according to the <a href="https://insights.stackoverflow.com/survey/2021#technology-most-popular-technologies">Stack Overflow 2021 Developer Survey</a>.</p><p>But despite its low market share, Rust is the most loved programming language of them all, with 86.98% of developers wishing to continue using it in their projects. Moreover, Rust has maintained this status for <a href="https://www.zdnet.com/article/developers-reveal-their-most-loved-programming-language-and-the-ones-they-dread-using/">six years in a row</a>!</p><p>The reason for the admiration is simple – Rust, albeit difficult to adopt due to its syntax and programming rules, has brilliant documentation, excellent performance, and makes developers more conscious about their code’s quality and safety. The end product of Rust’s business value is code that <a href="https://link.springer.com/chapter/10.1007/978-3-030-60939-9_9">executes quickly</a>, is secure, and has a minimum number of bugs.</p><p>As a general-purpose programming language, Rust – like Python – can again serve as the foundation for any project. It has libraries for mathematical computations, web development, game development, and text processing, among other things.</p><p>Additionally, Rust has recently found increased application in research. More and more researchers are choosing to program their scientific projects in Rust – primarily due to the language’s <a href="https://peerj.com/articles/cs-406/">performance and efficiency</a>.</p><h2>Rust vs Python – comparison of the languages’ key features</h2><p>Now that we have a clear idea of the features and purpose of Rust and Python, we can stack them up against each other. Below, we’ll <a href="https://www.academia.edu/download/66255305/ijatcse761022021.pdf">compare the languages</a> from the standpoints of their:</p><ul><li>Security.</li><li>Performance.</li><li>Ecosystems.</li><li>Ease of use.</li><li>Documentation and community.</li></ul><p></p><h3>Security</h3><p><strong><em>Rust is stronger than Python in the area of memory safety and secure programming. Python is no slouch either, but Rust, unlike Python, implements a set of strict rules to keep developers from crafting unsafe code.</em></strong></p><p>Rust is engineered with<a href="https://dl.acm.org/doi/pdf/10.1145/3319535.3354241"> memory safety and security </a>in mind. The standard implementation of Rust is even called <a href="https://dl.acm.org/doi/pdf/10.1145/3319535.3354241">Safe Rust</a> which clearly showcases the intentions of Rust’s creators with the language.</p><p>The safety of Rust is based on several unique features – including ownership and the borrow checker – whose function is to essentially make sure that variables don’t escape the scope they are intended to be used in. This helps <a href="https://www.usenix.org/system/files/soups2021-fulton.pdf">secure data</a> and prevents memory leaks and bugs.</p><p>Unlike Python, Rust does not have garbage collection. Instead, Rust performs memory management via its safety features. Additionally, Rust doesn’t compile code it deems unsafe – whenever its best practices are violated, the language throws an error message describing what went wrong.</p><p>This “handholding” can make Rust seem rather clunky to developers who are used to more traditional programming logic. However, once mastered, Rust offers a secure coding experience with few to no bugs.</p><p>This is not to say that Python is unsafe. As a high-level language, Python handles memory allocation and deallocation behind the scenes, leaving few opportunities for introducing vulnerabilities to code or triggering memory leaks. However, Python doesn’t have an equivalent to Rust’s strict coding ruleset and thus requires more care from users in terms of security.</p><p></p><h3>Performance</h3><p><strong><em>Rust is significantly faster and more efficient than Python. However, many Python packages are actually written in other, faster programming languages, which may somewhat compensate for the performance differences in some applications.</em></strong></p><p>Performance is another important “selling point” of the Rust programming language. Rust offers performance comparable to that of C and C++. In fact, in some applications, Rust often performs computations faster than C/C++!</p><p>Python, on the other hand, is quite notorious for its sluggishness. As an interpreted programming language, Python is one of the slower programming languages in the industry, and it’s lightyears behind Rust in terms of performance. Rust often achieves <a href="https://ieeexplore.ieee.org/abstract/document/8990314">tenfold faster execution</a> than Python!</p><p>With all that said, Python’s performance heavily depends on which library package is used. Take matrix calculations, for example –standard Python lists appear to be horribly inefficient at them. However, if the team were to use statistical libraries such as SciPy or NumPy instead, substantially better performance could be achieved. That’s because these Python packages (and many others) are implemented in faster low-level languages, which allows users to enjoy good-to-better performance while remaining in Python’s intuitive environment.</p><p>In the end, Rust will likely outperform Python in most situations, but Python can also be a viable choice when speed is important – it all depends on the applications and the library packages used.</p><p></p><h3>Ecosystem</h3><p><strong><em>Python’s ecosystem is richer, offering packages for a wide range of applications such as machine learning/deep learning, data science, statistics, scientific computing, data visualization, and web development.</em></strong></p><p>Python possesses a wealth of library packages intended for use in an extensive array of industries ranging from artificial intelligence to app building. PyPi, Python’s package index, contained over 300,000 packages as of September 2021, while Crates.io (Rust’s package registry) had about 67,000 packages (called “crates” in Rust-speak).</p><p>Python is particularly prominent in machine learning/deep learning thanks to popular packages such as <a href="http://forum.cambricon.com/uploadfile/user/file/20190709/1562668282856425.pdf">TensorFlow</a> (developed by Google), <a href="https://link.springer.com/chapter/10.1007/978-1-4842-4470-8_30">Keras </a>an API built on top of TensorFlow to facilitate its use), <a href="http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-highperformance-deep-learning-library.pdf">PyTorch</a> (developed by Facebook), and <a href="https://www.jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf">scikit-learn</a>. These libraries are unaffected by Python’s low performance because their backends aren’t written exclusively in Python – many of them are implemented in C++, C, or <a href="https://en.wikipedia.org/wiki/CUDA">CUDA</a> (an API that allows developers to use NVIDIA graphics processing units for general-purpose computing).</p><p>Rust’s packages also enable the language to be used virtually for anything and everything, but it’s <a href="https://www.sciencedirect.com/science/article/pii/S0950584921001051">weaker </a>than Python in certain areas. Python is arguably unparalleled in AI and is strong in data science &amp; statistics, and its immense treasury of packages can help organizations build the basis for any project.</p><p>That said if we consider the specific benefits of Rust – namely, memory safety and excellent performance – the smaller ecosystem may not seem problematic. When speed is a concern, it may be worth it to code in Rust, even if this requires some learning and implementing algorithms from scratch.</p><p></p><h3>Intuitiveness and ease of use</h3><p><strong><em>Python is a more beginner-friendly programming language and is a far better choice for those new to programming (as long as we don’t consider the differences in the use cases of Python and Rust).</em></strong></p><p>Python’s syntax is clean and simple, which significantly lessens the language’s learning curve.</p><p>Most notably, Python requires minimal punctuation (e.g., semicolons are not required to end statements) and doesn’t require users to specify the type of variables when declaring them. The language also makes heavy use of indentation to determine code execution order and improve readability.</p><p>Combined, these features lower the entry barrier to Python, which is one of the reasons for the language’s insane popularity. However, the reliance on indentation can make one’s programming experience in Python frustrating – one extra space in the code can cause errors that may be difficult to fix.</p><p>Rust, in contrast, has a syntax that resembles C and C++. Due to this, Rust will be familiar to many people, and those who despise Python’s design choices may favor <a href="https://arxiv.org/pdf/1806.04929">Rust’s syntax</a> as well.</p><p>But no matter what you think about Python’s design philosophy, Rust is most certainly more difficult to learn. Not only is this because of the syntax, but the memory-safe programming rules of Rust will likely make beginners scratch their heads in confusion very often.</p><p>One thing to keep in mind here – Rust’s error messages are actually more useful than Python’s in some cases. As an example, if you mistype a variable name in Rust, the compiler will throw an exception (as expected) and will hint that a variable with a name similar to what has been typed exists. In Python, you get a generic error that just tells you that a variable name is not defined.</p><p>But overall, Python is easier to get started with, and many of you will likely find Python code more visually pleasing.</p><p></p><h3>Documentation and community</h3><p><strong><em>Both Rust and Python have excellent documentation and thriving communities, but there are more resources available for Python due to its popularity and adoption.</em></strong></p><p>Neither of the languages lacks in the availability of educational resources. <a href="https://www.python.org/">Python </a>and <a href="https://doc.rust-lang.org/error-index.html">Rust </a>have well-developed, in-depth standard documentation, and their [communities ](Rust Communities “Rust Communities”)are extremely friendly to beginners. No matter which language you select for your project, you won’t experience a shortage of information, though Python is, all-in-all, better covered.</p><p>But considering how much smaller Rust is than Python, the Rust team’s monumental efforts in documenting their language deserve praise. Rust’s documentation is neatly and logically organized into sections covering aspects of <a href="https://dl.acm.org/doi/pdf/10.1145/3136000.3136010">Rust-like</a>, its package manager <a href="https://doc.rust-lang.org/cargo/">Cargo</a>, the basic functionality of the language, and its foundational features and concepts. The <a href="https://doc.rust-lang.org/error-index.html">Rust Compiler Error Index</a> is a particularly helpful resource that contains Rust’s error codes along with detailed explanations and examples of bad and working codes. The Error Index is interactive as well, with users being able to run, edit, and debug the presented code in their web browsers.</p><p>So, although Rust’s market penetration is no match to that of Python, its outstanding documentation makes the onboarding process quite smooth. Nonetheless, keep in mind that getting to grips with Rust may take much longer than with Python.</p><h2>When to use Rust? When to use Python?</h2><p>The Rust vs Python clash shows no declared winner – each language has distinct advantages that make it suitable for specific applications. To summarize the strengths and weaknesses of Rust and Python, some of the best use cases for each language are outlined in the following table.</p><table><thead><tr><th>Use Rust If:</th><th>Use Python If:</th></tr></thead><tbody><tr><td>The team is working with large datasets and need high performance</td><td>The team is building a project for AI, statistics, or data science purposes</td></tr><tr><td>The team is building a polished app with few to no bugs</td><td>The team needs to deploy a working product fast</td></tr><tr><td>The team requires an extra degree of security</td><td>The team values code readability and simplicity</td></tr></tbody></table><p>In summary, this table suggests:</p><ol><li>Rust is a great fit for projects where performance and memory safety are top priorities. It’s also a wonderful choice if you need to build a complex product with very few bugs.</li><li>Python is a great fit for projects aimed at solving AI, data science, or statistics problems. Its clean and intuitive syntax allows those with limited experience in programming to get a product up and running in-a-minute timeframe.</li></ol><h2>Next steps</h2><p>When trying to pick a programming language for your project, don’t get caught up in the eternal debates about the “best language in the world.” Programming languages are just tools intended to help us achieve specific goals, and no universal language can meet any and all of our needs.</p><p>Choosing a programming language comes down to understanding your project’s requirements and making sure that your toolset will be able to support you on your way to your goals. Be flexible, don’t fixate on a programming language only because it’s likable or popular, and ensure that you have a versatile arsenal of instruments (whether programmatic or not) to achieve your objectives!</p><p>If your team members are interested in a quick taste of either language, take a look at these free, self-paced introductions to <a href="https://www.edx.org/course/programming-for-everybody-getting-started-with-pyt?index=product&amp;queryID=677b68f62a761bf8f2d5bf3fbcf52fcd&amp;position=2">Python </a>or <a href="https://www.youtube.com/watch?v=agzf6ftEsLU">Rust</a>.</p><div><h3>References:</h3><p>Curcio, K., Navarro, T., Malucelli, A., &amp; Reinehr, S. (2018). Requirements engineering: A systematic mapping study in agile software development. <em>Journal of Systems and Software, 139</em>, 32-50.</p><p>Kasauli, R., Liebel, G., Knauss, E., Gopakumar, S., &amp; Kanagwa, B. (2017, September). Requirements engineering challenges in large-scale agile system development. In <em>2017 IEEE 25th International Requirements Engineering Conference (RE)</em> (pp. 352-361). IEEE.</p><p>Vogelsang, A., &amp; Borg, M. (2019, September). Requirements engineering for machine learning: Perspectives from data scientists. In <em>2019 IEEE 27th International Requirements Engineering Conference Workshops (REW)</em> (pp. 245-251). IEEE.</p><p>Zamudio, L., Aguilar, J. A., Tripp, C., &amp; Misra, S. (2017, July). A requirements engineering techniques review in agile software development methods. In <em>International Conference on Computational Science and Its Applications</em> (pp. 683-698). Springer, Cham.</p></div></div></div>||||I|||| Software Solutions Studio
 * Home
 * Studio
 * Getting Started
 * Resources
 Blog White Papers Video Tutorials Books New! AI Safety FAQs
 * About
 * Contact
 Free Trial

 Rust programming for Python developers

 * programming languages
 * rust
 * python

 October 9, 2021 2431 words 12 minutes to read Last Updated: January 15, 2022

 Which programming language should you choose for your next business project?

 In many situations, the right answer would be Python. Python is a hyper-popular programming language that can be used to build anything from a calculator to a robust website backend. With that said, Python can be lackluster for some applications.

 Here’s where Rust comes in. Rust is obviously not the only alternative to Python, but it possesses certain features that make it highly preferable in certain environments.

 Rust is a relatively young programming language. Rust has won over developers, researchers, and hobbyists with its emphasis on performance, efficiency, and security. Rust’s adoption is still in its infancy, but the language promises to become the prime tool for speed- and safety-oriented projects.

 So, in the context of the team’s needs, would Python be able to satisfy the requirements? Or should you instead consider incorporating Rust in the team’s IT workflows?

 To answer these questions, we need to better understand what makes these programming languages unique!

 If you happen to be looking for a programming language for a data science project, read our guide detailing the most popular statistical languages. This guide will help you come up with a structured approach to choosing a programming language, and it will also introduce you to the pros and cons of several languages, including Python, R, and MATLAB.

 What is Python?

 Python in a nutshell:

 * Possesses hundreds of thousands of libraries.
 * Retains a simple and easily readable syntax.
 * Demonstrates a very beginner-friendly environment.
 * Exhibits flexibility and can be used in virtually any situation.
 * Reveals its popularity in machine learning, deep learning, data science, and scientific computing.
 * Displays slow performance (mitigated in packages that are implemented in fast programming languages).

 Python is a general-purpose programming language that emphasizes simplicity and clarity in coding. According to the Stack Overflow 2021 Developer Survey, Python is the third most popular programming language out there, with 48.24% of respondents using it extensively in their projects.

 The straightforward syntax of Python has made it the go-to language for beginner programmers. What’s more, Python’s ecosystem contains hundreds of thousands of libraries (packages) for virtually any need. These libraries include modules for web development, game development, GUI (graphical user interface) building, web scraping, scientific computing, and artificial intelligence.

 The Python programming language is an especially strong player in the industries of machine learning and deep learning. Existing off-the-shelf packages hide away the math-intensive implementation details of AI algorithms, allowing anyone to build and use complex models to enhance their business operations.

 What is Rust?

 Rust in a nutshell:

 * Demonstrates excellent performance (on par with C and C++).
 * Heavily emphasizes memory safety.
 * Establishes strict rules to make code secure, safe, and bug-free.
 * Has tens of thousands of libraries.
 * Relatively difficult to learn due to the C++-like syntax and unique coding rules.

 Released in 2010, Rust is still in the early stages of adoption and is no match to Python when it comes to popularity. Only about 7.03% of developers use Rust, according to the Stack Overflow 2021 Developer Survey.

 But despite its low market share, Rust is the most loved programming language of them all, with 86.98% of developers wishing to continue using it in their projects. Moreover, Rust has maintained this status for six years in a row!

 The reason for the admiration is simple – Rust, albeit difficult to adopt due to its syntax and programming rules, has brilliant documentation, excellent performance, and makes developers more conscious about their code’s quality and safety. The end product of Rust’s business value is code that executes quickly, is secure, and has a minimum number of bugs.

 As a general-purpose programming language, Rust – like Python – can again serve as the foundation for any project. It has libraries for mathematical computations, web development, game development, and text processing, among other things.

 Additionally, Rust has recently found increased application in research. More and more researchers are choosing to program their scientific projects in Rust – primarily due to the language’s performance and efficiency.

 Rust vs Python – comparison of the languages’ key features

 Now that we have a clear idea of the features and purpose of Rust and Python, we can stack them up against each other. Below, we’ll compare the languages from the standpoints of their:

 * Security.
 * Performance.
 * Ecosystems.
 * Ease of use.
 * Documentation and community.

 Security

 Rust is stronger than Python in the area of memory safety and secure programming. Python is no slouch either, but Rust, unlike Python, implements a set of strict rules to keep developers from crafting unsafe code.

 Rust is engineered with memory safety and security in mind. The standard implementation of Rust is even called Safe Rust which clearly showcases the intentions of Rust’s creators with the language.

 The safety of Rust is based on several unique features – including ownership and the borrow checker – whose function is to essentially make sure that variables don’t escape the scope they are intended to be used in. This helps secure data and prevents memory leaks and bugs.

 Unlike Python, Rust does not have garbage collection. Instead, Rust performs memory management via its safety features. Additionally, Rust doesn’t compile code it deems unsafe – whenever its best practices are violated, the language throws an error message describing what went wrong.

 This “handholding” can make Rust seem rather clunky to developers who are used to more traditional programming logic. However, once mastered, Rust offers a secure coding experience with few to no bugs.

 This is not to say that Python is unsafe. As a high-level language, Python handles memory allocation and deallocation behind the scenes, leaving few opportunities for introducing vulnerabilities to code or triggering memory leaks. However, Python doesn’t have an equivalent to Rust’s strict coding ruleset and thus requires more care from users in terms of security.

 Performance

 Rust is significantly faster and more efficient than Python. However, many Python packages are actually written in other, faster programming languages, which may somewhat compensate for the performance differences in some applications.

 Performance is another important “selling point” of the Rust programming language. Rust offers performance comparable to that of C and C++. In fact, in some applications, Rust often performs computations faster than C/C++!

 Python, on the other hand, is quite notorious for its sluggishness. As an interpreted programming language, Python is one of the slower programming languages in the industry, and it’s lightyears behind Rust in terms of performance. Rust often achieves tenfold faster execution than Python!

 With all that said, Python’s performance heavily depends on which library package is used. Take matrix calculations, for example –standard Python lists appear to be horribly inefficient at them. However, if the team were to use statistical libraries such as SciPy or NumPy instead, substantially better performance could be achieved. That’s because these Python packages (and many others) are implemented in faster low-level languages, which allows users to enjoy good-to-better performance while remaining in Python’s intuitive environment.

 In the end, Rust will likely outperform Python in most situations, but Python can also be a viable choice when speed is important – it all depends on the applications and the library packages used.

 Ecosystem

 Python’s ecosystem is richer, offering packages for a wide range of applications such as machine learning/deep learning, data science, statistics, scientific computing, data visualization, and web development.

 Python possesses a wealth of library packages intended for use in an extensive array of industries ranging from artificial intelligence to app building. PyPi, Python’s package index, contained over 300,000 packages as of September 2021, while Crates.io (Rust’s package registry) had about 67,000 packages (called “crates” in Rust-speak).

 Python is particularly prominent in machine learning/deep learning thanks to popular packages such as TensorFlow (developed by Google), Keras an API built on top of TensorFlow to facilitate its use), PyTorch (developed by Facebook), and scikit-learn. These libraries are unaffected by Python’s low performance because their backends aren’t written exclusively in Python – many of them are implemented in C++, C, or CUDA (an API that allows developers to use NVIDIA graphics processing units for general-purpose computing).

 Rust’s packages also enable the language to be used virtually for anything and everything, but it’s weaker than Python in certain areas. Python is arguably unparalleled in AI and is strong in data science & statistics, and its immense treasury of packages can help organizations build the basis for any project.

 That said if we consider the specific benefits of Rust – namely, memory safety and excellent performance – the smaller ecosystem may not seem problematic. When speed is a concern, it may be worth it to code in Rust, even if this requires some learning and implementing algorithms from scratch.

 Intuitiveness and ease of use

 Python is a more beginner-friendly programming language and is a far better choice for those new to programming (as long as we don’t consider the differences in the use cases of Python and Rust).

 Python’s syntax is clean and simple, which significantly lessens the language’s learning curve.

 Most notably, Python requires minimal punctuation (e.g., semicolons are not required to end statements) and doesn’t require users to specify the type of variables when declaring them. The language also makes heavy use of indentation to determine code execution order and improve readability.

 Combined, these features lower the entry barrier to Python, which is one of the reasons for the language’s insane popularity. However, the reliance on indentation can make one’s programming experience in Python frustrating – one extra space in the code can cause errors that may be difficult to fix.

 Rust, in contrast, has a syntax that resembles C and C++. Due to this, Rust will be familiar to many people, and those who despise Python’s design choices may favor Rust’s syntax as well.

 But no matter what you think about Python’s design philosophy, Rust is most certainly more difficult to learn. Not only is this because of the syntax, but the memory-safe programming rules of Rust will likely make beginners scratch their heads in confusion very often.

 One thing to keep in mind here – Rust’s error messages are actually more useful than Python’s in some cases. As an example, if you mistype a variable name in Rust, the compiler will throw an exception (as expected) and will hint that a variable with a name similar to what has been typed exists. In Python, you get a generic error that just tells you that a variable name is not defined.

 But overall, Python is easier to get started with, and many of you will likely find Python code more visually pleasing.

 Documentation and community

 Both Rust and Python have excellent documentation and thriving communities, but there are more resources available for Python due to its popularity and adoption.

 Neither of the languages lacks in the availability of educational resources. Python and Rust have well-developed, in-depth standard documentation, and their [communities ](Rust Communities “Rust Communities”)are extremely friendly to beginners. No matter which language you select for your project, you won’t experience a shortage of information, though Python is, all-in-all, better covered.

 But considering how much smaller Rust is than Python, the Rust team’s monumental efforts in documenting their language deserve praise. Rust’s documentation is neatly and logically organized into sections covering aspects of Rust-like, its package manager Cargo, the basic functionality of the language, and its foundational features and concepts. The Rust Compiler Error Index is a particularly helpful resource that contains Rust’s error codes along with detailed explanations and examples of bad and working codes. The Error Index is interactive as well, with users being able to run, edit, and debug the presented code in their web browsers.

 So, although Rust’s market penetration is no match to that of Python, its outstanding documentation makes the onboarding process quite smooth. Nonetheless, keep in mind that getting to grips with Rust may take much longer than with Python.

 When to use Rust? When to use Python?

 The Rust vs Python clash shows no declared winner – each language has distinct advantages that make it suitable for specific applications. To summarize the strengths and weaknesses of Rust and Python, some of the best use cases for each language are outlined in the following table.

 Use Rust If: Use Python If: 
 The team is working with large datasets and need high performance The team is building a project for AI, statistics, or data science purposes
 The team is building a polished app with few to no bugs The team needs to deploy a working product fast 
 The team requires an extra degree of security The team values code readability and simplicity 
 

 In summary, this table suggests:

 1. Rust is a great fit for projects where performance and memory safety are top priorities. It’s also a wonderful choice if you need to build a complex product with very few bugs.
 2. Python is a great fit for projects aimed at solving AI, data science, or statistics problems. Its clean and intuitive syntax allows those with limited experience in programming to get a product up and running in-a-minute timeframe.

 Next steps

 When trying to pick a programming language for your project, don’t get caught up in the eternal debates about the “best language in the world.” Programming languages are just tools intended to help us achieve specific goals, and no universal language can meet any and all of our needs.

 Choosing a programming language comes down to understanding your project’s requirements and making sure that your toolset will be able to support you on your way to your goals. Be flexible, don’t fixate on a programming language only because it’s likable or popular, and ensure that you have a versatile arsenal of instruments (whether programmatic or not) to achieve your objectives!

 If your team members are interested in a quick taste of either language, take a look at these free, self-paced introductions to Python or Rust.

 References:

 Curcio, K., Navarro, T., Malucelli, A., & Reinehr, S. (2018). Requirements engineering: A systematic mapping study in agile software development. Journal of Systems and Software, 139, 32-50.

 Kasauli, R., Liebel, G., Knauss, E., Gopakumar, S., & Kanagwa, B. (2017, September). Requirements engineering challenges in large-scale agile system development. In 2017 IEEE 25th International Requirements Engineering Conference (RE) (pp. 352-361). IEEE.

 Vogelsang, A., & Borg, M. (2019, September). Requirements engineering for machine learning: Perspectives from data scientists. In 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW) (pp. 245-251). IEEE.

 Zamudio, L., Aguilar, J. A., Tripp, C., & Misra, S. (2017, July). A requirements engineering techniques review in agile software development methods. In International Conference on Computational Science and Its Applications (pp. 683-698). Springer, Cham.

 Search for content

 Get the latest tips, tricks, and white papers

 I agree to the Privacy Policy
 Subscribe
 Share this post

 Related posts

 * programming languages
 * research

 The benefits of the Rust programming language in research

 One of the sectors where Rust’s popularity continues to increase is science and research.

 September 18, 2021

 * statistics
 * data science
 * programming languages

 Selecting the optimal statistical programming language for a data science application

 Researchers and practitioners should carefully, but quickly, assess the needs of their projects before selecting a statistical programming language.

 August 14, 2021

 * stochastic processes
 * input modeling

 The best programming languages for stochastic modeling and analysis

 Stochastic methods have become a vital part of both scientific research and practice, across a wide variety of industries and domains.

 June 26, 2021

 Get full access to the simulator, for free, without creating an account

 Get Instant Access


 Software Solutions Studio

 A platform for the discovery, analysis, and design of software-centric systems

 At Nodematic, we contribute 20% of our revenue to carbon removal

 Product

 * Home
 * Studio
 * Tutorial

 Resources

 * Blog
 * White Papers
 * Video Tutorials
 * FAQs
 * Books

 Social Media

 * Tutorials Code
 * YouTube
 * Hacker News
 * Terms of Service
 * Cookie Policy
 * Privacy Policy

 © Copyright 2020 - 2023

 This part of the website is not ready yet. If you would like to get update notifications, please consider signing up for the newsletter.

 Cookie Policy

 The Software Solutions Studio uses cookies to improve your user experience. By using this website you consent to our cookie policy.

 Accept
Which programming language should you choose for your next business project?  In many situations, the right answer would be Python. Python is a hyper-popular programming language that can be used to build anything from a calculator to a robust website backend. With that said, Python can be lackluster for some applications. Here’s where Rust comes in. Rust is obviously not the only alternative to Python, but it possesses certain features that make it highly preferable in certain environments. Rust is a relatively young programming language. Rust has won over developers, researchers, and hobbyists with its emphasis on performance, efficiency, and security. Rust’s adoption is still in its infancy, but the language promises to become the prime tool for speed- and safety-oriented projects. So, in the context of the team’s needs, would Python be able to satisfy the requirements? Or should you instead consider incorporating Rust in the team’s IT workflows? To answer these questions, we need to better understand what makes these programming languages unique! If you happen to be looking for a programming language for a data science project, read our guide detailing the most popular statistical languages. This guide will help you come up with a structured approach to choosing a programming language, and it will also introduce you to the pros and cons of several languages, including Python, R, and MATLAB. What is Python?   Python in a nutshell:  Possesses hundreds of thousands of libraries. Retains a simple and easily readable syntax. Demonstrates a very beginner-friendly environment. Exhibits flexibility and can be used in virtually any situation. Reveals its popularity in machine learning, deep learning, data science, and scientific computing. Displays slow performance (mitigated in packages that are implemented in fast programming languages).  Python is a general-purpose programming language that emphasizes simplicity and clarity in coding. According to the Stack Overflow 2021 Developer Survey, Python is the third most popular programming language out there, with 48.24% of respondents using it extensively in their projects. The straightforward syntax of Python has made it the go-to language for beginner programmers. What’s more, Python’s ecosystem contains hundreds of thousands of libraries (packages) for virtually any need. These libraries include modules for web development, game development, GUI (graphical user interface) building, web scraping, scientific computing, and artificial intelligence. The Python programming language is an especially strong player in the industries of machine learning and deep learning. Existing off-the-shelf packages hide away the math-intensive implementation details of AI algorithms, allowing anyone to build and use complex models to enhance their business operations. What is Rust?   Rust in a nutshell:  Demonstrates excellent performance (on par with C and C++). Heavily emphasizes memory safety. Establishes strict rules to make code secure, safe, and bug-free. Has tens of thousands of libraries. Relatively difficult to learn due to the C++-like syntax and unique coding rules.  Released in 2010, Rust is still in the early stages of adoption and is no match to Python when it comes to popularity. Only about 7.03% of developers use Rust, according to the Stack Overflow 2021 Developer Survey. But despite its low market share, Rust is the most loved programming language of them all, with 86.98% of developers wishing to continue using it in their projects. Moreover, Rust has maintained this status for six years in a row! The reason for the admiration is simple – Rust, albeit difficult to adopt due to its syntax and programming rules, has brilliant documentation, excellent performance, and makes developers more conscious about their code’s quality and safety. The end product of Rust’s business value is code that executes quickly, is secure, and has a minimum number of bugs. As a general-purpose programming language, Rust – like Python – can again serve as the foundation for any project. It has libraries for mathematical computations, web development, game development, and text processing, among other things. Additionally, Rust has recently found increased application in research. More and more researchers are choosing to program their scientific projects in Rust – primarily due to the language’s performance and efficiency. Rust vs Python – comparison of the languages’ key features Now that we have a clear idea of the features and purpose of Rust and Python, we can stack them up against each other. Below, we’ll compare the languages from the standpoints of their:  Security. Performance. Ecosystems. Ease of use. Documentation and community.    Security   Rust is stronger than Python in the area of memory safety and secure programming. Python is no slouch either, but Rust, unlike Python, implements a set of strict rules to keep developers from crafting unsafe code.   Rust is engineered with memory safety and security in mind. The standard implementation of Rust is even called Safe Rust which clearly showcases the intentions of Rust’s creators with the language. The safety of Rust is based on several unique features – including ownership and the borrow checker – whose function is to essentially make sure that variables don’t escape the scope they are intended to be used in. This helps secure data and prevents memory leaks and bugs. Unlike Python, Rust does not have garbage collection. Instead, Rust performs memory management via its safety features. Additionally, Rust doesn’t compile code it deems unsafe – whenever its best practices are violated, the language throws an error message describing what went wrong. This “handholding” can make Rust seem rather clunky to developers who are used to more traditional programming logic. However, once mastered, Rust offers a secure coding experience with few to no bugs. This is not to say that Python is unsafe. As a high-level language, Python handles memory allocation and deallocation behind the scenes, leaving few opportunities for introducing vulnerabilities to code or triggering memory leaks. However, Python doesn’t have an equivalent to Rust’s strict coding ruleset and thus requires more care from users in terms of security.   Performance   Rust is significantly faster and more efficient than Python. However, many Python packages are actually written in other, faster programming languages, which may somewhat compensate for the performance differences in some applications.   Performance is another important “selling point” of the Rust programming language. Rust offers performance comparable to that of C and C++. In fact, in some applications, Rust often performs computations faster than C/C++! Python, on the other hand, is quite notorious for its sluggishness. As an interpreted programming language, Python is one of the slower programming languages in the industry, and it’s lightyears behind Rust in terms of performance. Rust often achieves tenfold faster execution than Python! With all that said, Python’s performance heavily depends on which library package is used. Take matrix calculations, for example –standard Python lists appear to be horribly inefficient at them. However, if the team were to use statistical libraries such as SciPy or NumPy instead, substantially better performance could be achieved. That’s because these Python packages (and many others) are implemented in faster low-level languages, which allows users to enjoy good-to-better performance while remaining in Python’s intuitive environment. In the end, Rust will likely outperform Python in most situations, but Python can also be a viable choice when speed is important – it all depends on the applications and the library packages used.   Ecosystem   Python’s ecosystem is richer, offering packages for a wide range of applications such as machine learning/deep learning, data science, statistics, scientific computing, data visualization, and web development.   Python possesses a wealth of library packages intended for use in an extensive array of industries ranging from artificial intelligence to app building. PyPi, Python’s package index, contained over 300,000 packages as of September 2021, while Crates.io (Rust’s package registry) had about 67,000 packages (called “crates” in Rust-speak). Python is particularly prominent in machine learning/deep learning thanks to popular packages such as TensorFlow (developed by Google), Keras an API built on top of TensorFlow to facilitate its use), PyTorch (developed by Facebook), and scikit-learn. These libraries are unaffected by Python’s low performance because their backends aren’t written exclusively in Python – many of them are implemented in C++, C, or CUDA (an API that allows developers to use NVIDIA graphics processing units for general-purpose computing). Rust’s packages also enable the language to be used virtually for anything and everything, but it’s weaker than Python in certain areas. Python is arguably unparalleled in AI and is strong in data science &amp; statistics, and its immense treasury of packages can help organizations build the basis for any project. That said if we consider the specific benefits of Rust – namely, memory safety and excellent performance – the smaller ecosystem may not seem problematic. When speed is a concern, it may be worth it to code in Rust, even if this requires some learning and implementing algorithms from scratch.   Intuitiveness and ease of use   Python is a more beginner-friendly programming language and is a far better choice for those new to programming (as long as we don’t consider the differences in the use cases of Python and Rust).   Python’s syntax is clean and simple, which significantly lessens the language’s learning curve. Most notably, Python requires minimal punctuation (e.g., semicolons are not required to end statements) and doesn’t require users to specify the type of variables when declaring them. The language also makes heavy use of indentation to determine code execution order and improve readability. Combined, these features lower the entry barrier to Python, which is one of the reasons for the language’s insane popularity. However, the reliance on indentation can make one’s programming experience in Python frustrating – one extra space in the code can cause errors that may be difficult to fix. Rust, in contrast, has a syntax that resembles C and C++. Due to this, Rust will be familiar to many people, and those who despise Python’s design choices may favor Rust’s syntax as well. But no matter what you think about Python’s design philosophy, Rust is most certainly more difficult to learn. Not only is this because of the syntax, but the memory-safe programming rules of Rust will likely make beginners scratch their heads in confusion very often. One thing to keep in mind here – Rust’s error messages are actually more useful than Python’s in some cases. As an example, if you mistype a variable name in Rust, the compiler will throw an exception (as expected) and will hint that a variable with a name similar to what has been typed exists. In Python, you get a generic error that just tells you that a variable name is not defined. But overall, Python is easier to get started with, and many of you will likely find Python code more visually pleasing.   Documentation and community   Both Rust and Python have excellent documentation and thriving communities, but there are more resources available for Python due to its popularity and adoption.   Neither of the languages lacks in the availability of educational resources. Python and Rust have well-developed, in-depth standard documentation, and their [communities ](Rust Communities “Rust Communities”)are extremely friendly to beginners. No matter which language you select for your project, you won’t experience a shortage of information, though Python is, all-in-all, better covered. But considering how much smaller Rust is than Python, the Rust team’s monumental efforts in documenting their language deserve praise. Rust’s documentation is neatly and logically organized into sections covering aspects of Rust-like, its package manager Cargo, the basic functionality of the language, and its foundational features and concepts. The Rust Compiler Error Index is a particularly helpful resource that contains Rust’s error codes along with detailed explanations and examples of bad and working codes. The Error Index is interactive as well, with users being able to run, edit, and debug the presented code in their web browsers. So, although Rust’s market penetration is no match to that of Python, its outstanding documentation makes the onboarding process quite smooth. Nonetheless, keep in mind that getting to grips with Rust may take much longer than with Python. When to use Rust? When to use Python? The Rust vs Python clash shows no declared winner – each language has distinct advantages that make it suitable for specific applications. To summarize the strengths and weaknesses of Rust and Python, some of the best use cases for each language are outlined in the following table.    Use Rust If: Use Python If:     The team is working with large datasets and need high performance The team is building a project for AI, statistics, or data science purposes   The team is building a polished app with few to no bugs The team needs to deploy a working product fast   The team requires an extra degree of security The team values code readability and simplicity    In summary, this table suggests:  Rust is a great fit for projects where performance and memory safety are top priorities. It’s also a wonderful choice if you need to build a complex product with very few bugs. Python is a great fit for projects aimed at solving AI, data science, or statistics problems. Its clean and intuitive syntax allows those with limited experience in programming to get a product up and running in-a-minute timeframe.  Next steps When trying to pick a programming language for your project, don’t get caught up in the eternal debates about the “best language in the world.” Programming languages are just tools intended to help us achieve specific goals, and no universal language can meet any and all of our needs. Choosing a programming language comes down to understanding your project’s requirements and making sure that your toolset will be able to support you on your way to your goals. Be flexible, don’t fixate on a programming language only because it’s likable or popular, and ensure that you have a versatile arsenal of instruments (whether programmatic or not) to achieve your objectives! If your team members are interested in a quick taste of either language, take a look at these free, self-paced introductions to Python or Rust.  References: Curcio, K., Navarro, T., Malucelli, A., &amp; Reinehr, S. (2018). Requirements engineering: A systematic mapping study in agile software development. Journal of Systems and Software, 139, 32-50. Kasauli, R., Liebel, G., Knauss, E., Gopakumar, S., &amp; Kanagwa, B. (2017, September). Requirements engineering challenges in large-scale agile system development. In 2017 IEEE 25th International Requirements Engineering Conference (RE) (pp. 352-361). IEEE. Vogelsang, A., &amp; Borg, M. (2019, September). Requirements engineering for machine learning: Perspectives from data scientists. In 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW) (pp. 245-251). IEEE. Zamudio, L., Aguilar, J. A., Tripp, C., &amp; Misra, S. (2017, July). A requirements engineering techniques review in agile software development methods. In International Conference on Computational Science and Its Applications (pp. 683-698). Springer, Cham.
Rust programming for Python developers
https://softwaresim.com/blog/rust-programming-for-python-developers/
https://www.codingame.com/playgrounds/54888/rust-for-python-developers---operators/introduction
Coding Games and Programming Challenges to Code Better
2019-01-01
None
<div><div><p>Loading...</p></div><div><div></div></div>
</div>||||I|||| Loading...
Loading...
Coding Games and Programming Challenges to Code Better
https://www.codingame.com/playgrounds/54888/rust-for-python-developers---operators/introduction