<h1><center>Investigating Tensors with PyTorch</center></h1>
<br><span><a href="https://floydhub.com/run?template=https://github.com/sayakpaul/DataCampTensorWithPyTorchBlog">
    
</a></span>

In deep learning it is common to see a lot of discussion around tensors as the cornerstone data structure. Tensor even appears in name of Google’s flagship machine learning library: “TensorFlow”. Tensors are a type of data structure used in linear algebra, and like vectors and matrices, you can calculate arithmetic operations with tensors.

On the other hand, <a href = "https://pytorch.org">PyTorch</a> is a python package built by <b>Facebook</b> that provides two high-level features: 1) Tensor computation (like Numpy) with strong GPU acceleration and 2) Deep Neural Networks built on a tape-based automatic differentiation system.

In this tutorial, you will discover what tensors are and how to manipulate them in Python with PyTorch.

In this tutorial you will cover the following:
- Introduction to Tensors
- Introduction to PyTorch
- Installation steps of PyTorch
- Few tensor operations with PyTorch
- A simple neural network with PyTorch

So, without further ado let's get started with the introduction to Tensors.

## Introduction to Tensors:

A tensor is a generalization of vectors and matrices and is easily understood as a multidimensional array. According to the popular deep learning book called "Deep Learning" (Goodfellow et al.) - 

<i>"In the general case, an array of numbers arranged on a regular grid with a variable number of axes is known as a tensor."</i>

A scalar is zero-order tensor or rank zero tensor. A vector is a one-dimensional or first order tensor and a matrix is a two-dimensional or second order tensor.

The following info-graphic describes tensors in a very convenient way: 

</img>

Let's build the intuition behind tensors in a more lucid way now. 

A tensor is the basic building block of modern machine learning. At its core its a data container. Mostly it contains numbers. Sometimes it even contains strings, but that is rare. So think of it as a bucket of numbers.

But often, people confuse tensors with multi-dimensional arrays. As per StackExchange:

Tensors and multidimensional arrays are different types of object. The first is a type of function, the second is a data structure suitable for representing a tensor in a coordinate system.

Mathematically, tensors are defined as a multi-linear function. A multi-linear function consists of various vector variables. A tensor field is a tensor-valued function. For a rigorous mathematical explanation you can read <a href = "http://math.stackexchange.com/questions/10282/an%C2%ADintroduction%C2%ADto%C2%ADtensors?%20noredirect=1&lq=1">here</a>.

So, tensors are functions or containers which you need to define. The actual calculation happens when there is data fed.  What you see as arrays or multi-dimensional (1D, 2D, …, ND) can be considered as generic tensors.


Now, let's talk a bit about <b>Tensor notation</b>. 

Tensor notation is much like matrix notation with a capital letter representing a tensor and lowercase letters with subscript integers representing scalar values within the tensor.

</img>

Many of the operations that can be performed with scalars, vectors, and matrices can be reformulated to be performed with tensors.

As a tool, tensors and tensor algebra is widely used in the fields of physics and engineering. It is a term and set of techniques known in machine learning in the training and operation of deep learning models can be described in terms of tensors.

## Introducing PyTorch:

PyTorch is a Python based scientific computing package targeted for:

- A replacement for NumPy to use the power of GPUs.
- A deep learning research platform that provides maximum flexibility and speed.

Let's quickly summarize the unique features of PyTorch - 

- PyTorch provides a wide variety of tensor routines to accelerate and fit your scientific computation needs such as slicing, indexing, math operations, linear algebra, reductions. And they are fast.

- PyTorch has a unique way of building neural networks: using and replaying a tape recorder.

- Most frameworks such as TensorFlow, Theano, Caffe and CNTK have a static view of the world. One has to build a neural network, and reuse the same structure again and again. Changing the way the network behaves means that one has to start from scratch.

- PyTorch uses a technique called Reverse-mode auto-differentiation, which allows you to change the way your network behaves arbitrarily with zero lag or overhead. Our inspiration comes from several research papers on this topic, as well as current and past work such as autograd, autograd, Chainer, etc.

(While this technique is not unique to PyTorch, it’s one of the fastest implementations of it to date. You get the best of speed and flexibility for your crazy research.)

- PyTorch has minimal framework overhead. We integrate acceleration libraries such as Intel MKL and NVIDIA (CuDNN, NCCL) to maximize speed. At the core, it’s CPU and GPU Tensor and Neural Network backends (TH, THC, THNN, THCUNN) are written as independent libraries with a C99 API.
They are mature and have been tested for years.

(Hence, PyTorch is quite fast – whether you run small or large neural networks.)

- The memory usage in PyTorch is extremely efficient compared to Torch or some of the alternatives. PyTorch's creators have written custom memory allocators for the GPU to make sure that your deep learning models are maximally memory efficient. This enables you to train bigger deep learning models than before.

This <a href="https://towardsdatascience.com/pytorch-vs-tensorflow-spotting-the-difference-25c75777377b">blog</a> does the comparison between PyTorch and Tensorflow very well. 

<b><i>P.S.: <a href = "https://www.datacamp.com/community/tutorials/tensorflow-tutorial">This DataCamp blog</a> is very good starter for getting started with Tensorflow. </i></b>

## Installing PyTorch

Installation of PyTorch is pretty straight forward. As PyTorch supports efficient GPU computation, it efficiently communicates with your Cuda drivers and performs things faster. 

You will be needing <i>torch</i> and <i>torchvision</i> for using PyTorch. Let's install them for a <b>Windows</b> environment. Later, you will see the steps for Linux and Mac as well. 

I have Python 3.5 with <i>Anaconda</i> setup. Also, I don't have CUDA. Run the following commands to install torch an torchvision:

- pip3 install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp35-cp35m-win_amd64.whl 
- pip3 install torchvision

(Please keep in mind that PyTorch does not support Python 2.7. So it is must have a Python version => 3.5.)

If you have CUDA enabled on your machine feel free and run the following (this is for CUDA 9.0): 
- pip3 install http://download.pytorch.org/whl/cu90/torch-0.4.1-cp35-cp35m-win_amd64.whl 
- pip3 install torchvision




Now, let's see the installation steps for a <b>Linux</b> environment with Python 3.5 and no CUDA support:

- pip3 install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp35-cp35m-linux_x86_64.whl 
- pip3 install torchvision

Yes, you guessed it right! It is the same as the Windows' one. 

Now, if you have CUDA support (9.0) then the step would be: 

- pip3 install torch torchvision

For a Mac environment with Python 3.5 and no CUDA support the steps would be: 
- pip3 install torch torchvision  

And, with CUDA support (9.0): 
- pip3 install torch torchvision 

(MacOS Binaries don't support CUDA, install from source if CUDA is needed)

Hope, you are done with the installation of PyTorch by now! 

Now, let's straight dive to some Tensor arithmetic with PyTorch.

## Tensor arithmetic with PyTorch:

First let's import all the required libraries.


```python
from __future__ import print_function
import torch
```

If the PyTorch installation was successful, then running the above lines of code won't give you any errors. 

Now, let's construct a 5x3 matrix, uninitialized:


```python
x = torch.rand(5, 3)
print(x)
```

    tensor([[ 0.5991,  0.9365,  0.6120],
            [ 0.3622,  0.1408,  0.8811],
            [ 0.6248,  0.4808,  0.0322],
            [ 0.2267,  0.3715,  0.8430],
            [ 0.0145,  0.0900,  0.3418]])


Construct a matrix filled zeros and of data type long:


```python
x = torch.zeros(5, 3, dtype=torch.long)
print(x)
```

    tensor([[ 0,  0,  0],
            [ 0,  0,  0],
            [ 0,  0,  0],
            [ 0,  0,  0],
            [ 0,  0,  0]])


Construct a tensor directly from data:




```python
x = torch.tensor([5.5, 3])
print(x)
```

    tensor([ 5.5000,  3.0000])


If you understood Tensors correctly, tell me what kind of Tensor <i>x</i> is in the comments section!

You can create a tensor based on an existing tensor. These methods will reuse properties of the input tensor, e.g. dtype (data type), unless new values are provided by user:


```python
x = x.new_ones(5, 3, dtype=torch.double)    
print(x)

x = torch.randn_like(x, dtype=torch.float)    
print(x)   
```

    tensor([[ 1.,  1.,  1.],
            [ 1.,  1.,  1.],
            [ 1.,  1.,  1.],
            [ 1.,  1.,  1.],
            [ 1.,  1.,  1.]], dtype=torch.float64)
    tensor([[-1.2174,  1.1807,  1.4249],
            [-1.1114, -0.8098,  0.4003],
            [ 0.0780, -0.5011, -1.0985],
            [ 1.8160, -0.3778, -0.8610],
            [-0.7109, -2.0509, -1.2079]])


Get its size:


```python
print(x.size())
```

    torch.Size([5, 3])


<p>Note that 
``torch.Size`` is in fact a tuple, so it supports all tuple operations.</p>

Now, let's do an addition operation on Tensors. 

### Tensor addition:
The element-wise addition of two tensors with the same dimensions results in a new tensor with the same dimensions where each scalar value is the element-wise addition of the scalars in the parent tensors.

</img>


```python
# Syntax 1 for Tensor addition in PyTorch
y = torch.rand(5, 3)
print(x)
print(y)
print(x + y)
```

    tensor([[-1.2174,  1.1807,  1.4249],
            [-1.1114, -0.8098,  0.4003],
            [ 0.0780, -0.5011, -1.0985],
            [ 1.8160, -0.3778, -0.8610],
            [-0.7109, -2.0509, -1.2079]])
    tensor([[ 0.8285,  0.7619,  0.1147],
            [ 0.1624,  0.8994,  0.6119],
            [ 0.2802,  0.2950,  0.7098],
            [ 0.8132,  0.3382,  0.4383],
            [ 0.6738,  0.2022,  0.3264]])
    tensor([[-0.3889,  1.9426,  1.5396],
            [-0.9490,  0.0897,  1.0122],
            [ 0.3583, -0.2061, -0.3887],
            [ 2.6292, -0.0396, -0.4227],
            [-0.0371, -1.8487, -0.8815]])



```python
# Syntax 2 for Tensor addition in PyTorch
print(torch.add(x, y))
```

    tensor([[-0.3889,  1.9426,  1.5396],
            [-0.9490,  0.0897,  1.0122],
            [ 0.3583, -0.2061, -0.3887],
            [ 2.6292, -0.0396, -0.4227],
            [-0.0371, -1.8487, -0.8815]])


Let's do Tensor subtraction now. 

### Tensor subtraction:
The element-wise subtraction of one tensor from another tensor with the same dimensions results in a new tensor with the same dimensions where each scalar value is the element-wise subtraction of the scalars in the parent tensors4.

</img>

Now let's take a look at <b>Tensor Product</b>:
### Tensor Product:

Performs a matrix multiplication of the matrices mat1 and mat2.

If mat1 is a (n×m) tensor, mat2 is a (m×p) tensor, out will be a (n×p) tensor.

You do Tensor products in PyTorch like the following:


```python
mat1 = torch.randn(2, 3)
mat2 = torch.randn(3, 3)
print(mat1)
print(mat2)
print(torch.mm(mat1, mat2))
```

    tensor([[ 1.9490, -0.6503, -1.9448],
            [-0.7126,  1.0519, -0.4250]])
    tensor([[ 0.0846,  0.4410, -0.0625],
            [-1.3264, -0.5265,  0.2575],
            [-1.3324,  0.6644,  0.3528]])
    tensor([[ 3.6185, -0.0901, -0.9753],
            [-0.8892, -1.1504,  0.1654]])


Note that torch.mm() does not broadcast. Let's discuss a bit about <i><b>broadcasting</b></i>.

### Broadcasting:
The term broadcasting describes how arrays are treated with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python. It does this without making needless copies of data and usually leads to efficient algorithm implementations. There are, however, cases where broadcasting is a bad idea because it leads to inefficient use of memory that slows computation.

Two tensors are “broadcastable” if the following rules hold:

- Each tensor has at least one dimension.
- When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist.

Let's understand this with PyTorch using the following code snippet: 


```python
x=torch.empty(5,7,3)
y=torch.empty(5,7,3)
# same shapes are always broadcastable (i.e. the above rules always hold)
```


```python
x=torch.empty((0,))
y=torch.empty(2,2)
# x and y are not broadcastable, because x does not have at least 1 dimension
```


```python
# can line up trailing dimensions
x=torch.empty(5,3,4,1)
y=torch.empty(  3,1,1)
# x and y are broadcastable.
# 1st trailing dimension: both have size 1
# 2nd trailing dimension: y has size 1
# 3rd trailing dimension: x size == y size
# 4th trailing dimension: y dimension doesn't exist
```


```python
# but:
>>> x=torch.empty(5,2,4,1)
>>> y=torch.empty(  3,1,1)
# x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3
```

Now that you have got a fair idea of broadcasting let's see if two tensors are "broadcastable" then what would be their resulting tensor.

If two tensors x, y are “broadcastable”, the resulting tensor size is calculated as follows:

If the number of dimensions of x and y are not equal, prepend 1 to the dimensions of the tensor with fewer dimensions to make them equal length.
Then, for each dimension size, the resulting dimension size is the max of the sizes of x and y along that dimension.
For Example:


```python
# can line up trailing dimensions to make reading easier
x=torch.empty(5,1,4,1)
y=torch.empty(  3,1,1)
(x+y).size()
torch.Size([5, 3, 4, 1])
```




    torch.Size([5, 3, 4, 1])




```python
# but not necessary:
x=torch.empty(1)
y=torch.empty(3,1,7)
(x+y).size()
torch.Size([3, 1, 7])
```




    torch.Size([3, 1, 7])




```python
x=torch.empty(5,2,4,1)
y=torch.empty(3,1,1)
(x+y).size()
```

You get the concept now !

The tensor product is the most common form of tensor multiplication that you may encounter, but there are many other types of tensor multiplications that exist, such as the <i>tensor dot product</i> and the <i>tensor contraction</i>.

Converting a Torch Tensor to a NumPy array and vice versa is a breeze. The concept is called <b>Numpy Bridge</b>. Let's take a look at that.

### Numpy Bridge: 
The Torch Tensor and NumPy array will share their underlying memory locations, and changing one will change the other.

Converting a Torch Tensor to a NumPy Array. 


```python
# A 1D tensor of 5 ones
a = torch.ones(5)
print(a)
```

    tensor([ 1.,  1.,  1.,  1.,  1.])



```python
# Convert the Torch tensor to a NumPy array
b = a.numpy()
print(b)
```

    [1. 1. 1. 1. 1.]


See how the NumPy array changed in value.

So, in the above sections you did some basic Tensor arithmetic like addition, subtraction and tensor products. Now in the following section you will implement a basic neural network in PyTorch.

## Implementing a simple neural network using PyTorch:
If you need a refresher on neural nets then <a href="https://www.datacamp.com/community/news/a-gentle-introduction-to-neural-networks-for-machine-learning-np2xaq5ew1">this DataCamp article</a> is the best to look for. You might want to check the following links as well:

- https://matrices.io/deep-neural-network-from-scratch/
- http://neuralnetworksanddeeplearning.com
- https://www.youtube.com/watch?v=bxe2T-V8XRs

Before the implementing, let's discuss a bit about a concept called <i>Automatic Differentiation</i> which is central to all neural networks in PyTorch. This is particularly useful for calculating gradients in course of doing backpropagation.

The <i>autograd</i> package provides automatic differentiation for all operations on Tensors. It is a define-by-run framework, which means that your backpropagation is defined by how your code is run, and that every single iteration can be different.

Let's see Automatic Differentiation in action with a very simple code example.


```python
# Create a Rank-2 tensor of all ones
x = torch.ones(2, 2, requires_grad=True)
print(x)
```

    tensor([[ 1.,  1.],
            [ 1.,  1.]])


Do an addition operation.


```python
y = x + 2
print(y)
```

    tensor([[ 3.,  3.],
            [ 3.,  3.]])


Do some more operations on <i>y</i>.


```python
z = y * y * 3
out = z.mean()

print(z, out)
```

    tensor([[ 27.,  27.],
            [ 27.,  27.]]) tensor(27.)


Let’s backprop now because <i>out</i> contains a single scalar, out.backward() is equivalent to out.backward(torch.tensor(1)).


```python
out.backward()

# print gradients d(out)/dx
print(x.grad)
```

    tensor([[ 4.5000,  4.5000],
            [ 4.5000,  4.5000]])


You should have got a matrix of 4.5. Let’s call the out Tensor “o”. We have that \begin{align}o = \frac{1}{4}\sum_i z_i  
\end{align} where \begin{align} z_i = 3(x_i+2)^2 \ \end{align}
and  \begin{align} z_i\bigr\rvert_{x_i=1} = 27 \end{align}.


Therefore, \begin{align}\frac{\partial o}{\partial x_i} = \frac{3}{2}(x_i+2)\end{align}, hence \begin{align}\frac{\partial o}{\partial x_i}\bigr\rvert_{x_i=1} = \frac{9}{2} = 4.5 \end{align}.

Now that you know about Automatic Differentiation and how PyTorch handles it, let's straight get to its implementation in PyTorch: 

You’ll create a simple neural network with one hidden layer and a single output unit. You will use <a href = "https://medium.com/alchemicai/activation-functions-demystified-661d1183f5f8">the ReLU activation in the hidden layer and the sigmoid activation in the output layer</a>.

First, you need to import the PyTorch library. Neural networks can be constructed using the <b>torch.nn</b> package.


```python
import torch
import torch.nn as nn
```

Then you e define the sizes of all the layers and the batch size:


```python
n_in, n_h, n_out, batch_size = 10, 5, 1, 10
```

And now, you will create some dummy input data <b>x</b> and some dummy target data <b>y</b>. You will use PyTorch Tensors to store this data. PyTorch Tensors can be used and manipulated just like NumPy arrays but with the added benefit that PyTorch tensors can be run on the GPUs. But you will simply run them on the CPU for this tutorial. Although, it is quite simple to transfer them to a GPU.


```python
x = torch.randn(batch_size, n_in)
y = torch.tensor([[1.0], [0.0], [0.0], [1.0], [1.0], [1.0], [0.0], [0.0], [1.0], [1.0]])
```

And now, you will define our model in one line of code.


```python
model = nn.Sequential(nn.Linear(n_in, n_h),
                     nn.ReLU(),
                     nn.Linear(n_h, n_out),
                     nn.Sigmoid())
```

This creates a model that looks like input -> linear -> relu -> linear -> sigmoid. There is another way to define your models which is used to define more complicated and custom models. It is done by defining our model in a class. You can read about it <a href = "https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules">here</a>.

Now, it is time to construct your loss function. You will use the <b>Mean Squared Error Loss</b>.


```python
criterion = torch.nn.MSELoss()
```

Also, don’t forget to define your optimizer. You will use the mighty Stochastic Gradient Descent in this one and a learning rate of 0.01. model.parameters() returns an iterator over your model’s parameters (weights and biases).


```python
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
```

Now, you will run Gradient Descent for 50 epochs. This does the forward propagation, loss computation, backward propagation and parameter updation in that sequence.


```python
for epoch in range(50):
    # Forward Propagation
    y_pred = model(x)
    # Compute and print loss
    loss = criterion(y_pred, y)
    print('epoch: ', epoch,' loss: ', loss.item())
    # Zero the gradients
    optimizer.zero_grad()
    
    # perform a backward pass (backpropagation)
    loss.backward()
    
    # Update the parameters
    optimizer.step()
```

    epoch:  0  loss:  0.2399429827928543
    epoch:  1  loss:  0.23988191783428192
    epoch:  2  loss:  0.23982088267803192
    epoch:  3  loss:  0.2397598922252655
    epoch:  4  loss:  0.23969893157482147
    epoch:  5  loss:  0.23963800072669983
    epoch:  6  loss:  0.23957709968090057
    epoch:  7  loss:  0.23951618373394012
    epoch:  8  loss:  0.23945537209510803
    epoch:  9  loss:  0.23939454555511475
    epoch:  10  loss:  0.23933371901512146
    epoch:  11  loss:  0.23927298188209534
    epoch:  12  loss:  0.23921218514442444
    epoch:  13  loss:  0.23915143311023712
    epoch:  14  loss:  0.2390907108783722
    epoch:  15  loss:  0.23903003334999084
    epoch:  16  loss:  0.23896940052509308
    epoch:  17  loss:  0.23890872299671173
    epoch:  18  loss:  0.23884813487529755
    epoch:  19  loss:  0.23878750205039978
    epoch:  20  loss:  0.23872694373130798
    epoch:  21  loss:  0.2386663407087326
    epoch:  22  loss:  0.2386058121919632
    epoch:  23  loss:  0.23854532837867737
    epoch:  24  loss:  0.23848481476306915
    epoch:  25  loss:  0.23842433094978333
    epoch:  26  loss:  0.2383638620376587
    epoch:  27  loss:  0.23830339312553406
    epoch:  28  loss:  0.2382429838180542
    epoch:  29  loss:  0.23818258941173553
    epoch:  30  loss:  0.2381247729063034
    epoch:  31  loss:  0.2380656749010086
    epoch:  32  loss:  0.23800739645957947
    epoch:  33  loss:  0.2379491776227951
    epoch:  34  loss:  0.2378900945186615
    epoch:  35  loss:  0.23783239722251892
    epoch:  36  loss:  0.23777374625205994
    epoch:  37  loss:  0.23771481215953827
    epoch:  38  loss:  0.23765745759010315
    epoch:  39  loss:  0.23759838938713074
    epoch:  40  loss:  0.23753997683525085
    epoch:  41  loss:  0.2374821901321411
    epoch:  42  loss:  0.23742322623729706
    epoch:  43  loss:  0.23736533522605896
    epoch:  44  loss:  0.23730707168579102
    epoch:  45  loss:  0.23724813759326935
    epoch:  46  loss:  0.23719079792499542
    epoch:  47  loss:  0.23713204264640808
    epoch:  48  loss:  0.23707345128059387
    epoch:  49  loss:  0.2370160073041916


- y_pred gets the predicted values from a forward pass of our model. You pass this, along with target values y to the criterion which calculates the loss. 

- Then, optimizer.zero_grad() zeroes out all the gradients. You need to do this so that previous gradients don’t keep on accumulating. 

- Then, loss.backward() is the main PyTorch magic that uses PyTorch’s Autograd feature. Autograd computes all the gradients w.r.t. all the parameters automatically based on the computation graph that it creates dynamically. Basically, this does the backward pass (backpropagation) of gradient descent. 

- Finally, you call optimizer.step() which does a single updation of all the parameters using the new gradients.

So, you have made till the end. In this post, you covered a whole bunch of things starting from Tensors to Automatic Differentiation and what not! You also implemented a simple neural net using PyTorch and its tensor system. 

If you want to learn more about PyTorch and want to dive deeper into it, take a look at PyTorch’s official documentation and tutorials. They are really well-written. You can find them <a href = "https://pytorch.org">here</a>.

Following are some references which helped me writing this tutorial: 

- A Student’s Guide to Vectors and Tensors, 2011. <a href = "http://amzn.to/2kmUvvF">link</a>
- PyTorch's official documentation (I already mentioned the link).
- A gentle introduction to Tensors - <a href = "https://machinelearningmastery.com/introduction-to-tensors-for-machine-learning/">Machine Learning Mastery blog</a>.

Let me know if you have something to ask or discuss in the comments section!

9e7f7082abbfeb51d602de34ad9af7633155cbc2
ipynb
Jupyter Notebook
DataCamp PyTorch blog/Investigating Tensors with PyTorch.ipynb
sayakpaul/DataCamp-blogs
902453e3f89485afb6a444db761e8209051bcd5f
MIT
2019-07-27T04:37:02.000Z
2020-06-11T07:22:01.000Z
DataCamp PyTorch blog/Investigating Tensors with PyTorch.ipynb
sayakpaul/DataCamp-blogs
902453e3f89485afb6a444db761e8209051bcd5f
MIT
DataCamp PyTorch blog/Investigating Tensors with PyTorch.ipynb
sayakpaul/DataCamp-blogs
902453e3f89485afb6a444db761e8209051bcd5f
MIT
2019-07-07T18:48:22.000Z
2019-07-07T18:48:22.000Z
Qwen/Qwen-72B
1. YES
2. YES