{
"cells": [
{
"cell_type": "markdown",
"id": "10d57acd",
"metadata": {},
"source": [
"# Pytorch: Machine Learning library\n",
"\n",
"[Pytorch](https://pytorch.org/) is one of open-source, modern deep learning libraries out there and what we will use in this workshop. Other popular libraries include [Tensorflow](https://www.tensorflow.org/), [Keras](https://keras.io), [MXNet](https://mxnet.apache.org), [Spark ML](https://spark.apache.org/mllib/), etc. ...\n",
"\n",
"All of those libraries works very similar in terms of implementing your neural network architecture. If you are new, probably any of Pytorch/Keras/Tensorflow would work well with lots of guidance/examples/discussion-forums online! Common things you have to learn include:\n",
"\n",
"1. [Data types](#datatype) (typically arbitrary dimension matrix, or _tensor_ )\n",
"2. [Data loading tools](#dataloader) (streamline prepping data into appropraite types from input files)\n",
"3. [Chaining operations](#graph) = a _computation graph_ \n",
"\n",
"In this notebook, we cover the basics part in each of topics above.\n",
"\n",
"\n",
"## 1. Tensor data types in PyTorch\n",
"In `pytorch`, we use `torch.Tensor` object to represent data matrix. It is a lot like `numpy` array but not quite the same. `torch` provide APIs to easily convert data between `numpy` array and `torch.Tensor`. Let's play a little bit."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "b43545df",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from __future__ import print_function\n",
"import numpy as np\n",
"import torch\n",
"SEED=123\n",
"np.random.seed(SEED)\n",
"torch.manual_seed(SEED)"
]
},
{
"cell_type": "markdown",
"id": "9b6d6b6e",
"metadata": {},
"source": [
"... yep, that's how we set pytorch random number seed! (see Python-03-Numpy if you don't know about a seed)\n",
"\n",
"### Creating a torch.Tensor\n",
"\n",
"Pytorch provides constructors similar to numpy (and named same way where possible to avoid users having to look-up function names). Here are some examples."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b2c53d00",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.zeros:\n",
" tensor([[0., 0., 0.],\n",
" [0., 0., 0.]])\n",
"\n",
"torch.ones:\n",
" tensor([[1., 1., 1.],\n",
" [1., 1., 1.]])\n",
"\n",
"torch.arange:\n",
" tensor([[0., 1., 2.],\n",
" [3., 4., 5.]])\n",
"\n",
"torch.randn:\n",
" tensor([[-0.1115, 0.1204, -0.3696],\n",
" [-0.2404, -1.1969, 0.2093]])\n"
]
}
],
"source": [
"# Tensor of 0s = numpy.zeros\n",
"t=torch.zeros(2,3)\n",
"print('torch.zeros:\\n',t)\n",
"\n",
"# Tensor of 1s = numpy.ones\n",
"t=torch.ones(2,3)\n",
"print('\\ntorch.ones:\\n',t)\n",
"\n",
"# Tensor from a sequential integers = numpy.arange\n",
"t=torch.arange(0,6,1).reshape(2,3).float()\n",
"print('\\ntorch.arange:\\n',t)\n",
"\n",
"# Normal distribution centered at 0.0 and sigma=1.0 = numpy.rand.randn\n",
"t=torch.randn(2,3)\n",
"print('\\ntorch.randn:\\n',t)"
]
},
{
"cell_type": "markdown",
"id": "0df698d2",
"metadata": {},
"source": [
"... or you can create from a simple list, tuple, and numpy arrays."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "900903b0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Numpy data\n",
" [[1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
" [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
" [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
" [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
" [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
" [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
" [0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
" [0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
" [0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
" [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]\n",
"\n",
"torch.Tensor data\n",
" tensor([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
" [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
" [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]])\n",
"\n",
"Python list : [1, 2, 3]\n",
"torch.Tensor: tensor([1., 2., 3.])\n"
]
}
],
"source": [
"# Create numpy array\n",
"data_np = np.zeros([10,10],dtype=np.float32)\n",
"# Fill something\n",
"np.fill_diagonal(data_np,1.)\n",
"print('Numpy data\\n',data_np)\n",
"\n",
"# Create torch.Tensor\n",
"data_torch = torch.Tensor(data_np)\n",
"print('\\ntorch.Tensor data\\n',data_torch)\n",
"\n",
"# One can make also from a list\n",
"data_list = [1,2,3]\n",
"data_list_torch = torch.Tensor(data_list)\n",
"print('\\nPython list :',data_list)\n",
"print('torch.Tensor:',data_list_torch)"
]
},
{
"cell_type": "markdown",
"id": "9523602d",
"metadata": {},
"source": [
"Converting back from `torch.Tensor` to a numpy array can be easily done"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "8601560d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Numpy data (converted back from torch.Tensor)\n",
" [[1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
" [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
" [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
" [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
" [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
" [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
" [0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
" [0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
" [0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
" [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]\n"
]
}
],
"source": [
"# Bringing back into numpy array\n",
"data_np = data_torch.numpy()\n",
"print('\\nNumpy data (converted back from torch.Tensor)\\n',data_np)"
]
},
{
"cell_type": "markdown",
"id": "c70565e3",
"metadata": {},
"source": [
"Ordinary operations to an array also exists like `numpy`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "039f532d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mean tensor(0.1000) std tensor(0.3015) sum tensor(10.)\n"
]
}
],
"source": [
"# mean & std\n",
"print('mean',data_torch.mean(),'std',data_torch.std(),'sum',data_torch.sum())"
]
},
{
"cell_type": "markdown",
"id": "b88fc09d",
"metadata": {},
"source": [
"We see the return of those functions (`mean`,`std`,`sum`) are tensor objects. If you would like a single scalar value, you can call `item` function."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "0c58c23a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mean 0.10000000149011612 std 0.30151134729385376 sum 10.0\n"
]
}
],
"source": [
"# mean & std\n",
"print('mean',data_torch.mean().item(),'std',data_torch.std().item(),'sum',data_torch.sum().item())"
]
},
{
"cell_type": "markdown",
"id": "087d126f",
"metadata": {},
"source": [
"### Tensor addition and multiplication\n",
"Common operations include element-wise multiplication, matrix multiplication, and reshaping. Read the [documentation](https://pytorch.org/docs/stable/tensors.html) to find the right function for what you want to do!"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "0dc316bf",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Two numpy matrices\n",
"[[1. 0. 0.]\n",
" [0. 1. 0.]\n",
" [0. 0. 1.]]\n",
"[[1. 1. 1.]\n",
" [0. 0. 0.]\n",
" [0. 0. 0.]] \n",
"\n",
"torch.Tensor element-wise multiplication:\n",
"tensor([[1., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]])\n",
"\n",
"torch.Tensor matrix multiplication:\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1., 1., 1.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]])\n",
"\n",
"torch.Tensor matrix addition:\n",
"tensor([[ 0., -1., -1.],\n",
" [ 0., 1., 0.],\n",
" [ 0., 0., 1.]])\n",
"\n",
"adding a scalar 1:\n",
"tensor([[2., 1., 1.],\n",
" [1., 2., 1.],\n",
" [1., 1., 2.]])\n"
]
}
],
"source": [
"# Two matrices \n",
"data_a = np.zeros([3,3],dtype=np.float32)\n",
"data_b = np.zeros([3,3],dtype=np.float32)\n",
"np.fill_diagonal(data_a,1.)\n",
"data_b[0,:]=1.\n",
"# print them\n",
"print('Two numpy matrices')\n",
"print(data_a)\n",
"print(data_b,'\\n')\n",
"\n",
"# Make torch.Tensor\n",
"torch_a = torch.Tensor(data_a)\n",
"torch_b = torch.Tensor(data_b)\n",
"\n",
"print('torch.Tensor element-wise multiplication:')\n",
"print(torch_a*torch_b)\n",
"\n",
"print('\\ntorch.Tensor matrix multiplication:')\n",
"print(torch_a.matmul(torch_b))\n",
"\n",
"print('\\ntorch.Tensor matrix addition:')\n",
"print(torch_a-torch_b)\n",
"\n",
"print('\\nadding a scalar 1:')\n",
"print(torch_a+1)"
]
},
{
"cell_type": "markdown",
"id": "2f59af61",
"metadata": {},
"source": [
"### Reshaping\n",
"\n",
"You can access the tensor shape via `.shape` attribute like numpy"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "516c5da9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch_a shape: torch.Size([3, 3])\n",
"The 0th dimension size: 3\n"
]
}
],
"source": [
"print('torch_a shape:',torch_a.shape)\n",
"print('The 0th dimension size:',torch_a.shape[0])"
]
},
{
"cell_type": "markdown",
"id": "ff0a62bb",
"metadata": {},
"source": [
"Similarly, there is a `reshape` function"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "e0331b60",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([1, 9])"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"torch_a.reshape(1,9).shape"
]
},
{
"cell_type": "markdown",
"id": "0aa8682e",
"metadata": {},
"source": [
"... and you can also use -1 in the same way you used for numpy"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "aab0d414",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([3, 3])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"torch_a.reshape(-1,3).shape"
]
},
{
"cell_type": "markdown",
"id": "dfc37a52",
"metadata": {},
"source": [
"### Indexing (Slicing)\n",
"\n",
"We can use a similar indexing trick like we tried with a numpy array"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "740fadce",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([1., 0., 0.])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"torch_a[0,:]"
]
},
{
"cell_type": "markdown",
"id": "da37ca14",
"metadata": {},
"source": [
"or a boolean mask generation"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "74e5bf4f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[False, True, True],\n",
" [ True, False, True],\n",
" [ True, True, False]])"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mask = torch_a == 0.\n",
"mask"
]
},
{
"cell_type": "markdown",
"id": "0b00649a",
"metadata": {},
"source": [
"... and slicing with it using `masked_select` function"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "0bba7fe5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([1., 1., 1.])"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"torch_a.masked_select(~mask)"
]
},
{
"cell_type": "markdown",
"id": "36423922",
"metadata": {},
"source": [
"\n",
"## 2. Data loading tools in Pytorch\n",
"\n",
"In Python-02-Python, we covered an iteratable class and how it could be useful to generalize a design of data access tools. Pytorch (and any other ML libraries out there) provides a generalized tool to interface such iteratable data instance called `DataLoader`. Desired capabilities of such tools include ability to choose random vs. ordered subset in data, parallelized workers to simultaneously prepare multiple batch data, etc..\n",
"\n",
"Let's practice the use of `DataLoader`. \n",
"\n",
"First, we define the same iteretable class mentioned in Python-02-Python notebook."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "c7d2e3d0",
"metadata": {},
"outputs": [],
"source": [
"class dataset:\n",
" \n",
" def __init__(self):\n",
" self._data = tuple(range(100))\n",
" \n",
" def __len__(self):\n",
" return len(self._data)\n",
" \n",
" def __getitem__(self,index):\n",
" return self._data[index]\n",
" \n",
"data = dataset()"
]
},
{
"cell_type": "markdown",
"id": "ee04ccfe",
"metadata": {},
"source": [
"Here is how you can instantiate a `DataLoader`. We construct an instance called `loader` that can automatically packs 10 elements of data (`batch_size=10`) that is randomly selected (`shuffle=True`) using 1 parallel worker to prepare such data (`num_workers=1`)."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "465d4285",
"metadata": {},
"outputs": [],
"source": [
"from torch.utils.data import DataLoader\n",
"loader = DataLoader(data,batch_size=10,shuffle=True,num_workers=1)"
]
},
{
"cell_type": "markdown",
"id": "c1bc9244",
"metadata": {},
"source": [
"The dataloader itself is an iterable object. We created a dataloader with batch size 10 where the dataset instance has the length 100. This means, if we iterate on the dataloader instance, we get 10 separate batch data."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "e84c9be2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Batch entry 0 ... batch data tensor([23, 14, 64, 51, 94, 25, 38, 44, 70, 28])\n",
"Batch entry 1 ... batch data tensor([37, 57, 66, 43, 53, 13, 72, 48, 74, 62])\n",
"Batch entry 2 ... batch data tensor([89, 3, 40, 92, 86, 65, 63, 95, 21, 97])\n",
"Batch entry 3 ... batch data tensor([ 9, 42, 45, 54, 31, 87, 99, 46, 98, 26])\n",
"Batch entry 4 ... batch data tensor([41, 80, 36, 90, 0, 59, 52, 69, 17, 56])\n",
"Batch entry 5 ... batch data tensor([16, 61, 82, 30, 77, 73, 96, 33, 6, 83])\n",
"Batch entry 6 ... batch data tensor([39, 5, 24, 32, 85, 35, 50, 60, 1, 78])\n",
"Batch entry 7 ... batch data tensor([18, 2, 71, 7, 34, 20, 49, 10, 8, 84])\n",
"Batch entry 8 ... batch data tensor([76, 93, 12, 81, 22, 55, 4, 19, 11, 27])\n",
"Batch entry 9 ... batch data tensor([29, 15, 47, 88, 75, 68, 67, 58, 79, 91])\n"
]
}
],
"source": [
"for index, batch_data in enumerate(loader):\n",
" print('Batch entry',index,'... batch data',batch_data)"
]
},
{
"cell_type": "markdown",
"id": "eb476dbc",
"metadata": {},
"source": [
"We can see that data elements are chosen randomly as we chose \"shuffle=True\". Does this cover all data elements in the dataset? Let's check this by combining all iterated data."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "d5f157f8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,\n",
" 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
" 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\n",
" 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,\n",
" 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,\n",
" 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99])"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_collection = []\n",
"for index,batch_data in enumerate(loader):\n",
" data_collection += [int(v) for v in batch_data]\n",
" \n",
"import numpy as np\n",
"np.unique(data_collection)"
]
},
{
"cell_type": "markdown",
"id": "1c8c7d79",
"metadata": {},
"source": [
"This covers the minimal concept of `DataLoader` you need to know in order to follow the workshop. You can read more about `DataLoader` in pytorch documentation [here](https://pytorch.org/docs/stable/data.html) and also more extended example in [their tutorial](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html) if you are interested in exploring yourself.\n",
"\n",
"\n",
"## 3. Computation graph\n",
"\n",
"The last point to cover is how to chain modularized mathematical operations. \n",
"\n",
"To get started, let's introduce a few, well used mathematical operations in pytorch.\n",
"\n",
"* `torch.nn.ReLU` ([link](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html#torch.nn.ReLU)) ... a function that takes an input tenor and outputs a tensor of the same shape where elements are 0 if the corresponding input element has a value below 0, and otherwise the same value.\n",
"* `torch.nn.Softmax` ([link](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html#torch.nn.Softmax)) ... a function that applies a [softmax function](https://en.wikipedia.org/wiki/Softmax_function) on the specified dimension of an input data.\n",
"* `torch.nn.MaxPool2d` ([link](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html#torch.nn.MaxPool2d)) ... a function that down-sample the input matrix by taking maximum value from sub-matrices of a specified shape.\n",
"\n",
"Let's see what each of these functions do first using a simple 2D matrix data."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "71015d84",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[ 1.5810, 1.3010, 1.2753, -0.2010, -0.1606],\n",
" [-0.4015, 0.6957, -1.8061, -1.1589, -0.4210],\n",
" [-0.9620, 1.2825, 0.8768, 1.6221, -1.4779],\n",
" [ 1.1331, -1.2203, -1.1285, 0.4135, 0.2892],\n",
" [ 2.2473, -0.8036, -0.2808, 0.7697, -0.6596]]])"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Create a 2D tensor of shape (1,5,5) with some negative and positive values\n",
"data = torch.randn(25).reshape(1,5,5)\n",
"data"
]
},
{
"cell_type": "markdown",
"id": "f5bce9ec",
"metadata": {},
"source": [
"Here's how `ReLU` works"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "981fba38",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[1.5810, 1.3010, 1.2753, 0.0000, 0.0000],\n",
" [0.0000, 0.6957, 0.0000, 0.0000, 0.0000],\n",
" [0.0000, 1.2825, 0.8768, 1.6221, 0.0000],\n",
" [1.1331, 0.0000, 0.0000, 0.4135, 0.2892],\n",
" [2.2473, 0.0000, 0.0000, 0.7697, 0.0000]]])"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"op0 = torch.nn.ReLU()\n",
"op0(data)"
]
},
{
"cell_type": "markdown",
"id": "2ece3a39",
"metadata": {},
"source": [
"Here's how `Softmax` works"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "f9235f14",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[0.3526, 0.2665, 0.2597, 0.0593, 0.0618],\n",
" [0.1757, 0.5264, 0.0431, 0.0824, 0.1723],\n",
" [0.0327, 0.3086, 0.2057, 0.4334, 0.0195],\n",
" [0.4725, 0.0449, 0.0492, 0.2301, 0.2032],\n",
" [0.7093, 0.0336, 0.0566, 0.1618, 0.0388]]])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"op1 = torch.nn.Softmax(dim=2)\n",
"op1(data)"
]
},
{
"cell_type": "markdown",
"id": "852172d6",
"metadata": {},
"source": [
"Here's how `MaxPool2d` works with a kernel shape (5,1)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "4883bf99",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)\n",
" return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n"
]
},
{
"data": {
"text/plain": [
"tensor([[[1.5810],\n",
" [0.6957],\n",
" [1.6221],\n",
" [1.1331],\n",
" [2.2473]]])"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"op2 = torch.nn.MaxPool2d(kernel_size=(1,5))\n",
"op2(data)"
]
},
{
"cell_type": "markdown",
"id": "3d69e28c",
"metadata": {},
"source": [
"So if we want to define a computation graph that applies these operations in a sequential order, we could try:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "1c44658e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[0.3444],\n",
" [0.3339],\n",
" [0.3874],\n",
" [0.3905],\n",
" [0.6472]]])"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"op2(op1(op0(data)))"
]
},
{
"cell_type": "markdown",
"id": "409a9e05",
"metadata": {},
"source": [
"Pytorch provides tools called _containers_ to make this easy. Let's try `torch.nn.Sequential` (see different type of containers [here](https://pytorch.org/docs/stable/nn.html#containers))."
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "2fb58c95",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[0.3444],\n",
" [0.3339],\n",
" [0.3874],\n",
" [0.3905],\n",
" [0.6472]]])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myop = torch.nn.Sequential(op0,op1,op2)\n",
"myop(data)"
]
},
{
"cell_type": "markdown",
"id": "c6c0c93d",
"metadata": {},
"source": [
"We might wonder \"Can I add a custom operation to this graph?\" Yes, we can add any _module_ that inherits from `torch.nn.Module` class. Let's define one for ourself."
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "d78f6278",
"metadata": {},
"outputs": [],
"source": [
"class AddOne(torch.nn.Module):\n",
"\n",
" # always call the base class constructor for defining your torch.nn.Module inherit class!\n",
" def __init__(self):\n",
" super().__init__()\n",
" \n",
" # forward needs to be defined. This is called by \"()\" function call.\n",
" def forward(self,input):\n",
" \n",
" return input + 1;"
]
},
{
"cell_type": "markdown",
"id": "dd067c9e",
"metadata": {},
"source": [
"Now let's add our operation"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "82f05305",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[1.3444],\n",
" [1.3339],\n",
" [1.3874],\n",
" [1.3905],\n",
" [1.6472]]])"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myop = torch.nn.Sequential(op0,op1,op2,AddOne())\n",
"myop(data)"
]
},
{
"cell_type": "markdown",
"id": "35483f45",
"metadata": {},
"source": [
"Of course, you can also embed `op0`, `op1`, and `op2` inside one module."
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "4abd813c",
"metadata": {},
"outputs": [],
"source": [
"class MyOp(torch.nn.Module):\n",
" \n",
" def __init__(self):\n",
" super().__init__()\n",
" self._sequence = torch.nn.Sequential(torch.nn.ReLU(), \n",
" torch.nn.Softmax(dim=2), \n",
" torch.nn.MaxPool2d(kernel_size=(1,5)),\n",
" AddOne(),\n",
" )\n",
" \n",
" def forward(self,input):\n",
" \n",
" return self._sequence(input)"
]
},
{
"cell_type": "markdown",
"id": "af2c4ef5",
"metadata": {},
"source": [
"Let's try using it."
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "e738dc25",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[1.3444],\n",
" [1.3339],\n",
" [1.3874],\n",
" [1.3905],\n",
" [1.6472]]])"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myop = MyOp()\n",
"myop(data)"
]
},
{
"cell_type": "markdown",
"id": "beee3891",
"metadata": {},
"source": [
"## Extra: GPU acceleration\n",
"\n",
"**This section only works if you run this notebook on a GPU-enabled machine (not on the binder unfortunately)**\n",
"\n",
"Putting `torch.Tensor` on GPU is as easy as calling `.cuda()` function (and if you want to bring it back to cpu, call `.cpu()` on a `cuda.Tensor`). Let's do a simple speed comparison. \n",
"\n",
"Create two arrays with an identical data type, shape, and values."
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "a571c4ab",
"metadata": {},
"outputs": [],
"source": [
"# Create 1000x1000 matrix\n",
"data_np=np.zeros([1000,1000],dtype=np.float32)\n",
"data_cpu = torch.Tensor(data_np).cpu()\n",
"#data_gpu = torch.Tensor(data_np).cuda()"
]
},
{
"cell_type": "markdown",
"id": "88baa5af",
"metadata": {},
"source": [
"Time fifth power of the matrix on CPU"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "fa3f7d0c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6.09 ms ± 195 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
]
}
],
"source": [
"%%timeit\n",
"mean = (data_cpu ** 5).mean().item()"
]
},
{
"cell_type": "markdown",
"id": "e184d4fe",
"metadata": {},
"source": [
"... and next on GPU"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "25033136",
"metadata": {},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'data_gpu' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_46365/2823993350.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'timeit'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'mean = (data_gpu ** 5).mean().item()\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m 2401\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2402\u001b[0m \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2403\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2404\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2405\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 189\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtimeit\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m 1167\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mindex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1168\u001b[0m \u001b[0mnumber\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m10\u001b[0m \u001b[0;34m**\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1169\u001b[0;31m \u001b[0mtime_number\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtimer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimeit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnumber\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1170\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtime_number\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m0.2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1171\u001b[0m \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtimeit\u001b[0;34m(self, number)\u001b[0m\n\u001b[1;32m 167\u001b[0m \u001b[0mgc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 168\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 169\u001b[0;31m \u001b[0mtiming\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mit\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 170\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 171\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mgcold\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m\u001b[0m in \u001b[0;36minner\u001b[0;34m(_it, _timer)\u001b[0m\n",
"\u001b[0;31mNameError\u001b[0m: name 'data_gpu' is not defined"
]
}
],
"source": [
"%%timeit\n",
"mean = (data_gpu ** 5).mean().item()"
]
},
{
"cell_type": "markdown",
"id": "02ec6e84",
"metadata": {},
"source": [
"... which is more than x10 faster than the cpu counter part :)\n",
"\n",
"But there's a catch you should be aware! Preparing a data on GPU does take time because data needs to be sent to GPU, which could take some time. Let's compare the time it takes to create a tensor on CPU v.s. GPU."
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "1cd61522",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"165 µs ± 1.02 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
]
}
],
"source": [
"%%timeit\n",
"data_np=np.zeros([1000,1000],dtype=np.float32)\n",
"data_cpu = torch.Tensor(data_np).cpu()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "9b9b6cb0",
"metadata": {},
"outputs": [
{
"ename": "ValueError",
"evalue": "empty body on For",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_46365/3126707622.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'timeit'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'#data_np=np.zeros([1000,1000],dtype=np.float32)\\n#data_gpu = torch.Tensor(data_np).cuda()\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m 2401\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2402\u001b[0m \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2403\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2404\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2405\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 189\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.8/dist-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtimeit\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m 1144\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1145\u001b[0m \u001b[0mt0\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1146\u001b[0;31m \u001b[0mcode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshell\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeit_ast\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"exec\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1147\u001b[0m \u001b[0mtc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mt0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1148\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/usr/lib/python3.8/codeop.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, source, filename, symbol)\u001b[0m\n\u001b[1;32m 141\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 142\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msource\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msymbol\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 143\u001b[0;31m \u001b[0mcodeob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msource\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msymbol\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflags\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 144\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mfeature\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_features\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 145\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcodeob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mco_flags\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0mfeature\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompiler_flag\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mValueError\u001b[0m: empty body on For"
]
}
],
"source": [
"%%timeit\n",
"#data_np=np.zeros([1000,1000],dtype=np.float32)\n",
"#data_gpu = torch.Tensor(data_np).cuda()"
]
},
{
"cell_type": "markdown",
"id": "e6fcef62",
"metadata": {},
"source": [
"As you can see, it takes nearly 10 times longer time to create this particular data tensor on our GPU. This speed depends on many factors including your hardware configuration (e.g. CPU-GPU communication via PCI-e or NVLINK). It makes sense to move computation that takes longer than this data transfer time to perform on GPU."
]
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "-all",
"formats": "md:myst",
"text_representation": {
"extension": ".md",
"format_name": "myst",
"format_version": 0.13,
"jupytext_version": "1.10.3"
}
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"source_map": [
14,
32,
39,
46,
62,
65,
81,
84,
88,
91,
94,
97,
100,
104,
130,
135,
138,
141,
143,
146,
148,
153,
155,
157,
160,
162,
164,
174,
187,
190,
193,
196,
199,
202,
209,
226,
230,
233,
236,
239,
242,
245,
248,
251,
253,
256,
259,
262,
273,
276,
279,
282,
296,
299,
302,
311,
316,
319,
322,
324,
327,
332,
338,
342
]
},
"nbformat": 4,
"nbformat_minor": 5
}