tutorials/intro_to_differentiable_programming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction To Basic API Operations for DiffKt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Copyright (c) Meta Platforms, Inc. and affiliates.**\n",
    "\n",
    "This source code is licensed under the MIT license found in the\n",
    "LICENSE file in the root directory of this source tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "Welcome to the Introduction To Basic API Operations for diffkt, (pronounced diff kit).\n",
    "\n",
    "Differentiable programming is a process of computing derivative over functions automatically. These functions can operate on floating point values, tensors, and user-defined data structures containing them. This tutorial will show you how to use the **DiffKt** API to create programs for functions with derivatives so that you can incorporate them into numerical algorithms for scientific computing, optimization, machine learning, and statistics.\n",
    "\n",
    "## Background on Differentiable Programming\n",
    "\n",
    "Below are some review papers and a book on differentiable programming. The field is also called automatic differentiation or algorithmic differentiation.\n",
    "\n",
    "__[A Review of Automatic Differentiation and its Efficient Implementation, (2019)](https://arxiv.org/pdf/1811.05031.pdf)__\n",
    "\n",
    "__[Automatic Differentiation in Machine Learning: A Survey, (2018)](https://www.jmlr.org/papers/volume18/17-468/17-468.pdf)__\n",
    "\n",
    "__[Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd Ed., (2008)](https://my.siam.org/Store/Product/viewproduct/?ProductId=1005)__\n",
    "\n",
    "### Some Housekeeping\n",
    "This notebook uses `api.jar` from the **DiffKt** project.<br>\n",
    "\n",
    "`@file:DependsOn(\"...\")` tells the Kotlin Jupyter notebook the path to a jar that it needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "@file:DependsOn(\"../kotlin/api/build/libs/api.jar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The notebook uses the following imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import org.diffkt.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Tensors\n",
    "\n",
    "In **DiffKt** there are many different types of differentiable tensors. Tensor means a multi-dimensional array. A float scalar is  a 0D tensor. A vector is a 1D tensor. A 2D array is a 2D tensor. A 3D array is a 3D tensor, and so on.\n",
    "\n",
    "__[DTensor](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/)__ is the interface for all differentiable tensors in **DiffKt**. A differentiable tensor can be a scalar, a 1D tensor, a 2D tensor, a 3D tensor, or have even more dimensions. Scalars also inherit from __[DTensor]( http://www.diffkt.org/api/api/org.diffkt/-d-tensor/)__. A tensor has a number of properties, functions, or extensions defined in the interface. Properties we will discuss about __[DTensor]( http://www.diffkt.org/api/api/org.diffkt/-d-tensor)__ are size, rank, shape, isScalar, and indexing.\n",
    "\n",
    "A tensor has a __[size](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/size.html)__, which is the number of elements in the tensor,\n",
    "\n",
    "A tensor has a __[rank](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/rank.html)__, which indicates the number of dimensions: rank 0 - scalar, rank 1 - 1D tensor, rank 2 - 2D tensor, rank 3 - 3D tensor, and so on.\n",
    "\n",
    "A tensor has a __[shape](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/shape.html)__, which indicates the number of axes and the length of each axis of the tensor.\n",
    "\n",
    "A tensor has an boolean property to see if it is a scalar, __[isScalar](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/is-scalar.html)__.\n",
    "\n",
    "Retrieve an element of a tensor use indexing, with the indices indicating the location of the element, such as [0,0] to get the first element of the 2D array.\n",
    "\n",
    "__[FloatTensor](http://www.diffkt.org/api/api/org.diffkt/-float-tensor/index.html)__ is an an abstract class for the implementation of __[DTensor](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/index.html)__ for floating point numbers. There are multiple types of implementations such as scalar, dense, and sparse tensors.\n",
    "\n",
    "__[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ is the interface for all differentiable scalars.\n",
    "\n",
    "__[FloatScalar](http://www.diffkt.org/api/api/org.diffkt/-float-scalar/index.html)__ is an implementation of the interfaces __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ and  __[FloatTensor](http://www.diffkt.org/api/api/org.diffkt/-float-tensor/index.html)__.\n",
    "\n",
    "__[tensorOf](http://www.diffkt.org/api/api/org.diffkt/tensor-of.html)__ is a factory function that creates a FloatTensor from a set of float numbers. The initial tensor is a 1D array. After creating a tensor with __[tensorOf](http://www.diffkt.org/api/api/org.diffkt/tensor-of.html)__, you may need to __[reshape](http://www.diffkt.org/api/api/org.diffkt/reshape.html)__ the tensor to the shape you want.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Scalar Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "fs.size = 1\n",
      "fs.rank = 0\n",
      "fs.shape = Shape()\n",
      "fs.isScalar = true\n"
     ]
    }
   ],
   "source": [
    "// Scalar Example\n",
    "\n",
    "val constant = 1f\n",
    "val fs = FloatScalar(constant)\n",
    "\n",
    "println(fs.toString())\n",
    "println(\"fs.size = ${fs.size}\")\n",
    "println(\"fs.rank = ${fs.rank}\")\n",
    "println(\"fs.shape = ${fs.shape}\")\n",
    "println(\"fs.isScalar = ${fs.isScalar}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be:\n",
    "\n",
    "`1.0`<br>\n",
    "`fs.size = 1`<br>\n",
    "`fs.rank = 0`<br>\n",
    "`fs.shape = Shape()`<br>\n",
    "`fs.isScalar = true`<br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The value is 1.0.<br>\n",
    "The tensor has only one element, so the `size` is one.<br>\n",
    "This is a scalar, so the `rank` is zero.<br>\n",
    "For a scalar, there are no arguments to `Shape()` indicating the length of the dimensions because the rank is 0. <br>\n",
    "`isScalar` is true because it is a scalar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 1D Tensor Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0, 2.0, 3.0]\n",
      "ft1.size = 3\n",
      "ft1.rank = 1\n",
      "ft1.shape = Shape(3)\n",
      "ft1.isScalar = false\n",
      "\n",
      "ft1[0] = 1.0\n",
      "ft1[1] = 2.0\n",
      "ft1[2] = 3.0\n"
     ]
    }
   ],
   "source": [
    "// 1D Tensor Example\n",
    "\n",
    "val ft1 = tensorOf(1f, 2f, 3f)\n",
    "\n",
    "println(ft1)\n",
    "println(\"ft1.size = ${ft1.size}\")\n",
    "println(\"ft1.rank = ${ft1.rank}\")\n",
    "println(\"ft1.shape = ${ft1.shape}\")\n",
    "println(\"ft1.isScalar = ${ft1.isScalar}\")\n",
    "println()\n",
    "// print the contents of the tensor\n",
    "println(\"ft1[0] = ${ft1[0]}\")\n",
    "println(\"ft1[1] = ${ft1[1]}\")\n",
    "println(\"ft1[2] = ${ft1[2]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be:\n",
    "\n",
    "`[1.0, 2.0, 3.0]`<br>\n",
    "`ft1.size = 3`<br>\n",
    "`ft1.rank = 1`<br>\n",
    "`ft1.shape = Shape(3)`<br>\n",
    "`ft1.isScalar = false`<br>\n",
    "\n",
    "`ft1[0] = 1.0`<br>\n",
    "`ft1[1] = 2.0`<br>\n",
    "`ft1[2] = 3.0`<br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The tensor is a 1D tensor with three elements.<br>\n",
    "The tensor has three elements so the `size` is 3.<br>\n",
    "This is a 1D tensor, so the `rank` is 1.<br>\n",
    "The shape is `Shape(3)`, which means 1 dimension with a length of 3.<br>\n",
    "`isScalar` is false because it is not a scalar.<br>\n",
    "The indexing gets the values at the locations indicated by the index.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 2D Tensor Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.0, 2.0], [3.0, 4.0]]\n",
      "ft2.size = 4\n",
      "ft2.rank = 2\n",
      "ft2.shape = Shape(2, 2)\n",
      "ft2.isScalar = false\n",
      "\n",
      "ft2[0.0] = 1.0\n",
      "ft2[0,1] = 2.0\n",
      "ft2[1,0] = 3.0\n",
      "ft2[1,1] = 4.0\n"
     ]
    }
   ],
   "source": [
    "// 2D Tensor Example\n",
    "\n",
    "var ft2 = tensorOf(1f, 2f, 3f, 4f).reshape(2,2)\n",
    "\n",
    "println(ft2)\n",
    "println(\"ft2.size = ${ft2.size}\")\n",
    "println(\"ft2.rank = ${ft2.rank}\")\n",
    "println(\"ft2.shape = ${ft2.shape}\")\n",
    "println(\"ft2.isScalar = ${ft2.isScalar}\")\n",
    "println()\n",
    "// print the contents of the tensor\n",
    "println(\"ft2[0.0] = ${ft2[0,0]}\")\n",
    "println(\"ft2[0,1] = ${ft2[0,1]}\")\n",
    "println(\"ft2[1,0] = ${ft2[1,0]}\")\n",
    "println(\"ft2[1,1] = ${ft2[1,1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be:\n",
    "\n",
    "`[[1.0, 2.0], [3.0, 4.0]]` <br>\n",
    "`ft2.size = 4` <br>\n",
    "`ft2.rank = 2` <br>\n",
    "`ft2.shape = Shape(2, 2)` <br>\n",
    "`ft2.isScalar = false` <br>\n",
    "\n",
    "`ft2[0,0] = 1.0`<br>\n",
    "`ft2[0,1] = 2.0`<br>\n",
    "`ft2[1,0] = 3.0`<br>\n",
    "`ft2[1,1] = 4.0`<br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The tensor is created as a tensor with 4 elements and then is reshaped into a 2x2 tensor.<br>\n",
    "The tensor has 4 elements, so the `size` is 4. <br>\n",
    "The tensor is a 2D tensor, so the `rank` is 2.<br>\n",
    "The `Shape(2,2)` indicates the tensor is 2 dimensional with the length of each axis equal to 2. <br>\n",
    "`isScalar` is false since this is a 3D tensor.<br>\n",
    "The multi-dimensional indexing gets the value of a tensor element at the location specified with a 2D index.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 3D Tensor Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]\n",
      "ft3.size = 12\n",
      "ft3.rank = 3\n",
      "ft3.shape = Shape(2, 2, 3)\n",
      "ft3.isScalar = false\n",
      "\n",
      "ft3[0,0,0] = 1.0\n",
      "ft3[0,0,1] = 2.0\n",
      "ft3[0,0,2] = 3.0\n",
      "ft3[0,1,0] = 4.0\n",
      "ft3[0,1,1] = 5.0\n",
      "ft3[0,1,2] = 6.0\n",
      "ft3[1,0,0] = 7.0\n",
      "ft3[1,0,1] = 8.0\n",
      "ft3[1,0,2] = 9.0\n",
      "ft3[1,1,0] = 10.0\n",
      "ft3[1,1,1] = 11.0\n",
      "ft3[1,1,2] = 12.0\n"
     ]
    }
   ],
   "source": [
    "// 3D Tensor Example\n",
    "\n",
    "var ft3 = tensorOf(1f, 2f, 3f, 4f, 5f, 6f, 7f, 8f, 9f, 10f, 11f, 12f).reshape(2,2,3)\n",
    "\n",
    "println(ft3)\n",
    "println(\"ft3.size = ${ft3.size}\")\n",
    "println(\"ft3.rank = ${ft3.rank}\")\n",
    "println(\"ft3.shape = ${ft3.shape}\")\n",
    "println(\"ft3.isScalar = ${ft3.isScalar}\")\n",
    "\n",
    "// print the contents of the tensor\n",
    "println()\n",
    "\n",
    "for (i in 0..1) {\n",
    "    for (j in 0..1) {\n",
    "        for (k in 0..2) {\n",
    "            println(\"ft3[${i},${j},${k}] = ${ft3[i,j,k]}\")\n",
    "        }\n",
    "    }\n",
    "} \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be\n",
    "\n",
    "`[[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]` <br>\n",
    "`ft3.size = 12`<br>\n",
    "`ft3.rank = 3`<br>\n",
    "`ft3.shape = Shape(2, 2, 3)`<br>\n",
    "`ft3.isScalar = false`<br>\n",
    "\n",
    "`ft3[0,0,0] = 1.0`<br>\n",
    "`ft3[0,0,1] = 2.0`<br>\n",
    "`ft3[0,0,2] = 3.0`<br>\n",
    "`ft3[0,1,0] = 4.0`<br>\n",
    "`ft3[0,1,1] = 5.0`<br>\n",
    "`ft3[0,1,2] = 6.0`<br>\n",
    "`ft3[1,0,0] = 7.0`<br>\n",
    "`ft3[1,0,1] = 8.0`<br>\n",
    "`ft3[1,0,2] = 9.0`<br>\n",
    "`ft3[1,1,0] = 10.0`<br>\n",
    "`ft3[1,1,1] = 11.0`<br>\n",
    "`ft3[1,1,2] = 12.0`<br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The tensor was created with 12 elements, so the `size` is 12.<br>\n",
    "The tensor is a 3D tensor, so the `rank` is 3.<br>\n",
    "The `Shape(2,2,3)` indicates the tensor has 3 dimensions. The length of the axes are 2, 2, and 3.<br>\n",
    "`isScalar` is false since this is a 3D tensor.<br>\n",
    "The multi-dimensional indexing gets the value of a tensor element at the location specified with a 3D index.<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Scalar Operations\n",
    "\n",
    "The __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ interface has many operations that can be applied to a differentiable scalar. Click on the **Extensions** tab in the Kotlin docs of __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ to see all the operations available. Note, some operations but not all, will allow you to use traditional arithmatic notation, which is an example of operator overloading. We will look at a few of the operations:<br>\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__,<br>\n",
    "__['-'](http://www.diffkt.org/api/api/org.diffkt/minus.html)__ or __[minus](http://www.diffkt.org/api/api/org.diffkt/minus.html)__,<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__,<br>\n",
    "__['/'](http://www.diffkt.org/api/api/org.diffkt/div.html)__ or __[div](http://www.diffkt.org/api/api/org.diffkt/div.html)__, and<br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Simple Polynomial over a Scalar\n",
    "We will create a simple polynomial over $x$, a diffentiable scalar variable. Constants $a$ and $b$ are `Float`. Constant $c$ is a __[FloatScalar](http://www.diffkt.org/api/api/org.diffkt/-float-scalar/index.html)__. You can mix both `Float` types and __[FloatScalar](http://www.diffkt.org/api/api/org.diffkt/-float-scalar/index.html)__ in the arithmetic operations. The equation is $ f(x) = (a + bx^2 - cx^3) / c$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = 3.0\n",
      "f(x) = -20.666666\n"
     ]
    }
   ],
   "source": [
    "// scalar polynomial\n",
    "\n",
    "val a = 1f\n",
    "val b = 2f\n",
    "val c = FloatScalar(3f)\n",
    "\n",
    "fun f(x : DScalar) : DScalar {\n",
    "    val y = (a + b * x.pow(2f) - c * x.pow(3f)) / c\n",
    "    return y\n",
    "}\n",
    "\n",
    "// Evaluation of the scalar polynomial\n",
    "\n",
    "val x = FloatScalar(3f)\n",
    "val f = f(x)\n",
    "\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be:\n",
    "\n",
    "`x = 3.0`<br>\n",
    "`f(x) = -20.666666`\n",
    "\n",
    "#### Observations\n",
    "\n",
    "Both `Float` and __[FloatScalar](http://www.diffkt.org/api/api/org.diffkt/-float-scalar/index.html)__  were mixed in the arithmatic operation.\n",
    "\n",
    "The following scalar operations were used:\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__,<br>\n",
    "__['-'](http://www.diffkt.org/api/api/org.diffkt/minus.html)__ or __[minus](http://www.diffkt.org/api/api/org.diffkt/minus.html)__,<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__,<br>\n",
    "__['/'](http://www.diffkt.org/api/api/org.diffkt/div.html)__ or __[div](http://www.diffkt.org/api/api/org.diffkt/div.html)__, and<br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Calculating the Derivative of a Scalar Function\n",
    "\n",
    "There are two different algorithms for calculating the derivative of a function over a __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ variable, the forward derivative algorithm and the reverse derivative algorithm. The forward derivative algorithm is more efficient for when a function has more output variables than input variables. The reverse derivative algorithm is more efficient for a function that has more input variables that output variables. For most situations of optimizing a scalar function, where the output of the function is a single variable, the reverse derivative algorithm is more efficient.\n",
    "\n",
    "In calling the below functions, one passes a scalar variable, a __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__, to be differentiated and a lambda of the function of the variable. In Kotlin, if you declare the function `fun f(x)` then the lambda is `::f`.\n",
    "\n",
    "__[forwardDerivative](http://www.diffkt.org/api/api/org.diffkt/forward-derivative.html)__ calculates the derivative of a function over a __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ evaluated at the __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ `x` using the forward derivative algorithm.\n",
    "\n",
    "__[reverseDerivative](http://www.diffkt.org/api/api/org.diffkt/reverse-derivative.html)__ calculates the derivative of a function over a __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ evaluated at the __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ `x` using the reverse derivative algorithm.\n",
    "\n",
    "In many cases it is more efficient to calculate the orignal scalar function and its derivative at the same time. In the below functions, they return a `Pair<DTensor, DTensor>` where the first value is called the `primal`, which is the value of a function evaluated at `x`, where `x` is a tensor, and the second value is called the `tangent`, which is the derivative of a function evaluated at `x`, where `x` is a tensor.\n",
    "\n",
    "__[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__ calculates a function over __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ and its derivative evaluated at the __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ `x` using the forward derivative algorithm.\n",
    "\n",
    "__[primalAndReverseDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-reverse-derivative.html)__ calculates a function over a __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ and its derivative evaluated at the __[DScalar](http://www.diffkt.org/api/api/org.diffkt/-d-scalar/index.html)__ `x` using the reverse derivative algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Derivative of a Simple Polynomial Function over a Scalar\n",
    "\n",
    "We will calculate the derivative a simple polynomial over $x$, a diffentiable scalar variable. The variables $a$ and $b$ are constants. The variable $c$ is a __[FloatScalar](http://www.diffkt.org/api/api/org.diffkt/-float-scalar/index.html)__. The equation is $ f(x) = (a + bx^2 -cx^3) / c$, and the derivative is $ \\frac {df(x)}{dx} = \\frac{2b}{c}x - 3x^2 $.\n",
    "\n",
    "The derivative will be computed with the __[forwardDerivative](http://www.diffkt.org/api/api/org.diffkt/forward-derivative.html)__ function. Since the function $ f(x) $ returns a scalar, the derivative,$ \\frac {df(x)}{dx} $ is a scalar.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = 3.0\n",
      "f(x) = -20.666666\n",
      "df(x)/dx = -23.0\n"
     ]
    }
   ],
   "source": [
    "// scalar polynomial\n",
    "\n",
    "val a = 1f\n",
    "val b = 2f\n",
    "val c = FloatScalar(3f)\n",
    "\n",
    "fun f(x : DScalar) : DScalar {\n",
    "       \n",
    "    val y = (a + b * x.pow(2f) - c * x.pow(3f)) / c\n",
    "    return y\n",
    "}\n",
    "\n",
    "// evaluation of the scalar polynomial and its derivative\n",
    "\n",
    "val x = FloatScalar(3f)\n",
    "val f = f(x)\n",
    "val df = forwardDerivative(x, ::f)\n",
    "\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${f}\")\n",
    "println(\"df(x)/dx = ${df}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be\n",
    "\n",
    "`x = 3.0`<br>\n",
    "`f(x) = -20.666666`<br>\n",
    "`df(x)/dx = -23.0`<br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "Both `Float` and __[FloatScalar](http://www.diffkt.org/api/api/org.diffkt/-float-scalar/index.html)__ were mixed in the arithmatic operation.\n",
    "\n",
    "The following scalar operations were used:\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__,<br>\n",
    "__['-'](http://www.diffkt.org/api/api/org.diffkt/minus.html)__ or __[minus](http://www.diffkt.org/api/api/org.diffkt/minus.html)__,<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__,<br>\n",
    "__['/'](http://www.diffkt.org/api/api/org.diffkt/div.html)__ or __[div](http://www.diffkt.org/api/api/org.diffkt/div.html)__, and<br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__.\n",
    "\n",
    "The following function was used to calculate the derivative:\n",
    "\n",
    "__[forwardDerivative](http://www.diffkt.org/api/api/org.diffkt/forward-derivative.html)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Using the `primalAndForwardDerivative` Function\n",
    "\n",
    "We will calculate the derivative of a simple polynomial over $x$, a diffentiable scalar variable. The variables $a$ and $b$ are constants. The equation is $ f(x) = a + bx^2 $, and the derivative is $ \\frac{df(x)}{dx} = 2bx $.\n",
    "\n",
    "We will use the __[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__  function to calculate both the value of the function and its derivative at the same time. \n",
    "\n",
    "Both $f(x)$ and $ \\frac{df(x)}{dx}$ are scalars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = 3.0\n",
      "f(x) = 19.0\n",
      "df(x)/dx = 12.0\n"
     ]
    }
   ],
   "source": [
    "// scalar polynomial\n",
    "\n",
    "val a = 1f\n",
    "val b = 2f\n",
    "\n",
    "fun f(x : DScalar) : DScalar {\n",
    "        \n",
    "    return a + b * x.pow(2f)\n",
    "}\n",
    "\n",
    "// evaluation of scalar polynomial\n",
    "\n",
    "val x = FloatScalar(3f)\n",
    "val (fx, df) = primalAndForwardDerivative(x, ::f)\n",
    "\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${fx}\")\n",
    "println(\"df(x)/dx = ${df}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be\n",
    "`x = 3.0`<br>\n",
    "`f(x) = 19.0`<br>\n",
    "`df(x)/dx = 12.0`<br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The following scalar operations were used:\n",
    "\n",
    "__[+](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__<br>\n",
    "__[*](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__<br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__<br>\n",
    "\n",
    "The following function was used to calculate both the function and the derivative:\n",
    "\n",
    "__[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Tensor  Operations\n",
    "\n",
    "\n",
    "The __[DTensor](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/index.html)__ interface has many operations that can be applied to a tensor. Click on the **Extentions** tab in the Kotlin docs of __[DTensor](http://www.diffkt.org/api/api/org.diffkt/-d-tensor/index.html)__ to see all the operations. Some of the operations allow the use of traditional arithmatic notation, or operator overloading. We will look at a few of the operations in the below examples:<br>\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__,<br>\n",
    "__['-'](http://www.diffkt.org/api/api/org.diffkt/minus.html)__ or __[minus](http://www.diffkt.org/api/api/org.diffkt/minus.html)__,<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__,<br>\n",
    "__['/'](http://www.diffkt.org/api/api/org.diffkt/div.html)__ or __[div](http://www.diffkt.org/api/api/org.diffkt/div.html)__, <br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__,<br>\n",
    "__[sin](http://www.diffkt.org/api/api/org.diffkt/sin.html)__,<br>\n",
    "__[cos](http://www.diffkt.org/api/api/org.diffkt/cos.html)__,<br>\n",
    "__[matmul](http://www.diffkt.org/api/api/org.diffkt/matmul.html)__,<br>\n",
    "__[sum](http://www.diffkt.org/api/api/org.diffkt/sum.html)__ and,<br>\n",
    "__[innerProduct](http://www.diffkt.org/api/api/org.diffkt/inner-product.html)__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Derivatives of a Function over a Tensor\n",
    "\n",
    "The symbol nabla, $\\nabla$, is an inverted greek symbol $\\Delta$. The gradient of a function over a vector of variables is $\\nabla f(\\mathbf x)$, and is the partial derivatives of the function with respect to each variable. The Jacobian of a vector valued function, either $ J(\\mathbf f(\\mathbf x))$ or $\\mathbf \\nabla \\mathbf f( \\mathbf x)$ is the gradient of each vector component of the function, or the partial derivatives of each vector component of the function with respect to each variable.\n",
    "\n",
    "The partial derivatives of a function with N inputs and 1 output at a point $ \\mathbf x $, where $ \\mathbf x $ is a vector of size N, or a function $ f(\\mathbf x):R^N \\rightarrow R^1 $, is the gradient of the function, which is a function $ \\nabla f(\\mathbf x): R^N \\rightarrow R^N $. The gradient of a function of N variables, where $ \\mathbf x = \\left [ x_1, x_2, \\cdots, x_n \\right ] $ is \n",
    "\n",
    "$ \\nabla f(\\mathbf x) = \\left [ \\frac {\\partial f(\\mathbf x)} {\\partial x_1}, \\frac {\\partial f(\\mathbf x)} {\\partial x_2}, \\cdots, \\frac {\\partial f(\\mathbf x)} {\\partial x_n} \\right ]^T$. \n",
    "\n",
    "For example, if $ f(x,y) = 4x^2 + 2y $ then $ \\nabla f(x, y) = \\left [ 8x, 2 \\right ]^T $, where $ \\nabla f(x,y) = \\left [\\frac {\\partial f(x,y)} {\\partial x}, \\frac {\\partial f(x,y)} {\\partial y} \\right ]^T$.\n",
    "\n",
    "\n",
    "The partial derivatives of a function with N inputs and M outputs at a point $ \\mathbf x $, where $ \\mathbf x $ is of size N, or a function $ \\mathbf f(\\mathbf x): R^N \\rightarrow R^M $, is the Jacobian of the function, or $ \\mathbf \\nabla \\mathbf f(\\mathbf x): R^N \\rightarrow R^{NxM} $. The point $ \\mathbf x $ is a vector of variables, $ \\mathbf x = \\left [ x_1, x_2, \\cdots, x_n \\right ] $. The function $ \\mathbf f(\\mathbf x) $ is a vector of functions evaluated at $ \\mathbf x $, $ \\mathbf f(\\mathbf x) = \\left [ f_1(\\mathbf x), f_2(\\mathbf x), \\cdots, f_m(\\mathbf x) \\right ]^T$. \n",
    "\n",
    "The Jacobian of a function is the partial derivatives of each component function by each variable. \n",
    "\n",
    "$ \\mathbf \\nabla \\mathbf f(\\mathbf x) = \\left [ \\begin {array} {3} \\frac {\\partial f_1} {\\partial x_1} \\cdots \\frac {\\partial f_1} {\\partial x_n} \\\\ \\hspace{0.5em} \\vdots \\hspace{0.3em} \\ddots \\hspace{0.3em} \\vdots \\\\ \\frac {\\partial f_m}{\\partial x_1} \\cdots \\frac {\\partial f_m}{\\partial x_n} \\end {array}\\right ] $. \n",
    "\n",
    "For example, if $ \\mathbf f(x,y) = \\left [ 4x^2 + 2y, 2x + 4y^2 \\right] $ then \n",
    "\n",
    "the Jacobian is $ \\mathbf \\nabla \\mathbf f(x,y) = \\left [ \\begin {array} {2} 8x, \\hspace{0.5em} 2\\\\ \\hspace{0.5em} 2, 8y \\end {array} \\right ]$.\n",
    "\n",
    "__[forwardDerivative](http://www.diffkt.org/api/api/org.diffkt/forward-derivative.html)__ calculates the derivative of a function over a tensor, evaluated at the tensor `x`, using the forward derivative algorithm.\n",
    "\n",
    "__[reverseDerivative](http://www.diffkt.org/api/api/org.diffkt/reverse-derivative.html)__ calculates the derivative of a function over a tensor, evaluated at the tensor `x`, using the reverse derivative algorithm. The reverse derivative algorithm returns the transpose of the derivative calculation, compared to the forward derivative algorithm, when the result is a Jacobian or 2D tensor.\n",
    "\n",
    "In many cases it is more efficient to calculate the orignal function and its partial derivatives at the same time. In the below functions, they return a `Pair<DTensor, DTensor>`. The first value is called the `primal`, which is the value of a function evaluated at `x`, where `x` is a tensor. The second value is called the `tangent`, which is the derivative of a function evaluated at `x`, where `x` is a tensor.\n",
    "\n",
    "__[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__ calculates a function over a tensor `x` and its derivative, evaluated at the tensor `x,` using the forward derivative algorithm.\n",
    "\n",
    "__[primalAndReverseDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-reverse-derivative.html)__ calculates a function over a tensor `x` and its derivative, evaluated at the tensor `x`, using the reverse derivative algorithm. The reverse derivative algorithm returns the transpose of the derivative calculation, compared to the forward derivative algorithm, when the result is a Jacobian or 2D tensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Derivatives of a Polynomial Function over a 1D Tensor\n",
    "\n",
    "We will calculate the partial derivatives of a polynomial over $ {\\mathbf x} $, a 1D __[FloatTensor](http://www.diffkt.org/api/api/org.diffkt/-float-tensor/index.html)__ differentiable tensor. The constants, $\\mathbf a $ and $\\mathbf b $, are 1D __[FloatTensor](http://www.diffkt.org/api/api/org.diffkt/-float-tensor/index.html)__ tensors. In this example, the addition and multiplication are element-wise.\n",
    "\n",
    "We will use the __[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__ function to calculate both the value of the function and its derivative at the same time.\n",
    "\n",
    "The Jacobian is the partial derivatives of vector valued function, a 2D tensor. \n",
    "\n",
    "Recall, $ \\mathbf \\nabla \\mathbf f(\\mathbf x) = \\left [ \\begin {array} {3} \\frac {\\partial f_1} {\\partial x_1} \\cdots \\frac {\\partial f_1} {\\partial x_n} \\\\ \\hspace{0.5em} \\vdots \\hspace{0.3em} \\ddots \\hspace{0.3em} \\vdots \\\\ \\frac {\\partial f_m}{\\partial x_1} \\cdots \\frac {\\partial f_m}{\\partial x_n} \\end {array}\\right ] $. \n",
    "\n",
    "For this example, \n",
    "\n",
    "$ {\\mathbf f}({\\mathbf x}) = \\left [ a_1 + b_1x_1^2,\\hspace{0.5em} a_2 + b_2x_2^2,\\hspace{0.5em} a_3 + b_3x_3^2\\right ] $, \n",
    "\n",
    "so, \n",
    "\n",
    "$ \\mathbf \\nabla {\\mathbf f}({\\mathbf x}) = \\left [ \\begin {array} {*{3}{c@{{},{},{}}c}} 2b_1 & 0 & 0 \\\\ 0 & 2b_2 & 0\\\\ 0 &  0 & 2b_3 \\end {array}\\right ] $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = [1.0, 2.0, 3.0]\n",
      "f(x) = [2.0, 10.0, 30.0]\n",
      "Jacobian(f(x)) = [[2.0, 0.0, 0.0], [0.0, 8.0, 0.0], [0.0, 0.0, 18.0]]\n"
     ]
    }
   ],
   "source": [
    "// polynomial over a 1D tensor\n",
    "\n",
    "val a = tensorOf(1f, 2f, 3f)\n",
    "val b = tensorOf(1f, 2f, 3f)\n",
    "\n",
    "fun f(x : DTensor) : DTensor {   \n",
    "    val y = a + (b * (x.pow(2f)))\n",
    "    return y\n",
    "}\n",
    "\n",
    "// evaluation of the polynomial\n",
    "\n",
    "val x = tensorOf(1f, 2f, 3f)\n",
    "val (fx, jacobian) = primalAndForwardDerivative(x, ::f)\n",
    "\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${fx}\")\n",
    "println(\"Jacobian(f(x)) = ${jacobian}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be \n",
    "\n",
    "`x = [1.0, 2.0, 3.0]` <br>\n",
    "`f(x) = [2.0, 10.0, 30.0]` <br>\n",
    "`Jacobian(f(x)) = [[2.0, 0.0, 0.0], [0.0, 8.0, 0.0], [0.0, 0.0, 18.0]]`\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The tensor variables are created with __[tensorOf](http://www.diffkt.org/api/api/org.diffkt/tensor-of.html)__<br>\n",
    "\n",
    "The following tensor operations were used:\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__<br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__<br>\n",
    "\n",
    "The following function was used to calculate both the function and the derivative:\n",
    "\n",
    "__[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__\n",
    "\n",
    "The Jacobian is a 2D tensor. For this function, the partial derivatives are on the diagonal.<br>\n",
    "\n",
    "$ \\mathbf \\nabla{\\mathbf f}(\\left[ 1, 2, 3 \\right]) = \\left [ \\begin {array} {*{3}{c@{{},{},{}}c}} 2 & 0 & 0 \\\\ 0 & 8 & 0\\\\ 0 &  0 & 18 \\end {array}\\right ] $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Difference Between Forward Derivative and Backward Derivative Algorithms\n",
    "\n",
    "For a vector valued function where you have N inputs and M outputs, the Jacobian that is returned by the __[forwardDerivative](http://www.diffkt.org/api/api/org.diffkt/forward-derivative.html)__ and __[primalAndForwardDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-forward-derivative.html)__ is an M x N matrix. For the reverse derivative algorithms, __[reverseDerivative](http://www.diffkt.org/api/api/org.diffkt/reverse-derivative.html)__ and __[primalAndReverseDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-reverse-derivative.html)__, the transpose of the Jacobian is returned, which is a N x M Matrix.\n",
    "\n",
    "For this example there are 3 inputs and 2 outputs.\n",
    "\n",
    "Let $\\mathbf x $ be a 1D tensor with three elements.\n",
    "\n",
    "Let $g(\\mathbf x) = 2 x_0 + 2 x_1^2 + 3 x_1 x_2^3$\n",
    "\n",
    "Let $h(\\mathbf x) = 3 x_0^3 x_1 + 2 x_1^2 + 3 x_2^3$\n",
    "\n",
    "Then $\\mathbf f(\\mathbf x) = (g(\\mathbf x), h(\\mathbf x)) = 2 x_0 + 2 x_1^2 + 3 x_1 x_2^3, 3 x_0^3 x_1 + 2 x_1^2 + 3 x_2^3$ \n",
    "\n",
    "The derivative, $\\mathbf \\nabla \\mathbf f(\\mathbf x) = \\left [ \\begin {array} {*{2}{c@{{},{},{}}c}} 2 & 4 x_1 + 3x_2^3 & 9x_1x_2^2 \\\\ 9x_0^2x_1 & 3x_0^3 + 4x_1 & 9x_2^2 \\end {array} \\right ]$\n",
    "\n",
    "The transpose of the derivative, ($\\mathbf \\nabla \\mathbf f(\\mathbf x))^T = \\left [ \\begin {array} {*{3}{c@{{},{}}c}} 2 & 9x_0^2x_1 \\\\ 4 x_1 + 3x_2^3 & 3x_0^3 + 4x_1 \\\\ 9x_1x_2^2  & 9x_2^2 \\end {array} \\right ]$\n",
    "\n",
    "if $ \\mathbf x = \\left [ 1, 2, 3 \\right ]$,\n",
    "\n",
    "then $ \\mathbf f(\\mathbf x) = \\left [ 172, 95 \\right ] $\n",
    "\n",
    "and \n",
    "\n",
    "$ \\mathbf \\nabla \\mathbf f(\\mathbf x) = \\left [ \\begin {array} {*{2}{c@{{},{},{}}c}} 2 & 89 & 162 \\\\ 18 & 11 & 81 \\end {array} \\right ]$\n",
    "\n",
    "$ (\\mathbf \\nabla \\mathbf f(\\mathbf x))^T = \\left [ \\begin {array} {*{3}{c@{{},{}}c}} 2 &  18 \\\\ 89 & 11 \\\\ 162 & 81 \\end {array} \\right ]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = [1.0, 2.0, 3.0]\n",
      "forward f(x) = [172.0, 95.0]\n",
      "forward Jacobian(f(x)) = [[2.0, 89.0, 162.0], [18.0, 11.0, 81.0]]\n",
      "reverse f(x) = [172.0, 95.0]\n",
      "reverse Jacobian(f(x)) = [[2.0, 18.0], [89.0, 11.0], [162.0, 81.0]]\n"
     ]
    }
   ],
   "source": [
    "fun f(x: DTensor) : DTensor {\n",
    "    \n",
    "    val g  = 2f * x[0] + 2f * x[1].pow(2f) + 3f * x[1] * x[2].pow(3f) \n",
    "    val h  = 3f * x[0].pow(3f) * x[1] + 2f * x[1].pow(2f) + 3f * x[2].pow(3f)\n",
    "  \n",
    "    return tensorOf(g as DScalar, h as DScalar) \n",
    "}\n",
    "\n",
    "val x = tensorOf(1f, 2f, 3f)\n",
    "val (forwardFx, forwardJacobian) = primalAndForwardDerivative(x, ::f)\n",
    "val (reverseFx, reverseJacobian) = primalAndReverseDerivative(x, ::f)\n",
    "\n",
    "println(\"x = ${x}\")\n",
    "println(\"forward f(x) = ${forwardFx}\")\n",
    "println(\"forward Jacobian(f(x)) = ${forwardJacobian}\")\n",
    "println(\"reverse f(x) = ${reverseFx}\")\n",
    "println(\"reverse Jacobian(f(x)) = ${reverseJacobian}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be\n",
    "\n",
    "`x = [1.0, 2.0, 3.0]` <br>\n",
    "`forward f(x) = [172.0, 95.0]` <br>\n",
    "`forward Jacobian(f(x)) = [[2.0, 89.0, 162.0], [18.0, 11.0, 81.0]]` <br>\n",
    "`reverse f(x) = [172.0, 95.0]` <br>\n",
    "`reverse Jacobian(f(x)) = [[2.0, 18.0], [89.0, 11.0], [162.0, 81.0]]` <br>\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The reverse derivative algorithm returns the transpose of the Jacobian."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Derivative of a Dot Product Function\n",
    "\n",
    "Give a 1D tensor of constants, $\\mathbf c = \\left[ c_1, c_2, \\cdots, c_n \\right] $, and a 1D tensor of variables, $ \\mathbf x = \\left[x_1^2, x_2^2, \\cdots, x_n^2 \\right] $ then the dot product of $ \\mathbf c $ and $ \\mathbf x $, or $\\mathbf c \\cdot \\mathbf x $ is defined as $  f(\\mathbf x) = \\sum c_1 \\cdot x_1^2 + c_2 \\cdot x_2^2 + \\cdots + c_n + x_n^2 $. \n",
    "\n",
    "The gradient, as this is a function with one output, or a scalar output, is $ \\nabla f(\\mathbf x) = \\left[ \\frac {df(\\mathbf x)}{dx_1}, \\frac {df(\\mathbf x)}{dx_2}, \\cdots, \\frac {df(\\mathbf x)}{dx_n} \\right]$ and $ \\frac {df(\\mathbf x)}{dx_i} = 2 * c_i * x_i $. \n",
    "\n",
    "In our case where $ \\mathbf c = \\left [ 1, 2, 3 \\right ]$ and $ \\mathbf x = \\left [1, 2, 3 \\right ]$ then $ \\nabla f(\\mathbf x) = \\left[ 2, 8, 18 \\right]$.\n",
    "\n",
    "We will demostrate calculating the derivative of a dot product three different ways: using a loop, using the sum function, and using the interproduct function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c = [1.0, 2.0, 3.0]\n",
      "x = [1.0, 2.0, 3.0]\n",
      "f(x) = 36.0\n",
      "grad(f(x)) = [2.0, 8.0, 18.0]\n"
     ]
    }
   ],
   "source": [
    "// Dot Product using a loop\n",
    "\n",
    "val c = tensorOf(1f, 2f, 3f)  \n",
    "    \n",
    "fun f(x: DTensor) : DTensor {\n",
    "    \n",
    "    val len = x.size - 1\n",
    "    \n",
    "    // note the need to declare the type DTensor\n",
    "    // all arithmatic operations return type DTensor\n",
    "    \n",
    "    var y : DTensor = FloatScalar(0f)\n",
    "    \n",
    "    for (i in 0..len) {\n",
    "        y = y + c[i] * x[i].pow(2f)\n",
    "    }\n",
    "    return y\n",
    "}\n",
    "\n",
    "val x = tensorOf(1f, 2f, 3f)\n",
    "\n",
    "val (fx, grad) = primalAndReverseDerivative(x, ::f)\n",
    "\n",
    "println(\"c = ${c}\")\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${fx}\")\n",
    "println(\"grad(f(x)) = ${grad}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c = [1.0, 2.0, 3.0]\n",
      "x = [1.0, 2.0, 3.0]\n",
      "f(x) = 36.0\n",
      "grad(f(x)) = [2.0, 8.0, 18.0]\n"
     ]
    }
   ],
   "source": [
    "// Dot Product using sum\n",
    "\n",
    "val c = tensorOf(1f, 2f, 3f)  \n",
    "    \n",
    "fun f(x: DTensor) : DTensor {\n",
    "    \n",
    "    val y = (c * x.pow(2f)).sum()\n",
    "    return y\n",
    "}\n",
    "\n",
    "val x = tensorOf(1f, 2f, 3f)\n",
    "\n",
    "val (fx, grad) = primalAndReverseDerivative(x, ::f)\n",
    "\n",
    "println(\"c = ${c}\")\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${fx}\")\n",
    "println(\"grad(f(x)) = ${grad}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c = [1.0, 2.0, 3.0]\n",
      "x = [1.0, 2.0, 3.0]\n",
      "f(x) = 36.0\n",
      "grad(f(x)) = [2.0, 8.0, 18.0]\n"
     ]
    }
   ],
   "source": [
    "// Dot Product using innerProduct\n",
    "\n",
    "val c = tensorOf(1f, 2f, 3f)  \n",
    "    \n",
    "fun f(x: DTensor) : DTensor {\n",
    "    \n",
    "    val y = c.innerProduct(Shape(3), x.pow(2f))\n",
    "    return y\n",
    "}\n",
    "\n",
    "val x = tensorOf(1f, 2f, 3f)\n",
    "\n",
    "val (fx, grad) = primalAndReverseDerivative(x, ::f)\n",
    "\n",
    "println(\"c = ${c}\")\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${fx}\")\n",
    "println(\"grad(f(x)) = ${grad}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be:\n",
    "\n",
    "`c = [1.0, 2.0, 3.0]` <br>\n",
    "`x = [1.0, 2.0, 3.0]` <br>\n",
    "`f(x) = 36.0` <br>\n",
    "`grad(f(x)) = [2.0, 8.0, 18.0]`\n",
    "\n",
    "#### Observations\n",
    "\n",
    "The tensor variables are created with __[tensorOf](http://www.diffkt.org/api/api/org.diffkt/tensor-of.html)__<br>\n",
    "\n",
    "The following tensor operations were used:\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__<br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__<br>\n",
    "__[sum](http://www.diffkt.org/api/api/org.diffkt/sum.html)__<br>\n",
    "__[innerProduct](http://www.diffkt.org/api/api/org.diffkt/inner-product.html)__<br>\n",
    "\n",
    "The following function was used to calculate both the function and the derivative:\n",
    "\n",
    "__[primalAndReverseDerivative](http://www.diffkt.org/api/api/org.diffkt/primal-and-reverse-derivative.html)__\n",
    "\n",
    "The actual calculations:\n",
    "\n",
    "$ f(\\mathbf x) = 1 \\cdot 1 + 2 \\cdot 4 + 3 \\cdot 9 = 36 $<br>\n",
    "$ \\nabla f(\\mathbf x) = \\left [  2, 8, 18 \\right]$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### A More Complex Example\n",
    "\n",
    "The purpose of this example is to show more tensor operations.\n",
    "\n",
    "__['+'](http://www.diffkt.org/api/api/org.diffkt/plus.html)__ or __[plus](http://www.diffkt.org/api/api/org.diffkt/plus.html)__,<br>\n",
    "__['-'](http://www.diffkt.org/api/api/org.diffkt/minus.html)__ or __[minus](http://www.diffkt.org/api/api/org.diffkt/minus.html)__,<br>\n",
    "__['*'](http://www.diffkt.org/api/api/org.diffkt/times.html)__ or __[times](http://www.diffkt.org/api/api/org.diffkt/times.html)__,<br>\n",
    "__['/'](http://www.diffkt.org/api/api/org.diffkt/div.html)__ or __[div](http://www.diffkt.org/api/api/org.diffkt/div.html)__, <br>\n",
    "__[pow](http://www.diffkt.org/api/api/org.diffkt/pow.html)__,<br>\n",
    "__[sin](http://www.diffkt.org/api/api/org.diffkt/sin.html)__,<br>\n",
    "__[cos](http://www.diffkt.org/api/api/org.diffkt/cos.html)__ and,<br>\n",
    "__[matmul](http://www.diffkt.org/api/api/org.diffkt/matmul.html)__,<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x = [[1.0, 2.0], [3.0, 4.0]]\n",
      "f(x) = [[1.1677711, 4.485937], [6.0297217, 8.905575]]\n",
      "Jacobian(f(x)) = [[[[1.7507683, 0.0], [0.0, 0.0]], [[0.0, 0.3049898], [0.0, 0.0]]], [[[0.0, 0.0], [-0.41488642, 0.0]], [[0.0, 0.0], [0.0, 0.930223]]]]\n"
     ]
    }
   ],
   "source": [
    "// A more complex example\n",
    " \n",
    "val c = tensorOf(1f, 2f, 3f, 4f).reshape(2, 2)\n",
    "\n",
    "fun f(x:DTensor) : DTensor {\n",
    "        \n",
    "    var y = tensorOf(0f, 0f, 0f, 0f).reshape(2, 2)\n",
    "    \n",
    "    for (i in 0..2) {\n",
    "        if (i % 2 == 0) \n",
    "            y = y + c * sin(x).pow(i)\n",
    "        else\n",
    "            y = y - c * cos(x).pow(i)\n",
    "    }\n",
    "    \n",
    "    y = y / 2f\n",
    "    \n",
    "    val scale : DTensor = tensorOf(2f, 0f, 0f, 2f).reshape(2, 2)\n",
    "    y = y.matmul(scale)\n",
    "    \n",
    "    return y\n",
    "    \n",
    "}\n",
    "\n",
    "val x = tensorOf(1f, 2f, 3f, 4f).reshape(2, 2)\n",
    "val (fx, jacobian) = primalAndReverseDerivative(x, ::f)\n",
    "\n",
    "println(\"x = ${x}\")\n",
    "println(\"f(x) = ${fx}\")\n",
    "println(\"Jacobian(f(x)) = ${jacobian}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### The output should be:\n",
    "\n",
    "`x = [[1.0, 2.0], [3.0, 4.0]]` <br>\n",
    "`f(x) = [[1.1677711, 4.485937], [6.0297217, 8.905575]]` <br>\n",
    "`Jacobian(f(x)) = [[[[1.7507683, 0.0], [0.0, 0.0]], [[0.0, 0.3049898], [0.0, 0.0]]], [[[0.0, 0.0], [-0.41488642, 0.0]], [[0.0, 0.0], [0.0, 0.930223]]]]`\n",
    "\n",
    "#### Observations\n",
    "\n",
    "$ \\mathbf f(\\mathbf x)$ is a vector valued function, so the derivative is the Jacobian."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## The End\n",
    "\n",
    "This notebook demonstrated how to create a differentable tensor, how to construct a function that applies operations to the tensor, and how to take the derivative of the function."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Kotlin",
   "language": "kotlin",
   "name": "kotlin"
  },
  "language_info": {
   "codemirror_mode": "text/x-kotlin",
   "file_extension": ".kt",
   "mimetype": "text/x-kotlin",
   "name": "kotlin",
   "nbconvert_exporter": "",
   "pygments_lexer": "kotlin",
   "version": "1.7.0-dev-3303"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}