tutorials/neural_network_user_defined_types.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bbed776e",
   "metadata": {},
   "source": [
    "# Neural Networks with User-Defined Types\n",
    "\n",
    "Copyright (c) Meta Platforms, Inc. and affiliates.\n",
    "\n",
    "This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.\n",
    "\n",
    "\n",
    "In this notebook we will learn how to build a neural network using DiffKt with user-defined types. Neural networks are not exactly simple, but they are composed of simple mathematical techniques working in orchestration. However the calculus behind neural networks can be tedious, as derivatives for each layer need to be calculated for gradient descent purposes. Because weights and biases are applied in nested functions from each layer, it's mathematically like pulling apart an onion layer-by-layer. Thankfully DiffKt can take care of this task of calculating gradients for weight and bias layers, and leave out the messiness of solving derivatives by hand. Along the way, we will use custom types and demonstrate DiffKt's capabilities with its `Wrapper` interface. \n",
    "\n",
    "To get started, first bring in DiffKt to use in this notebook as well as these imports. Then we will talk about the structure of a neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7b713c9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "@file:DependsOn(\"../kotlin/api/build/libs/api.jar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "09f0ce45",
   "metadata": {},
   "outputs": [],
   "source": [
    "import org.diffkt.*\n",
    "import java.net.URL\n",
    "import org.diffkt.random.DiffktRandom"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "509bfda5",
   "metadata": {},
   "source": [
    "## The Anatomy of a Neural Network \n",
    "\n",
    "Let's present a problem adapted from Chapter 7 in the book [*Essential Math for Data Science (O'Reilly)*](https://learning.oreilly.com/library/view/essential-math-for/9781098102920/). We want to train a neural network to predict a light/dark font for a given background color. For example, a <span style=\"background-color:DarkBlue; color:white\"><text color='white'>Dark Blue</text></span> background would warrant a light font and a <span style=\"background-color:pink; color:black\"><text color='white'>Pink</text></span> background would warrant a dark font. We could solve this with a logistic regression or even a [known heuristic](https://stackoverflow.com/questions/1855884/determine-font-color-based-on-background-color), but this will be a nice toy example to discover the workings of a neural network and applying DiffKt.  \n",
    "\n",
    "Let's get to building the neural network. This visual below does not show the activation functions, a critical component to make a neural network work. We will get to that. Let's look at the nodes first. \n",
    "\n",
    "<img src=\"./resources/sGQdjdjUMw.png\" style=\"width: 600px;\"/>\n",
    "\n",
    "The first layer is simply an input of the three variables (R, G, and B values for a given color). In the hidden layer (which resides in the middle), notice that we have three **nodes**, or functions of weights and biases, between the inputs and outputs. There is a weight $ w_i $ between each input node and hidden node, and another set of weights between each hidden node and output node. Each hidden and output node gets an additional bias $ b_i $ added.\n",
    "\n",
    "The output node repeats the same operation, taking the resulting weighted and summed outputs from the hidden layer and making them inputs into the output layer, where another set of weights and biases will be applied. I put \"repeat weighting and summing\" instead of the mathematical expressions because the expressions propagating from the hidden layer is too long to display in the graphic. But here is the expression for the final node. \n",
    "\n",
    "$ \\text{output} = w_{10}(x_1w_1 + x_2w_2 + x_3w_3 + b_1) $ \n",
    "\n",
    "$ + w_{11}(x_1w_4 + x_2w_5 + x_3w_6 + b_2) $ \n",
    "\n",
    "$ + w_{12}(x_1w_7 + x_2w_8 + x_3w_9 + b_3) + b_4 $ \n",
    "\n",
    "We need to solve for each of these weight and bias values, and this is what we call **training** the neural network. But before we get to that later in this notebook, there is one more critical component we need to add. The **activation function** is a nonlinear function that transforms the weighted and summed values in a node, helping separate the weighted data so it can be classified. For this neural network, we will use the _ReLU_ function for the hidden layer and the sigmoid function for the output layer. \n",
    "\n",
    "<img src=\"./resources/PvLebFIsiT.png\" style=\"width: 600px;\"/>\n",
    "\n",
    "The **ReLU \"rectified linear unit\" function** will take a given numeric input and turn it to $ 0 $ if negative. ReLU is commonly used for hidden layers in neural networks because of its speed. It also mitigates the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem) where partial derivative gradients get so small they prematurely approach $ 0 $ and bring training to a halt. DiffKt comes packaged with a `relu()` function that is compatible with its tensors and scalar types. It can be built from scratch simply using a maximum function between 0 and the input value. \n",
    "\n",
    "$$\n",
    "ReLU(x) = max(0, x)\n",
    "$$\n",
    "\n",
    "<img src=\"./resources/tKkerIrVkt.png\" style=\"width: 600px;\"/>\n",
    "\n",
    "\n",
    "The output layer consolidates all the inputs from the hidden layer and turns them into interpretable results in the output layer. In this case where our output is binary (light/dark font) we only have one output node. We use the **sigmoid function** to compress values between 0 and 1 using a logistic curve. This can be interpreted as a probability between 0 and 1, where closer to 0 indicates a dark font recommendation and closer to 1 recommends a light font. We can use $ 0.5 $ as our threshhold so anything less than $ 0.5 $ is considered a light font recommendation, and anything equal or higher is dark font. DiffKt also comes with a `sigmoid()` function, which is mathematically defined below: \n",
    "\n",
    "$$\n",
    "sigmoid(x) = \\frac{1}{1 + e^{-x}}\n",
    "$$\n",
    "\n",
    "<img src=\"./resources/DllsJpEMCJ.png\" style=\"width: 600px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "876151bc",
   "metadata": {},
   "source": [
    "## Declaring the Types\n",
    "\n",
    "We will need to declare our classes in advance that will be targeted for differentiation. The first class `Color` will hold a red, green, and blue value as properties. It will also contain an `asFloatArray()` function for convenience to package the three values as a float array. We make it a `data class` so it prints the object type and its properties. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a527f4e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "data class Color(val r: Float, val g: Float, val b: Float)  {\n",
    "    fun asFloatArray() = floatArrayOf(r,g,b)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c9387bc",
   "metadata": {},
   "source": [
    "The `FontShade` is a simple boolean indicator indicating light (0) or dark (1) font. We express this an an enumerable for explicitness and store the `value` as a float for mathematical operations. We also can convert a floating value of `1f` or `0f` using the `valueOf()` companion function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aec50d53",
   "metadata": {},
   "outputs": [],
   "source": [
    "enum class FontShade(val value: Float) {\n",
    "    LIGHT(0f),\n",
    "    DARK(1f);\n",
    "\n",
    "    companion object {\n",
    "        fun valueOf(value: Float) = values().first { it.value == value }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b3ec97f",
   "metadata": {},
   "source": [
    "A `ColorFontAndShade` class pairs a `Color` and a `FontShade` together. It also has a constructor build both of those items at once. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a223a174",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class ColorAndFontShade(val color: Color, val fontShade: FontShade) {\n",
    "\n",
    "    constructor(r: Float, g: Float, b: Float, fontShadeValue: Float):\n",
    "            this(Color(r,g,b), FontShade.valueOf(fontShadeValue))\n",
    "\n",
    "    override fun toString() = \"(${color.r},${color.g},${color.b})-${fontShade}\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4985ea0d",
   "metadata": {},
   "source": [
    "Now let's look at the neural network components, each declared as a user-defined type. We are going to use DiffKt to calculate differentials on each layer, which contains nodes with weights and biases. The `NodeDiff` will contain the `weightDiffs` with regard to each weight, and a `biasDiff` to target the bias. We will also implement a `times()` function to multiply the diffs by a scalar. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e9998e03",
   "metadata": {},
   "outputs": [],
   "source": [
    "data class NodeDiff(val weightDiffs: List<DScalar>, val biasDiff: DScalar)  {\n",
    "    operator fun times(scalar: Float) = NodeDiff(weightDiffs.map { it * scalar}, biasDiff * scalar)\n",
    "}\n",
    "\n",
    "data class LayerDiff(val nodeDiffs: List<NodeDiff>) {\n",
    "    operator fun times(scalar: Float) = LayerDiff(nodeDiffs.map { it * scalar })\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c50f9c83",
   "metadata": {},
   "source": [
    "We will then implement a `Node` type which contains the weights and biases for a given node on a layer. This will implement the `Differentiable` interface and make it wrappable to the DiffKt API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ae09f002",
   "metadata": {},
   "outputs": [],
   "source": [
    "data class Node(val weights: List<DScalar>,\n",
    "                val bias: DScalar = DiffktRandom.fromTimeOfDay().nextUniform()\n",
    "): Differentiable<Node> {\n",
    "\n",
    "    fun forward(values: List<DScalar>) =\n",
    "        values.zip(weights) { v, w -> v * w }\n",
    "            .reduce { x, y -> x + y } + bias\n",
    "\n",
    "    constructor(connectionCount: Int) : this(\n",
    "        (0 until connectionCount).map { (DiffktRandom.fromTimeOfDay().nextUniform() * 2f) - 1f }\n",
    "    )\n",
    "\n",
    "    override fun wrap(wrapper: Wrapper) =\n",
    "        Node(wrapper.wrap(weights), wrapper.wrap(bias))\n",
    "\n",
    "    private fun bound(value: DScalar, lower: Float, upper: Float) : DScalar {\n",
    "        val l = FloatScalar(lower)\n",
    "        val u = FloatScalar(upper)\n",
    "        val trimLower = ifThenElse(value lt l, l, value)\n",
    "        val trimUpper = ifThenElse(trimLower gt u, u, trimLower)\n",
    "        return trimUpper\n",
    "    }\n",
    "\n",
    "    operator fun minus(nodeDiff: NodeDiff) =\n",
    "        Node(weights.zip(nodeDiff.weightDiffs) { w, d ->\n",
    "            bound(w - d, -1f, 1f)\n",
    "        }, bound(bias - nodeDiff.biasDiff, 0f, 1f))\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aa1830a",
   "metadata": {},
   "source": [
    "A `forward()` function will take incoming values and apply the weights and bias through a dot product operation. The `constructor` will randomly initialize a node with the provided number of connections, resulting in corresponding weights and a bias. The `wrap()` instructs DiffKt how to pass the weights and bias to a new instance of `Node` and enable differentiation. The `bound()` is a utility function to keep a value inside a range, and that is used in `minus()` when a `NodeDiff` is subtracted from a `Node` during gradient descent where the weight and bias values are adjusted based on the gradients. \n",
    "\n",
    "The `Layer` contains an ordered collection of nodes and an activation function, which is defined as a functional argument taking a `DScalar` and converting into another `DScalar`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d90052f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "data class Layer(val nodes: List<Node>,\n",
    "                 val activation: (DScalar) -> DScalar): Wrappable<Layer> {\n",
    "\n",
    "    constructor(previousNodeCount: Int,\n",
    "                nodeCount: Int,\n",
    "                activation: (DScalar) -> DScalar): this(\n",
    "        (0 until nodeCount).map { Node(previousNodeCount) },\n",
    "        activation\n",
    "    )\n",
    "\n",
    "    fun forward(values: List<DScalar>) = nodes.map { node ->\n",
    "        activation(node.forward(values))\n",
    "    }\n",
    "\n",
    "    operator fun minus(layerDiff: LayerDiff) =\n",
    "        Layer(nodes.zip(layerDiff.nodeDiffs) { n,d -> n - d }, activation)\n",
    "\n",
    "    override fun toString() = nodes.joinToString(\",\")\n",
    "\n",
    "    override fun wrap(wrapper: Wrapper) = Layer(wrapper.wrap(nodes), activation)\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c25f5f",
   "metadata": {},
   "source": [
    "The `Layer` implements `Wrappable` so it can be differentiated on effectively and implements the `wrap()` functionon the nodes. The alternate `constructor` takes as argument the previous layer's number of nodes so it can calculate the number of connections, and thus weights, from the previous layer. Otherwise if it is the input layer the the primary constructor is used for an input layer. The `forward()` function will apply the nodes (weights and biases) and activation function to a given input. The `minus()` will subtract a `LayerDiff` to apply differentials on a layer of nodes.\n",
    "\n",
    "Finally, a `NeuralNetwork` will be constructed off a list of layers and will implement a `Wrappable` interface as well. Notice that there is a wrappable hierarchy with `Neural Network` ⮕ `Layer` ⮕ `Node`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bdb1f2ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "data class NeuralNetwork(val layers: List<Layer>): Wrappable<NeuralNetwork> {\n",
    "    constructor(vararg layers: Layer): this(layers.toList())\n",
    "\n",
    "    fun forward(input: FloatArray): List<DScalar> {\n",
    "        var forwardProp: List<DScalar> = input.map(::FloatScalar)\n",
    "\n",
    "        for (layer in layers) {\n",
    "            forwardProp = layer.forward(forwardProp)\n",
    "        }\n",
    "        return forwardProp\n",
    "    }\n",
    "\n",
    "    operator fun minus(layerDiffs: List<LayerDiff>) =\n",
    "        NeuralNetwork(\n",
    "            layers.zip(layerDiffs) { layer, diff -> layer - diff }\n",
    "        )\n",
    "\n",
    "    override fun wrap(wrapper: Wrapper) = NeuralNetwork(wrapper.wrap(layers))\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e107bd",
   "metadata": {},
   "source": [
    "This will make differentiating an entire neural network possible with respect to each individual weight and bias. When we call the `forward()` function it will propogate the values through each layer including applying the weights, biases, summations, and activation functions. We return the result as a list of `DScalar` which will contain the prediction. This `forward()` function is what we will differentiate in a loss function. \n",
    "\n",
    "The `minus()` subtracts layer differentials from the entire neural network, and this will aid us in gradient descent easily by subtracting a fraction of the gradients across so many iterations. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d88f868",
   "metadata": {},
   "source": [
    "## Importing the Data\n",
    "\n",
    "\n",
    "\n",
    "Let's first explore our data stored [here](https://tinyurl.com/y2qmhfsr). It contains 3 input variable columns (red, green, and blue) and the output light/dark font indicator which is a boolean we want to predict. We have 1345 records in this training data. Here is a sample: \n",
    "\n",
    "| RED | GREEN | BLUE | LIGHT_OR_DARK_FONT_IND |\n",
    "|-----|-------|------|------------------------|\n",
    "| 0   | 0     | 0    | 0                      |\n",
    "| 0   | 0     | 128  | 0                      |\n",
    "| 0   | 139   | 69   | 0                      |\n",
    "| 0   | 154   | 205  | 0                      |\n",
    "| 0   | 178   | 238  | 1                      |\n",
    "| 0   | 197   | 205  | 1                      |\n",
    "| 0   | 199   | 140  | 1                      |\n",
    "| 0   | 201   | 87   | 1                      |\n",
    "| 0   | 205   | 0    | 0                      |\n",
    "| 0   | 205   | 102  | 1                      |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d77bfa7",
   "metadata": {},
   "source": [
    "To bring in this data, let's use the Java `URL` interface to read the CSV from GitHub. We will use some [regular expressions](https://www.oreilly.com/content/an-introduction-to-regular-expressions/) to split the lines, and [Sequence](https://kotlinlang.org/docs/sequences.html) operations to clean up the lines like an assembly line. We will map the whole CSV into a list of `ColorAndfontShade` objects. Note we will also rescale the color values down by a factor of 255, to compress the red, green, and blue values to be between 0 and 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0d21ad23",
   "metadata": {},
   "outputs": [],
   "source": [
    "val colorsAndFontShades = URL(\"https://tinyurl.com/y2qmhfsr\")\n",
    "    .readText().split(Regex(\"\\\\r?\\\\n\"))\n",
    "    .asSequence()\n",
    "    .drop(1)\n",
    "    .filter { it.isNotBlank() }\n",
    "    .map { s ->\n",
    "        s.split(\",\").map { it.toFloat()  }\n",
    "    }.map {\n",
    "        ColorAndFontShade(it[0] / 255, it[1] / 255, it[2] / 255, it[3])\n",
    "    }.toList()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80aa0e89",
   "metadata": {},
   "source": [
    "## Performing Gradient Descent\n",
    "\n",
    "Let's declare an instance of our neural network to predict a light or dark font. We will use expect 3 inputs that will be passed to a middle layer with 12 nodes, and uses a ReLU function. The output layer will take those 12 incoming values as inputs and then weight/bias them again and sum them into a single value, hence why there is only 1 node. That final layer will take that single value and pass it through the `sigmoid()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "92170450",
   "metadata": {},
   "outputs": [],
   "source": [
    "var nn = NeuralNetwork(\n",
    "    Layer(3,12,::relu),\n",
    "    Layer(12,1,::sigmoid)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a7f63e3",
   "metadata": {},
   "source": [
    "Obviously our neural network is going to perform poorly as the weights and biases are randomly initialized and not optimized yet. The first step in optimizing them is to declare a `loss()` function that measures how far off our neural network outputs are from our training outputs. Let's use a simple squared loss function, where $ C $ is the sum of squares between the predicted outputs $ A_2 $ and the actual outputs from the data $ Y $. \n",
    "\n",
    "$ C = (Y - A_2)^2 $ \n",
    "\n",
    "Let's create this `loss()` function as specified below. We will perform batch gradient descent here and iterate the entire batch, calculating the sum of squared differences between the neural network prediction and actual output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7ba30d45",
   "metadata": {},
   "outputs": [],
   "source": [
    "var count = 0\n",
    "\n",
    "fun loss(nn: NeuralNetwork): DScalar {\n",
    "    var loss: DScalar = FloatScalar.ZERO\n",
    "\n",
    "    for (trainExample in colorsAndFontShades) {\n",
    "        val (trainInput, trainOutput) = trainExample.let { it.color.asFloatArray() to it.fontShade.value }\n",
    "        val diff = nn.forward(trainInput)[0] - trainOutput\n",
    "        loss += diff.pow(2)\n",
    "    }\n",
    "\n",
    "    println(\"$count. ${sqrt(loss.basePrimal() / colorsAndFontShades.size)}\")\n",
    "    count++\n",
    "    return loss\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bc76d0c",
   "metadata": {},
   "source": [
    "Finally let's move forward with our neural network. We will implement a learning rate of $ .001 $ and 100 itertions. On each iteration we will traverse the entire dataset as implemented in our `loss()` function, and pass the whole `NeuralNetwork` as an input object into `reverseDerivative()`. The `extractDerivative` lambda will map each layer from the neural network to a `LayerDiff`, which then contains nodes mapped to `NodeDiff` instances containing the `extractTangent` calculations for the weights and tensors. We can then take this resulting list of `LayerDiff` objects, multiply them by our learning rate, and subtract them from our neural network. \n",
    "\n",
    "We can then calculate the proportion of accurate predictions by running all the data through the neural network, and comparing to the actual values. Reading the output below, you should get high accuracy around $ .96 $, give or take. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9a79e60b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. 0.54920614\n",
      "1. 0.5178369\n",
      "2. 0.50264174\n",
      "3. 0.48897183\n",
      "4. 0.47608778\n",
      "5. 0.46379277\n",
      "6. 0.45166457\n",
      "7. 0.44003513\n",
      "8. 0.42941603\n",
      "9. 0.42018256\n",
      "10. 0.4114745\n",
      "11. 0.4035245\n",
      "12. 0.39601928\n",
      "13. 0.38903233\n",
      "14. 0.38250408\n",
      "15. 0.37636676\n",
      "16. 0.37055892\n",
      "17. 0.365396\n",
      "18. 0.36049214\n",
      "19. 0.35582516\n",
      "20. 0.3513578\n",
      "21. 0.3470943\n",
      "22. 0.3429735\n",
      "23. 0.33900622\n",
      "24. 0.33527622\n",
      "25. 0.3318883\n",
      "26. 0.32862973\n",
      "27. 0.32542607\n",
      "28. 0.32231262\n",
      "29. 0.31930184\n",
      "30. 0.31638476\n",
      "31. 0.3137104\n",
      "32. 0.3112201\n",
      "33. 0.30891708\n",
      "34. 0.30702764\n",
      "35. 0.30517948\n",
      "36. 0.3033503\n",
      "37. 0.3015414\n",
      "38. 0.29976398\n",
      "39. 0.29803848\n",
      "40. 0.29634753\n",
      "41. 0.29468715\n",
      "42. 0.29304844\n",
      "43. 0.29143605\n",
      "44. 0.28982726\n",
      "45. 0.28823754\n",
      "46. 0.28667375\n",
      "47. 0.28511363\n",
      "48. 0.2836921\n",
      "49. 0.28238967\n",
      "50. 0.28111345\n",
      "51. 0.2799777\n",
      "52. 0.27886778\n",
      "53. 0.27777228\n",
      "54. 0.27669948\n",
      "55. 0.27567115\n",
      "56. 0.27465108\n",
      "57. 0.27364227\n",
      "58. 0.27264416\n",
      "59. 0.27165627\n",
      "60. 0.2706783\n",
      "61. 0.26971057\n",
      "62. 0.26876372\n",
      "63. 0.2679001\n",
      "64. 0.26705843\n",
      "65. 0.2663019\n",
      "66. 0.26557568\n",
      "67. 0.2648647\n",
      "68. 0.26416805\n",
      "69. 0.26348498\n",
      "70. 0.26281464\n",
      "71. 0.26215592\n",
      "72. 0.26150844\n",
      "73. 0.2608741\n",
      "74. 0.26025307\n",
      "75. 0.25964338\n",
      "76. 0.25904503\n",
      "77. 0.25845766\n",
      "78. 0.25787994\n",
      "79. 0.25734797\n",
      "80. 0.25708765\n",
      "81. 0.25684455\n",
      "82. 0.256613\n",
      "83. 0.25639006\n",
      "84. 0.2561754\n",
      "85. 0.25596428\n",
      "86. 0.25576273\n",
      "87. 0.2555938\n",
      "88. 0.25543553\n",
      "89. 0.2552821\n",
      "90. 0.2551304\n",
      "91. 0.25497937\n",
      "92. 0.2548348\n",
      "93. 0.25469875\n",
      "94. 0.25459203\n",
      "95. 0.254492\n",
      "96. 0.25439516\n",
      "97. 0.254301\n",
      "98. 0.25420898\n",
      "99. 0.25411925\n",
      "100. 0.25403133\n",
      "NeuralNetwork(layers=[Node(weights=[-0.27542502, -0.8973492, -0.13545553], bias=1.0),Node(weights=[-0.0887655, 1.0, 0.3513219], bias=0.0),Node(weights=[0.17563678, 0.10888315, -0.694091], bias=0.017594248),Node(weights=[-0.5863736, -0.7214906, 0.3964951], bias=1.0),Node(weights=[-0.8628719, -0.007164005, -0.29888514], bias=1.0),Node(weights=[-0.4189871, -0.5151518, -0.35790804], bias=1.0),Node(weights=[0.45788673, 1.0, -0.15261371], bias=0.0),Node(weights=[-0.56738377, 0.23330118, -0.976017], bias=0.42044497),Node(weights=[-0.485411, -0.4530675, -0.056381956], bias=0.15412267),Node(weights=[-0.0099647315, -0.6021924, -0.64941114], bias=0.13867453),Node(weights=[-0.1778316, 1.0, 0.20152433], bias=0.0),Node(weights=[-0.06606455, 0.62878627, -0.34489495], bias=0.42887074), Node(weights=[-1.0, 1.0, 0.062779166, -1.0, -1.0, -1.0, 1.0, 0.051147085, 0.20965675, 0.25179556, 1.0, 0.079933494], bias=0.0)])\n",
      "ACCURACY: 0.9672862\n"
     ]
    }
   ],
   "source": [
    "// The learning rate\n",
    "val lr = .001F\n",
    "\n",
    "// The number of iterations to perform gradient descent\n",
    "val iterations = 100\n",
    "\n",
    "    // Perform gradient descent\n",
    "    for (i in 0..iterations) {\n",
    "\n",
    "        val tangents = reverseDerivative(\n",
    "            x = nn,\n",
    "            f = ::loss,\n",
    "            extractDerivative= { input, output, extractTangent ->\n",
    "                input.layers.map { layer ->\n",
    "                    LayerDiff(\n",
    "                        layer.nodes.map { node ->\n",
    "                            NodeDiff(\n",
    "                                node.weights.map { extractTangent(it, output) as DScalar },\n",
    "                                extractTangent(node.bias, output) as DScalar\n",
    "                            )\n",
    "                        }\n",
    "                    )\n",
    "                }\n",
    "            }\n",
    "        )\n",
    "\n",
    "        nn -= tangents.map { it * lr }\n",
    "    }\n",
    "\n",
    "    // calculate accuracy\n",
    "    val accuracy = colorsAndFontShades.map {\n",
    "        (nn.forward(it.color.asFloatArray())[0].ge(.5f).eq(it.fontShade.value) as FloatScalar).value\n",
    "    }.sum() / colorsAndFontShades.count()\n",
    "\n",
    "    println(nn)\n",
    "    println(\"ACCURACY: $accuracy\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Kotlin",
   "language": "kotlin",
   "name": "kotlin"
  },
  "language_info": {
   "codemirror_mode": "text/x-kotlin",
   "file_extension": ".kt",
   "mimetype": "text/x-kotlin",
   "name": "kotlin",
   "nbconvert_exporter": "",
   "pygments_lexer": "kotlin",
   "version": "1.6.20-dev-6372"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}