Extend docs by adding info about @do_not_convert for NUMBA and Py…

…thon ops (#5488) - adds info about ``do_not_convert`` decorator in the context of the numba and python operators, to avoid AutoGraph transformations on the callbacks - aligns wording inside external source operator and its examples Signed-off-by: Janusz Lisiecki <[email protected]>
NVIDIA · Jun 4, 2024 · 794e45d · 794e45d
1 parent 08c8f05
commit 794e45d
Show file tree

Hide file tree

Showing 9 changed files with 166 additions and 53 deletions.
diff --git a/dali/operators/numba_function/numba_func.cc b/dali/operators/numba_function/numba_func.cc
@@ -48,6 +48,20 @@ If no setup function provided, the output shape and data type will be the same a
 .. note::
     This operator is experimental and its API might change without notice.
 
+.. warning::
+    When the pipeline has conditional execution enabled, additional steps must be taken to
+    prevent the ``run_fn`` and ``setup_fn`` functions from being rewritten by AutoGraph.
+    There are two ways to achieve this:
+
+        1. Define the functions at global scope (i.e. outside of ``pipeline_def`` scope).
+
+        2. If functions are a result of another "factory" function, then the factory function
+           must be defined outside pipeline definition function and decorated with
+           :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`.
+
+    More details can be found in :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`
+    documentation.
+
 **Example 1:**
 
 The following example shows a simple setup function which permutes the order of dimensions in the shape.

diff --git a/dali/operators/python_function/python_function.cc b/dali/operators/python_function/python_function.cc
@@ -42,6 +42,20 @@ The function should not modify input tensors.
 
 .. warning::
   This operator is not compatible with TensorFlow integration.
+
+.. warning::
+    When the pipeline has conditional execution enabled, additional steps must be taken to
+    prevent the ``function`` from being rewritten by AutoGraph.
+    There are two ways to achieve this:
+
+        1. Define the function at global scope (i.e. outside of ``pipeline_def`` scope).
+
+        2. If function is a result of another "factory" function, then the factory function
+           must be defined outside pipeline definition function and decorated with
+           :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`.
+
+    More details can be found in :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`
+    documentation.
 )code")
         .NumInput(0, 256)
         .AllowSequences()

diff --git a/dali/python/nvidia/dali/external_source.py b/dali/python/nvidia/dali/external_source.py
@@ -528,16 +528,15 @@ class ExternalSource:
 
     |
     .. warning::
-        The ``source``, when :ref:`conditional mode <conditional_execution>` is enabled, must not
-        be transformed by the AutoGraph. There are two conditions to prevent that:
+        When the pipeline has conditional execution enabled, additional steps must be taken to
+        prevent the ``source`` from being rewritten by AutoGraph.
+        There are two ways to achieve this:
 
-            1. The ``source`` or a factory creating the ``source`` must be defined at a global scope
-               (i.e. outside of ``pipeline_def`` scope).
+            1. Define the function at global scope (i.e. outside of ``pipeline_def`` scope).
 
-            2. If the ``source`` is created by a factory function that is invoked within pipeline
-               definition, the factory function must be decorated with
-               :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`. Otherwise it will be
-               recursively converted when the pipeline definition is traced.
+            2. If function is a result of another "factory" function, then the factory function
+               must be defined outside pipeline definition function and decorated with
+               :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`.
 
         More details can be found in :meth:`@do_not_convert <nvidia.dali.pipeline.do_not_convert>`
         documentation.

diff --git a/docs/examples/custom_operations/gpu_python_operator.ipynb b/docs/examples/custom_operations/gpu_python_operator.ipynb
@@ -69,6 +69,23 @@
     ")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "<b>Warning</b>\n",
+    "\n",
+    "When the pipeline has conditional execution enabled, additional steps must be taken to prevent the `function` from being rewritten by AutoGraph.\n",
+    "There are two ways to achieve this:\n",
+    "1. Define the function at global scope (i.e. outside of ``pipeline_def`` scope).\n",
+    "2. If function is a result of another \"factory\" function, then the factory function must be defined outside pipeline definition function and decorated with `<nvidia.dali.pipeline.do_not_convert>`.\n",
+    "\n",
+    "\n",
+    "More details can be found in `nvidia.dali.pipeline.do_not_convert` documentation.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -270,7 +287,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -284,7 +301,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.9"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

diff --git a/docs/examples/custom_operations/numba_function.ipynb b/docs/examples/custom_operations/numba_function.ipynb
diff --git a/docs/examples/custom_operations/python_operator.ipynb b/docs/examples/custom_operations/python_operator.ipynb
@@ -55,6 +55,23 @@
     "**Note:** Both input images are copied, because the input data should not be modified."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "<b>Warning</b>\n",
+    "\n",
+    "When the pipeline has conditional execution enabled, additional steps must be taken to prevent the `function` from being rewritten by AutoGraph.\n",
+    "There are two ways to achieve this:\n",
+    "1. Define the function at global scope (i.e. outside of ``pipeline_def`` scope).\n",
+    "2. If function is a result of another \"factory\" function, then the factory function must have `nvidia.dali.pipeline.do_not_convert` attribute.\n",
+    "\n",
+    "\n",
+    "More details can be found in `nvidia.dali.pipeline.do_not_convert` documentation.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -263,7 +280,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },

diff --git a/docs/examples/general/data_loading/external_input.ipynb b/docs/examples/general/data_loading/external_input.ipynb
@@ -66,6 +66,22 @@
     "        return (batch, labels)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "<b>Warning</b>\n",
+    "\n",
+    "When the pipeline has conditional execution enabled, additional steps must be taken to prevent the `source` from being rewritten by AutoGraph.\n",
+    "There are two ways to achieve this:\n",
+    "1. Define the function at global scope (i.e. outside of `pipeline_def` scope).\n",
+    "2. If function is a result of another \"factory\" function, then the factory function must be defined outside pipeline definition function and decorated with `<nvidia.dali.pipeline.do_not_convert>`.\n",
+    "\n",
+    "More details can be found in `nvidia.dali.pipeline.do_not_convert` documentation.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -332,7 +348,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -346,7 +362,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.9"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

diff --git a/docs/examples/general/data_loading/parallel_external_source.ipynb b/docs/examples/general/data_loading/parallel_external_source.ipynb
@@ -43,6 +43,22 @@
     "Because the parallel sample mode can provide the biggest speed up, we present how to adapt iterable source to run in parallel sample mode."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "<b>Warning</b>\n",
+    "\n",
+    "When the pipeline has conditional execution enabled, additional steps must be taken to prevent the `source` from being rewritten by AutoGraph.\n",
+    "There are two ways to achieve this:\n",
+    "1. Define the function at global scope (i.e. outside of `pipeline_def` scope).\n",
+    "2. If function is a result of another \"factory\" function, then the factory function must be defined outside pipeline definition function and decorated with `<nvidia.dali.pipeline.do_not_convert>`.\n",
+    "\n",
+    "More details can be found in `nvidia.dali.pipeline.do_not_convert` documentation.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -1229,7 +1245,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,

diff --git a/docs/examples/general/data_loading/parallel_external_source_fork.ipynb b/docs/examples/general/data_loading/parallel_external_source_fork.ipynb
@@ -36,6 +36,22 @@
     "Let's take the `source` callable defined in the previous tutorial. Compared to the `'spawn'` method, all the code from this process ends up copied into the forked process, thus we do not experience the problems with serialization. "
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-warning\">\n",
+    "<b>Warning</b>\n",
+    "\n",
+    "When the pipeline has conditional execution enabled, additional steps must be taken to prevent the `source` from being rewritten by AutoGraph.\n",
+    "There are two ways to achieve this:\n",
+    "1. Define the function at global scope (i.e. outside of `pipeline_def` scope).\n",
+    "2. If function is a result of another \"factory\" function, then the factory function must be defined outside pipeline definition function and decorated with `<nvidia.dali.pipeline.do_not_convert>`.\n",
+    "\n",
+    "More details can be found in `nvidia.dali.pipeline.do_not_convert` documentation.\n",
+    "</div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -269,7 +285,7 @@
    "hash": "767d51c1340bd893661ea55ea3124f6de3c7a262a8b4abca0554b478b1e2ff90"
   },
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -283,7 +299,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.9"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,