operations: deploy: Continuous Delivery of DataFlows usage example

- ffmpeg example operations
- Deploy operations
- Continuous Delivery of DataFlows usage example

Fixes: intel#608
Fixes: intel#532
Fixes: intel#534

Signed-off-by: John Andersen <[email protected]>

.github/workflows/testing.yml

@@ -52,7 +52,7 @@ jobs:
       fail-fast: false
       max-parallel: 40
       matrix:
-        plugin: [., examples/shouldi, model/tensorflow, model/tensorflow_hub, model/transformers, model/scratch, model/scikit, model/vowpalWabbit, operations/binsec, source/mysql, feature/git, feature/auth, service/http, configloader/yaml, configloader/png]
+        plugin: [., examples/shouldi, model/tensorflow, model/tensorflow_hub, model/transformers, model/scratch, model/scikit, model/vowpalWabbit, operations/binsec, operations/deploy, source/mysql, feature/git, feature/auth, service/http, configloader/yaml, configloader/png]
         python-version: [3.7, 3.8]
 
     steps:
@@ -124,6 +124,7 @@ jobs:
         feature/git=${{ secrets.PYPI_FEATURE_GIT }}
         feature/auth=${{ secrets.PYPI_FEATURE_AUTH }}
         operations/binsec=${{ secrets.PYPI_OPERATIONS_BINSEC }}
+        operations/deploy=${{ secrets.PYPI_OPERATIONS_DEPLOY }}
         service/http=${{ secrets.PYPI_SERVICE_HTTP }}
         configloader/yaml=${{ secrets.PYPI_CONFIG_YAML }}
         configloader/png=${{ secrets.PYPI_CONFIG_PNG }}

CHANGELOG.md

@@ -21,7 +21,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Ability to export any object with `dffml service dev export`
 - Complete example for dataflow run cli command
 - Tests for default configs instantiation.
+- Example ffmpeg operation.
+- Operations to deploy docker container on receving github webhook.
+- New use case `Redeploying dataflow on webhook` in docs.
 - Documentation for creating Source for new File types taking `.ini` as an example.
+- New input modes, output modes for HTTP API dataflow registration.
 ### Changed
 - `Edit on Github` button now hidden for plugins.
 - Doctests now run via unittests

dffml/__init__.py

@@ -101,11 +101,7 @@ class DuplicateName(Exception):
                     f"new: {module}) "
                 )
         # Add to dict to ensure no duplicates
-        cls_func_all[obj.__qualname__] = (
-            import_name_no_package,
-            module,
-            obj,
-        )
+        cls_func_all[obj.__qualname__] = (import_name_no_package, module, obj)
 
 for name, (_import_name, _module, obj) in cls_func_all.items():
     setattr(sys.modules[__name__], name, obj)

dffml/model/slr.py

@@ -4,10 +4,7 @@
 
 from ..base import config, field
 from ..util.entrypoint import entrypoint
-from .model import (
-    SimpleModel,
-    ModelNotTrained,
-)
+from .model import SimpleModel, ModelNotTrained
 from .accuracy import Accuracy
 from ..feature.feature import Feature, Features
 from ..source.source import Sources

dffml/skel/operations/Dockerfile

@@ -1,6 +1,7 @@
 # Usage
 # docker build -t REPLACE_ORG_NAME/REPLACE_IMPORT_PACKAGE_NAME .
-# docker run --rm -ti -p 80:8080 intelotc/operations -insecure -log debug
+# docker run --rm -ti -p 80:8080 REPLACE_ORG_NAME/REPLACE_IMPORT_PACKAGE_NAME  -insecure -log debug
+#
 # curl -v http://127.0.0.1:80/list/sources
 FROM ubuntu:20.04
 

dffml/source/ini.py

@@ -41,7 +41,7 @@ async def load_fd(self, fileobj):
                 temp_dict[k] = parser_helper(v)
             # Each section used as a record
             self.mem[str(section)] = Record(
-                str(section), data={"features": temp_dict},
+                str(section), data={"features": temp_dict}
             )
 
         self.logger.debug("%r loaded %d sections", self, len(self.mem))

dffml/util/data.py

@@ -138,9 +138,7 @@ def type_lookup(typename):
 def export_value(obj, key, value):
     # export and _asdict are not classmethods
     if hasattr(value, "ENTRY_POINT_ORIG_LABEL") and hasattr(value, "config"):
-        obj[key] = {
-            "plugin": value.ENTRY_POINT_ORIG_LABEL,
-        }
+        obj[key] = {"plugin": value.ENTRY_POINT_ORIG_LABEL}
         export_value(obj[key], "config", value.config)
     elif inspect.isclass(value):
         obj[key] = value.__qualname__

docs/usage/index.rst

@@ -11,3 +11,4 @@ The following are some example use cases of DFFML.
     dataflows
     mnist
     io
+    webhook/index

docs/usage/webhook/deploy.rst

@@ -0,0 +1,160 @@
+.. _usage_ffmpeg_deploy:
+
+Deploying with the HTTP Service
+===============================
+
+In this tutorial we will deploy a dataflow(ffmpeg dataflow) which converts a video to gif over an HTTP service. We'll
+also see how to deploy the same in a docker container. Finally in :ref:`usage_ffmpeg_deploy_serve`
+we'll setup another HTTP service which waits on GitHub webhooks to rebuilt and deploy the ffmpeg dataflow.
+
+.. note::
+
+    All the code for this example is located under the
+    `examples/ffmpeg <https://github.com/intel/dffml/blob/master/examples/ffmpeg/>`_
+    directory of the DFFML source code.
+
+We'll be using additional plugins from dffml, ``dffml-yaml-config`` and ``dffml-http-service``.
+
+.. code-block:: console
+
+    $ pip install dffml-yaml-config dffml-http-service
+
+Create the Package
+------------------
+
+To create a new operation we first create a new Python package. DFFML has a script to do it for you.
+
+.. code-block:: console
+
+    $ dffml service dev create operations ffmpeg
+    $ cd ffmpeg
+
+Write operations and definitions to convert videos files to gif by calling
+``ffmpeg`` (Make sure you `download and install <https://www.ffmpeg.org/download.html>`_ it).
+The operation accepts bytes (of the video), converts it to gif and outputs it as bytes.
+We will start writing our operation in ``./ffmpeg/operations.py``
+
+**ffmpeg/operations.py**
+
+.. literalinclude:: /../examples/ffmpeg/ffmpeg/operations.py
+
+Add the operation to ``ffmpeg/setup.py``
+
+.. code-block:: python
+
+    common.KWARGS["entry_points"] = {
+        "dffml.operation": [
+            f"convert_to_gif = {common.IMPORT_NAME}.operations:convert_to_gif"
+        ]
+    }
+
+Install the package
+
+.. code-block:: console
+
+    $ pip install -e .
+
+Dataflow and Config files
+-------------------------
+
+**ffmpeg/dataflow.py**
+
+.. literalinclude:: /../examples/ffmpeg/ffmpeg/dataflow.py
+
+.. code-block:: console
+
+    $ mkdir -p deploy/mc/http deploy/df
+    $ dffml service dev export -config yaml ffmpeg.dataflow:DATAFLOW > deploy/df/ffmpeg.yaml
+
+Create the config file for the HTTP service
+in ``deploy/mc/http/ffmpeg.yaml``
+
+.. code-block:: console
+
+    $ cat > ./deploy/mc/http/ffmpeg.yaml <<EOF
+    path: /ffmpeg
+    input_mode: bytes:input_file
+    output_mode: bytes:image/gif:post_input.output_file
+    EOF
+
+- ``input_mode``
+
+    - ``bytes:input_file``
+
+        - We want the input from the request to be treated as bytes with definition ``input_file``.
+
+- ``output_mode``
+
+    - ``bytes:image/gif:post_input.output_file``
+
+        - We want the response (content_type = image/gif) to be bytes taken from  ``results["post_input"]["output_file"]``,
+          where ``results`` is the output of the dataflow.
+
+For more details see `HttpChannelConfig <../../plugins/service/http/dataflow.html#HttpChannelConfig>`__ .
+
+.. _usage_ffmpeg_deploy_serve:
+
+Serving the DataFlow
+--------------------
+
+Serving the dataflow on port 8080
+
+.. code-block:: console
+
+    $ dffml service http server -insecure -mc-config deploy -port 8080
+
+.. warning::
+
+    The ``-insecure`` flag is only being used here to speed up this
+    tutorial. See documentation on HTTP API
+    :doc:`/plugins/service/http/security` for more information.
+
+Now from another terminal, we can send post requests to the dataflow running at this port.
+
+.. code-block:: console
+
+    $ curl -v --request POST --data-binary @input.mp4 http://localhost:8080/ffmpeg -o output.gif
+
+You should replace ``input.mp4`` with path to your video file and ``output.gif`` to where you want the converted gif
+to be output to. An example video is available `here <https://github.com/intel/dffml/raw/master/examples/ffmpeg/tests/input.mp4>`_ .
+
+Deploying via container
+=======================
+
+A ``Dockerfile`` is already generated in ffmpeg folder. We need to modify it to include ``ffmpeg``.
+
+**Dockerfile**
+
+.. literalinclude:: /../examples/ffmpeg/Dockerfile
+
+.. note::
+
+    The run command in the comment section of the Dockerfile will be used to execute
+    the container after receving webhooks, so make sure you change it to your usecase.
+
+For this tutorial we will change it to
+
+.. code-block:: Dockerfile
+
+    # docker run --rm -ti -p 8080:8080 $USER/ffmpeg -mc-config deploy -insecure -log debug
+
+.. note::
+
+    The image built after pulling the contaier will be taged ``USERNAME/REPONAME``, where USERNAME and REPONAME
+    are gathered from the github html url, received in the webhook.
+
+We can run the container and sent a post request to verify that the container is working.
+
+.. code-block:: console
+
+    $ docker build -t $USER/ffmpeg .
+    $ docker run --rm -ti -p 8080:8080 $USER/ffmpeg -mc-config deploy -insecure -log debug
+
+Now in another terminal
+
+.. code-block:: console
+
+    $ curl -v --request POST --data-binary @input.mp4 http://localhost:8080/ffmpeg -o output.gif
+
+Now in :ref:`usage_webhook` we'll setup this container to be automatically redeployed
+whenever we push to the Git repo containing this code.

docs/usage/webhook/index.rst

@@ -0,0 +1,12 @@
+Continuous Deployment of DataFlows
+==================================
+
+This example shows how to deploy your dataflow and to set it up so that it gets redployed
+on receiving webhooks from GitHub.
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Contents:
+
+    deploy
+    webhook

docs/usage/webhook/webhook.rst

@@ -0,0 +1,114 @@
+.. _usage_webhook:
+
+Redeploying on receving GitHub webhook
+======================================
+
+We'll move ``ffmpeg`` to a GitHub repo, and set up a webhook DataFlow such that whenever
+we push to the default branch, the new version is pulled and its docker container is built and run.
+
+Webhook Dataflow
+----------------
+
+We'll be using operations from ``dffml-operations-deploy``, ``dffml-feature-git``, ``dffml-config-yaml``.
+
+.. code-block:: console
+
+    $ pip install dffml-operations-deploy dffml-feature-git dffml-config-yaml
+
+Setup a http server in ``ffmpeg/deploy/webhook``, to receive webhook and redploy ffmpeg
+
+.. code-block:: console
+
+    $ mkdir -p deploy/webhook/df deploy/webhook/mc/http
+    $ cat > /tmp/operations <<EOF
+    get_url_from_payload
+    clone_git_repo
+    check_if_default_branch
+    get_image_tag
+    get_running_containers
+    get_status_running_containers
+    parse_docker_commands
+    docker_build_image
+    restart_running_containers
+    cleanup_git_repo
+    EOF
+    $ dffml dataflow create -config yaml $(cat /tmp/operations) > deploy/webhook/df/webhook.yaml
+
+Config
+
+**deploy/webhook/mc/http/webhook.yaml**
+
+.. code-block:: console
+
+    $ cat > ./deploy/webhook/mc/http/webhook.yaml <<EOF
+    path: /webhook/github
+    output_mode: json
+    input_mode: json:git_payload
+    EOF
+
+Note that the input_mode is ``json:git_payload``, this means that inputs from post request will
+be parsed as JSON and then forwarded to dataflow as the ``git_payload`` definition..
+
+Deploy it in port 8081 as 8080 is being used by ffmpeg http service
+
+.. code-block:: console
+
+    $ dffml service http server -insecure -mc-config deploy/webhook -port 8081
+
+.. note::
+
+    If you're not setting this up on a server directly accessible on the internet,
+    here are two methods of exposing the webhook,
+
+    Using `localhost.run <https://localhost.run>`_
+
+    .. code-block:: console
+
+        $ ssh -R 80:localhost:8081 [email protected]
+
+    .. image:: ./images/localhost_run.png
+
+    Using ngrok
+
+    .. code-block:: console
+
+        $ ~/ngrok http 8081
+
+    .. image:: ./images/ngrok_out.png
+
+Copy paste the output url to ``Payload URL`` in webhook settings of ffmpeg repo.
+
+.. image:: ./images/github_settings.png
+
+Now whenever there's a push to the default branch of the repo, the ffmpeg container
+which is running gets redeployed from the fresh pull. To check this we will modify the
+end time of the conversion from 10 to 12 in ``ffmpeg/operations.py`` by changing
+
+.. code-block:: python
+
+    proc = await asyncio.create_subprocess_exec(
+        "ffmpeg",
+        "-ss",
+        "0.3",
+        "-t",
+        "10",
+        ..
+        ..
+    )
+
+to
+
+.. code-block:: python
+
+    proc = await asyncio.create_subprocess_exec(
+        "ffmpeg",
+        "-ss",
+        "0.3",
+        "-t",
+        "12",
+        ..
+        ..
+    )
+
+on pushing the changes to our repo, the container will be redeployed. To verify this run
+``docker ps`` and check the up time of the container.

examples/ffmpeg/.coveragerc

@@ -0,0 +1,13 @@
+[run]
+source =
+    ffmpeg
+    tests
+branch = True
+
+[report]
+exclude_lines =
+    no cov
+    no qa
+    noqa
+    pragma: no cover
+    if __name__ == .__main__.: