examples: ffmpeg: Use entrypoint loading

- service: http : Use dot seperated path to index dataflow output
- operations: deploy: Operation to verify GitHub secret
- examples: ffmpeg: Transform from package to single file
- docs: usage: webhook: Update docs
- operations: deploy: Use config for secret

Fixes: intel#658

CHANGELOG.md

@@ -45,6 +45,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Support for entrypoint style loading of operations and seed inputs in `dataflow create`.
 - Definition for output of the function that `op` wraps.
 - Expose high level load, run and save functions to noasync.
+- Operation to verify secret for GitHub webhook.
 - Option to modify flow and add config in `dataflow create`.
 ### Changed
 - Update MNIST use case to normalize image arrays.

dffml/df/base.py

@@ -364,7 +364,6 @@ def wrap(func):
                 name_list = [func.__qualname__, "inputs", name]
                 if func.__module__ != "__main__":
                     name_list.insert(0, func.__module__)
-
                 kwargs["inputs"][name] = create_definition(
                     ".".join(name_list), param.annotation
                 )
@@ -377,7 +376,6 @@ def wrap(func):
                 name_list = [func.__qualname__, "outputs", "result"]
                 if func.__module__ != "__main__":
                     name_list.insert(0, func.__module__)
-
                 kwargs["outputs"] = {
                     "result": create_definition(
                         ".".join(name_list), return_type

docs/usage/webhook/deploy.rst

@@ -19,52 +19,32 @@ We'll be using additional plugins from dffml, ``dffml-yaml-config`` and ``dffml-
 
     $ pip install dffml-yaml-config dffml-http-service
 
-Create the Package
-------------------
-
-To create a new operation we first create a new Python package. DFFML has a script to do it for you.
-
-.. code-block:: console
-
-    $ dffml service dev create operations ffmpeg
-    $ cd ffmpeg
-
-Write operations and definitions to convert videos files to gif by calling
-``ffmpeg`` (Make sure you `download and install <https://www.ffmpeg.org/download.html>`_ it).
-The operation accepts bytes (of the video), converts it to gif and outputs it as bytes.
-We will start writing our operation in ``./ffmpeg/operations.py``
-
-**ffmpeg/operations.py**
-
-.. literalinclude:: /../examples/ffmpeg/ffmpeg/operations.py
-
-Add the operation to ``ffmpeg/setup.py``
-
-.. code-block:: python
-
-    common.KWARGS["entry_points"] = {
-        "dffml.operation": [
-            f"convert_to_gif = {common.IMPORT_NAME}.operations:convert_to_gif"
-        ]
-    }
+Writing the function
+--------------------
+We'll first write the function to convert videos files to gif by calling
+``ffmpeg`` (Make sure you `download and install <https://www.ffmpeg.org/download.html>`_ it) in
+``operation.py``. The function accepts bytes (of the video), converts it to gif and outputs it as
+bytes.
 
-Install the package
 
-.. code-block:: console
+**operations.py**
 
-    $ pip install -e .
+.. literalinclude:: /../examples/ffmpeg/operations.py
 
 Dataflow and Config files
 -------------------------
 
-**ffmpeg/dataflow.py**
-
-.. literalinclude:: /../examples/ffmpeg/ffmpeg/dataflow.py
+DFFML can create a dataflow out of our python function.
 
 .. code-block:: console
 
-    $ mkdir -p deploy/mc/http deploy/df
-    $ dffml service dev export -configloader yaml ffmpeg.dataflow:DATAFLOW > deploy/df/ffmpeg.yaml
+    $ dffml dataflow create -config yaml operations:convert_to_gif get_single \
+        -seed 480=operations.convert_to_gif.inputs.resolution \
+        "['operations.convert_to_gif.outputs.result']=get_single_spec" \
+         > deploy/df/ffmpeg.yaml
+
+Here through the seed argument, we set the default resolution to 480 and output
+of the dataflow to result of `convert_to_gif`.
 
 Create the config file for the HTTP service
 in ``deploy/mc/http/ffmpeg.yaml``
@@ -73,21 +53,22 @@ in ``deploy/mc/http/ffmpeg.yaml``
 
     $ cat > ./deploy/mc/http/ffmpeg.yaml <<EOF
     path: /ffmpeg
-    input_mode: bytes:input_file
-    output_mode: bytes:image/gif:post_input.output_file
+    input_mode: bytes:operations.convert_to_gif.inputs.input_file
+    output_mode: bytes:image/gif:post_input."operations.convert_to_gif.outputs.result".output_file
     EOF
 
 - ``input_mode``
 
-    - ``bytes:input_file``
+    - ``bytes:operations.convert_to_gif.inputs.input_file``
+
+        - We want the input from the request to be treated as bytes with definition ``operations.convert_to_gif.inputs.input_file``.
 
-        - We want the input from the request to be treated as bytes with definition ``input_file``.
 
 - ``output_mode``
 
-    - ``bytes:image/gif:post_input.output_file``
+    - ``bytes:image/gif:post_input."operations.convert_to_gif.outputs.result".output_file``
 
-        - We want the response (content_type = image/gif) to be bytes taken from  ``results["post_input"]["output_file"]``,
+        - We want the response (content_type = image/gif) to be bytes taken from  ``results["post_input"]["operations.convert_to_gif.outputs.result"][output_file]``,
           where ``results`` is the output of the dataflow.
 
 For more details see `HttpChannelConfig <../../plugins/service/http/dataflow.html#HttpChannelConfig>`__ .
@@ -116,13 +97,11 @@ Now from another terminal, we can send post requests to the dataflow running at
     $ curl -v --request POST --data-binary @input.mp4 http://localhost:8080/ffmpeg -o output.gif
 
 You should replace ``input.mp4`` with path to your video file and ``output.gif`` to where you want the converted gif
-to be output to. An example video is available `here <https://github.com/intel/dffml/raw/master/examples/ffmpeg/tests/input.mp4>`_ .
+to be output to. An example video is available `here <https://github.com/intel/dffml/raw/master/examples/ffmpeg/input.mp4>`_ .
 
 Deploying via container
 =======================
 
-A ``Dockerfile`` is already generated in ffmpeg folder. We need to modify it to include ``ffmpeg``.
-
 **Dockerfile**
 
 .. literalinclude:: /../examples/ffmpeg/Dockerfile
@@ -132,7 +111,7 @@ A ``Dockerfile`` is already generated in ffmpeg folder. We need to modify it to
     The run command in the comment section of the Dockerfile will be used to execute
     the container after receving webhooks, so make sure you change it to your usecase.
 
-For this tutorial we will change it to
+For this tutorial we will set it to
 
 .. code-block:: Dockerfile
 

docs/usage/webhook/webhook.rst

@@ -21,6 +21,7 @@ Setup a http server in ``ffmpeg/deploy/webhook``, to receive webhook and redploy
 
     $ mkdir -p deploy/webhook/df deploy/webhook/mc/http
     $ cat > /tmp/operations <<EOF
+    check_secret_match
     get_url_from_payload
     clone_git_repo
     check_if_default_branch
@@ -32,7 +33,19 @@ Setup a http server in ``ffmpeg/deploy/webhook``, to receive webhook and redploy
     restart_running_containers
     cleanup_git_repo
     EOF
-    $ dffml dataflow create -configloader yaml $(cat /tmp/operations) > deploy/webhook/df/webhook.yaml
+    $ dffml dataflow create -configloader yaml $(cat /tmp/operations) \
+    -config \
+        ini=check_secret_match.secret.plugin \
+        "./deploy/webhook/secret.ini"=check_secret_match.secret.config.filename \
+    > deploy/webhook/df/webhook.yaml
+
+Through config we specify the dataflow to use ini file plugin and use the ini file
+located at deploy/webhook/secret.ini that contains the secret token, which we’ll
+setup in GitHub.
+
+**deploy/webhook/secret.ini**
+
+.. literalinclude:: /../examples/ffmpeg/deploy/webhook/secret.ini
 
 Config
 
@@ -43,11 +56,12 @@ Config
     $ cat > ./deploy/webhook/mc/http/webhook.yaml <<EOF
     path: /webhook/github
     output_mode: json
-    input_mode: json:git_payload
+    input_mode: bytes:payload
+    forward_headers: webhook_headers
     EOF
 
-Note that the input_mode is ``json:git_payload``, this means that inputs from post request will
-be parsed as JSON and then forwarded to dataflow as the ``git_payload`` definition..
+Note that the input_mode is ``bytes:payload``, this means that inputs from post request will
+be passed as bytes to the dataflow with ``payload`` definition..
 
 Deploy it in port 8081 as 8080 is being used by ffmpeg http service
 
@@ -76,7 +90,8 @@ Deploy it in port 8081 as 8080 is being used by ffmpeg http service
 
     .. image:: ./images/ngrok_out.png
 
-Copy paste the output url to ``Payload URL`` in webhook settings of ffmpeg repo.
+Copy paste the output url to ``Payload URL`` in webhook settings of ffmpeg repo and set
+the secret token.
 
 .. image:: ./images/github_settings.png
 

examples/ffmpeg/deploy/df/ffmpeg.yaml

@@ -1,50 +1,51 @@
 definitions:
-  input_file:
-    name: input_file
-    primitive: bytes
-  output_file:
-    name: output_file
-    primitive: bytes
-  Resolution:
-    name: Resolution
-    primitive: int
   get_single_output:
     name: get_single_output
     primitive: map
   get_single_spec:
     name: get_single_spec
     primitive: array
+  operations.convert_to_gif.inputs.input_file:
+    name: operations.convert_to_gif.inputs.input_file
+    primitive: bytes
+  operations.convert_to_gif.inputs.resolution:
+    name: operations.convert_to_gif.inputs.resolution
+    primitive: int
+  operations.convert_to_gif.outputs.result:
+    name: operations.convert_to_gif.outputs.result
+    primitive: bytes
 flow:
-  convert_to_gif:
+  get_single:
+    inputs:
+      spec:
+      - seed
+  operations:convert_to_gif:
     inputs:
       input_file:
       - seed
       resolution:
       - seed
-  get_single:
-    inputs:
-      spec:
-      - seed
 linked: true
 operations:
-  convert_to_gif:
-    inputs:
-      input_file: input_file
-      resolution: Resolution
-    name: convert_to_gif
-    outputs:
-      output_file: output_file
-    stage: processing
   get_single:
     inputs:
       spec: get_single_spec
     name: get_single
     outputs:
       output: get_single_output
     stage: output
+  operations:convert_to_gif:
+    inputs:
+      input_file: operations.convert_to_gif.inputs.input_file
+      resolution: operations.convert_to_gif.inputs.resolution
+    name: operations:convert_to_gif
+    outputs:
+      result: operations.convert_to_gif.outputs.result
+    stage: processing
 seed:
+- definition: operations.convert_to_gif.inputs.resolution
+  value: 440
 - definition: get_single_spec
   value:
-  - output_file
-- definition: Resolution
-  value: 480
+  - operations.convert_to_gif.outputs.result
+

examples/ffmpeg/deploy/mc/http/ffmpeg.yaml

@@ -1,3 +1,3 @@
 path: /ffmpeg
-input_mode: bytes:input_file
-output_mode: bytes:image/gif:post_input.output_file
+input_mode: bytes:operations.convert_to_gif.inputs.input_file
+output_mode: bytes:image/gif:post_input."operations.convert_to_gif.outputs.result".output_file

examples/ffmpeg/deploy/webhook/df/webhook.yaml

@@ -1,3 +1,9 @@
+configs:
+  check_secret_match:
+    secret:
+      config:
+        filename: ./deploy/webhook/secret.ini
+      plugin: ini
 definitions:
   URL:
     name: URL
@@ -8,12 +14,15 @@ definitions:
   docker_image_tag:
     name: docker_image_tag
     primitive: str
+  docker_restarted_containers:
+    name: docker_restarted_containers
+    primitive: str
   docker_running_containers:
     name: docker_running_containers
     primitive: List[str]
   git_payload:
     name: git_payload
-    primitive: Dict[Any]
+    primitive: Dict[str,Any]
   git_repository:
     lock: true
     name: git_repository
@@ -34,14 +43,30 @@ definitions:
   is_image_built:
     name: is_image_built
     primitive: bool
+  payload:
+    name: payload
+    primitive: bytes
   valid_git_repository_URL:
     name: valid_git_repository_URL
     primitive: boolean
+  webhook_headers:
+    name: webhook_headers
+    primitive: Dict[str,Any]
 flow:
   check_if_default_branch:
     inputs:
       payload:
+      - check_secret_match: git_payload
+  check_secret_match:
+    inputs:
+      body:
+      - seed
+      headers:
       - seed
+  cleanup_git_repo:
+    inputs:
+      repo:
+      - clone_git_repo: repo
   clone_git_repo:
     conditions:
     - seed
@@ -58,7 +83,7 @@ flow:
   get_image_tag:
     inputs:
       payload:
-      - seed
+      - check_secret_match: git_payload
   get_running_containers:
     inputs:
       tag:
@@ -70,7 +95,7 @@ flow:
   get_url_from_payload:
     inputs:
       payload:
-      - seed
+      - check_secret_match: git_payload
   parse_docker_commands:
     inputs:
       image_tag:
@@ -94,6 +119,20 @@ operations:
     outputs:
       is_default_branch: is_default_branch
     stage: processing
+  check_secret_match:
+    inputs:
+      body: payload
+      headers: webhook_headers
+    name: check_secret_match
+    outputs:
+      git_payload: git_payload
+    stage: processing
+  cleanup_git_repo:
+    inputs:
+      repo: git_repository
+    name: cleanup_git_repo
+    outputs: {}
+    stage: cleanup
   clone_git_repo:
     conditions:
     - valid_git_repository_URL
@@ -156,6 +195,7 @@ operations:
       containers: docker_running_containers
       docker_commands: docker_commands
     name: restart_running_containers
-    outputs: {}
+    outputs:
+      containers: docker_restarted_containers
     stage: processing
 

examples/ffmpeg/deploy/webhook/mc/http/webhook.yaml

@@ -1,3 +1,4 @@
 path: /webhook/github
 output_mode: json
-input_mode: json:git_payload
+input_mode: bytes:payload
+forward_headers: webhook_headers