Add discord integration (scrapinghub#348)

* Add Discord integration

Co-authored-by: Ana Paula Gomes <[email protected]>
Co-authored-by: Ana Paula Gomes <[email protected]>
Co-authored-by: Roy <[email protected]>

CONTRIBUTING.rst

@@ -82,7 +82,7 @@ Ready to contribute? Here's how to set up `spidermon` for local development.
    including testing other Python versions with tox::
 
     $ pip install -r requirements.txt
-    $ pip install -r requirements-test.txt
+    $ pip install -r requirements-docs.txt
     $ tox
 
 #. Make sure that your code is correctly formatted using `black`_ . No code will

docs/source/actions/discord-action.rst

@@ -0,0 +1,59 @@
+Discord action
+===============
+
+This action allows you to send custom messages to a `Discord`_ channel
+using a bot when your monitor suites finish their execution.
+
+To use this action you need to provide the `Discord webhook URL`_ in your ``settings.py`` file as follows:
+
+.. code-block:: python
+
+    # settings.py
+    SPIDERMON_DISCORD_WEBHOOK_URL = "<DISCORD_WEBHOOK_URL>"
+
+A notification will look like the following:
+
+.. image:: /_static/discord_notification.png
+   :scale: 50 %
+   :alt: Discord Notification
+
+Follow :ref:`these steps <configuring-discord-bot>` to configure your Discord bot.
+
+The following settings are the minimum needed to make this action work:
+
+SPIDERMON_DISCORD_WEBHOOK_URL
+-----------------------------
+
+`Webhook URL` of your bot.
+
+.. warning::
+
+    Be careful when using bot webhooks URL in Spidermon. Do not publish them in public code repositories.
+
+Other settings available:
+
+.. _SPIDERMON_DISCORD_FAKE:
+
+_SPIDERMON_DISCORD_FAKE
+-----------------------
+
+Default: ``False``
+
+If set `True`, the Discord message content will be in the logs but nothing will be sent.
+
+.. _SPIDERMON_DISCORD_MESSAGE:
+
+SPIDERMON_DISCORD_MESSAGE
+-------------------------
+
+The message to be sent, it supports Jinja2 template formatting.
+
+.. _SPIDERMON_DISCORD_MESSAGE_TEMPLATE:
+
+SPIDERMON_DISCORD_MESSAGE_TEMPLATE
+----------------------------------
+
+Path to a Jinja2 template file to format messages sent by the Discord Action.
+
+.. _`Discord`: https://discord.com/
+.. _`Discord webhook URL`: https://discord.com/developers/docs/resources/webhook

docs/source/actions/index.rst

@@ -14,6 +14,7 @@ You can define your own actions or use one of the existing built-in actions.
    email-action
    slack-action
    telegram-action
+   discord-action
    job-tags-action
    file-report-action
    sentry-action

docs/source/getting-started.rst

@@ -233,7 +233,7 @@ Telegram notifications
 ----------------------
 
 Here we will configure a built-in Spidermon action that sends a pre-defined message to
-a Telegram use, group or channel using a bot when a monitor fails.
+a Telegram user, group or channel using a bot when a monitor fails.
 
 .. code-block:: python
 
@@ -268,6 +268,48 @@ If a monitor fails, the recipients provided will receive a message in Telegram:
    :scale: 50 %
    :alt: Telegram Notification
 
+Discord notifications
+---------------------
+
+Here we will configure a built-in Spidermon action that sends a pre-defined message to
+a Discord channel using a bot when a monitor fails.
+
+.. code-block:: python
+
+    # tutorial/monitors.py
+    from spidermon.contrib.actions.discord.notifiers import SendDiscordMessageSpiderFinished
+
+    # (...your monitors code...)
+
+    class SpiderCloseMonitorSuite(MonitorSuite):
+        monitors = [
+            ItemCountMonitor,
+        ]
+
+        monitors_failed_actions = [
+            SendDiscordMessageSpiderFinished,
+        ]
+
+After enabling the action, you need to provide the `Discord Webhook URL`_. You can
+learn more about how to create and configure a webhook :ref:`configuring-discord-bot`.
+Later, fill the required information in your `settings.py` as follows:
+
+.. code-block:: python
+
+    # tutorial/settings.py
+    (...)
+    SPIDERMON_DISCORD_WEBHOOK_URL = "<DISCORD_WEBHOOK_URL>"
+
+If a monitor fails, the recipients provided will receive a message in Discord:
+
+.. image:: /_static/discord_notification.png
+   :scale: 50 %
+   :alt: Discord Notification
+
+The target channel is configured during the webhook creation on Discord.
+
+In case you want to see the messages only in the terminal, set as `True` the environment
+variable `SPIDERMON_DISCORD_FAKE`.
 
 Item validation
 ---------------
@@ -438,3 +480,4 @@ The resulted item will look like this:
 .. _`Scrapy project`: https://doc.scrapy.org/en/latest/intro/tutorial.html?#creating-a-project
 .. _`Slack credentials`: https://api.slack.com/docs/token-types
 .. _`Telegram bot token`: https://core.telegram.org/bots
+.. _`Discord Webhook URL`: https://discord.com/developers/docs/resources/webhook

docs/source/howto/configuring-discord-for-spidermon.rst

@@ -0,0 +1,26 @@
+.. _configuring-discord-bot:
+
+How do I configure a Discord bot for Spidermon?
+===============================================
+
+What are bots?
+--------------
+
+A bot is a type of Discord user designed to interact with users via conversation.
+
+To work with :doc:`Discord Actions </actions/discord-action>`,
+you will need a Discord webhook which would send notifications to Discord from Spidermon.
+
+Steps
+-----
+
+#. Create a Discord webhook <https://discord.com/developers/docs/resources/webhook>`_.
+
+#. Once your webhook is created, you will receive its URL. This is what we use for ``SPIDERMON_DISCORD_WEBHOOK_URL``.
+
+#. Add your Discord bot credential to your Scrapy project's settings. That's it.
+
+.. code-block:: python
+
+    # settings.py
+    SPIDERMON_DISCORD_WEBHOOK_URL = "DISCORD_WEBHOOK_URL"

docs/source/howto/configuring-telegram-for-spidermon.rst

@@ -9,7 +9,7 @@ What are bots?
 A bot is a type of Telegram user designed to interact with users via conversation.
 
 To work with :doc:`Telegram Actions </actions/telegram-action>`, you will need a Telegram bot which would
-send notificationsto Telegram from Spidermon.
+send notifications to Telegram from Spidermon.
 
 Steps
 -----

docs/source/howto/index.rst

@@ -7,4 +7,5 @@
    required-fields-coverage-validation
    configuring-slack-for-spidermon
    configuring-telegram-for-spidermon
+   configuring-discord-for-spidermon
    stats-collection

docs/source/index.rst

@@ -18,7 +18,7 @@ following features:
   * Schematics: `<https://github.com/schematics/schematics>`_
 * It allows you to define conditions that should trigger an alert based on
   Scrapy stats.
-* It supports notifications via email, Slack and Telegram.
+* It supports notifications via email, Slack, Telegram and Discord.
 * It can generate custom reports.
 
 Contents

spidermon/contrib/actions/discord/__init__.py

@@ -0,0 +1,76 @@
+from __future__ import absolute_import
+
+import logging
+
+import requests
+from spidermon.contrib.actions.templates import ActionWithTemplates
+from spidermon.exceptions import NotConfigured
+
+logger = logging.getLogger(__name__)
+
+
+class DiscordMessageManager:
+    sender_token = None
+
+    def __init__(self, webhook_url, fake=False):
+        if not webhook_url:
+            raise NotConfigured("You must provide a discord webhook URL.")
+        self.webhook_url = webhook_url
+        self.fake = fake
+
+    def send_message(self, text):
+        if self.fake:
+            logger.info(text)
+            return
+
+        body = {"content": text}
+        response = requests.post(self.webhook_url, json=body)
+        response.raise_for_status()
+
+        if not response.ok:
+            logger.error(
+                f"Failed to send message. Discord API error: {response.reason}"
+            )
+
+
+class SendDiscordMessage(ActionWithTemplates):
+    webhook_url = None
+    message = None
+    message_template = "discord/default/message.jinja"
+    fake = False
+
+    def __init__(
+        self,
+        webhook_url=None,
+        message=None,
+        message_template=None,
+        fake=None,
+    ):
+        super(SendDiscordMessage, self).__init__()
+
+        self.fake = fake or self.fake
+        self.manager = DiscordMessageManager(
+            webhook_url or self.webhook_url, fake=self.fake
+        )
+        self.message = message or self.message
+        self.message_template = message_template or self.message_template
+
+    @classmethod
+    def from_crawler_kwargs(cls, crawler):
+        return {
+            "webhook_url": crawler.settings.get("SPIDERMON_DISCORD_WEBHOOK_URL"),
+            "message": crawler.settings.get("SPIDERMON_DISCORD_MESSAGE"),
+            "message_template": crawler.settings.get(
+                "SPIDERMON_DISCORD_MESSAGE_TEMPLATE"
+            ),
+            "fake": crawler.settings.getbool("SPIDERMON_DISCORD_FAKE"),
+        }
+
+    def run_action(self):
+        self.manager.send_message(self.get_message())
+
+    def get_message(self):
+        if self.message:
+            return self.render_text_template(self.message)
+        else:
+            return self.render_template(self.message_template)

spidermon/contrib/actions/discord/notifiers.py

@@ -0,0 +1,53 @@
+from __future__ import absolute_import
+
+from . import SendDiscordMessage
+
+
+class SendDiscordMessageSpiderStarted(SendDiscordMessage):
+    message_template = "discord/spider/notifier/start/message.jinja"
+
+
+class SendDiscordMessageSpiderFinished(SendDiscordMessage):
+    message_template = "discord/spider/notifier/finish/message.jinja"
+    include_ok_messages = False
+    include_error_messages = True
+
+    def __init__(
+        self, include_ok_messages=None, include_error_messages=None, *args, **kwargs
+    ):
+        super(SendDiscordMessageSpiderFinished, self).__init__(*args, **kwargs)
+        self.include_ok_messages = include_ok_messages or self.include_ok_messages
+        self.include_error_messages = (
+            include_error_messages or self.include_error_messages
+        )
+
+    @classmethod
+    def from_crawler_kwargs(cls, crawler):
+        kwargs = super(SendDiscordMessageSpiderFinished, cls).from_crawler_kwargs(
+            crawler
+        )
+        kwargs.update(
+            {
+                "include_ok_messages": crawler.settings.get(
+                    "SPIDERMON_DISCORD_NOTIFIER_INCLUDE_OK_MESSAGES"
+                ),
+                "include_error_messages": crawler.settings.get(
+                    "SPIDERMON_DISCORD_NOTIFIER_INCLUDE_ERROR_MESSAGES"
+                ),
+            }
+        )
+        return kwargs
+
+    def get_template_context(self):
+        context = super(SendDiscordMessageSpiderFinished, self).get_template_context()
+        context.update(
+            {
+                "include_ok_messages": self.include_ok_messages,
+                "include_error_messages": self.include_error_messages,
+            }
+        )
+        return context
+
+
+class SendDiscordMessageSpiderRunning(SendDiscordMessageSpiderFinished):
+    message_template = "discord/spider/notifier/periodic/message.jinja"

spidermon/contrib/actions/discord/templates/discord/default/message.jinja

@@ -0,0 +1 @@
+This is the default message...

spidermon/contrib/actions/discord/templates/discord/spider/notifier/finish/message.jinja

@@ -0,0 +1,16 @@
+{% import 'discord/spider/notifier/macros.jinja' as renderer with context %}
+{% if monitors_failed %}
+{{ "💀" }} "`{{ renderer.render_spider_name() }}`" spider finished with errors! {{ renderer.render_url() }} _(errors={{ result.monitors_failed_results|length }})_ {{ "\n" }}
+{% else %}
+{{ "🎉"}} "`{{ renderer.render_spider_name() }}`" spider finished! {{ renderer.render_url() }} {{ "\n" }}
+{% endif %}
+{%- if include_ok_messages and monitors_passed -%}
+```
+{{ renderer.render_passed() -}}
+```
+{% endif %}
+{%- if include_error_messages and monitors_failed -%}
+```
+{{ renderer.render_failed() -}}
+```
+{% endif %}

spidermon/contrib/actions/discord/templates/discord/spider/notifier/macros.jinja

@@ -0,0 +1,19 @@
+{% macro render_result(emoji, result) %}
+{{ emoji }}  {{ result.monitor.name }}
+{% endmacro %}
+
+{% macro render_results(emoji, results) %}
+{% for result in results %} {{- render_result(emoji, result) -}} {% endfor %}
+{% endmacro %}
+
+{% macro render_passed() %}
+{{- render_results("✔️", result.monitors_passed_results) -}}
+{% endmacro %}
+
+{% macro render_failed() %}
+{{ render_results("❌", result.monitors_failed_results) }}
+{% endmacro %}
+
+{% macro render_job_url() %}{% if data.job %} / [view job in Scrapy Cloud](https://app.scrapinghub.com/p/{{ data.job.key }}){% endif %}{% endmacro %}
+{% macro render_url() %}{{ render_job_url() }}{% endmacro %}
+{% macro render_spider_name() %}{% if data.spider %}{{ data.spider.name }}{% elif data.job %}{{ data.job.metadata['spider'] }}{% else %}??{% endif %}{% endmacro %}

spidermon/contrib/actions/discord/templates/discord/spider/notifier/periodic/message.jinja

@@ -0,0 +1,12 @@
+{% import 'discord/spider/notifier/macros.jinja' as renderer with context %}
+{{ "💀" }} "`{{ renderer.render_spider_name() }}`" Detected errors on running spider!{{ renderer.render_job_url() }} _(errors={{ result.monitors_failed_results|length }})_ {{ "\n\n" }}
+{%- if include_ok_messages and monitors_passed -%}
+```
+{{ renderer.render_passed() -}}
+```
+{% endif %}
+{%- if include_error_messages and monitors_failed -%}
+```
+{{ renderer.render_failed() -}}
+```
+{% endif %}

spidermon/contrib/actions/discord/templates/discord/spider/notifier/start/message.jinja

@@ -0,0 +1,2 @@
+{% import 'discord/spider/notifier/macros.jinja' as renderer with context %}
+{{ "🕒" }} "`{{ renderer.render_spider_name() }}`" *spider started!* {{ renderer.render_job_url() }}