Merge branch 'master' into purge_models

* master: (26 commits)
  Remove unused function - this is a base class which should never be used and instantiated directly.
  Also teach it to handle null (none) types.
  Also teach pylint about other possible types.
  Update pylint API model plugin so it correctly sets property type for dicts and lists.
  Use pylint 1.4.4.
  Cleaning up test alias in `chatops` pack
  Prettify
  Removing skip_message
  Pin astroid pylint depenency to a working version. New version which was just released breaks everything.
  Fixed more links
  New endpoint
  fixed a minor typo
  ChatOps: Avoid failing the whole action chain when result output is disabled
  Return empty string if result output is disabled
  Fix typo
  Update CHANGELOG
  Allow namespacing in jinja templating for parameters and execution results in notification
  Add docs for jinja support in notifications
  Improve example
  Some refactor
  ...

Conflicts:
	CHANGELOG.rst

CHANGELOG.rst

@@ -36,6 +36,7 @@ in development
   in default etc. (improvement)
 * Fix action chain so it doesn't end up in an infinite loop if an action which is part of the chain
   is canceled. (bug fix)
+* Allow jinja templating to be used in ``message`` and ``data`` field for notifications.(new feature)
 * Add tools for purging executions (also, liveactions with it) and trigger instances older than
   certain UTC timestamp from the db in bulk.
 

contrib/chatops/actions/format_result.py

@@ -36,9 +36,9 @@ def run(self, execution):
 
             result = getattr(alias, 'result', None)
             if result:
-                enabled = result.get('enabled', True)
-
-                if enabled and 'format' in alias.result:
+                if not result.get('enabled', True):
+                    raise Exception("Output of this template is disabled.")
+                if 'format' in alias.result:
                     template = alias.result['format']
 
         return self.jinja.from_string(template).render(context)

contrib/examples/actions/local.yaml

@@ -7,7 +7,7 @@ notify:
   on-complete:
     routes:
     - slack
-    message: '"@channel: Action succeeded."'
+    message: '"@channel: Action run by {{action_context.user}} succeeded. Cmd run was {{cmd}}."'
 parameters:
   cmd:
     description: Arbitrary Linux command to be executed on the remote host(s).

docs/source/chatops/notifications.rst

@@ -56,7 +56,6 @@ is presented below.
         - slack
         message: '"@channel: Woohoo!"'
 
-The message doesn't support jinja templating yet. This support will be added in the future.
 Also, when the notification triggers are sent out, the message supplied along with a ``data``
 field containing the results of the execution are sent out. The rule can use these two fields -
 ``message`` and ``data`` - and send it out as part of the action.
@@ -90,11 +89,34 @@ with when you setup chatops. An example is below:
 As you can see, this rule is setup for notification route ``slack``. The action section shows
 that ``slack.post_message`` is the one what would be kicked off. We are skipping the ``data`` part
 of the trigger for brevity. If you had a slack action that also consumed some data as JSON string,
-you could pass ``data: "{{data}}"`` as a parameter. Again, selecting specific fields from the
-output (via jinja) is not supported yet.
+you could pass ``data: "{{data}}"`` as a parameter.
+
+Jinja templating in ``message`` and ``data``
+--------------------------------------------
+
+As of release 1.2, Jinja templating is supported for both ``message`` and ``data``. The jinja
+context available for use are parameters of action and runner ({{action_parameters.cmd}}),
+keys in execution results ({{action_results.stdout}}, {{action_results.stderr}} for example),
+anything in action context ({{action_context.user}})
+and anything in key-value store ({{system.foo}}). Some examples are shown below.
+
+::
+
+  on-success:
+    routes:
+      - slack
+    message: '"@channel: Woohoo!". Action run by user {{action_context.user}} succeeded.'
+
+  on-success:
+    routes:
+      - email
+    message: '"@channel: Woohoo!". Action run by user {{action_context.user}} succeeded.'
+    data:
+      cmd: "{{action_parameters.cmd}}"
+      stdout: "{{action_results.stdout}}"
 
 How do I setup notifications in action chain?
------------st2api/st2api/controllers/v1/ruleviews.py----------------------------------
+---------------------------------------------
 
 The procedure here is the same if you want the same notify for all tasks in the chain. You would
 register an action meta with notify section. For example:
@@ -129,7 +151,7 @@ How do I setup different notifications for different tasks in the chain?
 
 The ``notify`` subsection is the same format as you have seen in examples above. You basically
 place the subsection in action chain tasks. If you have a notify section for the action meta
-and there is a notify section in the task, the task one will override. The relvant section of chain
+and there is a notify section in the task, the task one will override. The relevant section of chain
 action with task notify is shown below.
 
 ::

docs/source/install/all_in_one.rst

@@ -8,7 +8,7 @@ That's OK! You're busy, we get it. How do you just get started? Get your own box
 
 ::
 
-   curl -sSL http://stackstorm.com/install.sh | sudo sh
+   curl -sSL https://install.stackstorm.com/ | sudo sh
 
 If you're installing *Enterprise Edition*, enter your license key when prompted. With no enterprise key, StackStorm community will be installed.
 
@@ -39,7 +39,7 @@ Optionally, you can also provide an Enterprise License Key and get access to the
 
 * |st2| FLOW - HTML5 based Graphical Workflow editor. Use this to visualize, edit, and share workflows.
 * |st2| Role Based Access Control. Apply fine-grained controls to actions and rules to fit into the most complex of environments.
-* |st2| LDAP Authentication Support. Integrate with your existing authentication directory.G
+* |st2| LDAP Authentication Support. Integrate with your existing authentication directory.
 
 Supported Platforms
 ###################
@@ -92,7 +92,7 @@ Bring Your Own Box
 
 ::
 
-   curl -sSL http://stackstorm.com/install.sh | sudo sh
+   curl -sSL https://install.stackstorm.com/ | sudo sh
 
 You will need elevated privileges in order to run this script. This will boot up a fresh |st2| installation along with the Mistral workflow engine on Ubuntu 14.04 LTS. While loading, some console output in red is expected and can be safely ignored. Once completed, you will see the following console output.
 
@@ -313,7 +313,7 @@ The answers file is formatted in standard YAML. Below, we will discuss the vario
 
 ::
 
-   curl -sSL http://stackstorm.com/install.sh | sudo sh -s -- -a <answers_file>.yaml
+   curl -sSL https://install.stackstorm.com/ | sudo sh -s -- -a <answers_file>.yaml
 
 
 If you have already installed using this method, you can find and update your answers file at `/opt/puppet/hieradata/answers.yaml`

docs/source/install/index.rst

@@ -14,7 +14,7 @@ run the bootstrap script:
 
 ::
 
-    curl -sSL http://stackstorm.com/install.sh | sudo sh
+    curl -sSL https://install.stackstorm.com/ | sudo sh
 
 You will need elevated privileges in order to run this script. This will download and deploy the stable release of |st2| (currently |release|). Check out :doc:`all_in_one` to learn how to provide answer file, get latest development version, and other details. Installation should take about 20 min. *Yes, we are working on making it faster!* Grab a coffee and watch :doc:`/video` while it is being installed. Once completed, you will see the following console output. Read it :)
 

docs/source/install/vagrant.rst

@@ -79,5 +79,5 @@ Below is an an example of a Vagrantfile capable of loading StackStorm. This mini
         v.memory = 1024
       end
       config.vm.provision "shell",
-          inline: "curl -sSL http://stackstorm.com/install.sh | sudo su"
+          inline: "curl -sSL https://install.stackstorm.com/ | sudo su"
     end

pylint_plugins/api_models.py

@@ -21,7 +21,10 @@
 constructor.
 """
 
+import six
+
 from astroid import MANAGER
+from astroid import nodes
 from astroid import scoped_nodes
 
 # A list of class names for which we want to skip the checks
@@ -53,11 +56,33 @@ def transform(cls):
             # Not a class we are interested in
             return
 
-        properties = schema.get('properties', {}).keys()
-
-        for property_name in properties:
+        properties = schema.get('properties', {})
+        for property_name, property_data in six.iteritems(properties):
             property_name = property_name.replace('-', '_')  # Note: We do the same in Python code
-            cls.locals[property_name] = [scoped_nodes.Class(property_name, None)]
+            property_type = property_data.get('type', None)
+
+            if isinstance(property_type, (list, tuple)):
+                # Hack for attributes with multiple types (e.g. string, null)
+                property_type = property_type[0]
+
+            if property_type == 'object':
+                node = nodes.Dict()
+            elif property_type == 'array':
+                node = nodes.List()
+            elif property_type == 'integer':
+                node = scoped_nodes.builtin_lookup('int')[1][0]
+            elif property_type == 'number':
+                node = scoped_nodes.builtin_lookup('float')[1][0]
+            elif property_type == 'string':
+                node = scoped_nodes.builtin_lookup('str')[1][0]
+            elif property_type == 'boolean':
+                node = scoped_nodes.builtin_lookup('bool')[1][0]
+            elif property_type == 'null':
+                node = scoped_nodes.builtin_lookup('None')[1][0]
+            else:
+                node = scoped_nodes.Class(property_name, None)
+
+            cls.locals[property_name] = [node]
 
 
 MANAGER.register_transform(scoped_nodes.Class, transform)

st2actions/st2actions/notifier/notifier.py

@@ -34,6 +34,13 @@
 from st2common.transport import consumers, liveaction, publishers
 from st2common.transport import utils as transport_utils
 from st2common.transport.reactor import TriggerDispatcher
+from st2common.util import isotime
+from st2common.util import jinja as jinja_utils
+from st2common.constants.action import ACTION_CONTEXT_KV_PREFIX
+from st2common.constants.action import ACTION_PARAMETERS_KV_PREFIX
+from st2common.constants.action import ACTION_RESULTS_KV_PREFIX
+from st2common.constants.system import SYSTEM_KV_PREFIX
+from st2common.services.keyvalues import KeyValueLookup
 
 __all__ = [
     'Notifier',
@@ -56,6 +63,8 @@ class Notifier(consumers.MessageHandler):
 
     def __init__(self, connection, queues, trigger_dispatcher=None):
         super(Notifier, self).__init__(connection, queues)
+        if not trigger_dispatcher:
+            trigger_dispatcher = TriggerDispatcher(LOG)
         self._trigger_dispatcher = trigger_dispatcher
         self._notify_trigger = ResourceReference.to_string_reference(
             pack=NOTIFY_TRIGGER_TYPE['pack'],
@@ -74,61 +83,76 @@ def process(self, liveaction):
                       (live_action_id), extra=extra)
             return
 
-        execution_id = self._get_execution_id_for_liveaction(liveaction)
+        execution = self._get_execution_for_liveaction(liveaction)
 
-        if not execution_id:
+        if not execution:
             LOG.exception('Execution object corresponding to LiveAction %s not found.',
                           live_action_id, extra=extra)
             return None
 
-        self._apply_post_run_policies(liveaction=liveaction, execution_id=execution_id)
+        self._apply_post_run_policies(liveaction=liveaction)
 
         if liveaction.notify is not None:
-            self._post_notify_triggers(liveaction=liveaction, execution_id=execution_id)
+            self._post_notify_triggers(liveaction=liveaction, execution=execution)
 
-        self._post_generic_trigger(liveaction=liveaction, execution_id=execution_id)
+        self._post_generic_trigger(liveaction=liveaction, execution=execution)
 
-    def _get_execution_id_for_liveaction(self, liveaction):
+    def _get_execution_for_liveaction(self, liveaction):
         execution = ActionExecution.get(liveaction__id=str(liveaction.id))
 
         if not execution:
             return None
 
-        return str(execution.id)
+        return execution
 
-    def _post_notify_triggers(self, liveaction=None, execution_id=None):
+    def _post_notify_triggers(self, liveaction=None, execution=None):
         notify = getattr(liveaction, 'notify', None)
 
         if not notify:
             return
 
         if notify.on_complete:
             self._post_notify_subsection_triggers(
-                liveaction=liveaction, execution_id=execution_id,
+                liveaction=liveaction, execution=execution,
                 notify_subsection=notify.on_complete,
                 default_message_suffix='completed.')
         if liveaction.status == LIVEACTION_STATUS_SUCCEEDED and notify.on_success:
             self._post_notify_subsection_triggers(
-                liveaction=liveaction, execution_id=execution_id,
+                liveaction=liveaction, execution=execution,
                 notify_subsection=notify.on_success,
                 default_message_suffix='succeeded.')
         if liveaction.status in LIVEACTION_FAILED_STATES and notify.on_failure:
             self._post_notify_subsection_triggers(
-                liveaction=liveaction, execution_id=execution_id,
+                liveaction=liveaction, execution=execution,
                 notify_subsection=notify.on_failure,
                 default_message_suffix='failed.')
 
-    def _post_notify_subsection_triggers(self, liveaction=None, execution_id=None,
+    def _post_notify_subsection_triggers(self, liveaction=None, execution=None,
                                          notify_subsection=None,
                                          default_message_suffix=None):
         routes = (getattr(notify_subsection, 'routes') or
                   getattr(notify_subsection, 'channels', None))
 
+        execution_id = str(execution.id)
+
         if routes and len(routes) >= 1:
             payload = {}
             message = notify_subsection.message or (
                 'Action ' + liveaction.action + ' ' + default_message_suffix)
-            data = notify_subsection.data or {}  # XXX: Handle Jinja
+            data = notify_subsection.data or {}
+
+            jinja_context = self._build_jinja_context(liveaction=liveaction, execution=execution)
+
+            try:
+                message = self._transform_message(message=message,
+                                                  context=jinja_context)
+            except:
+                LOG.exception('Failed (Jinja) transforming `message`.')
+
+            try:
+                data = self._transform_data(data=data, context=jinja_context)
+            except:
+                LOG.exception('Failed (Jinja) transforming `data`.')
 
             # At this point convert result to a string. This restricts the rulesengines
             # ability to introspect the result. On the other handle atleast a json usable
@@ -142,8 +166,8 @@ def _post_notify_subsection_triggers(self, liveaction=None, execution_id=None,
             payload['data'] = data
             payload['execution_id'] = execution_id
             payload['status'] = liveaction.status
-            payload['start_timestamp'] = str(liveaction.start_timestamp)
-            payload['end_timestamp'] = str(liveaction.end_timestamp)
+            payload['start_timestamp'] = isotime.format(liveaction.start_timestamp)
+            payload['end_timestamp'] = isotime.format(liveaction.end_timestamp)
             payload['action_ref'] = liveaction.action
             payload['runner_ref'] = self._get_runner_ref(liveaction.action)
 
@@ -165,6 +189,22 @@ def _post_notify_subsection_triggers(self, liveaction=None, execution_id=None,
             if len(failed_routes) > 0:
                 raise Exception('Failed notifications to routes: %s' % ', '.join(failed_routes))
 
+    def _build_jinja_context(self, liveaction, execution):
+        context = {SYSTEM_KV_PREFIX: KeyValueLookup()}
+        context.update({ACTION_PARAMETERS_KV_PREFIX: liveaction.parameters})
+        context.update({ACTION_CONTEXT_KV_PREFIX: liveaction.context})
+        context.update({ACTION_RESULTS_KV_PREFIX: execution.result})
+        return context
+
+    def _transform_message(self, message, context=None):
+        mapping = {'message': message}
+        context = context or {}
+        return (jinja_utils.render_values(mapping=mapping, context=context)).get('message',
+                                                                                 message)
+
+    def _transform_data(self, data, context=None):
+        return jinja_utils.render_values(mapping=data, context=context)
+
     def _get_trace_context(self, execution_id):
         trace_db = trace_service.get_trace_db_by_action_execution(
             action_execution_id=execution_id)
@@ -174,11 +214,12 @@ def _get_trace_context(self, execution_id):
         # it shall be created downstream. Sure this is impl leakage of some sort.
         return None
 
-    def _post_generic_trigger(self, liveaction=None, execution_id=None):
+    def _post_generic_trigger(self, liveaction=None, execution=None):
         if not ACTION_SENSOR_ENABLED:
             LOG.debug('Action trigger is disabled, skipping trigger dispatch...')
             return
 
+        execution_id = str(execution.id)
         payload = {'execution_id': execution_id,
                    'status': liveaction.status,
                    'start_timestamp': str(liveaction.start_timestamp),
@@ -197,7 +238,7 @@ def _post_generic_trigger(self, liveaction=None, execution_id=None):
         self._trigger_dispatcher.dispatch(self._action_trigger, payload=payload,
                                           trace_context=trace_context)
 
-    def _apply_post_run_policies(self, liveaction=None, execution_id=None):
+    def _apply_post_run_policies(self, liveaction=None):
         # Apply policies defined for the action.
         policy_dbs = Policy.query(resource_ref=liveaction.action)
         LOG.debug('Applying %s post_run policies' % (len(policy_dbs)))

st2actions/st2actions/runners/actionchainrunner.py

@@ -22,7 +22,7 @@
 
 from st2actions.runners import ActionRunner
 from st2common import log as logging
-from st2common.constants.action import ACTION_KV_PREFIX
+from st2common.constants.action import ACTION_CONTEXT_KV_PREFIX
 from st2common.constants.action import LIVEACTION_STATUS_SUCCEEDED
 from st2common.constants.action import LIVEACTION_STATUS_TIMED_OUT
 from st2common.constants.action import LIVEACTION_STATUS_FAILED
@@ -472,7 +472,7 @@ def _resolve_params(action_node, original_parameters, results, chain_vars, chain
         context.update(chain_vars)
         context.update({RESULTS_KEY: results})
         context.update({SYSTEM_KV_PREFIX: KeyValueLookup()})
-        context.update({ACTION_KV_PREFIX: chain_context})
+        context.update({ACTION_CONTEXT_KV_PREFIX: chain_context})
         try:
             rendered_params = jinja_utils.render_values(mapping=action_node.params,
                                                         context=context)