Merge pull request kapicorp#33 from deepmind/python3

Moving to python3

.travis.yml

@@ -1,7 +1,7 @@
 language: python
 
 python:
-  - "2.7"
+  - "3.6"
 
 branches:
   only:
@@ -12,13 +12,14 @@ notifications:
     - [email protected]
     - [email protected]
     - [email protected]
+    - [email protected]
   email:
     on_success: change
     on_failure: always
 
 before_install:
   - sudo apt-get -qq update
-  - sudo apt-get install -y gnupg2
+  - sudo apt-get install -y gnupg2 git
 
 # command to install dependencies
 install:

Dockerfile

@@ -1,12 +1,15 @@
-FROM alpine
+FROM python:3.6-alpine
+
+RUN apk --update add git g++ make libstdc++ gnupg musl-dev \
+    && rm -rf /var/cache/apk/* \
+    && mkdir /kapitan
 
-RUN apk --update add --virtual build-dependencies build-base python-dev py-pip && \ 
-    apk --update add python libstdc++ gnupg && mkdir /kapitan
 WORKDIR /kapitan
 COPY kapitan/ kapitan/
 COPY requirements.txt ./
-RUN pip install --no-cache-dir -r requirements.txt && apk del build-dependencies
 
+RUN pip install --upgrade --no-cache-dir pip \
+    && pip install --no-cache-dir -r requirements.txt
 
 ENV PYTHONPATH="/kapitan/"
 ENV SEARCHPATH="/src"

README.md

@@ -27,7 +27,7 @@ How is it different from [`Helm`](https://github.com/kubernetes/helm)? Please lo
 
 # Main Features
 
-* Use [reclass](https://github.com/madduck/reclass) to build an inventory of data/variables for all your deployments as a whole, avoiding dangerous duplications, and improving readibility.
+* Use [reclass](https://github.com/salt-formulas/reclass) to build an inventory of data/variables for all your deployments as a whole, avoiding dangerous duplications, and improving readibility.
 * Use [Jsonnet](https://github.com/google/jsonnet) to create json/yaml based configurations (e.g. Kubernetes, Terraform);
 * Use [Jinja2](http://jinja.pocoo.org/docs/2.9/) to create text based templates for scripts and documentation;
 * Manage secrets by defining who can see them, without compromising collaboration with other users.
@@ -37,14 +37,18 @@ How is it different from [`Helm`](https://github.com/kubernetes/helm)? Please lo
 
 # Installation
 
-Kapitan needs Python 2.7+ and can be installed with pip.
+Kapitan needs Python 3.6+ (it still works with Python 2.7 but support has been removed in v0.12.0).
 
+Install Python 3:
+<br>Linux: `sudo apt-get update && sudo apt-get install -y python3.6-dev`
+<br>Mac: `brew install python3`
+
+Install Kapitan via pip:
 ```
-$ pip install git+https://github.com/deepmind/kapitan.git
+$ pip3 install git+https://github.com/deepmind/kapitan.git --process-dependency-links
 ```
 
 
-
 # Example
 
 The example below _compiles_ 3 targets inside the `examples/kubernetes` folder.
@@ -408,7 +412,7 @@ $ kapitan searchvar parameters.elasticsearch.replicas
 
 * [Jsonnet](https://github.com/google/jsonnet)
 * [Jinja2](http://jinja.pocoo.org/docs/2.9/)
-* [reclass](https://github.com/madduck/reclass)
+* [reclass](https://github.com/salt-formulas/reclass)
 
 # FAQ
 
@@ -429,7 +433,7 @@ With Kapitan, we worked to de-compose several problems that most of the other so
 
 1) ***Kubernetes manifests***: We like the jsonnet approach of using json as the working language. Jsonnet allows us to use inheritance and composition, and hide complexity at higher levels.
 2) ***Configuration files***: Most solutions will assume this problem is solved somewhere else. We feel Jinja (or your template engine of choice) have the upper hand here.
-3) ***Hierarchical inventory***: This is the feature that sets us apart from other solutions. We use the inventory (based on [reclass](https://github.com/madduck/reclass)) to define variables and properties that can be reused across different projects/deployments. This allows us to limit repetition, but also to define a nicer interface with developers (or CI tools) which will only need to understand YAML to operate changes.
+3) ***Hierarchical inventory***: This is the feature that sets us apart from other solutions. We use the inventory (based on [reclass](https://github.com/salt-formulas/reclass)) to define variables and properties that can be reused across different projects/deployments. This allows us to limit repetition, but also to define a nicer interface with developers (or CI tools) which will only need to understand YAML to operate changes.
 4) ***Canned scripts***: We treat scripts as text templates, so that we can craft pre-canned scripts for the specific target we are working on. This can be used for instance to define scripts that setup clusters, contexts or allow to run kubectl with all the correct settings. Most other solutions require you to define contexts and call kubectl with the correct settings. We take care of that for you. Less ambiguity, less mistakes.
 5) ***Documentation***: We also use templates to create documentation for the targets we deploy. Documentation lived alongside everything else and it is treated as a first class citizen.
 We feel most other solutions are pushing the limits of their capacity in order to provide for the above problems.

bin/kapitan

@@ -1,4 +1,4 @@
 #!/bin/bash
 
 BINPATH=`dirname $0`
-PYTHONPATH="$BINPATH/../:$PYTHONPATH" python -m kapitan $@
+PYTHONPATH="$BINPATH/../:$PYTHONPATH" python3 -m kapitan $@

kapitan/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #

kapitan/__main__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #

kapitan/cli.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #
@@ -14,6 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+
 "command line module"
 
 import argparse
@@ -141,7 +143,7 @@ def main():
             json_obj = json.loads(json_output)
             yaml.safe_dump(json_obj, sys.stdout, default_flow_style=False)
         elif json_output:
-            print json_output
+            print(json_output)
 
     elif cmd == 'compile':
         if args.verbose:

kapitan/resources.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #
@@ -21,6 +21,7 @@
 import json
 import logging
 import os
+import io
 import reclass
 import reclass.core
 from reclass.errors import ReclassException, NotFoundError
@@ -83,7 +84,7 @@ def read_file(search_path, name):
     logger.debug("read_file trying file %s", full_path)
     if os.path.exists(full_path):
         logger.debug("read_file found file at %s", full_path)
-        with open(full_path) as f:
+        with io.open(full_path, newline='') as f:
             return f.read()
     raise IOError("Could not find file %s" % name)
 

kapitan/secrets.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #
@@ -16,6 +16,8 @@
 
 "secrets module"
 
+from six import string_types
+
 import base64
 import errno
 from functools import partial
@@ -75,7 +77,7 @@ def secret_gpg_read(gpg_obj, secrets_path, token, **kwargs):
             dec = secret_gpg_decrypt(gpg_obj, data_decoded, **kwargs)
             logger.debug("Read secret %s at %s", token, full_secret_path)
             if dec.ok:
-                return dec.data
+                return dec.data.decode("UTF-8")
             else:
                 raise GPGError(dec.status)
     except IOError as ex:
@@ -166,7 +168,7 @@ def secret_gpg_write(gpg_obj, secrets_path, token, data, encode_base64, recipien
         encoding = "original"
         _data = data
         if encode_base64:
-            _data = base64.b64encode(data)
+            _data = base64.b64encode(data.encode("UTF-8"))
             encoding = "base64"
         enc = secret_gpg_encrypt(gpg_obj, _data, recipients, **kwargs)
         if enc.ok:
@@ -203,7 +205,7 @@ def reveal_gpg_replace(gpg_obj, secrets_path, match_obj, verify=True, **kwargs):
     if verify:
         _, token_path, token_hash = secret_token_compiled_attributes(token)
         secret_raw_obj = secret_gpg_raw_read(secrets_path, token)
-        secret_hash = hashlib.sha256("%s%s" % (token_path, secret_raw_obj["data"])).hexdigest()
+        secret_hash = hashlib.sha256("%s%s".encode("UTF-8") % (token_path, secret_raw_obj["data"])).hexdigest()
         secret_hash = secret_hash[:8]
         logger.debug("Attempting to reveal token %s with secret hash %s", token, token_hash)
         if secret_hash != token_hash:
@@ -253,11 +255,11 @@ def sub_reveal_data(data):
         return re.sub(SECRET_TOKEN_TAG_PATTERN, _reveal_gpg_replace, data)
 
     if isinstance(obj, dict):
-        for k, v in obj.iteritems():
+        for k, v in obj.items():
             obj[k] = secret_gpg_reveal_obj(gpg_obj, secrets_path, v, verify, **kwargs)
     elif isinstance(obj, list):
         obj = [secret_gpg_reveal_obj(gpg_obj, secrets_path, item, verify, **kwargs) for item in obj]
-    elif isinstance(obj, basestring):  # XXX this is python 2 specific
+    elif isinstance(obj, string_types):
         obj = sub_reveal_data(obj)
 
     return obj

kapitan/targets.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #
@@ -16,6 +16,8 @@
 
 "kapitan targets"
 
+from six import string_types
+
 import logging
 import os
 import errno
@@ -189,7 +191,7 @@ def compile_jinja2(path, context, compile_path, **kwargs):
     secrets_reveal = kwargs.get('secrets_reveal', False)
     gpg_obj = kwargs.get('gpg_obj', None)
 
-    for item_key, item_value in render_jinja2(path, context).iteritems():
+    for item_key, item_value in render_jinja2(path, context).items():
         full_item_path = os.path.join(compile_path, item_key)
         try:
             os.makedirs(os.path.dirname(full_item_path))
@@ -233,7 +235,7 @@ def compile_jsonnet(file_path, compile_path, search_path, ext_vars, **kwargs):
         json_output = prune_empty(json_output)
         logger.debug("Pruned output for: %s", file_path)
 
-    for item_key, item_value in json_output.iteritems():
+    for item_key, item_value in json_output.items():
         # write each item to disk
         if output == 'json':
             file_path = os.path.join(compile_path, '%s.%s' % (item_key, output))
@@ -342,23 +344,23 @@ def write_json(self, obj):
     def sub_token_compiled_obj(self, obj):
         "recursively find and replace tokens with hashed tokens in obj"
         if isinstance(obj, dict):
-            for k, v in obj.iteritems():
+            for k, v in obj.items():
                 obj[k] = self.sub_token_compiled_obj(v)
         elif isinstance(obj, list):
-            obj = map(self.sub_token_compiled_obj, obj)
-        elif isinstance(obj, basestring):  # XXX this is python 2 specific
+            obj = list(map(self.sub_token_compiled_obj, obj))
+        elif isinstance(obj, string_types):
             obj = self.sub_token_compiled_data(obj)
 
         return obj
 
     def sub_token_reveal_obj(self, obj):
         "recursively find and reveal token tags in data"
         if isinstance(obj, dict):
-            for k, v in obj.iteritems():
+            for k, v in obj.items():
                 obj[k] = self.sub_token_reveal_obj(v)
         elif isinstance(obj, list):
             obj = map(self.sub_token_reveal_obj, obj)
-        elif isinstance(obj, basestring):  # XXX this is python 2 specific
+        elif isinstance(obj, string_types):
             obj = self.sub_token_reveal_data(obj)
 
         return obj
@@ -376,7 +378,7 @@ def hash_token_tag(self, token_tag):
         token = secret_token_from_tag(token_tag)
         secret_raw_obj = secret_gpg_raw_read(secrets_path, token)
         backend, token_path = secret_token_attributes(token)
-        sha256 = hashlib.sha256("%s%s" % (token_path, secret_raw_obj["data"])).hexdigest()
+        sha256 = hashlib.sha256("%s%s".encode("UTF-8") % (token_path.encode("UTF-8"), secret_raw_obj["data"].encode("UTF-8"))).hexdigest()
         sha256 = sha256[:8]
         return "?{%s:%s:%s}" % (backend, token_path, sha256)
 

kapitan/utils.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #
@@ -14,7 +14,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from __future__ import print_function
+
 "random utils"
+
 import functools
 from hashlib import sha256
 import logging
@@ -44,7 +47,7 @@ def render_jinja2_template(content, context):
 
 def jinja2_sha256_hex_filter(string):
     "Returns hex digest for string"
-    return sha256(string).hexdigest()
+    return sha256(string.encode("UTF-8")).hexdigest()
 
 def jinja2_yaml_filter(obj):
     "Returns yaml for object"

kapitan/version.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 #
 # Copyright 2017 The Kapitan Authors
 #
@@ -17,7 +17,7 @@
 "Project description variables"
 
 PROJECT_NAME = 'kapitan'
-VERSION = '0.11.0'
+VERSION = '0.12.0'
 DESCRIPTION = 'Kapitan is a tool to manage kubernetes configuration using jsonnet templates'
 AUTHOR = 'Ricardo Amaro'
 AUTHOR_EMAIL = '[email protected]'

requirements.txt

@@ -1,6 +1,11 @@
 jsonnet==0.10.0
 PyYAML==3.12
-Jinja2==2.9.4
-jsonschema==2.5.1
-reclass==1.4.1
-python-gnupg==0.4.1
+Jinja2>=2.10
+# Latest commit from salt-formulas/reclass - python3 branch
+# TODO: Change commit hash to release tag, once python3 branch is merged in
+git+https://github.com/salt-formulas/reclass.git@31770c6#egg=reclass
+jsonschema>=2.6.0
+# Closest commit to official python-gnupg==0.4.1 + fix for https://bitbucket.org/vinay.sajip/python-gnupg/issues/84/on-osx-version-detection-fails-then-raises
+# TODO: Change to python-gnupg==0.4.2 once released
+git+https://github.com/vsajip/python-gnupg.git@73b5d8d#egg=python-gnupg
+six>=1.11.0