Secrets in kv store phase #0 #2631

lakshmi-kannan · 2016-04-19T23:44:05Z

What?

This PR allows encrypted items to be stored in key value store. The encryption is symmetric AES256 encryption and we use keyczar to do this. The encryption key is global for all keys. So StackStorm admin can decrypt all the keys. However, with RBAC, we'll make sure only intended user (and admin) can decrypt the user's keys.

TODO

Encrypted values are non UTF-8. Mongoengine complains about that.
CLI changes
- st2 keyvalue get changes
- st2 keyvalue list changes
Tests
Tool for generating AES key in tools/ 249340e
Docs Document 'securing key value store' st2docs#125

CLI examples

PUT

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key set foo bar                                                                                                                                                       [ruby-1.9.3p484]
+------------------+-------+
| Property         | Value |
+------------------+-------+
| name             | foo   |
| value            | bar   |
| expire_timestamp |       |
+------------------+-------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key set secret_foo secret_bar --encrypt                                                                                                                               [ruby-1.9.3p484]
+------------------+--------------------------------------------------------------+
| Property         | Value                                                        |
+------------------+--------------------------------------------------------------+
| name             | secret_foo                                                   |
| value            | 0083E1033A7C286818517F1DDD2190042C6F6161A4A6318CA7A7767B97E8 |
|                  | DCD2984F36B070A60BA3BFE0ACF0C583F6DE8233B7C842E2EAD2D6       |
| expire_timestamp |                                                              |
+------------------+--------------------------------------------------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$

GET

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key get secret_foo --decrypt                                                                                                                                          [ruby-1.9.3p484]
+------------------+------------+
| Property         | Value      |
+------------------+------------+
| name             | secret_foo |
| value            | secret_bar |
| secret           | True       |
| encrypted        | False      |
| expire_timestamp |            |
+------------------+------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key get secret_foo                                                                                                                                                    [ruby-1.9.3p484]
+------------------+--------------------------------------------------------------+
| Property         | Value                                                        |
+------------------+--------------------------------------------------------------+
| name             | secret_foo                                                   |
| value            | 0083E1033A90EB483B7368A414E9456928FDAB3808830B553CA63691E045 |
|                  | 3990D8B4BEB0263B3F9363F1D19CF06BF70EC9B2D81F7F7F0F6199       |
| secret           | True                                                         |
| encrypted        | True                                                         |
| expire_timestamp |                                                              |
+------------------+--------------------------------------------------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key get foo                                                                                                                                                           [ruby-1.9.3p484]
+------------------+-------+
| Property         | Value |
+------------------+-------+
| name             | foo   |
| value            | bar   |
| secret           | False |
| encrypted        | False |
| expire_timestamp |       |
+------------------+-------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$

List

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key list --decrypt                                                                                                                                                   [ruby-1.9.3p484]

+------------+------------+--------+-----------+------------------+
| name       | value      | secret | encrypted | expire_timestamp |
+------------+------------+--------+-----------+------------------+
| secret_foo | secret_bar | True   | False     |                  |
| foo        | bar        | False  | False     |                  |
+------------+------------+--------+-----------+------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$                                                                                                                                                                          [ruby-1.9.3p484]
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ st2 key list                                                                                                                                                             [ruby-1.9.3p484]
+------------+----------------------------------------------------+--------+-----------+------------------+
| name       | value                                              | secret | encrypted | expire_timestamp |
+------------+----------------------------------------------------+--------+-----------+------------------+
| secret_foo | 0083E1033A040EA17E5D5A61E1087675EEFED8C7A1BFF0927F | True   | True      |                  |
|            | EDEF0BE9C948044A87BE35DE9238FC67E707D377999C3FEA43 |        |           |                  |
|            | 17B4EF32371572                                     |        |           |                  |
| foo        | bar                                                | False  | False     |                  |
+------------+----------------------------------------------------+--------+-----------+------------------+
(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$

API examples

No secret case

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ http PUT http://127.0.0.1:9101/v1/keys/foo < kv_nosecret.json                                                                                                             [ruby-1.9.3p484]
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 97
Content-Type: application/json; charset=UTF-8
Date: Thu, 21 Apr 2016 20:12:54 GMT
Server: gunicorn/19.4.5
X-Request-ID: 6244e7c9-bdd3-4040-9da5-2941e84391eb

{
    "encrypted": false,
    "name": "foo",
    "secret": false,
    "uid": "key_value_pair:foo",
    "value": "bar"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$

Secret case. No encryption key setup by admin

XXX: Note I return 400 BAD REQUEST. Opinions welcome.

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●)$ http PUT http://127.0.0.1:9101/v1/keys/secret_foo < kv.json                                                                                                               [ruby-1.9.3p484]
HTTP/1.1 400 Bad Request
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 131
Content-Type: application/json
Date: Tue, 19 Apr 2016 23:43:26 GMT
Server: gunicorn/19.4.5
X-Request-ID: 14ae1cf2-f6be-472b-b1e4-8fa46b4d6148

{
    "faultstring": "Crypto key not found in conf/st2_kvstore_demo.crypto.key.json. Unable to encrypt value for key secret_foo."
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●)$

Secret case. Encryption key setup.

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ http PUT http://127.0.0.1:9101/v1/keys/secret_foo < kv.json                                                                                                               [ruby-1.9.3p484]
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 220
Content-Type: application/json; charset=UTF-8
Date: Thu, 21 Apr 2016 20:13:26 GMT
Server: gunicorn/19.4.5
X-Request-ID: 41cc580c-7557-4e1b-9e32-df4672da7a63

{
    "encrypted": true,
    "name": "secret_foo",
    "secret": true,
    "uid": "key_value_pair:secret_foo",
    "value": "0083E1033A66FF9293F6715BCF5B3918E3A18E2281B6A45B08603032571E3812C6D9CEE25BD1E060DADD8E951C85E4479C1C1BE91835BACEF1"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ http GET "http://127.0.0.1:9101/v1/keys/secret_foo?decrypt=False"                                                                                                         [ruby-1.9.3p484]
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 220
Content-Type: application/json; charset=UTF-8
Date: Thu, 21 Apr 2016 20:13:58 GMT
Server: gunicorn/19.4.5
X-Request-ID: 8ee5d905-5878-47ca-98d1-118ccddc5a3d

{
    "encrypted": true,
    "name": "secret_foo",
    "secret": true,
    "uid": "key_value_pair:secret_foo",
    "value": "0083E1033A66FF9293F6715BCF5B3918E3A18E2281B6A45B08603032571E3812C6D9CEE25BD1E060DADD8E951C85E4479C1C1BE91835BACEF1"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$ http GET "http://127.0.0.1:9101/v1/keys/secret_foo?decrypt=True"                                                                                                          [ruby-1.9.3p484]
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Content-Type,Authorization,X-Auth-Token,St2-Api-Key,X-Request-ID
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: http://172.168.50.50:3000
Access-Control-Expose-Headers: Content-Type,X-Limit,X-Total-Count,X-Request-ID
Connection: keep-alive
Content-Length: 117
Content-Type: application/json; charset=UTF-8
Date: Thu, 21 Apr 2016 20:14:03 GMT
Server: gunicorn/19.4.5
X-Request-ID: 3a5ad5a8-bb68-472b-854f-3d3764b22b28

{
    "encrypted": false,
    "name": "secret_foo",
    "secret": true,
    "uid": "key_value_pair:secret_foo",
    "value": "secret_bar"
}

(virtualenv)vagrant@st2dev /mnt/src/storm/st2 (secrets_kv_store●●)$

> db.key_value_pair_d_b.find({'name': 'secret_foo'})
{ "_id" : ObjectId("5717e445d9d7ed0ea5b07ec7"), "uid" : "key_value_pair:secret_foo", "name" : "secret_foo", "value" : "0083E1033AE5F3D1B045398FC51F1046927971A46502A5A5BEB72ED37FB32073709569E37D081D9D38BF3E96F8A9F4D1EA5FCE349D6D18B3EB", "encrypted" : true }
>

manasdk · 2016-04-22T01:24:15Z

Secret case. No encryption key setup by admin

I am ok with 400. I guess we are saying for the current configuration state this request is bad.

manasdk · 2016-04-22T01:30:49Z

st2common/st2common/models/api/keyvalue.py

@@ -57,8 +76,39 @@ class KeyValuePairAPI(BaseAPI):
        'additionalProperties': False
    }

+    @staticmethod
+    def _setup_crypto():


I would imagine encryption and decryption for kvp being handled in the persistence or the db modules. This way its not an API only feature and it would be possible to get decrypted keys even if not using the API i.e. from services do not use the API layer.

Yeah, I can move the code around when we address that.

manasdk · 2016-04-22T19:00:31Z

st2 key list --decrypt

Would it be safer to not support a list level decrypt? I wonder if there is value in providing this in the first place.

manasdk · 2016-04-22T19:01:11Z

Overall +1 to design and impl.

Once you add some tests and docs around this good to go :)

lakshmi-kannan · 2016-04-22T19:21:40Z

Would it be safer to not support a list level decrypt?

Well, with RBAC this might get even harder but it is something we need to solve I think. I was debating it myself too. Let's wait for another input to break the tie.

manasdk · 2016-04-23T00:16:08Z

st2common/st2common/util/crypto.py

+    'Initialization Vector' per run and the IV is part of the output.
+
+    :param encrypt_key: Symmetric AES key to use for encryption.
+    :type encrypt_key: :class:`keyczar.keys.AesKey`


for this method does it matter if its an AesKey? Is there is higher interface/superclass?

Yep. Unfortunately it does. The Encrypt method is on the key object which is keyczar and AES specific implementation.

Kami · 2016-04-25T11:04:28Z

Tool for generating AES key in tools/

Keyczar already provides a tool for managing and rotating keys. Do you plan to build a wrapper around that or something else?

Kami · 2016-04-25T11:05:32Z

conf/st2_kvstore_demo.crypto.key.json

@@ -0,0 +1 @@
+{"hmacKey": {"hmacKeyString": "685EWJ8U2U_c3Ulv9ibB6aa0Y9aYm-0bCwe2mqjLJuw", "size": 256}, "aesKeyString": "eKGFrfNEUhAgX0uofaU1wQ", "mode": "CBC", "size": 128}


For fixtures that's a fine path, but for actual key on production, we should advise users to put in a path with limited access (e.g. /etc/private or similar).

Yep, will say that in docs.

Kami · 2016-04-25T11:14:53Z

I'm not totally against this approach, but as mentioned on Slack, I would prefer to use public key cryptography.

This will make it easier for us to eventually move to a safer model, where we have a single service (or similar) which is responsible for decrypting data (this would be the only service which has access to the private key, reducing the surface area), but other services (e.g. API) will still be able to encrypt data using the public key...

lakshmi-kannan · 2016-04-25T18:57:37Z

I'm not totally against this approach, but as mentioned on Slack, I would prefer to use public key cryptography.

We can swap it out if needed. I don't see this as a big problem now that we have API & CLI changes.

lakshmi-kannan · 2016-04-25T18:58:55Z

st2client/tests/unit/test_commands.py

-    @mock.patch.object(
-        models.ResourceManager, 'get_by_id',
-        mock.MagicMock(return_value=None))
-    def test_command_get_by_name(self):


I removed this bogus test. The underlying code was anyways calling get_by_id and we are already testing that in previous test case.

lakshmi-kannan · 2016-04-25T19:00:14Z

Keyczar already provides a tool for managing and rotating keys. Do you plan to build a wrapper around that or something else?

Correct - a wrapper. Some argparse + keyczar + fwrite()

dzimine · 2016-04-25T23:14:55Z

👍 for design, thanks! didn't look into impl. yet.

st2 key list --decrypt

Would it be safer to not support a list level decrypt? I wonder if there is value in providing this in the first place.

The use case is, for instance, migration by admin to another node. As admin, I dump all KV with st2 key list --decrypt -j and recreate it on the other one. Here having "encrypted" value helps too, to hint to decrypt it when json is loaded.

manasdk · 2016-04-26T00:17:58Z

As admin, I dump all KV with st2 key list --decrypt -j and recreate it on the other one.

Would migrating the encrypted data as well as the key work just as well?

Kami · 2016-04-26T03:53:02Z

Re saving and serializing encrypted data - I can't recall if Keyczar already does that, but if doesn't we should update serialized data so it includes header with the encryption information (which algorithm was used, etc.).

This will makes this such as migrating to a different algorithm or similar easier.

Kami · 2016-05-02T10:26:05Z

st2client/st2client/models/core.py

@@ -240,7 +240,8 @@ def query(self, **kwargs):
        if 'limit' in kwargs and kwargs.get('limit') <= 0:
            kwargs.pop('limit')
        token = kwargs.get('token', None)
-        params = {}
+        params = kwargs.get('params', {})
+        print(params)


Probably need to remove print here?

Removed it in 406a701.

lakshmi-kannan · 2016-05-02T18:48:05Z

Re saving and serializing encrypted data - I can't recall if Keyczar already does that, but if doesn't we should update serialized data so it includes header with the encryption information (which algorithm was used, etc.).

It does not. It assumes symmetric crypto key to be always AES. The library doesn't support other symmetric crypto algos. https://github.com/google/keyczar/blob/master/python/src/keyczar/keys.py#L425

Lakshmi Kannan added 3 commits April 19, 2016 16:34

Data model changes

8a7741a

API Controller changes

2060ad5

Utility methods

75be11b

lakshmi-kannan added the WIP label Apr 19, 2016

Lakshmi Kannan added 5 commits April 20, 2016 13:22

Convert crypto to actual string before saving it in db

dae6c18

By default, GET /keys/secret_key returns encrypted value.

2ffa51f

Initial CLI changes - corresponding API touch ups

69f82c4

Add a make target for building CLI

83a1981

Support decrypt in list operation for keyvalue - API & CLI

6256063

manasdk reviewed Apr 22, 2016
View reviewed changes

Lakshmi Kannan added 2 commits April 22, 2016 16:34

First set of tests

c5ad5b3

Add db model test for kv

09fe92d

manasdk reviewed Apr 23, 2016
View reviewed changes

Kami reviewed Apr 25, 2016
View reviewed changes

Fix st2client tests

c4ac445

lakshmi-kannan reviewed Apr 25, 2016
View reviewed changes

Tool to generate secure key

249340e

Lakshmi Kannan added 2 commits April 25, 2016 14:16

Remove backup option

5bb6c67

Cleanup imports

195b1be

Tests and dev env to supply key file, default is no key file

1ffa9e6

lakshmi-kannan changed the title ~~WIP: Secrets in kv store~~ Secrets in kv store phase #0 Apr 26, 2016

lakshmi-kannan merged commit d0345b7 into master Apr 26, 2016

lakshmi-kannan deleted the secrets_kv_store branch April 26, 2016 00:07

Kami reviewed May 2, 2016
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Secrets in kv store phase #0 #2631

Secrets in kv store phase #0 #2631

lakshmi-kannan commented Apr 19, 2016 •

edited

Loading

manasdk commented Apr 22, 2016

manasdk Apr 22, 2016

lakshmi-kannan Apr 22, 2016

manasdk commented Apr 22, 2016

manasdk commented Apr 22, 2016

lakshmi-kannan commented Apr 22, 2016

manasdk Apr 23, 2016

lakshmi-kannan Apr 23, 2016

Kami commented Apr 25, 2016

Kami Apr 25, 2016

lakshmi-kannan Apr 25, 2016

Kami commented Apr 25, 2016

lakshmi-kannan commented Apr 25, 2016

lakshmi-kannan Apr 25, 2016

lakshmi-kannan commented Apr 25, 2016

dzimine commented Apr 25, 2016

manasdk commented Apr 26, 2016

Kami commented Apr 26, 2016

Kami May 2, 2016

Kami May 5, 2016

lakshmi-kannan commented May 2, 2016

		@@ -0,0 +1 @@
		{"hmacKey": {"hmacKeyString": "685EWJ8U2U_c3Ulv9ibB6aa0Y9aYm-0bCwe2mqjLJuw", "size": 256}, "aesKeyString": "eKGFrfNEUhAgX0uofaU1wQ", "mode": "CBC", "size": 128}

Secrets in kv store phase #0 #2631

Secrets in kv store phase #0 #2631

Conversation

lakshmi-kannan commented Apr 19, 2016 • edited Loading

What?

TODO

CLI examples

PUT

GET

List

API examples

No secret case

Secret case. No encryption key setup by admin

Secret case. Encryption key setup.

manasdk commented Apr 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manasdk commented Apr 22, 2016

manasdk commented Apr 22, 2016

lakshmi-kannan commented Apr 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kami commented Apr 25, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kami commented Apr 25, 2016

lakshmi-kannan commented Apr 25, 2016

Choose a reason for hiding this comment

lakshmi-kannan commented Apr 25, 2016

dzimine commented Apr 25, 2016

manasdk commented Apr 26, 2016

Kami commented Apr 26, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lakshmi-kannan commented May 2, 2016

lakshmi-kannan commented Apr 19, 2016 •

edited

Loading