Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add migration script for migrating old execution field data to the new format #5255

Merged
merged 35 commits into from
May 13, 2021
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
b9fc5e0
Add WIP change for migrating old executions field data which utilizes…
Kami May 7, 2021
7fe8da3
Bump apt cache key.
Kami May 7, 2021
c22fa56
Add support for migrating other objects which utilize new data type and
Kami May 8, 2021
70261e0
Fix query syntax, remove unncessary variable.
Kami May 8, 2021
3a3693d
Print message if we fail to migrate "related" LiveAction (non-fatal
Kami May 8, 2021
10b920c
Fix log statements.
Kami May 8, 2021
f849763
Add symlink needed to be able to import module in tests.
Kami May 8, 2021
9c600fd
Remove duplicated comment.
Kami May 8, 2021
928add0
Add basic unit tests for migration script.
Kami May 8, 2021
a917e4d
Fix operator syntax.
Kami May 8, 2021
3f0c184
Update changelog.
Kami May 8, 2021
f2bd224
Update script name.
Kami May 8, 2021
8c79a14
Use correct path, use better test module name.
Kami May 8, 2021
e48aca4
Re-organize the class definitions + allow_inheritance override to avoid
Kami May 8, 2021
05e3c04
Pin python version to a full version including patch version.
Kami May 9, 2021
9fc2af9
Bump apt cache version.
Kami May 9, 2021
86afbb1
Bump eventlet version due to security vulnerability (which doesn't
Kami May 9, 2021
e836198
Temporary disable apt cache since it seems to be failing the build.
Kami May 9, 2021
23ddc7e
Revert "Bump eventlet version due to security vulnerability (which do…
Kami May 9, 2021
7a117bc
Update migration script so we only load maximum of one whole database
Kami May 11, 2021
91cee9c
Merge branch 'master' of github.com:StackStorm/st2 into db_field_type…
Kami May 11, 2021
3e3743f
Also print duration at the end.
Kami May 11, 2021
b5376cb
Ignore DB object doesn't exist errors since those should not be fatal -
Kami May 11, 2021
a5964e2
Update the output to make it easier to see how far along we are.
Kami May 11, 2021
94ed6b0
Address review feedback.
Kami May 11, 2021
db132ac
Don't include patch version in the job name.
Kami May 11, 2021
ab93e86
Optimize the script and operate with raw pymongo values when retrieving
Kami May 11, 2021
85149e7
Add new --start-ts=<unix ts> argument and default it to now - 30 days.
Kami May 11, 2021
2a0969f
Make sure we also set execution.result_size field on migration and add
Kami May 12, 2021
3bc46f0
Add --start-dt, --end-dt argument to the migration script, update
Kami May 12, 2021
f9c3536
Remove temporary change which is not needed anymore.
Kami May 12, 2021
5f24607
Update formatting.
Kami May 12, 2021
613bf2d
Support special now notation for --end-dt.
Kami May 12, 2021
9d8d981
Use isotime.parse() since it supports more formats.
Kami May 12, 2021
ff37b4f
Remove unused constant.
Kami May 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make sure we also set execution.result_size field on migration and add
temporary workaround for AJ which will be removed after he runs this
script and confirms it works.
  • Loading branch information
Kami committed May 12, 2021
commit 2a0969f6458ac8fdbe9bf02c5e197cd102f327bf
42 changes: 41 additions & 1 deletion st2common/bin/migrations/v3.5/st2-migrate-db-dict-field-values
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ def migrate_executions(start_dt: datetime.datetime) -> None:
# works correctly - with PersistanceClass.query().only() all the fields will still be retrieved.

# 1. Migrate ActionExecutionDB objects
# 1.1 Find all executions where field type is not binData
result = (
ActionExecutionDB.objects(
__raw__={
Expand All @@ -102,7 +103,36 @@ def migrate_executions(start_dt: datetime.datetime) -> None:
.only("id")
.as_pymongo()
)
execution_ids = [str(item["_id"]) for item in result]
execution_ids = set([str(item["_id"]) for item in result])

# 1.2 Find all executions which don't contain "result_size" attribute, but field type is
# aready binData
# This is just aa workaround for migration script issue during development which has been
# resolved.
# https://github.com/StackStorm/st2/pull/5255#issuecomment-839920397
# This code path can be removed once @AJ runs that code on his setup.
result = (
ActionExecutionDB.objects(
__raw__={
"result": {
"$type": "binData",
},
"status": {
"$in": LIVEACTION_COMPLETED_STATES,
},
},
result_size__not__exists=True,
start_timestamp__gte=start_dt,
)
.only("id")
.as_pymongo()
)

for item in result:
execution_ids.add(str(item["_id"]))

execution_ids = list(execution_ids)

objects_count = result.count()

if not execution_ids:
Expand Down Expand Up @@ -135,6 +165,16 @@ def migrate_executions(start_dt: datetime.datetime) -> None:
execution_db._mark_as_changed("result")
execution_db._mark_as_changed("result_size")

# We need to explicitly set result_size attribute since Document.save() code path doesn't
# populate it (but other code paths we utilize elsewhere do).
# Technically we could do it on document clean() / validate() method, but we don't want that
# since execution update code in action runner and elsewhere is optimized to make partial
# updates more efficient.
result_size = len(
ActionExecutionDB.result._serialize_field_value(execution_db.result or {})
)
execution_db.result_size = result_size

# NOTE: If you want to view changed fields, you can access execution_db._changed_fields

execution_db.save()
Expand Down