-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue 6304 - RFE when memberof is enabled, defer updates of members f… #6305
Conversation
54510f5
to
2e89f70
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, there are a few minor comments for test parts.
For the core server part, it looks good as far as I can see.
I have a few questions, though.
- Did you make an ASAN biuld/run?
- How will the dereffered tasks act with the nester groups? Like if we add/remove users to/from different levels of the hierarchy?
- In general, I think we might need to extend the tests for the RFE (not necessary by you). Things that come to my mind:
- Modify the same group from multiple clients concurrently AND Perform simultaneous add/remove operations on different groups;
- Configure one supplier with deferred updates and one without (is it valid?)
- Measure the performance impact;
- Multi-valued RDNs for groups and users;
- Might worth to add more restarts/kills into the above scenarious.
dirsrvtests/tests/suites/memberof_plugin/memberof_deferred_repl_test.py
Outdated
Show resolved
Hide resolved
@droideck , first of all a big thanks for your review. I did the changes regarding the CI tests and push them. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great to me! Thanks! Ack
…rom the update of the group Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: 389ds#6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group (#6305) Bug Description: When an update of a static group changes impacts a large portion of its members, memberof triggers a large number of internal updates on members. All the updates are done a same TXN that may hold sensitive DB pages and block others SRCH req waiting for those pages. In extreme condition, all workers are stuck on SRCH req waiting for the completion of the update. Then the server appears unresponsive. Fix Description: The fix is to defer the update of the members. Memberof tests: - for the verification of the membership, if deferred update is set, then adds a delay before checking - automember_plugin/automember_test.py - automember_plugin/basic_test.py - memberof_plugin/memberof_include_scopes_test.py - plugins/acceptance_test.py - plugins/entryusn_test.py - plugins/memberof_test.py - lib389/plugins.py - original update (group) succeeds even if deferred updates fails (multiple TXN) - betxns/betxn_test.py - Check replication of memberof - memberof_plugin/memberof_deferred_repl_test.py - Check deferred update and shutdown - memberof_plugin/memberof_include_scopes_test.py Core implementation: - Make sure that direct update (not internal) wait for deferred update before returning a result - back-ldbm/ldbm_add.c - back-ldbm/ldbm_delete.c - back-ldbm/ldbm_modify.c - back-ldbm/ldbm_modrdn.c - Implementation of the deferred update - memberof/memberof.h - memberof/memberof.c - memberof/memberof_config.c - slapd/pblock.c - slapd/pblock_v3.h - slapd/schema.c - slapd/slapi-plugin.h memberof_be_postop_init registers memberof_push_deferred_task that: push deferred update to deferred thread task taken from pblock (SLAPI_MEMBEROF_DEFERRED_TASK) push to the memberof config deferred_list deferred thread (deferred_thread_func) if 'memberOfNeedFixup: on' then run fixup task loop until shutdown fetch task (remove_deferred_task) from the memberof config deferred_list proceed with the task if it exits abruptly, it logs an alert and add 'memberOfNeedFixup: on' to the config memberof_postop_start if deferred update is configured then it creates a deferred_list in the config it spawn the deferred thread + CV Each postop_operation memberof_postop_modrdn, memberof_postop_del, memberof_postop_modify, memberof_postop_add if deferred update it allocates a task and add it in the pblock (SLAPI_MEMBEROF_DEFERRED_TASK) Related: #6304 Reviewed by: Simon Pichugin (Thanks!!)
…rom the update of the group
Bug Description:
When an update of a static group changes
impacts a large portion of its members, memberof triggers
a large number of internal updates on members.
All the updates are done a same TXN that may hold
sensitive DB pages and block others SRCH req
waiting for those pages.
In extreme condition, all workers are stuck on
SRCH req waiting for the completion of the update.
Then the server appears unresponsive.
Fix Description:
The fix is to defer the update of the members.
Related: #6304
Reviewed by: