Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rest_api: use @defer decorator on transformer loop for parallel execution #1883

Open
rudolfix opened this issue May 21, 2024 · 0 comments
Open

Comments

@rudolfix
Copy link
Collaborator

Source name

rest_api

Describe the data you'd like to see

Dependent resources loop over all data items in a batch sent from the connected parent. We can easily parallelize this loop.

  1. detect if parallelize flag is enabled on a transformer when generating loop over items
  2. if so use decorated version of the function (via dlt.defer)
    in essence such decorator wraps transformer function in callables and many of those can be executed at the same time

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

Yes, I'm ready.

dlt destination

No response

Additional information

No response

@burnash burnash transferred this issue from dlt-hub/verified-sources Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Planned
Status: Planned
Development

No branches or pull requests

1 participant