Feature request for --include and --exclude filters

[echan5]

Hi, this is a feature request. awscli has --include and --exclude filters and it would be great if s5cmd also had those options!

[seruman]

Hi @echan5. It would be great for us if you and other requester could share what kind of use cases you have in mind? That would help us to better grasp it and decide how to put it with the current feature set (e.g wildcard support).

[echan5]

@seruman - essentially, feature parity with awcli would be great! https://docs.aws.amazon.com/cli/latest/reference/s3/#use-of-exclude-and-include-filters
Examples include using --include "*.txt" to include/exclude certain file types, or --exclude "abc*" to include/exclude any file that starts with prefix "abc". Thank you for looking into this request!

[nelhage]

I'd also have a use case for this; I tried to work around using cp commands in s5cmd run mode, but have been blocked by #301 from actually deploying it in our environment.

In our particular case, we're storing ML models, and we store parameters, and some optimizer states (ADAM moments) alongside the parameters. When we're restoring a model to do inference, we would like to be able to s5cmd cp the relevant parameters to local disk, but ignore the optimizer moments, which are stored with consistent filenames. In our case, that looks something like --exclude=*_moment.dat.

[seruman]

Thank you both @echan5, @nelhage, what's your thoughts on adding include/exclude options those support regex patterns?

[nelhage]

I'd be fine with regexes for my use case. I would find it a bit surprising -- include/exclude in every other command I can think of offhand uses some form of glob syntax -- but I could live with it and it would potentially be a nice bit of flexibility.

[echan5]

It would be the same for me - I don't think I'd need regex patterns for my current use case, but it could be nice to have that flexibility

[seruman]

I would find it a bit surprising -- include/exclude in every other command I can think of offhand uses some form of glob syntax --

Given s5cmd already supports glob syntax, regex would have been weird as you mentioned. --exclude option is implemented thanks to the @ocakhasan, could you give it a try from master branch?