Publish: register publishes without copying them

[kalisp]

Brief description

This PR introduces 'source' publish template which could be used in 'in-place' publish, eg. files are published from their current location without copying them into 'publish' folder.

Description

This kind of publishing helps with massive files as pointcaches or renders when files are already created in proper location. Without copying publish process takes less time.

Additional logic to check that publish path is inside of project data folder (eg. in at least one configured root in Anatomy) to protect from publishing to "my C:/ folder".

!!! ExtractThumbnail* plugins are now adding _thumb suffix to names (my_project_seq02_shot01_renderCompositingMain_v018.1004_thumb.jpg)

Additional details

originalDirname was added to collector for resources, but some subset might be created only during Extract phase, so it was added also into integrate plugin, which might not be optimal. Comments welcomed.

I tested it in DCC with render which ended up in work/.../renders subfolders which doesn't seem too right, as they might be purged (they weren't in Maya and Nuke though).

Testing notes:

1. Double check that your project has 'source' template filled in project_anatomy/templates/others
   
2. Configure which hosts and families should use this template in project_settings/global/publish/IntegrateAssetNew or better in project_settings/global/tools/publish/template_name_profiles (see deprecation warning and move template profiles to later location) (I tested in on combination Traypublisher - render and Maya - render
   
3. In TP it should create representation containing data.path: "URL_OF_DRAGGED_FILE", no new version subfolder should be created in publish folder

[ynbot]

Task linked: OP-4504 Publishing: Register publishes without copying them

[antirotor]

Tested it in Tray Publisher and found strange problem with Extract Thumbnail (from source) - this is producing jpg file from input, but it is placed in staging directory like C:\Users\annat\AppData\Local\Temp\pyblish_tmp_tduppdsg\VFX025_00001.jpg in my testing data.

Now integrator will skip copying it:

WARNING: Destination 'e:/projects/foo/bar/vfx025/vfx_out/v10/vfx025_00001.jpg' is not in project folder. Skipping

but since it is in staging tmp directory it will never be find by integrator when registering.

To wrap it up: We need to handle moving stuff produced during the publishing process that is generated into temporary destinations even if the source template is used.

Updated

[iLLiCiTiT]

I would say original dirname and basename is representation specific and not instance specific.
EDITED:
The dirname is used for all representations. But I think it should be always set to staging dir from instance and never from representation.

[kalisp]

Ok, used instance_stagingdir instead.

[iLLiCiTiT]

We must be carefull about original basename because thumbnail extractors use same name as source so if .jpeg would be published the thumbnail would "override" the source file. So we should modify thumbnail extractors to add _thumbnail suffix to avoid conflicts with source file.

I'm 90% convinced that template in anatomy should look like {root[work]}/{originalDirname}/{originalBasename}<.{@frame}><_{udim}>.{ext} with the {root[...]} and we should look for exactly the one root. OR if not, then we should "change" the template for representation to contain {root[...]} because of multiplatform support (or potentially site sync if necessary).

Another question is if the originalBasename should not be originalFilename to template would be {root[work]}/{originalDirname}/{originalFilename} without anything else. The original basename is anyway defined by "files" and may can contain the frame already, or?

[hound]

continuation line over-indented for hanging indent

[kalisp]

Added explicit collector and validator to check if source file is in project folder.

[antirotor]

it will publish, but during the publishing it does strange things with the name. Here is a piece of log from the integrator:

DEBUG: Source: P:\PROJECTS\OP01_CG_demo\tests\tree-736885_960_720.jpg
DEBUG: Creating new version ...
INFO: Prepared version: v002
DEBUG: Anatomy template name: source
DEBUG: Anatomy template name: source
WARNING: File transfer in queue replaced..
DEBUG: Removed from queue: P:\PROJECTS\OP01_CG_demo\tests\tree-736885_960_720.jpg -> P:\PROJECTS\op01_cg_demo\tests\tree-736885_960_720.jpg.jpg replaced by C:\Users\annat\AppData\Local\Temp\pyblish_tmp_3fx1tx9p\tree-736885_960_720.jpg -> P:\PROJECTS\op01_cg_demo\tests\tree-736885_960_720.jpg.jpg
DEBUG: Anatomy template name: source
INFO: Subset renderAnimationMain and Version 2 written to database..
DEBUG: Integrating source files to destination ...
DEBUG: Copying file ... C:\Users\annat\AppData\Local\Temp\pyblish_tmp_3fx1tx9p\tree-736885_960_720.jpg -> P:\PROJECTS\op01_cg_demo\tests\tree-736885_960_720.jpg.jpg
DEBUG: Copying file ... P:\PROJECTS\OP01_CG_demo\tests\tree-736885_960_720_png.png -> P:\PROJECTS\op01_cg_demo\tests\tree-736885_960_720_png.png.png

Again, note the extensions...

based on this template:

[iLLiCiTiT]

I would use full name for suffix.

Suggested change

	jpeg_file = filename + "_thumb.jpg"
	jpeg_file = filename + "_thumbnail.jpg"

Opinions @antirotor @kalisp @mkolar @jakubjezek001 ?

What about other thumbnail extractors (e.g. in Maya)?

Updated

[antirotor]

Happy to report that it works as expected!