-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mmap for discontiguous data? #587
Comments
Also, data might be split across separate files (due to the 4gb limit of TIFF). I know the file from where a specific XY plane is located. I guess it might be possible to do the loading lazily. |
Awesome. I haven't read that document carefully, but first impression is that you might need to create something a bit like https://github.com/tanmaykm/ChainedVectors.jl. (There may be other examples of similar packages around.) I don't know whether you can |
That's an interesting idea. How would that work with AxisArrays? Would it be a subtype of that or a completely separate thing? |
Also, I'm going to rope @quinnj into this discussion since he was the main architect of the |
Completely separate. You could put the "chained array" inside an AxisArray wrapper, if you wanted. But shoot for completely orthogonal design. I'd even advocate for doing this in two pieces (i.e, two packages):
The TIFF-OME parser would create and return the "chained array," possibly wrapped inside an AxisArray. The advantage of the two-part split is reusability; it's not hard to imagine that other file formats (or other applications) might want to re-use that part. |
For reference, I will link my Discourse answer about using Dagger.jl to do this here: https://discourse.julialang.org/t/mmapping-a-discontiguous-file/2016/2?u=shashi But this should give you the "chained array" bit of this puzzle. :) You might have to create a |
For anyone interested in TIFF-OME, see https://discourse.julialang.org/t/bioformats-in-julia/2440. |
I'm still interested in writing a pure-julia OME-TIFF reader, I think it would be fine to have both. |
@tlnagy could you help check if this issue is already solved? |
I wasn't sure where to post this, but I'm hoping to add support for reading OME-TIFF files to Images.jl. I would also like to add mmap support as these files can get quite large, quite fast. I tried understanding NRRD.jl's usage of mmap to see how I can use that, but the issue I'm running into is how to use mmap for discontiguous array blocks. The general layout of OME-TIFFs is that data is stored in separate 2d XY arrays with labels specifying CZT information. @timholy do you have any suggestions on how to handle this with Images.jl's architecture/mmaping?
ref https://www.micro-manager.org/wiki/Micro-Manager_File_Formats#Image_file_stack_specification
The text was updated successfully, but these errors were encountered: