Suggested way of decoupling large Pluto notebooks in smaller "pieces" ?

[gianmariomanca]

Once Pluto notebook start becoming too large to be managed comfortably, what would be the best way to split/decouple pieces of code?

For example move function definitions to separate file, into modules, and the including the files in a Pluto cell and importing the module?

How does Pluto react to a changes in the "external" modules. Would reloading the cell with the "include" and "import" work? How would you re-trigger evaluation of the entire notebook? Would one have to turn the notebook off and re-start it? That could be very slow. Can we add a button for it that can re-trigger the evaluation of all the cells leaving to Julia to figure out what needs to be re-compiled? Would that be possible and/or faster?

Partly related to: #1344

[disberd]

I stumbled upon a similar problem not long ago where I was generating a notebook that had both descriptive parts in markdown and a lot of computations based on functions defined within the notebook itself.

Things were starting to become quite laggy with the notebook (I had >1500 lines of code including all the comments and markdown, but still a significant portion of it was functions).

I decided to move all the function definitions into a separate notebook where I could still use pluo reactivity to add and debug functions and then simply include this file in the other descriptive notebook where I was just basically creating a report with the help of the functions defined in the other file.

The straight use of include does not work as you get an error when trying to re-evaluate a cell that contains an include statement in Pluto. This is a known issue and a custom ingredients function was suggested in #1101 to deal with this.
The problem with the ingredients function (it was by design there, it was just a convenience problem for me) is that it puts all functions in a module of its own so everything has to be called by prepending the module name.

For this reason, when macro analysis capabilities were brought into Pluto, I created a new @ingredients macro in my unregistered PlutoUtils package.
With the macro exported (or all defined) variables and functions can be brought into scope from another notebook and the definitions can be refreshed by just re-executing manually the cell where the file is included with @ingredients.

If you end up going the separate notebook for functions, you might also want to have some internal testing for your functions in the function definition notebook which might be time consuming (I often use @benchmark when developing functions in notebooks) and you don't want these cells to execute everytime you load the definition notebook from somewhere else.

For this reason I also added a functionality to my Pluto fork to tag some cells as notebook exclusive so that they are only executed directly from the notebook they are defined in, and they are commented out in the actual notebook file so they are skipped when including it from other places.
I found this capability extremely useful for direct development solely using Pluto notebooks and I basically implemented it following the pull request #1209 for reflecting cell disable status on the notebook file.

[BrettKnoss]

I proposed this some time ago, and if it went the other way to breakdown markdown blocks sentence and clause. I was thinking of modeling it on COBOL grammar, with the ability to reuse valiable names in each division, and a play button for a paragraph, consisting of a multiple blocks.

[fonsp]

Update: #2018

[SachaRG]

I have also been hoping for a feature like this. In my case, I end up using cumbersome parameter definitions to ensure that a simulation I run in one part of the notebook is decoupled from a simulation run previously. This is necessary not only for runtime, but also to allow me to demonstrate different effects in different sections of the notebook (e.g. I want some variable Δ to be a constant in one section and be defined on a slider in another section. I end up naming them Δ1, Δ2, etc., but it is cumbersome.)

One feature I would love to see implemented in Pluto is the ability to define and decouple sections of the notebook. Essentially, to be able to have an intermediate scope between local and global such that a variable is only defined within a section. Any variables that need to be shared between sections could then be manually given a global scope.

[gianmariomanca]

@disberd Thank you. I like your @ingredients macro idea, and your development model with multiple Pluto notebooks. Sounds ideal to me. Especially if you are splitting a pre-existing notebook (otherwise you will need to manually modify a bunch of things, at the very least it would be nice to be able to select the desired behavior, prepend module name, or not). My only concern are stability, portability and future support. Are you planning to register your PlutoUtil package? Any chance to port your other suggestions from your fork to the main Pluto package?

[disberd]

Hi, nice to hear that you like the @ingredients macro idea!
For the moment PlutoUtils is still quite unstable and experimental so I did not plan on registering it yet.
I also realized that there is also another existing (currently empty placeholder) package with the same name so I should also consider changing mine though didn't go into the process of understanding how that is done yet.

My model of developing on using Pluto is still evolving and the more I try with it the more quirks and adaptations I go through.
I actually told @fonsp that I was gonna write a post on my experience with package development with Pluto but it's going slowish as I am still trying to wrap my head around some things.

So while I still plan for the foreseeable future to keep heavily using Pluto for my package development and thus working on PlutoUtils, I wanted to wait till I have some more stable idea myself of what is the most appropriate way forward before setting something in stone.

Currently the only significant feature that I have on my fork that is not on Pluto is the ability to comment out some cells in the notebook file so that they are not executed when the file is included (and have some routine to make sure they are instead executed in the notebook where they are defined, so this notebook exclusivity thing).
This functionality implemented via commenting out cells in the file will not likely be accepted on main Pluto and I myself am probably migrating to just prepending some simple macros to lines that are marked as exclusive so that they are not executed outside of the generating notebooks (this was the original implementation as well and the macros are found here).

This thing of might in some way become a PR to pluto once I try doing that.

While registering is still not super near-future, if you plan on trying out the macro and find some suggestions or issues please feel free to file an issue on PlutoUtils and I'll definitely try to help