Issue 1185 jupyter notebook #59

RichieBrady · 2021-04-12T15:20:58Z

@PekkaSavolainen This branch is up to date with the latest from today. The dependency for papermill is still in spine items, I am starting to move this to the engine. Note that this requires an update to the toolbox. I will make a pull request for that now.

…ion to tool item

…into notebook_executor

…ls to expand cmd args correctly.

…use_db_server False

…into notebook_executor

…olbox.

…into notebook_executor

… LabelArg.

…into notebook_executor

…into issue_1185_jupyter_notebook

ptsavol · 2021-04-12T15:54:51Z

Ok thanks, got it working and ran my first .ipynb file. I'll test some more and post comments tomorrow.

ptsavol · 2021-04-13T07:50:07Z

About the icon. How about this:

https://fontawesome.com/icons/book?style=solid

ptsavol · 2021-04-13T07:52:21Z

There's been some changes to the Tools spec editor window. Do you mind updating the Notebook spec editor window so they are consistent.

EDIT
Nevermind, this new Notebook spec editor has Jupyter Notebook input variables and Jupyter Notebook output variables lists, which are not in Tool spec editor of course. There's room for improvement but there's going to be changes to the Tool spec editor in the near future, so I think it's ok to make the notebook spec editor window more consistent after the merge.

ptsavol · 2021-04-13T08:32:51Z

@ererkka, @soininen, @manuelma

Anybody interested in testing the functionality/usability of this new item? I can check the code but I'm not a Jupyter Notebook user so I need input about the user experience.

ptsavol · 2021-04-13T09:24:12Z

The logger in papermill.execute_notebook() is the regular python logging module, so the [INFO] and [DEBUG] messages go to the console (i.e. where print() messages go). I was wondering what is your plan for these?

Can we use our own LoggerInterface to catch the messages coming from papermill.execute_notebook() so that we could print them to our Execution Log QTextBrowser?
The messages seem to be in a format that our kernel manager could interpret. Can we execute a Python notebook using papermill in our own Python Console?

manuelma · 2021-04-13T10:07:51Z

I definitely want to give this one a try but I'm a bit busy at the moment. I wouldn't mind if you merge it before I get to try it out, too.

soininen · 2021-04-13T10:34:27Z

papermill package seems to be missing from Toolbox's setup.py causing the unit tests to fail.

ptsavol · 2021-04-13T10:47:47Z

papermill package seems to be missing from Toolbox's setup.py causing the unit tests to fail.

Shouldn't it be in spine_engine setup.py instead?

soininen · 2021-04-13T11:04:40Z

Shouldn't it be in spine_engine setup.py instead?

I missed the fact that the package is imported only in the executable item so I guess spine_engine it is, then.

ptsavol · 2021-04-13T11:57:17Z

@RichieBrady FYI to make testing different spine_items branches easier, I moved category.py from toolbox to spine_items. Please update the spine_items/category.py for the notebook branch. In addition, please add this "item_type": ItemInfo.item_type() to notebook_specifications.py to_dict() function. After these updates, there's no need to change anything in ui_main.py in Spine Toolbox and the PR spine-tools/Spine-Toolbox#1365 can be about the new tests and the 'preferred' spine_items version bump. (if I am correct here, of course).

EDIT

And of course I did not test it enough. You need to add an additional new_spec_dict["item_type"] = ItemInfo.item_type()
to notebook_specification_editor_window.py _save() method. Sorry about the hassle.

ptsavol · 2021-04-14T07:02:24Z

The code needs a bit of house cleaning. There are some leftover methods from Tool that are surely not used by Notebook (for example, upgrade_v1_to_v2() in notebook.py). Also, there are a lot of docstrings that refer to Tool when they should say Notebook.

soininen · 2021-04-14T07:37:55Z

I wouldn't mind if my name was dropped from the :author: fields in the headers of new files. I don't really feel I authored anything here. (I'd rather drop the :author: and :date: fields in every single file since we live in a version controlled world but that is another discussion for another time.)

ptsavol · 2021-04-14T07:37:58Z

spine_items/notebook/notebook.py

+            spec_model_index = self._toolbox.specification_model.specification_index(self.specification().name)
+            spec_menu = NotebookSpecificationMenu(self._toolbox, spec_model_index)
+            actions = {a.text(): a for a in spec_menu.actions()}
+            self._actions = [actions["Edit specification"], actions["Open jupyter notebook"]]


Context-menu don't work because of a typo here. Should be Jupyter (capital J)

ptsavol · 2021-04-14T07:48:42Z

Execution in Source directory fails with this traceback

2021-04-14 10:47:18 INFO: Input Notebook:  shopping_list.ipynb
2021-04-14 10:47:18 INFO: Output Notebook: C:/Users/ttepsa/OneDrive - Teknologian Tutkimuskeskus VTT/Documents/SpineToolboxProjects/Notebook project/.spinetoolbox/items/notebook_2\shopping_list_out.ipynb
Exception in thread Thread-8:
Traceback (most recent call last):
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\papermill\iorw.py", line 197, in read
    json.loads(path)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\spine_engine\spine_engine.py", line 400, in _execute_item_filtered
    item_success = item.execute(filtered_forward_resources, filtered_backward_resources)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\spine_items\notebook\executable_item.py", line 231, in execute
    return_code = self._notebook_instance.execute()
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\spine_items\notebook\notebook_instance.py", line 94, in execute
    return self._console_execute()
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\spine_items\notebook\notebook_instance.py", line 105, in _console_execute
    parameters=self._nb_parameters
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\papermill\execute.py", line 90, in execute_notebook
    nb = load_notebook_node(input_path)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\papermill\iorw.py", line 410, in load_notebook_node
    nb = nbformat.reads(papermill_io.read(notebook_path), as_version=4)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\papermill\iorw.py", line 101, in read
    notebook_metadata = self.get_handler(path).read(path)
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\papermill\iorw.py", line 201, in read
    raise e
  File "C:\Users\ttepsa\.julia\conda\3\envs\notebook\lib\site-packages\papermill\iorw.py", line 191, in read
    with io.open(path, 'r', encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'shopping_list.ipynb'

ptsavol · 2021-04-14T07:53:29Z

A nice to have improvement. Double-clicking a Notebook item on Design View should open the notebook spec editor window (Like a Tool does).

…ecutionManager to execute papermill command. Currently supports python and julia notebooks.

…into issue_1185_jupyter_notebook

codecov-commenter · 2021-04-20T09:13:17Z

Codecov Report

Merging #59 (16d0b13) into master (e66afe5) will decrease coverage by 0.38%.
The diff coverage is 56.40%.

@@            Coverage Diff             @@
##           master      #59      +/-   ##
==========================================
- Coverage   60.57%   60.19%   -0.39%     
==========================================
  Files         241      265      +24     
  Lines       18244    19992    +1748     
==========================================
+ Hits        11052    12034     +982     
- Misses       7192     7958     +766

Impacted Files	Coverage Δ
spine_items/notebook/executable_item.py	`16.90% <16.90%> (ø)`
...ok/widgets/notebook_specification_editor_window.py	`18.28% <18.28%> (ø)`
spine_items/notebook/notebook_instance.py	`19.35% <19.35%> (ø)`
spine_items/notebook/output_resources.py	`23.52% <23.52%> (ø)`
spine_items/notebook/widgets/custom_menus.py	`36.11% <36.11%> (ø)`
spine_items/notebook/utils.py	`37.50% <37.50%> (ø)`
spine_items/notebook/commands.py	`50.00% <50.00%> (ø)`
spine_items/notebook/notebook_icon.py	`60.78% <60.78%> (ø)`
spine_items/notebook/notebook.py	`60.93% <60.93%> (ø)`
...pine_items/notebook/widgets/add_notebook_widget.py	`63.63% <63.63%> (ø)`
... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e66afe5...16d0b13. Read the comment docs.

manuelma · 2021-09-14T10:25:31Z

It looks like @RichieBrady is unable to continue this? How do we proceed then? @PekkaSavolainen @soininen @ererkka @jkiviluo ?

ptsavol · 2021-09-14T14:25:50Z

To be continued in v0.8. See spine-tools/Spine-Toolbox#1185.

RichieBrady added 14 commits January 15, 2021 22:38

Initial commit

bc7f57d

Notebook item updated to send and receive resources in a similar fash…

728759c

…ion to tool item

Merge branch 'master' of https://github.com/Spine-project/spine-items …

e5c0698

…into notebook_executor

Adding ui and widget related files. Testing in toolbox, currently fai…

c2da49a

…ls to expand cmd args correctly.

expand cmd line args now works with new labelled_resource_args() and …

f786ee1

…use_db_server False

Merge branch 'master' of https://github.com/Spine-project/spine-items …

88d8cc7

…into notebook_executor

Changed named from notebook_executor to notebook

4260fbc

Merge branch 'master' of https://github.com/Spine-project/spine-items …

5f8c40d

…into notebook_executor

Adding tests for notebook. Notebook specification editor broken in to…

9836d7d

…olbox.

Adding kernel select in notebook spec editor

c1c3ecf

Merge branch 'master' of https://github.com/Spine-project/spine-items …

ebb482a

…into notebook_executor

Update to latest toolbox and items. This item now uses CmdLineArg and…

4abfbac

… LabelArg.

Merge branch 'master' of https://github.com/Spine-project/spine-items …

94394ad

…into notebook_executor

Merge branch 'master' of https://github.com/Spine-project/spine-items …

05b6f5a

…into issue_1185_jupyter_notebook

Adding readme for notebook item.

8e1bd7a

ptsavol reviewed Apr 14, 2021

View reviewed changes

RichieBrady added 7 commits April 19, 2021 16:51

Removing papermill as dependency. Notebook instance now uses KernelEx…

8786102

…ecutionManager to execute papermill command. Currently supports python and julia notebooks.

Fix KeyValueError breaking context menu

e3641c2

Merge branch 'master' of https://github.com/Spine-project/spine-items …

df89e54

…into issue_1185_jupyter_notebook

Updating category and adding item type to notebook specification

80c4147

Update notebook icon and notebook factory with QColor()

5445e18

Update notebook factory and notebook icon correctly

bb7f91f

Small clean up of notebook instance

16d0b13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 1185 jupyter notebook #59

Issue 1185 jupyter notebook #59

RichieBrady commented Apr 12, 2021

ptsavol commented Apr 12, 2021

ptsavol commented Apr 13, 2021

ptsavol commented Apr 13, 2021 •

edited

Loading

ptsavol commented Apr 13, 2021

ptsavol commented Apr 13, 2021 •

edited

Loading

manuelma commented Apr 13, 2021

soininen commented Apr 13, 2021

ptsavol commented Apr 13, 2021

soininen commented Apr 13, 2021

ptsavol commented Apr 13, 2021 •

edited

Loading

ptsavol commented Apr 14, 2021

soininen commented Apr 14, 2021

ptsavol Apr 14, 2021

ptsavol commented Apr 14, 2021

ptsavol commented Apr 14, 2021

codecov-commenter commented Apr 20, 2021 •

edited

Loading

manuelma commented Sep 14, 2021

ptsavol commented Sep 14, 2021

Issue 1185 jupyter notebook #59

Are you sure you want to change the base?

Issue 1185 jupyter notebook #59

Conversation

RichieBrady commented Apr 12, 2021

ptsavol commented Apr 12, 2021

ptsavol commented Apr 13, 2021

ptsavol commented Apr 13, 2021 • edited Loading

ptsavol commented Apr 13, 2021

ptsavol commented Apr 13, 2021 • edited Loading

manuelma commented Apr 13, 2021

soininen commented Apr 13, 2021

ptsavol commented Apr 13, 2021

soininen commented Apr 13, 2021

ptsavol commented Apr 13, 2021 • edited Loading

ptsavol commented Apr 14, 2021

soininen commented Apr 14, 2021

ptsavol Apr 14, 2021

Choose a reason for hiding this comment

ptsavol commented Apr 14, 2021

ptsavol commented Apr 14, 2021

codecov-commenter commented Apr 20, 2021 • edited Loading

Codecov Report

manuelma commented Sep 14, 2021

ptsavol commented Sep 14, 2021

ptsavol commented Apr 13, 2021 •

edited

Loading

ptsavol commented Apr 13, 2021 •

edited

Loading

ptsavol commented Apr 13, 2021 •

edited

Loading

codecov-commenter commented Apr 20, 2021 •

edited

Loading