From Python, import Hy module from string with importlib

[valconius]

First, a bare-bones example from regular Python. Given a very simple Python program,

def main_py():
    print("This was from Python!")

…, we load this if it is in a file named hello.py by,

import hello
hello.main_py()

…, or we can feed it as a string to importlib and load the result with exec:

import sys, importlib.util
module_name = "hello_module"
module_source = """
def main_py():
    print("This was from Python!")
"""
spec = importlib.util.spec_from_loader(module_name, loader=None)
module = importlib.util.module_from_spec(spec)
exec(module_source, module.__dict__)
sys.modules[module_name] = module
globals()[module_name] = module
hello_module.main_py() # "This was from Python!"

I am not terribly familiar with Python, so afaik exec' accepts both a GLOBALS and LOCALS dict, so I wonder, might there any significance to doing the above with exec(module_source, None, module.dict)or would that not make sense? Wouldglobals()[module_name] = module` need to be changed? Anyway, that was but an aside; the actual problem comes next.

Given a very simple Hy program,

(defn main_hy []
    (print "This was from Hy!"))

…, we load this if it is in a file named hello.hy by,

import hy
import hello
hello.main_hy() # "This was from Hy!"

…, however, there is no obvious way to infer how to modify importlib.util.spec_from_loader's loader parameter to tell importlib that the provided source code should use something like hy/importer.py's modification of importlib.machinery.SOURCE_SUFFIXES.insert(0, ".hy") or the like.

[Kodiologist]

I converted this issue to a discussion because it doesn't appear to be a bug report or feature request.

What are you trying to accomplish here? Your example of how to load a Python module from a string seems like a very elaborate way to get what will ultimately be pretty similar to a SimpleNamespace. Unless your actual need is very arcane, there's probably a simple way to fulfill it in Hy, too.

[valconius]

Hi, thanks, I didn't realize github had discussions. Wow there's a lot of stuff under this tab.

The use case relates to different little prototypes of notebooking environments in the sense of jupyter notebook or emacs org mode, where a block is granted its own, for lack of a better way of putting it, lexical namespace. So a snippet of code lives across multiple physical blocks and at some trigger is compiled with various substitution rules into a virtual block that is, usually, either sent to a jupyter-style kernel or else sent to a REPL. I thought Hy's built-in way to load Hy code from a file, while nice, would lead to make clashes, for it is common to have a list of nearly-identical blocks right next to each other. Packing the blocks into their own respective namespaces as modules and re-loading all these modules on demand in a single python REPL would probably be a lot more convenient that the presently popular solution of using zeromq to communicate across disparate python REPLs.

I will look at your library, thanks. I do think however that using importlibb directly is the easiest thing, as it seems like it would work as-is if only I knew what loader was for, though now that I think of it, the answer is probably in the Hy function definition responsible for execing Hy code as a string from within Python.

[Kodiologist]

The approach usually taken by notebook-like interfaces, including Jupyter and Org-mode, is to evaluate every code block in a common module rather than try to isolate them into separate modules, or separate namespaces of some other kind. This is desirable because it allows code in one block to refer to variables defined in an earlier block. The way to do it in Hy is to call hy.eval with the same module object (or just a dictionary) each time. I recommend passing that object in as both locals and module for the most intuitive semantics.