bpo-36721: Add --embed option to python-config

[vstinner]

To embed Python into an application, a new --embed option must be
passed to "python3-config --libs --embed" to get "-lpython3.8" (link
the application to libpython). To support Python 3.7 and older, try
"python3-config --libs" (without --embed) on failure.

Add a pkg-config "python-3.8-embed" module to embed Python into an
application: "pkg-config python-3.8-embed --libs" includes
"-lpython3.8". To support Python 3.7 and older, try "pkg-config
python-X.Y --libs" (without "-embed") on failure (replace X.Y with
the Python version).

On the other hand, "pkg-config python3.8 --libs" no longer contains
"-lpython3.8". C extensions must not be linked to libpython (except
on Android, case handled by the script); this change is backward
incompatible on purpose.

"make install" now also installs "python-3.8-embed.pc".

https://bugs.python.org/issue36721

[vstinner]

I tested with:

./configure CFLAGS="-O0" --enable-shared --prefix=/opt/py38
make
make install

Commands without -lpython, build a C extension:

$ /opt/py38/bin/python3.8-config --libs
 -lcrypt -lpthread -ldl  -lutil -lm -lm 

# empty output (expected result),
$ pkg-config python-3.8 --libs 

Commands with -lpython, embed Python into an applicaton:

$ /opt/py38/bin/python3.8-config --libs --embed
-lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm -lm 

$ pkg-config python-3.8-embed --libs 
-L/opt/py38/lib -lpython3.8

[vstinner]

@hroncok @xdegaye: Would you mind to review (may also test?) this change? I marked https://bugs.python.org/issue36721 as a release blocker.

[vstinner]

cc @doko42

[vstinner]

I amended (and rebased) my commit to be able to update the commit to address @hroncok review.

[xdegaye]

As expected, all python-config commands print the argument to link against libpython on Android when python is built with this PR (in case you wonder, pthread is not printed as it is part of libc on Android):

generic_x86_64:/data/local/tmp/python $ sh bin/python3.8-config --libs
-lpython3.8d -ldl  -lm -lm
generic_x86_64:/data/local/tmp/python $ sh bin/python3.8-config --libs --embed
-lpython3.8d -ldl  -lm -lm
generic_x86_64:/data/local/tmp/python $ bin/python lib/python3.8/config-3.8d/python-config.py --libs
-lpython3.8d -ldl -lm -lm
generic_x86_64:/data/local/tmp/python $ bin/python lib/python3.8/config-3.8d/python-config.py --libs --embed
-lpython3.8d -ldl -lm -lm

pkg-config is not installed, so could not be tested.

[xdegaye]

Just a nit-pick

[vstinner]

As expected, all python-config commands print the argument to link against libpython on Android when python is built with this PR (...)

Yeah, it's the expected behavior. But I asked you to double check just in case :-) I'm not confident when I modify the build system. Thanks!

[hroncok]

Should we try to implement the waf part before this is merged to see how possible it is?

[vstinner]

Should we try to implement the waf part before this is merged to see how possible it is?

Are you talking about the code to handle Python 3.8 and older version? The fallback described in the documentation should be trivial to implement, no? Why would it be not possible?

[hroncok]

It should be entirely possible. It was just an idea.

[vstinner]

Even if I'm not confident in my change (add --embed option), I chose to merge it anyway since at least "waf" build system is broken by my other changes (no longer link C extensions to libpython). I would like to get this change into Python 3.8 beta1 to attempt to fix most applications embedding Python.

Anyway, if something goes wrong, we still have plenty of time to decide what to do before 3.8.0 final, scheduled for 2019-10-21: https://www.python.org/dev/peps/pep-0569/

[hroncok]

This is what it gave me in Fedora, feels a bit inconsistent:

   /usr/lib64/pkgconfig/python-3.8-embed.pc
   /usr/lib64/pkgconfig/python-embed-3.8d.pc
   /usr/lib64/pkgconfig/python3-embed.pc

[vstinner]

Should be fixed by PR #13551 :-)