Linking libboost_python.so to libpython

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Linking libboost_python.so to libpython

Boost - Dev mailing list
tools/build/src/tools/python.jam says:

    # On *nix, we do not want to link either Boost.Python or Python
extensions
    # to libpython, because the Python interpreter itself provides all those
    # symbols. If we linked to libpython, we would get duplicate symbols. So
    # declare two targets -- one for building extensions and another for
    # embedding.

Why would that give duplicate symbols? Wouldn't ELF symbol interposition
mean that only one is used? Why would it be a problem to link to libpython?

In Fedora's Boost RPM packages we *do* link libboost_python.so to the
system libpython (and we don't get any problems). But to do this, we have
these two local patches:

https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch

and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
we have this patch to pass the "m" into the Boost build:

https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch

I'd prefer not to need these local hacks. Would the upstream Boost.Python
consider making changes so that libboost_python.so can be (optionally)
linked to libpython?

Is there a cleaner way to do this than those patches?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
Hi Jonathan,

On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:

> tools/build/src/tools/python.jam says:
>
>      # On *nix, we do not want to link either Boost.Python or Python
> extensions
>      # to libpython, because the Python interpreter itself provides all those
>      # symbols. If we linked to libpython, we would get duplicate symbols. So
>      # declare two targets -- one for building extensions and another for
>      # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?

Do all *nix variants use ELF these days ?

> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems).

What is your rationale for this, given that (according to the above
reasoning) symbols will be resolved from the application that's loading
the module ?

>   But to do this, we have
> these two local patches:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?

I fully agree that whatever seems the right choice of build (including
link) process for a given platform, we (boost) should incorporate that
into our own infrastructure, so downstream package maintainers don't
need to patch it. I'll leave it to the Boost.Build maintainers to answer
the technical details of that process.

Thanks,

Stefan
--

       ...ich hab' noch einen Koffer in Berlin...


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
On Fri, 3 Apr 2020 at 16:53, Stefan Seefeld <[hidden email]> wrote:

> Hi Jonathan,
> On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
>
> tools/build/src/tools/python.jam says:
>
>     # On *nix, we do not want to link either Boost.Python or Python
> extensions
>     # to libpython, because the Python interpreter itself provides all those
>     # symbols. If we linked to libpython, we would get duplicate symbols. So
>     # declare two targets -- one for building extensions and another for
>     # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
>
> Do all *nix variants use ELF these days ?
>

All the ones in common usage, yes. And for my purposes, I only care about
one specific OS (Fedora Linux) which is definitely ELF.



> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems).
>
> What is your rationale for this, given that (according to the above
> reasoning) symbols will be resolved from the application that's loading the
> module ?
>

The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667

That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
people agree that linking to libpython is correct. Dave Abrahams seems to
be the origin of the "linking will cause duplicate symbols" issue, which
might only be the case when the python interpreter is statically linked to
libpython.a. Since linux distros usually (maybe always?) dynamically link
the python interpreter to libpythonX.Ym.so, that isn't an issue, and
Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
distros having to kluge around it.

 But to do this, we have

> these two local patches:
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?
>
> I fully agree that whatever seems the right choice of build (including
> link) process for a given platform, we (boost) should incorporate that into
> our own infrastructure, so downstream package maintainers don't need to
> patch it. I'll leave it to the Boost.Build maintainers to answer the
> technical details of that process.
>

I should note that the linked patches worked fine up to Boost 1.69.0 but
since 1.70.0 they break the build somehow, so that the mpi.so python
extensions are no longer built. I am unable to figure out the problem,
because I find jamfiles completely impenetrable. I'd love to try using
cmake instead, but that doesn't seem to be supported for Boost.MPI.

Boost 1.70.0 and 1.71.0 fail to build with:

...updating 24 targets...
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71.0
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.a
...updated 24 targets...
+ mkdir -p /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost
+ touch /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/__init__.py
+ mv /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
mv: cannot stat
'/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
No such file or directory

And 1.72.0 fails even earlier, with what looks like a conflict between
building Boost.Serialization with "python" and "python-for-extensions"
properties (if I'm reading python.jam right, which is unlikely):


+ echo ============================= build openmpi-x86_64 ==================
+ ./b2 -d+2 -q -j8 --with-mpi --with-graph_parallel
--build-dir=openmpi-x86_64 variant=release threading=multi
debug-symbols=on pch=off python=3.8 stage
Performing configuration checks

    - default address-model    : 64-bit
    - default architecture     : x86
warning: non-free usage requirements <threading>multi ignored
warning: in main-target mpi at
/builddir/build/BUILD/boost_1_72_0/tools/build/src/user-config.jam:6
warning: non-free usage requirements <threading>multi ignored
warning: in main-target boost_mpi at libs/mpi/build/Jamfile.v2:120
warning: non-free usage requirements <threading>multi ignored
warning: in main-target boost_mpi_python at libs/mpi/build/Jamfile.v2:145
error: Name clash for
'<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
error:
error: Tried to build the target twice, with property sets having
error: these incompatible properties:
error:
error:     -  none
error:     -  <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
error:
error: Please make sure to have consistent requirements for these
error: properties everywhere in your project, especially for install
error: targets.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Strong objection from me.

No python extensions nor applications that embed a python interpreter
should link with libpythonX.Y. The modules are effectively plugins,
and are compiled to be dlopened by the interpret and interpreter is
the one that must provide the symbols.

How an interpreter has been loaded, can be different from how an
application/extenion is linked, and thus symbols can be different.
Duplicate symbols are not a problem from ELF when they are the same,
it can become a problem when they are conflicting yet compatible. And
at that point modules can fail to load. The current schema allows
sharing compiled extensions across different builds of the python
interpret that may be shipped by different vendors.

Neither Ubuntu nor Debian links libpythonX.Y for any extension
modules, or apps. Please continue not linking against libpythonX.Y and
keep the binaries with unresolved python symbols.

For example, _ssl.so compiled extensions shipped by the python itself
has all the Py* symbols undefined, and is not linked against
libpythonX.Y.

Boost upstream, should follow Python upstream and do what they
recommend, which is not linking with libpythonX.Y.

On Fri, 3 Apr 2020 at 16:43, Jonathan Wakely via Boost
<[hidden email]> wrote:

>
> tools/build/src/tools/python.jam says:
>
>     # On *nix, we do not want to link either Boost.Python or Python
> extensions
>     # to libpython, because the Python interpreter itself provides all those
>     # symbols. If we linked to libpython, we would get duplicate symbols. So
>     # declare two targets -- one for building extensions and another for
>     # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
>
> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems). But to do this, we have
> these two local patches:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost



--
Regards,

Dimitri.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 03/04/2020 17:17, Jonathan Wakely via Boost wrote:

> The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667
>
> That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
> people agree that linking to libpython is correct. Dave Abrahams seems to
> be the origin of the "linking will cause duplicate symbols" issue, which
> might only be the case when the python interpreter is statically linked to
> libpython.a. Since linux distros usually (maybe always?) dynamically link
> the python interpreter to libpythonX.Ym.so, that isn't an issue, and
> Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
> distros having to kluge around it.

Dave got that originally from me.

The problem was this:

1. Python extension A is built against Boost.Python v1.X.

2. Python extension B is built against Boost.Python v1.Y.

3. What happens if end user imports Python extensions A and B into the
same Python?

If Boost.Python insists on a Python link for Extension A, and a
different Python link for Extension B, now what happens?

The generally accepted hack was to load Python extensions using
dlopen(RTLD_LOCAL), which Python accepted as the least worst of all
options back in the early 2000s, and incorporated that into CPython as
the default semantic for loading extensions. Therefore there is no
global symbol resolution at play here, instead Boost.Python ends up
loading multiple copies of libpython.so, which is for obvious reasons
very bad. So we don't do that.

I have no idea what has happened since then, and all the above
information may be out of date. But that was the rationale ~20 years ago
or so. It made a lot of sense at that time because of the fundamental
brokenness that is ELF. Note that neither PE nor MachO have any of these
problems.

Since ~2002, I patched GCC with -fvisibility in ~2005, and Dave
committed a patchset for exporting visible symbols for a good chunk of
Boost. So it may now be possible to completely change how Boost.Python
does things, and meet your request.

Niall

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Jonathan Wakely wrote:
> Boost 1.70.0 and 1.71.0 fail to build with:
> ...updating 24 targets...
...
> ...updated 24 targets...
...
> + mv
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
> mv: cannot stat
> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
> No such file or directory

This is not failure to build; the error comes from the script, it can't find
mpi.so.

> And 1.72.0 fails even earlier, with what looks like a conflict between
> building Boost.Serialization with "python" and "python-for-extensions"
> properties (if I'm reading python.jam right, which is unlikely):
...
> error: Name clash for
> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
> error:
> error: Tried to build the target twice, with property sets having
> error: these incompatible properties:
> error:
> error:     -  none
> error:     -  <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config

I don't think this is a conflict between `python` and
`python-for-extensions`. Rather, it is a conflict between
libboost_serialization.so by itself (without rpath), and
libboost_serialization.so built as a dependency of libboost_mpi.so (for
which someone tries to set rpath.)

What does user-config.jam contain?

(Also... before spending too much time on this, consider trying 1.73 as
well.)


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, 3 Apr 2020 at 17:29, Dimitri John Ledkov <[hidden email]> wrote:

> Strong objection from me.
>
> No python extensions nor applications that embed a python interpreter
> should link with libpythonX.Y. The modules are effectively plugins,
> and are compiled to be dlopened by the interpret and interpreter is
> the one that must provide the symbols.
>
> How an interpreter has been loaded, can be different from how an
> application/extenion is linked, and thus symbols can be different.
> Duplicate symbols are not a problem from ELF when they are the same,
> it can become a problem when they are conflicting yet compatible. And
> at that point modules can fail to load. The current schema allows
> sharing compiled extensions across different builds of the python
> interpret that may be shipped by different vendors.
>
> Neither Ubuntu nor Debian links libpythonX.Y for any extension
> modules, or apps. Please continue not linking against libpythonX.Y and
> keep the binaries with unresolved python symbols.
>
> For example, _ssl.so compiled extensions shipped by the python itself
> has all the Py* symbols undefined, and is not linked against
> libpythonX.Y.
>
> Boost upstream, should follow Python upstream and do what they
> recommend, which is not linking with libpythonX.Y.
>

That's fine as the default, why do you object to making it configurable, so
that distros that do link python modules to libpython can do so for
Boost.Python too?

On Fedora, I see

$ ldd /usr/lib64/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffde6583000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007fb5deefb000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007fb5dec1b000)
*        libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0
(0x00007fb5de8cf000)*
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb5de8ad000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb5de6e4000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fb5de6ca000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb5de6c1000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007fb5de6bc000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fb5de576000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb5defed000)

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, 3 Apr 2020 at 18:07, Peter Dimov via Boost <[hidden email]>
wrote:

> Jonathan Wakely wrote:
> > Boost 1.70.0 and 1.71.0 fail to build with:
> > ...updating 24 targets...
> ...
> > ...updated 24 targets...
> ...
> > + mv
> >
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
> >
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
> > mv: cannot stat
> >
> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
> > No such file or directory
>
> This is not failure to build; the error comes from the script, it can't
> find
> mpi.so.
>

Right, but mpi.so doesn't appear to get built. Maybe it did, and is just
not where our build script expects it, I should check that.



>
> > And 1.72.0 fails even earlier, with what looks like a conflict between
> > building Boost.Serialization with "python" and "python-for-extensions"
> > properties (if I'm reading python.jam right, which is unlikely):
> ...
> > error: Name clash for
> >
> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
> > error:
> > error: Tried to build the target twice, with property sets having
> > error: these incompatible properties:
> > error:
> > error:     -  none
> > error:     -  <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
>
> I don't think this is a conflict between `python` and
> `python-for-extensions`. Rather, it is a conflict between
> libboost_serialization.so by itself (without rpath), and
> libboost_serialization.so built as a dependency of libboost_mpi.so (for
> which someone tries to set rpath.)
>

Ah, we also patch out the setting of rpaths, maybe we need to make that
happen in an additional place now. Thanks for the clues what to look at
next.


> What does user-config.jam contain?
>
>
import os ;
local RPM_OPT_FLAGS = [ os.environ RPM_OPT_FLAGS ] ;
local RPM_LD_FLAGS = [ os.environ RPM_LD_FLAGS ] ;
using gcc : : : <compileflags>$(RPM_OPT_FLAGS) <linkflags>$(RPM_LD_FLAGS) ;
using mpi ;
using python : 3.8 : /usr/bin/python3 : /usr/include/python3.8m : : : ;





> (Also... before spending too much time on this, consider trying 1.73 as
> well.)
>
>
I will, although I would be surprised if these problems have just gone away
by themselves, given that they seem to be related to downstream
customizations for Fedora, and so not reported upstream to Boost.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, 3 Apr 2020 at 18:13, Niall Douglas via Boost <[hidden email]>
wrote:

> On 03/04/2020 17:17, Jonathan Wakely via Boost wrote:
> > The change was made for
> https://bugzilla.redhat.com/show_bug.cgi?id=1102667
> >
> > That links to https://svn.boost.org/trac10/ticket/2615 which suggests
> other
> > people agree that linking to libpython is correct. Dave Abrahams seems to
> > be the origin of the "linking will cause duplicate symbols" issue, which
> > might only be the case when the python interpreter is statically linked
> to
> > libpython.a. Since linux distros usually (maybe always?) dynamically link
> > the python interpreter to libpythonX.Ym.so, that isn't an issue, and
> > Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
> > distros having to kluge around it.
>
> Dave got that originally from me.
>
> The problem was this:
>
> 1. Python extension A is built against Boost.Python v1.X.
>
> 2. Python extension B is built against Boost.Python v1.Y.
>
> 3. What happens if end user imports Python extensions A and B into the
> same Python?
>
> If Boost.Python insists on a Python link for Extension A, and a
> different Python link for Extension B, now what happens?
>
> The generally accepted hack was to load Python extensions using
> dlopen(RTLD_LOCAL), which Python accepted as the least worst of all
> options back in the early 2000s, and incorporated that into CPython as
> the default semantic for loading extensions. Therefore there is no
> global symbol resolution at play here, instead Boost.Python ends up
> loading multiple copies of libpython.so, which is for obvious reasons
> very bad. So we don't do that.
>
> I have no idea what has happened since then, and all the above
> information may be out of date. But that was the rationale ~20 years ago
> or so. It made a lot of sense at that time because of the fundamental
> brokenness that is ELF. Note that neither PE nor MachO have any of these
> problems.
>
> Since ~2002, I patched GCC with -fvisibility in ~2005, and Dave
> committed a patchset for exporting visible symbols for a good chunk of
> Boost. So it may now be possible to completely change how Boost.Python
> does things, and meet your request.
>

Additionally, in a distro like Fedora everything is built with the same
versions of Boost and Python anyway.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, 3 Apr 2020 at 18:11, Jonathan Wakely <[hidden email]>
wrote:

>
>
> On Fri, 3 Apr 2020 at 18:07, Peter Dimov via Boost <[hidden email]>
> wrote:
>
>> Jonathan Wakely wrote:
>> > Boost 1.70.0 and 1.71.0 fail to build with:
>> > ...updating 24 targets...
>> ...
>> > ...updated 24 targets...
>> ...
>> > + mv
>> >
>> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
>> >
>> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
>> > mv: cannot stat
>> >
>> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
>> > No such file or directory
>>
>> This is not failure to build; the error comes from the script, it can't
>> find
>> mpi.so.
>>
>
> Right, but mpi.so doesn't appear to get built. Maybe it did, and is just
> not where our build script expects it, I should check that.
>
>
>
>>
>> > And 1.72.0 fails even earlier, with what looks like a conflict between
>> > building Boost.Serialization with "python" and "python-for-extensions"
>> > properties (if I'm reading python.jam right, which is unlikely):
>> ...
>> > error: Name clash for
>> >
>> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
>> > error:
>> > error: Tried to build the target twice, with property sets having
>> > error: these incompatible properties:
>> > error:
>> > error:     -  none
>> > error:     -  <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
>>
>> I don't think this is a conflict between `python` and
>> `python-for-extensions`. Rather, it is a conflict between
>> libboost_serialization.so by itself (without rpath), and
>> libboost_serialization.so built as a dependency of libboost_mpi.so (for
>> which someone tries to set rpath.)
>>
>
> Ah, we also patch out the setting of rpaths, maybe we need to make that
> happen in an additional place now. Thanks for the clues what to look at
> next.
>
>
>> What does user-config.jam contain?
>>
>>
> import os ;
> local RPM_OPT_FLAGS = [ os.environ RPM_OPT_FLAGS ] ;
> local RPM_LD_FLAGS = [ os.environ RPM_LD_FLAGS ] ;
> using gcc : : : <compileflags>$(RPM_OPT_FLAGS) <linkflags>$(RPM_LD_FLAGS) ;
> using mpi ;
> using python : 3.8 : /usr/bin/python3 : /usr/include/python3.8m : : : ;
>
>
I should add that we build in stages, because we want to build Boost.MPI
twice, for openmpi and MPICH, so we do a build without MPI first:

./b2 -d+2 -q \
--without-mpi --without-graph_parallel --build-dir=serial \
variant=release threading=multi debug-symbols=on pch=off \
python=3.8 \
stage

and then for each MPI backend we do:

./b2 -d+2  \
--with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
variant=release threading=multi debug-symbols=on pch=off \
python=3.8 stage

so that's probably why libboost_serialization ends up with different sets
of properties.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Jonathan Wakely wrote:
> > (Also... before spending too much time on this, consider trying 1.73 as
> > well.)
>
> I will, although I would be surprised if these problems have just gone
> away by themselves, given that they seem to be related to downstream
> customizations for Fedora, and so not reported upstream to Boost.

They won't... but new ones may appear :-) as we changed the directory for
mpi.so because it wasn't appropriate for Boost to install a file named
`mpi.so` into the default destination /usr/lib.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Jonathan Wakely wrote:

> I should add that we build in stages, because we want to build Boost.MPI
> twice, for openmpi and MPICH, so we do a build without MPI first:
>
> ./b2 -d+2 -q \
> --without-mpi --without-graph_parallel --build-dir=serial \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 \
> stage
>
> and then for each MPI backend we do:
>
> ./b2 -d+2  \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
>
> so that's probably why libboost_serialization ends up with different sets
> of properties.

That's why you weren't getting the error before 1.72, but not the reason why
libboost_serialization ends up with the different property sets.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
> That's why you weren't getting the error before 1.72, but not the reason
> why libboost_serialization ends up with the different property sets.

FWIW, my Travis build/install of --with-mpi works:

https://travis-ci.org/github/boostorg/boost_install/jobs/669821455

and I can't find the place that's setting <dll-path>/usr/lib
<dll-path>/usr/lib/python3.8/config.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Jonathan Wakely wrote:
> > error:     -  none
> > error:     -  <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
...
> and then for each MPI backend we do:
>
> ./b2 -d+2  \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage

I think that what you need to do here is find the place setting <dll-path>
on (I assume) libboost_mpi.so, remove it, and then pass dll-path on the
command line, so that it applies to the whole build:

> ./b2 -d+2  \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
  dll-path=/usr/lib dll-path=/usr/lib/python3.8/config



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Linking libboost_python.so to libpython

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, 3 Apr 2020, 19:26 Peter Dimov via Boost, <[hidden email]>
wrote:

> > That's why you weren't getting the error before 1.72, but not the reason
> > why libboost_serialization ends up with the different property sets.
>
> FWIW, my Travis build/install of --with-mpi works:
>
> https://travis-ci.org/github/boostorg/boost_install/jobs/669821455
>
> and I can't find the place that's setting <dll-path>/usr/lib
> <dll-path>/usr/lib/python3.8/config.
>


I think that's our local patch, I'll check later.




>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost