Boost cannot find numpy37 on Dockerized Ubuntu 20.04

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Boost cannot find numpy37 on Dockerized Ubuntu 20.04

Boost - Users mailing list
Hi there,

I've been trying to get GP-GOMEA running in a dockerized environment, which is a project that is built using Boost.

As part of the build process, the source code is compiled by the g++ compiler, using the flag -lboost_numpy37, which fails because (it seems) numpy cannot be found.

I've reported this also as an issue also in the boost repository, but mclow advised me to send this to the boost-users mailinglist too.

I've create the following Dockerfile as a minimal test case, which runs into the same problem. I'm a bit of a noob when it comes to Boost and numpy, but from what I found so far on the web, this should work (but it doesn't)

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get upgrade -y && apt-get clean
RUN apt-get install -y curl python3.7 python3-dev python3-distutils python3-numpy

# Install latest boost version
RUN apt-get -y install wget g++
RUN wget https://dl.bintray.com/boostorg/release/1.75.0/source/boost_1_75_0.tar.bz2 && \
    tar --bzip2 -xf boost_1_75_0.tar.bz2 && \
    cd boost_1_75_0 && \
    # this will generate ./b2
    ./bootstrap.sh  --prefix=/usr/local && \
    ./b2 --with=all install && \
    sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/local.conf' && \
    ldconfig

# Check numpy installation
RUN find / -name "numpy.hpp"
RUN find / -name "libboost_numpy37.so"
RUN find / -name "libboost_numpy3.so"
RUN find / -name "libboost_numpy.so"
RUN ld -lboost_numpy37 --verbose

This eventually results in the following error:
The command '/bin/sh -c ld -lboost_numpy37 --verbose' returned a non-zero code: 1, because it could not find any libboost_numpy37.so or libboost_numpy37.o file.

The full logs are attached in the GH issue linked above.
Did anyone encounter something similar, or does anyone have an idea what could cause this?

Any help is much appreciated!

-Roy


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost cannot find numpy37 on Dockerized Ubuntu 20.04

Boost - Users mailing list
The boost libraries shipped in Ubuntu 18.04 (bionic) use a different naming scheme for the boost python libraries:

It looks like the python libraries are built with the system default python interpreter when 18.04 got released, not the newer version you are installing.


On Wed, Feb 17, 2021, 10:05 Roy de Bokx via Boost-users <[hidden email]> wrote:
Hi there,

I've been trying to get GP-GOMEA running in a dockerized environment, which is a project that is built using Boost.

As part of the build process, the source code is compiled by the g++ compiler, using the flag -lboost_numpy37, which fails because (it seems) numpy cannot be found.

I've reported this also as an issue also in the boost repository, but mclow advised me to send this to the boost-users mailinglist too.

I've create the following Dockerfile as a minimal test case, which runs into the same problem. I'm a bit of a noob when it comes to Boost and numpy, but from what I found so far on the web, this should work (but it doesn't)

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get upgrade -y && apt-get clean
RUN apt-get install -y curl python3.7 python3-dev python3-distutils python3-numpy

# Install latest boost version
RUN apt-get -y install wget g++
RUN wget https://dl.bintray.com/boostorg/release/1.75.0/source/boost_1_75_0.tar.bz2 && \
    tar --bzip2 -xf boost_1_75_0.tar.bz2 && \
    cd boost_1_75_0 && \
    # this will generate ./b2
    ./bootstrap.sh  --prefix=/usr/local && \
    ./b2 --with=all install && \
    sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/local.conf' && \
    ldconfig

# Check numpy installation
RUN find / -name "numpy.hpp"
RUN find / -name "libboost_numpy37.so"
RUN find / -name "libboost_numpy3.so"
RUN find / -name "libboost_numpy.so"
RUN ld -lboost_numpy37 --verbose

This eventually results in the following error:
The command '/bin/sh -c ld -lboost_numpy37 --verbose' returned a non-zero code: 1, because it could not find any libboost_numpy37.so or libboost_numpy37.o file.

The full logs are attached in the GH issue linked above.
Did anyone encounter something similar, or does anyone have an idea what could cause this?

Any help is much appreciated!

-Roy

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost cannot find numpy37 on Dockerized Ubuntu 20.04

Boost - Users mailing list


Op wo 17 feb. 2021 om 11:04 schreef Anonymous Maarten via Boost-users <[hidden email]>:
The boost libraries shipped in Ubuntu 18.04 (bionic) use a different naming scheme for the boost python libraries:

It looks like the python libraries are built with the system default python interpreter when 18.04 got released, not the newer version you are installing.
Thanks for your answer Maarten!
Please bare with me as I'm a little new to this. Is there some way I can change the interpreter that is used for installing boost? Or do you think that adapting the --prefix flag may also work?  


On Wed, Feb 17, 2021, 10:05 Roy de Bokx via Boost-users <[hidden email]> wrote:
Hi there,

I've been trying to get GP-GOMEA running in a dockerized environment, which is a project that is built using Boost.

As part of the build process, the source code is compiled by the g++ compiler, using the flag -lboost_numpy37, which fails because (it seems) numpy cannot be found.

I've reported this also as an issue also in the boost repository, but mclow advised me to send this to the boost-users mailinglist too.

I've create the following Dockerfile as a minimal test case, which runs into the same problem. I'm a bit of a noob when it comes to Boost and numpy, but from what I found so far on the web, this should work (but it doesn't)

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get upgrade -y && apt-get clean
RUN apt-get install -y curl python3.7 python3-dev python3-distutils python3-numpy

# Install latest boost version
RUN apt-get -y install wget g++
RUN wget https://dl.bintray.com/boostorg/release/1.75.0/source/boost_1_75_0.tar.bz2 && \
    tar --bzip2 -xf boost_1_75_0.tar.bz2 && \
    cd boost_1_75_0 && \
    # this will generate ./b2
    ./bootstrap.sh  --prefix=/usr/local && \
    ./b2 --with=all install && \
    sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/local.conf' && \
    ldconfig

# Check numpy installation
RUN find / -name "numpy.hpp"
RUN find / -name "libboost_numpy37.so"
RUN find / -name "libboost_numpy3.so"
RUN find / -name "libboost_numpy.so"
RUN ld -lboost_numpy37 --verbose

This eventually results in the following error:
The command '/bin/sh -c ld -lboost_numpy37 --verbose' returned a non-zero code: 1, because it could not find any libboost_numpy37.so or libboost_numpy37.o file.

The full logs are attached in the GH issue linked above.
Did anyone encounter something similar, or does anyone have an idea what could cause this?

Any help is much appreciated!

-Roy

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost cannot find numpy37 on Dockerized Ubuntu 20.04

Boost - Users mailing list


Op wo 17 feb. 2021 om 20:30 schreef Anonymous Maarten <[hidden email]>:
Thanks for your answer Maarten!
Please bare with me as I'm a little new to this. Is there some way I can change the interpreter that is used for installing boost? Or do you think that adapting the --prefix flag may also work?  
 
When using the pre-built boost packages of a linux distribution (ubuntu), you're generally stuck with the versions/options that its packagers have chosen.
A distribution freezes versions to make a more robust experience.

Adding a `--prefix` will not help because, first of all, it's not a valid gcc option, and second, ubuntu 18.04 does not provide a libboost-python-py37 shared/static library.
Do you really need python 3.7? Doesn't 3.6 suffice?

Thanks! This helped me understanding the issue a bit further. I think 3.6 should suffice indeed, so I tried using 3.6 by removing the 3.7 installation from the Dockerfile and using the -lboost_numpy3 flag instead.
It seems I ran into some linking issue after this. It was able to find the right numpy3.so and libboost_python3-py36.so, but I'm getting a lot of errors like these:

//usr/lib/x86_64-linux-gnu/libboost_numpy3.so: undefined reference to `PyExc_ValueError'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyLong_AsLong'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyNumber_InPlaceFloorDivide'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyBool_Type'

etc...

I've pasted the full docker file and logs in https://github.com/boostorg/boost/issues/462
Thanks again for any help.

If you really want to use Boost.Python + python 3.7, you can do 2 things:
- stay on ubuntu bionic and build boost yourself (or use an alternative c/c++ package manager, e.g. [conan](https://conan.io/) has a [boost package](https://conan.io/center/boost))
- more to a more recent ubuntu release. e.g. ubuntu 20.10 has python 3.8: https://packages.ubuntu.com/groovy/amd64/libboost-python1.71-dev/filelist
I've also given 20.10 a try, however I ran into the same linking issues as mentioned above. 

 


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost cannot find numpy37 on Dockerized Ubuntu 20.04

Boost - Users mailing list
The missing symbols you quoted come from python itself. It looks like you're missing a "-lpython" linker flag.

Regards, 
Nathan

On Fri, Feb 19, 2021, 4:47 AM Roy de Bokx via Boost-users <[hidden email]> wrote:


Op wo 17 feb. 2021 om 20:30 schreef Anonymous Maarten <[hidden email]>:
Thanks for your answer Maarten!
Please bare with me as I'm a little new to this. Is there some way I can change the interpreter that is used for installing boost? Or do you think that adapting the --prefix flag may also work?  
 
When using the pre-built boost packages of a linux distribution (ubuntu), you're generally stuck with the versions/options that its packagers have chosen.
A distribution freezes versions to make a more robust experience.

Adding a `--prefix` will not help because, first of all, it's not a valid gcc option, and second, ubuntu 18.04 does not provide a libboost-python-py37 shared/static library.
Do you really need python 3.7? Doesn't 3.6 suffice?

Thanks! This helped me understanding the issue a bit further. I think 3.6 should suffice indeed, so I tried using 3.6 by removing the 3.7 installation from the Dockerfile and using the -lboost_numpy3 flag instead.
It seems I ran into some linking issue after this. It was able to find the right numpy3.so and libboost_python3-py36.so, but I'm getting a lot of errors like these:

//usr/lib/x86_64-linux-gnu/libboost_numpy3.so: undefined reference to `PyExc_ValueError'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyLong_AsLong'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyNumber_InPlaceFloorDivide'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyBool_Type'

etc...

I've pasted the full docker file and logs in https://github.com/boostorg/boost/issues/462
Thanks again for any help.

If you really want to use Boost.Python + python 3.7, you can do 2 things:
- stay on ubuntu bionic and build boost yourself (or use an alternative c/c++ package manager, e.g. [conan](https://conan.io/) has a [boost package](https://conan.io/center/boost))
- more to a more recent ubuntu release. e.g. ubuntu 20.10 has python 3.8: https://packages.ubuntu.com/groovy/amd64/libboost-python1.71-dev/filelist
I've also given 20.10 a try, however I ran into the same linking issues as mentioned above. 

 

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost cannot find numpy37 on Dockerized Ubuntu 20.04

Boost - Users mailing list


Op vr 19 feb. 2021 om 22:05 schreef Nathan Ernst via Boost-users <[hidden email]>:
The missing symbols you quoted come from python itself. It looks like you're missing a "-lpython" linker flag.
Thanks! That definitely put me on the right track. Eventually I got it working using the "-lpython3.6m" flag, since I found that the Python3.6 .so file was situated at /usr/lib/x86_64-linux-gnu/libpython3.6m.so
 Many thanks, this is much appreciated! I've pasted the full Dockerfile in https://github.com/boostorg/boost/issues/462, in case anyone is interested.

Regards, 
Nathan

On Fri, Feb 19, 2021, 4:47 AM Roy de Bokx via Boost-users <[hidden email]> wrote:


Op wo 17 feb. 2021 om 20:30 schreef Anonymous Maarten <[hidden email]>:
Thanks for your answer Maarten!
Please bare with me as I'm a little new to this. Is there some way I can change the interpreter that is used for installing boost? Or do you think that adapting the --prefix flag may also work?  
 
When using the pre-built boost packages of a linux distribution (ubuntu), you're generally stuck with the versions/options that its packagers have chosen.
A distribution freezes versions to make a more robust experience.

Adding a `--prefix` will not help because, first of all, it's not a valid gcc option, and second, ubuntu 18.04 does not provide a libboost-python-py37 shared/static library.
Do you really need python 3.7? Doesn't 3.6 suffice?

Thanks! This helped me understanding the issue a bit further. I think 3.6 should suffice indeed, so I tried using 3.6 by removing the 3.7 installation from the Dockerfile and using the -lboost_numpy3 flag instead.
It seems I ran into some linking issue after this. It was able to find the right numpy3.so and libboost_python3-py36.so, but I'm getting a lot of errors like these:

//usr/lib/x86_64-linux-gnu/libboost_numpy3.so: undefined reference to `PyExc_ValueError'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyLong_AsLong'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyNumber_InPlaceFloorDivide'

//usr/lib/x86_64-linux-gnu/libboost_python3-py36.so.1.65.1: undefined reference to `PyBool_Type'

etc...

I've pasted the full docker file and logs in https://github.com/boostorg/boost/issues/462
Thanks again for any help.

If you really want to use Boost.Python + python 3.7, you can do 2 things:
- stay on ubuntu bionic and build boost yourself (or use an alternative c/c++ package manager, e.g. [conan](https://conan.io/) has a [boost package](https://conan.io/center/boost))
- more to a more recent ubuntu release. e.g. ubuntu 20.10 has python 3.8: https://packages.ubuntu.com/groovy/amd64/libboost-python1.71-dev/filelist
I've also given 20.10 a try, however I ran into the same linking issues as mentioned above. 

 

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users