Test Failures on develop

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Test Failures on develop

Boost - Testing mailing list
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

GCC ends with this:

gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o

   "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
I"/var/boost/run/boost_root" -c -o "../../bin/common/build/gcc-8/release/process_jam_log.o" "../../common/build/../src/p
rocess_jam_log.cpp"

../../common/build/../src/process_jam_log.cpp: In function 'std::__cxx11::string {anonymous}::revision(const string&)':
../../common/build/../src/process_jam_log.cpp:359:16: warning: ignoring return value of 'int system(const char*)', decla
red with attribute warn_unused_result [-Wunused-result]
    std::system("git rev-parse --short=6 HEAD >.short-sha");
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v2/libs/f
ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...
...skipped <p/var/boost/run/boost_regression/stage/bin>process_jam_log for lack of <p../../bin/testing/build/gcc-8/relea
se>process_jam_log...
...failed updating 8 targets...
...skipped 4 targets...
...updated 608 targets...
# Searching for "process_jam_log" in "/var/boost/run/boost_regression/stage/bin"...
Traceback (most recent call last):
 File "run.py", line 71, in <module>
   runner(root)
 File "/var/boost/run/boost_regression_src/regression.py", line 286, in __init__
   self.main()
 File "/var/boost/run/boost_regression_src/regression.py", line 623, in main
   getattr(self,action_m)()
 File "/var/boost/run/boost_regression_src/regression.py", line 580, in command_regression
   self.command_setup()
 File "/var/boost/run/boost_regression_src/regression.py", line 351, in command_setup
   self.build_if_needed(self.process_jam_log,self.pjl_toolset)
 File "/var/boost/run/boost_regression_src/regression.py", line 714, in build_if_needed
   tool[ 'build_path' ] = self.tool_path( tool )
 File "/var/boost/run/boost_regression_src/regression.py", line 739, in tool_path
   , '\n'.join( [ name_or_spec[ 'path' ], build_dir ] )
Exception: Cannot find "process_jam_log" in any of the following locations:
/var/boost/run/boost_regression/stage/bin/process_jam_log
/var/boost/run/boost_regression/stage/bin



Clang ends with this:

clang-linux.compile.c++.without-pch ../../bin/common/build/clang-linux-8/release/process_jam_log.o

 "clang++-8" -c -x c++ -Wno-c99-extensions -fPIC -O3 -Wall -Wno-inline  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_
WARNINGS -I"/var/boost/run/boost_root" -o "../../bin/common/build/clang-linux-8/release/process_jam_log.o" "../../common
/build/../src/process_jam_log.cpp"

...skipped <p../../bin/testing/build/clang-linux-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v
2/libs/filesystem/build/clang-linux-8/release/link-static/visibility-hidden>libboost_filesystem.a...
...skipped <p/var/boost/run/boost_regression/stage/bin>process_jam_log for lack of <p../../bin/testing/build/clang-linux
-8/release>process_jam_log...
...failed updating 8 targets...
...skipped 4 targets...
...updated 608 targets...
# Searching for "process_jam_log" in "/var/boost/run/boost_regression/stage/bin"...
Traceback (most recent call last):
 File "run.py", line 71, in <module>
   runner(root)
 File "/var/boost/run/boost_regression_src/regression.py", line 286, in __init__
   self.main()
 File "/var/boost/run/boost_regression_src/regression.py", line 623, in main
   getattr(self,action_m)()
 File "/var/boost/run/boost_regression_src/regression.py", line 580, in command_regression
   self.command_setup()
 File "/var/boost/run/boost_regression_src/regression.py", line 351, in command_setup
   self.build_if_needed(self.process_jam_log,self.pjl_toolset)
 File "/var/boost/run/boost_regression_src/regression.py", line 714, in build_if_needed
   tool[ 'build_path' ] = self.tool_path( tool )
 File "/var/boost/run/boost_regression_src/regression.py", line 739, in tool_path
   , '\n'.join( [ name_or_spec[ 'path' ], build_dir ] )
Exception: Cannot find "process_jam_log" in any of the following locations:
/var/boost/run/boost_regression/stage/bin/process_jam_log
/var/boost/run/boost_regression/stage/bin


Anyone know what has changed, what is up with this?

Thanks,
Tom

_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list
On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

Are you running the tests in a Linux container?
 
--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Fri, Jun 14, 2019 at 9:14 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:
On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

Are you running the tests in a Linux container?


Yes. 

Container images created from here:
https://github.com/teeks99/boost-cpp-docker

Tom 

_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Fri, Jun 14, 2019 at 9:50 AM Tom Kent via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:14 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:
On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

Are you running the tests in a Linux container?


Yes. 

Container images created from here:
https://github.com/teeks99/boost-cpp-docker

Then the likely issue is a change to b2 that defaults to using parallel jobs to match the available system cores and launches so many compile commands that it runs out of memory. It seems most container virtualization systems are terrible at virtualizing the system information and leak the host information. Only immediate option is to pass b2 an appropriate "-jN" option. Or wait until I fix the problem (I already tried one solution, will keep trying).

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Fri, Jun 14, 2019 at 10:13 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:50 AM Tom Kent via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:14 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:
On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

Are you running the tests in a Linux container?


Yes. 

Container images created from here:
https://github.com/teeks99/boost-cpp-docker

Then the likely issue is a change to b2 that defaults to using parallel jobs to match the available system cores and launches so many compile commands that it runs out of memory. It seems most container virtualization systems are terrible at virtualizing the system information and leak the host information. Only immediate option is to pass b2 an appropriate "-jN" option. Or wait until I fix the problem (I already tried one solution, will keep trying).


Hmm, I'm not sure that is it. I was already running with -j8, on a 8 core machine.

I run:
run.py --runner=teeks99-dkr-dc8-2a --toolsets=clang-8~c++2a                    \
  "--bjam-options=-j8 address-model=64 --remove-test-targets"                  \
  --comment=info.html --tag=develop 
 

_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Fri, Jun 14, 2019 at 11:20 AM Tom Kent <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 10:13 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:50 AM Tom Kent via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:14 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:
On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

Are you running the tests in a Linux container?


Yes. 

Container images created from here:
https://github.com/teeks99/boost-cpp-docker

Then the likely issue is a change to b2 that defaults to using parallel jobs to match the available system cores and launches so many compile commands that it runs out of memory. It seems most container virtualization systems are terrible at virtualizing the system information and leak the host information. Only immediate option is to pass b2 an appropriate "-jN" option. Or wait until I fix the problem (I already tried one solution, will keep trying).


Hmm, I'm not sure that is it. I was already running with -j8, on a 8 core machine.

I run:
run.py --runner=teeks99-dkr-dc8-2a --toolsets=clang-8~c++2a                    \
  "--bjam-options=-j8 address-model=64 --remove-test-targets"                  \
  --comment=info.html --tag=develop 
 

I also have one physical machine (arm based) that is running the same(-ish) thing, with the same failure.

_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Fri, Jun 14, 2019 at 12:58 PM Tom Kent <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 11:20 AM Tom Kent <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 10:13 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:50 AM Tom Kent via Boost-Testing <[hidden email]> wrote:


On Fri, Jun 14, 2019 at 9:14 AM Rene Rivera via Boost-Testing <[hidden email]> wrote:
On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

Are you running the tests in a Linux container?


Yes. 

Container images created from here:
https://github.com/teeks99/boost-cpp-docker

Then the likely issue is a change to b2 that defaults to using parallel jobs to match the available system cores and launches so many compile commands that it runs out of memory. It seems most container virtualization systems are terrible at virtualizing the system information and leak the host information. Only immediate option is to pass b2 an appropriate "-jN" option. Or wait until I fix the problem (I already tried one solution, will keep trying).


Hmm, I'm not sure that is it. I was already running with -j8, on a 8 core machine.

I run:
run.py --runner=teeks99-dkr-dc8-2a --toolsets=clang-8~c++2a                    \
  "--bjam-options=-j8 address-model=64 --remove-test-targets"                  \
  --comment=info.html --tag=develop 
 

I also have one physical machine (arm based) that is running the same(-ish) thing, with the same failure.

Any luck tracking this down? I haven't been able to figure out a mitigation from my side. 

If the changes on develop are causing problems like this, can we revert them until we figure it out?

Tom 

_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list
In reply to this post by Boost - Testing mailing list
Since it's not due to the b2 -j change..

On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

GCC ends with this:

gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o

   "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
I"/var/boost/run/boost_root" -c -o "../../bin/common/build/gcc-8/release/process_jam_log.o" "../../common/build/../src/p
rocess_jam_log.cpp"

../../common/build/../src/process_jam_log.cpp: In function 'std::__cxx11::string {anonymous}::revision(const string&)':
../../common/build/../src/process_jam_log.cpp:359:16: warning: ignoring return value of 'int system(const char*)', decla
red with attribute warn_unused_result [-Wunused-result]
    std::system("git rev-parse --short=6 HEAD >.short-sha");
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v2/libs/f
ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...

I guess you'd need to find out what it says for trying to build filesystem lib.
 

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Thu, Jun 20, 2019 at 7:15 AM Rene Rivera <[hidden email]> wrote:
Since it's not due to the b2 -j change..

On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

GCC ends with this:

gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o

   "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
I"/var/boost/run/boost_root" -c -o "../../bin/common/build/gcc-8/release/process_jam_log.o" "../../common/build/../src/p
rocess_jam_log.cpp"

../../common/build/../src/process_jam_log.cpp: In function 'std::__cxx11::string {anonymous}::revision(const string&)':
../../common/build/../src/process_jam_log.cpp:359:16: warning: ignoring return value of 'int system(const char*)', decla
red with attribute warn_unused_result [-Wunused-result]
    std::system("git rev-parse --short=6 HEAD >.short-sha");
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v2/libs/f
ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...

I guess you'd need to find out what it says for trying to build filesystem lib.

 Here's what I'm seeing with the filesystem error:

   ln -f -s "../../libs/detail/include/boost/detail/has_default_constructor.hpp" "../../../boost_root/boost/detail/has_ 
default_constructor.hpp" 

gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL 
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin. 
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o" "/var/boost/run/boost_root/libs/file 
system/src/path_traits.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/path_traits.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f 
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o... 
gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin.
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o" "/var/boost/run/boost_root/libs/file
system/src/portability.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/portability.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o... 


And several more like that.

However, I'm also suspicious of the timing with the processor change.

Looking back at the last successful builds of develop on linux. I have a few from linux/x86_64/docker machines. The last one I saw was building with gcc-8 on June 4 around 6:30am CDT. There should have been a successful clang run about 5hrs after that (11:30am CDT), and another successful gcc around 10hrs after the last one that was actually successful (4:30pm CDT). 

This seems like it would correspond to the change to build here:
https://github.com/boostorg/build/commit/451059949d92b52b21f802562a50587f1eeecb82

The other failure I started seeing was only my linux/arm64 machine. This doesn't run docker, so might not be affected in the same way. The last successful run on there was at 4:50pm CDT on June 4...but it typically takes around 9hrs to run so it probably got the pre 451059949 commit. However here I can't rule out that it was impacted by a8ab76ef973.

I'm going to try to standup some normal VMs soon and run on them to see if they are affected in the same way. It would also be nice if we could try backing out the usage of _SC_NPROCESSORS_ONLN and see if that fixes things, is that feasible to try for a couple days?

Thanks,
Tom



_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list
On Thu, Jun 20, 2019 at 10:28 PM Tom Kent <[hidden email]> wrote:

On Thu, Jun 20, 2019 at 7:15 AM Rene Rivera <[hidden email]> wrote:
Since it's not due to the b2 -j change..

On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

GCC ends with this:

gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o

   "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
I"/var/boost/run/boost_root" -c -o "../../bin/common/build/gcc-8/release/process_jam_log.o" "../../common/build/../src/p
rocess_jam_log.cpp"

../../common/build/../src/process_jam_log.cpp: In function 'std::__cxx11::string {anonymous}::revision(const string&)':
../../common/build/../src/process_jam_log.cpp:359:16: warning: ignoring return value of 'int system(const char*)', decla
red with attribute warn_unused_result [-Wunused-result]
    std::system("git rev-parse --short=6 HEAD >.short-sha");
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v2/libs/f
ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...

I guess you'd need to find out what it says for trying to build filesystem lib.

 Here's what I'm seeing with the filesystem error:

   ln -f -s "../../libs/detail/include/boost/detail/has_default_constructor.hpp" "../../../boost_root/boost/detail/has_ 
default_constructor.hpp" 

gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL 
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin. 
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o" "/var/boost/run/boost_root/libs/file 
system/src/path_traits.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/path_traits.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f 
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o... 
gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin.
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o" "/var/boost/run/boost_root/libs/file
system/src/portability.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/portability.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o... 


And several more like that.

However, I'm also suspicious of the timing with the processor change.

Looking back at the last successful builds of develop on linux. I have a few from linux/x86_64/docker machines. The last one I saw was building with gcc-8 on June 4 around 6:30am CDT. There should have been a successful clang run about 5hrs after that (11:30am CDT), and another successful gcc around 10hrs after the last one that was actually successful (4:30pm CDT). 

This seems like it would correspond to the change to build here:
https://github.com/boostorg/build/commit/451059949d92b52b21f802562a50587f1eeecb82

The other failure I started seeing was only my linux/arm64 machine. This doesn't run docker, so might not be affected in the same way. The last successful run on there was at 4:50pm CDT on June 4...but it typically takes around 9hrs to run so it probably got the pre 451059949 commit. However here I can't rule out that it was impacted by a8ab76ef973.

I'm going to try to standup some normal VMs soon and run on them to see if they are affected in the same way. It would also be nice if we could try backing out the usage of _SC_NPROCESSORS_ONLN and see if that fixes things, is that feasible to try for a couple days?

Again.. It's not a b2 issue. Specifying the "-jX" option entirely obviates the changed code. It's almost certainly something else. Places to look would be in the filesystem lib, or the root "boost" project for changes.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Fri, Jun 21, 2019 at 5:11 AM Tom Kent <[hidden email]> wrote:


On Thu, Jun 20, 2019, 10:35 PM Rene Rivera <[hidden email]> wrote:
On Thu, Jun 20, 2019 at 10:28 PM Tom Kent <[hidden email]> wrote:

On Thu, Jun 20, 2019 at 7:15 AM Rene Rivera <[hidden email]> wrote:
Since it's not due to the b2 -j change..

On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

GCC ends with this:

gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o

   "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
I"/var/boost/run/boost_root" -c -o "../../bin/common/build/gcc-8/release/process_jam_log.o" "../../common/build/../src/p
rocess_jam_log.cpp"

../../common/build/../src/process_jam_log.cpp: In function 'std::__cxx11::string {anonymous}::revision(const string&)':
../../common/build/../src/process_jam_log.cpp:359:16: warning: ignoring return value of 'int system(const char*)', decla
red with attribute warn_unused_result [-Wunused-result]
    std::system("git rev-parse --short=6 HEAD >.short-sha");
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v2/libs/f
ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...

I guess you'd need to find out what it says for trying to build filesystem lib.

 Here's what I'm seeing with the filesystem error:

   ln -f -s "../../libs/detail/include/boost/detail/has_default_constructor.hpp" "../../../boost_root/boost/detail/has_ 
default_constructor.hpp" 

gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL 
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin. 
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o" "/var/boost/run/boost_root/libs/file 
system/src/path_traits.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/path_traits.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f 
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o... 
gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin.
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o" "/var/boost/run/boost_root/libs/file
system/src/portability.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/portability.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o... 


And several more like that.

However, I'm also suspicious of the timing with the processor change.

Looking back at the last successful builds of develop on linux. I have a few from linux/x86_64/docker machines. The last one I saw was building with gcc-8 on June 4 around 6:30am CDT. There should have been a successful clang run about 5hrs after that (11:30am CDT), and another successful gcc around 10hrs after the last one that was actually successful (4:30pm CDT). 

This seems like it would correspond to the change to build here:
https://github.com/boostorg/build/commit/451059949d92b52b21f802562a50587f1eeecb82

The other failure I started seeing was only my linux/arm64 machine. This doesn't run docker, so might not be affected in the same way. The last successful run on there was at 4:50pm CDT on June 4...but it typically takes around 9hrs to run so it probably got the pre 451059949 commit. However here I can't rule out that it was impacted by a8ab76ef973.

I'm going to try to standup some normal VMs soon and run on them to see if they are affected in the same way. It would also be nice if we could try backing out the usage of _SC_NPROCESSORS_ONLN and see if that fixes things, is that feasible to try for a couple days?

Again.. It's not a b2 issue. Specifying the "-jX" option entirely obviates the changed code. It's almost certainly something else. Places to look would be in the filesystem lib, or the root "boost" project for changes.

Looking at regression.py it looks like the command that builds process_jam_log doesn't use the bjam options. Can you verify if this is/isn't the case?

Tom

Somewhere between  9:57am and 12:02pm CDT today this bug migrated to master.

It can trivially be re-produced.

On a bare linux vm...Azure F8s_v2, Ubuntu 18.04, standard SSD.
$ sudo apt update && sudo apt install build-essential gcc-8 g++-8 python python3 git wget
...
echo using gcc : 8 : g++-8 : \; > ~/user-config.jam
$ python run.py --runner=teeks99-test01 --toolsets=gcc-8 --tag=develop --bjam-options="-j8 address-model=64"
...fails with the above errors.

You can also do the same thing to reproduce in a docker image (ubuntu:bionic). --bjam-options of -j1 -j8 and absent all fail the same.

This does not correspond to any changes between the filesystem library that indicates the error. But it *does* correspond exactly with the release of the new B2. https://github.com/boostorg/build/commit/04efaac148671479b514db882b1c520dcaafe89e

There have been no successful regression tests by any linux runners since that time.

Tom



 

_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list


On Sun, Jun 23, 2019 at 7:37 PM Tom Kent <[hidden email]> wrote:


On Fri, Jun 21, 2019 at 5:11 AM Tom Kent <[hidden email]> wrote:


On Thu, Jun 20, 2019, 10:35 PM Rene Rivera <[hidden email]> wrote:
On Thu, Jun 20, 2019 at 10:28 PM Tom Kent <[hidden email]> wrote:

On Thu, Jun 20, 2019 at 7:15 AM Rene Rivera <[hidden email]> wrote:
Since it's not due to the b2 -j change..

On Fri, Jun 14, 2019 at 8:25 AM Tom Kent via Boost-Testing <[hidden email]> wrote:
For the last few days (Since June 4?) I've been having problems with all my linux tests running against the develop branch. 

GCC ends with this:

gcc.compile.c++ ../../bin/common/build/gcc-8/release/process_jam_log.o

   "g++-8"   -fPIC -O3 -finline-functions -Wno-inline -Wall  -DBOOST_ALL_NO_LIB=1 -DNDEBUG -D_CRT_SECURE_NO_WARNINGS  -
I"/var/boost/run/boost_root" -c -o "../../bin/common/build/gcc-8/release/process_jam_log.o" "../../common/build/../src/p
rocess_jam_log.cpp"

../../common/build/../src/process_jam_log.cpp: In function 'std::__cxx11::string {anonymous}::revision(const string&)':
../../common/build/../src/process_jam_log.cpp:359:16: warning: ignoring return value of 'int system(const char*)', decla
red with attribute warn_unused_result [-Wunused-result]
    std::system("git rev-parse --short=6 HEAD >.short-sha");
    ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...skipped <p../../bin/testing/build/gcc-8/release>process_jam_log for lack of <p/var/boost/run/boost_root/bin.v2/libs/f
ilesystem/build/gcc-8/release/link-static/visibility-hidden>libboost_filesystem.a...

I guess you'd need to find out what it says for trying to build filesystem lib.

 Here's what I'm seeing with the filesystem error:

   ln -f -s "../../libs/detail/include/boost/detail/has_default_constructor.hpp" "../../../boost_root/boost/detail/has_ 
default_constructor.hpp" 

gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL 
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin. 
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o" "/var/boost/run/boost_root/libs/file 
system/src/path_traits.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/path_traits.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f 
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/path_traits.o... 
gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o 

   "g++-8"   -fvisibility-inlines-hidden -m64 -O3 -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL
_NO_LIB=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -DNDEBUG  -I"/var/boost/run/boost_root" -c -o "/var/boost/run/boost_root/bin.
v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o" "/var/boost/run/boost_root/libs/file
system/src/portability.cpp" 

/var/boost/run/boost_root/libs/filesystem/src/portability.cpp:20:10: fatal error: boost/filesystem/config.hpp: No such f
ile or directory 
#include <boost/filesystem/config.hpp> 
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
compilation terminated. 
...failed gcc.compile.c++ /var/boost/run/boost_root/bin.v2/libs/filesystem/build/gcc-8/release/link-static/visibility-hidden/portability.o... 


And several more like that.

However, I'm also suspicious of the timing with the processor change.

Looking back at the last successful builds of develop on linux. I have a few from linux/x86_64/docker machines. The last one I saw was building with gcc-8 on June 4 around 6:30am CDT. There should have been a successful clang run about 5hrs after that (11:30am CDT), and another successful gcc around 10hrs after the last one that was actually successful (4:30pm CDT). 

This seems like it would correspond to the change to build here:
https://github.com/boostorg/build/commit/451059949d92b52b21f802562a50587f1eeecb82

The other failure I started seeing was only my linux/arm64 machine. This doesn't run docker, so might not be affected in the same way. The last successful run on there was at 4:50pm CDT on June 4...but it typically takes around 9hrs to run so it probably got the pre 451059949 commit. However here I can't rule out that it was impacted by a8ab76ef973.

I'm going to try to standup some normal VMs soon and run on them to see if they are affected in the same way. It would also be nice if we could try backing out the usage of _SC_NPROCESSORS_ONLN and see if that fixes things, is that feasible to try for a couple days?

Again.. It's not a b2 issue. Specifying the "-jX" option entirely obviates the changed code. It's almost certainly something else. Places to look would be in the filesystem lib, or the root "boost" project for changes.

Looking at regression.py it looks like the command that builds process_jam_log doesn't use the bjam options. Can you verify if this is/isn't the case?

Tom

Somewhere between  9:57am and 12:02pm CDT today this bug migrated to master.

It can trivially be re-produced.

On a bare linux vm...Azure F8s_v2, Ubuntu 18.04, standard SSD.
$ sudo apt update && sudo apt install build-essential gcc-8 g++-8 python python3 git wget
...
echo using gcc : 8 : g++-8 : \; > ~/user-config.jam
$ python run.py --runner=teeks99-test01 --toolsets=gcc-8 --tag=develop --bjam-options="-j8 address-model=64"
...fails with the above errors.

You can also do the same thing to reproduce in a docker image (ubuntu:bionic). --bjam-options of -j1 -j8 and absent all fail the same.

This does not correspond to any changes between the filesystem library that indicates the error. But it *does* correspond exactly with the release of the new B2. https://github.com/boostorg/build/commit/04efaac148671479b514db882b1c520dcaafe89e

There have been no successful regression tests by any linux runners since that time.


--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: Test Failures on develop

Boost - Testing mailing list
On Sun, Jun 23, 2019 at 8:34 PM Rene Rivera <[hidden email]> wrote:

See also this <https://github.com/boostorg/admin/issues/170> for other relevant PRs.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Boost-Testing mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-testing