linking and running tests in parallel

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

linking and running tests in parallel

Boost - Build mailing list
I have a Jamfile defining a bunch of tests with run rules. When I run those tests via b2, they are all run one at a time (essentially), regardless of what I pass as -j argument. This makes my tests take a very long time to run. (this is my Jamfile https://github.com/arvidn/libtorrent/blob/RC_1_2/test/Jamfile#L163 )

Looking into this problem a bit, I suspect that it's really linking the test binaries that's serialised, the tests just happen to take about as long as it takes to link.

In the gcc toolset, there's mentioning of JAM_SEMAPHORE, in the link rules. I don't fully understand how this affect the linking, but I suspect it means only one target can be linked at a time.


Is my assessment correct?
If so, is there a way to tell b2 that I really *do* want to link in parallel?
If not, is there some other mechanism at play, that I can work around to allow parallel linking?

thanks!
-- 
Arvid Norberg

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: linking and running tests in parallel

Boost - Build mailing list
AMDG

On 3/29/19 2:57 AM, Arvid Norberg via Boost-build wrote:
> <snip>
> In the gcc toolset, there's mentioning of JAM_SEMAPHORE, in the link rules.
> I don't fully understand how this affect the linking, but I suspect it
> means only one target can be linked at a time.
>
> https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064
>
> Is my assessment correct?

Yes.

> If so, is there a way to tell b2 that I really *do* want to link in
> parallel?

The only way is to edit gcc.jam and remove the
semaphore.

> If not, is there some other mechanism at play, that I can work around to
> allow parallel linking?
>

This is a very old optimization which is almost
certainly out-dated.  If no one objects, I'm
just going to remove it.

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: linking and running tests in parallel

Boost - Build mailing list
On Sat, Mar 30, 2019 at 4:45 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
On 3/29/19 2:57 AM, Arvid Norberg via Boost-build wrote:
> <snip>
> In the gcc toolset, there's mentioning of JAM_SEMAPHORE, in the link rules.
> I don't fully understand how this affect the linking, but I suspect it
> means only one target can be linked at a time.
>
> https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064
> [...]

This is a very old optimization which is almost
certainly out-dated.  If no one objects, I'm
just going to remove it.

I'm still experiencing this issue and seeing the JAM_SEMAPHORE in gcc.jam and clang-linux.jam in boost-1.74.0

--
Arvid Norberg

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: linking and running tests in parallel

Boost - Build mailing list

On 28/08/2020 11:46, Arvid Norberg via Boost-build wrote:

> On Sat, Mar 30, 2019 at 4:45 PM Steven Watanabe via Boost-build
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     On 3/29/19 2:57 AM, Arvid Norberg via Boost-build wrote:
>     > <snip>
>     > In the gcc toolset, there's mentioning of JAM_SEMAPHORE, in the
>     link rules.
>     > I don't fully understand how this affect the linking, but I
>     suspect it
>     > means only one target can be linked at a time.
>     >
>     >
>     https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064
>     <https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064>
>     > [...]
>
>     This is a very old optimization which is almost
>     certainly out-dated.  If no one objects, I'm
>     just going to remove it.
>
>

Yes please!!!

I've complained about this before, it's a huge bottleneck building for
MIngw - basically the obj files get built first, and then it sits doing
pretty much nothing linking one test at a time :(

Cheers, John.


--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: linking and running tests in parallel

Boost - Build mailing list
On Fri, Aug 28, 2020 at 9:52 AM John Maddock via Boost-build <[hidden email]> wrote:

On 28/08/2020 11:46, Arvid Norberg via Boost-build wrote:
> On Sat, Mar 30, 2019 at 4:45 PM Steven Watanabe via Boost-build
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     On 3/29/19 2:57 AM, Arvid Norberg via Boost-build wrote:
>     > <snip>
>     > In the gcc toolset, there's mentioning of JAM_SEMAPHORE, in the
>     link rules.
>     > I don't fully understand how this affect the linking, but I
>     suspect it
>     > means only one target can be linked at a time.
>     >
>     >
>     https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064
>     <https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064>
>     > [...]
>
>     This is a very old optimization which is almost
>     certainly out-dated.  If no one objects, I'm
>     just going to remove it.
>
>

Yes please!!!

I've complained about this before, it's a huge bottleneck building for
MIngw - basically the obj files get built first, and then it sits doing
pretty much nothing linking one test at a time :(

Is there an issue or PR for this? I'll fix it regardless :-) Just wondering if I missed something.

--
-- René Ferdinand Rivera Morell
-- Don't Assume Anything  -- No Supone Nada
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: linking and running tests in parallel

Boost - Build mailing list
On Fri, Aug 28, 2020 at 10:23 AM René Ferdinand Rivera Morell <[hidden email]> wrote:
On Fri, Aug 28, 2020 at 9:52 AM John Maddock via Boost-build <[hidden email]> wrote:

On 28/08/2020 11:46, Arvid Norberg via Boost-build wrote:
> On Sat, Mar 30, 2019 at 4:45 PM Steven Watanabe via Boost-build
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     On 3/29/19 2:57 AM, Arvid Norberg via Boost-build wrote:
>     > <snip>
>     > In the gcc toolset, there's mentioning of JAM_SEMAPHORE, in the
>     link rules.
>     > I don't fully understand how this affect the linking, but I
>     suspect it
>     > means only one target can be linked at a time.
>     >
>     >
>     https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064
>     <https://github.com/boostorg/build/blob/develop/src/tools/gcc.jam#L1064>
>     > [...]
>
>     This is a very old optimization which is almost
>     certainly out-dated.  If no one objects, I'm
>     just going to remove it.
>
>

Yes please!!!

I've complained about this before, it's a huge bottleneck building for
MIngw - basically the obj files get built first, and then it sits doing
pretty much nothing linking one test at a time :(

Is there an issue or PR for this? I'll fix it regardless :-) Just wondering if I missed something.



--
-- René Ferdinand Rivera Morell
-- Don't Assume Anything  -- No Supone Nada
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build