Parallel builds and runs

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel builds and runs

Belcourt, Kenneth
Hi,

I'm trying to determine if boost build has support for or is
knowledgeable about things such as

1) compiling for MPI using the compiler wrapper scripts (mpicc, mpiCC)
2) honoring the requested <toolset> by loading the appropriate module(s)
3) running parallel unit-test applications with mpirun
4) support for submitting unit-tests to a queueing system like, for
example, PBS

I didn't find any information in the documentation so that's why I ask.

Thanks.

-- Noel Belcourt


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

Vladimir Prus
Hi Noel,

> I'm trying to determine if boost build has support for or is
> knowledgeable about things such as
>
>
> I didn't find any information in the documentation so that's why I ask.

Essentially, you ask if Boost.Build supports MPI. I think nobody ever tried,
so we can't claim such support. However:

> 1) compiling for MPI using the compiler wrapper scripts (mpicc, mpiCC)

If the question in replacing 'gcc' with 'mpicc' in command lines? Well, you
can specify a custom compiler name in 'gcc' invocation, or you can inherit
new "mpicc" toolset from gcc.jam and change it as needed.

> 2) honoring the requested <toolset> by loading the appropriate module(s)

I don't understand what you mean here, sorry.

> 3) running parallel unit-test applications with mpirun

It's possible to specify a command that's used to run unit tests, so the
command will be:

   mpirun whatever_unit_test_binary

> 4) support for submitting unit-tests to a queueing system like, for
> example, PBS

Again, you can specify "lancher" for unit tests, like this:

   unit-test whatever : whatever.cpp : <testing.launcher>PBS ;

and the command line will be:

   PBS whatever



I guess most of required machinery is needed, but since you're the one with
domain knowledge you'd need to try some things and then say what works and
what not and what should be fixed. I'd be happy if Boost.Build gets support
for MPI programs.

- Volodya
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

Belcourt, Kenneth

On Feb 21, 2006, at 12:02 AM, Vladimir Prus wrote:

> Hi Noel,
>
> Essentially, you ask if Boost.Build supports MPI. I think nobody ever
> tried,
> so we can't claim such support. However:
>
>> 1) compiling for MPI using the compiler wrapper scripts (mpicc, mpiCC)
>
> If the question in replacing 'gcc' with 'mpicc' in command lines?

Yes.

> Well, you
> can specify a custom compiler name in 'gcc' invocation, or you can
> inherit
> new "mpicc" toolset from gcc.jam and change it as needed.

Okay, I'll try this out.

>> 2) honoring the requested <toolset> by loading the appropriate
>> module(s)
>
> I don't understand what you mean here, sorry.

Sorry, I was a bit too terse.  Consider Darwin (Tiger actually) where I
can have
both gcc 3.3 and 4.0 compilers installed and can use the gcc_select
command
to swap which gcc compiler is picked up.  The ability to swap the
default compiler
is what I mean by loading a module.  On many of our systems, we have
numerous
compilers installed and need to load the appropriate compiler module(s)
before we
build.

For example, if on Darwin I asked for a gcc 3.3 build but the last
invocation of
gcc_select left the 4.0 compilers as the defaults, would bjam detect
that the wrong
compilers were being run?  I guess my question is does bjam perform a
version
check on the compiler if the user has requested a specific compiler
version?

> I guess most of required machinery is needed, but since you're the one
> with
> domain knowledge you'd need to try some things and then say what works
> and
> what not and what should be fixed. I'd be happy if Boost.Build gets
> support
> for MPI programs.

Great, I'll do just that and let you know how it goes.

Thanks for the help.

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

David Abrahams
"Noel Belcourt" <[hidden email]> writes:

> On Feb 21, 2006, at 12:02 AM, Vladimir Prus wrote:
>
>> Hi Noel,
>> Essentially, you ask if Boost.Build supports MPI. I think nobody
>> ever tried, so we can't claim such support. However:
>>
>>> 1) compiling for MPI using the compiler wrapper scripts (mpicc,
>>> mpiCC)
>> If the question in replacing 'gcc' with 'mpicc' in command lines?
>
> Yes.
>
>> Well, you can specify a custom compiler name in 'gcc' invocation, or
>> you can inherit new "mpicc" toolset from gcc.jam and change it as
>> needed.
>
> Okay, I'll try this out.
>
>>> 2) honoring the requested <toolset> by loading the appropriate
>>> module(s)
>> I don't understand what you mean here, sorry.
>
> Sorry, I was a bit too terse.  Consider Darwin (Tiger actually) where
> I can have both gcc 3.3 and 4.0 compilers installed and can use the
> gcc_select command to swap which gcc compiler is picked up.  The
> ability to swap the default compiler is what I mean by loading a
> module.  On many of our systems, we have numerous compilers installed
> and need to load the appropriate compiler module(s) before we build.
>
> For example, if on Darwin I asked for a gcc 3.3 build but the last
> invocation of gcc_select left the 4.0 compilers as the defaults, would
> bjam detect that the wrong compilers were being run?  I guess my
> question is does bjam perform a version check on the compiler if the
> user has requested a specific compiler version?

I know that's a common configuration technique, but Boost.Build
doesn't work that way.  It's designed so you can build with as many
versions of the compiler as you like, in one invocation.  So you
should arrange your

       using gcc : ... ;

invocations in user-config.jam (or site-config.jam) so that they can
run the compilers without using gcc_select.  For example, my
user-config.jam contains:

  using gcc : 4.0.2 : /usr/local/gcc-4.0.2/bin/g++.exe ;

  using gcc : 3.4.4 ;  # this one is what you get when you just say
                       # "gcc" at the command line.

  using gcc : 4.0.0 : /usr/local/gcc-4.0.0/bin/g++.exe ;

  using gcc : 2.95 : /bin/g++-2.exe ;


--
Dave Abrahams
Boost Consulting
www.boost-consulting.com

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

Belcourt, Kenneth

On Feb 21, 2006, at 12:38 PM, David Abrahams wrote:

> "Noel Belcourt" <[hidden email]> writes:
>
>> On Feb 21, 2006, at 12:02 AM, Vladimir Prus wrote:
>>
>>> Hi Noel,
>>> Essentially, you ask if Boost.Build supports MPI. I think nobody
>>> ever tried, so we can't claim such support. However:
>>>
>>>> 1) compiling for MPI using the compiler wrapper scripts (mpicc,
>>>> mpiCC)
>>> If the question in replacing 'gcc' with 'mpicc' in command lines?
>>
>> Yes.
>>
>>> Well, you can specify a custom compiler name in 'gcc' invocation, or
>>> you can inherit new "mpicc" toolset from gcc.jam and change it as
>>> needed.
>>
>> Okay, I'll try this out.
>>
>>>> 2) honoring the requested <toolset> by loading the appropriate
>>>> module(s)
>>> I don't understand what you mean here, sorry.
>>
>> Sorry, I was a bit too terse.  Consider Darwin (Tiger actually) where
>> I can have both gcc 3.3 and 4.0 compilers installed and can use the
>> gcc_select command to swap which gcc compiler is picked up.  The
>> ability to swap the default compiler is what I mean by loading a
>> module.  On many of our systems, we have numerous compilers installed
>> and need to load the appropriate compiler module(s) before we build.
>>
>> For example, if on Darwin I asked for a gcc 3.3 build but the last
>> invocation of gcc_select left the 4.0 compilers as the defaults, would
>> bjam detect that the wrong compilers were being run?  I guess my
>> question is does bjam perform a version check on the compiler if the
>> user has requested a specific compiler version?
>
> I know that's a common configuration technique, but Boost.Build
> doesn't work that way.  It's designed so you can build with as many
> versions of the compiler as you like, in one invocation.  So you
> should arrange your
>
>        using gcc : ... ;
>
> invocations in user-config.jam (or site-config.jam) so that they can
> run the compilers without using gcc_select.  For example, my
> user-config.jam contains:
>
>   using gcc : 4.0.2 : /usr/local/gcc-4.0.2/bin/g++.exe ;
>
>   using gcc : 3.4.4 ;  # this one is what you get when you just say
>                        # "gcc" at the command line.
>
>   using gcc : 4.0.0 : /usr/local/gcc-4.0.0/bin/g++.exe ;
>
>   using gcc : 2.95 : /bin/g++-2.exe ;

Dave,

This solution would be ideal if, on our other systems, the paths to the
compilers, linkers, and communication libraries were known and fixed.  
Unfortunately the paths are only known each time we login and load the
necessary modules as sysadmin can and does move the installations
around with regularity.  Since we can't use the above solution we're
reduced to using bjam to build for a single toolset at a time on those
systems, which really isn't a problem.

I'll definitely use this technique for the systems we control though.

Thanks.

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

David Abrahams
"Noel Belcourt" <[hidden email]> writes:

> On Feb 21, 2006, at 12:38 PM, David Abrahams wrote:
>
>> "Noel Belcourt" <[hidden email]> writes:
>>
>>> For example, if on Darwin I asked for a gcc 3.3 build but the last
>>> invocation of gcc_select left the 4.0 compilers as the defaults, would
>>> bjam detect that the wrong compilers were being run?  I guess my
>>> question is does bjam perform a version check on the compiler if the
>>> user has requested a specific compiler version?
>>
>> I know that's a common configuration technique, but Boost.Build
>> doesn't work that way.  It's designed so you can build with as many
>> versions of the compiler as you like, in one invocation.  So you
>> should arrange your
>>
>>        using gcc : ... ;
>>
>> invocations in user-config.jam (or site-config.jam) so that they can
>> run the compilers without using gcc_select.  For example, my
>> user-config.jam contains:
>>
>>   using gcc : 4.0.2 : /usr/local/gcc-4.0.2/bin/g++.exe ;
>>
>>   using gcc : 3.4.4 ;  # this one is what you get when you just say
>>                        # "gcc" at the command line.
>>
>>   using gcc : 4.0.0 : /usr/local/gcc-4.0.0/bin/g++.exe ;
>>
>>   using gcc : 2.95 : /bin/g++-2.exe ;
>
> Dave,
>
> This solution would be ideal if, on our other systems, the paths to the
> compilers, linkers, and communication libraries were known and fixed.  
> Unfortunately the paths are only known each time we login and load the
> necessary modules as sysadmin can and does move the installations
> around with regularity.  


That's the reason for site-config.jam.  Your sysadmin should just edit
that when he/she moves the installations.

--
Dave Abrahams
Boost Consulting
www.boost-consulting.com

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

Vladimir Prus
In reply to this post by Belcourt, Kenneth
On Wednesday 22 February 2006 09:17, Noel Belcourt wrote:

> >   using gcc : 4.0.2 : /usr/local/gcc-4.0.2/bin/g++.exe ;
> >
> >   using gcc : 3.4.4 ;  # this one is what you get when you just say
> >                        # "gcc" at the command line.
> >
> >   using gcc : 4.0.0 : /usr/local/gcc-4.0.0/bin/g++.exe ;
> >
> >   using gcc : 2.95 : /bin/g++-2.exe ;
>
> Dave,
>
> This solution would be ideal if, on our other systems, the paths to the
> compilers, linkers, and communication libraries were known and fixed.
> Unfortunately the paths are only known each time we login and load the
> necessary modules as sysadmin can and does move the installations
> around with regularity.  Since we can't use the above solution we're
> reduced to using bjam to build for a single toolset at a time on those
> systems, which really isn't a problem.
>
> I'll definitely use this technique for the systems we control though.

How about creating a script for each g++ version, in some location you
control, for example, in /home/noel/local/bin/g++-4.0.2.

The script would contain:

   #!/bin/bash

   gcc-select 4.0.2  
   g++ "$@"


and then given the path to this script in "using"?

- Volodya


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: Parallel builds and runs

David Abrahams
Vladimir Prus <[hidden email]> writes:

>> Dave,
>>
>> This solution would be ideal if, on our other systems, the paths to the
>> compilers, linkers, and communication libraries were known and fixed.
>> Unfortunately the paths are only known each time we login and load the
>> necessary modules as sysadmin can and does move the installations
>> around with regularity.  Since we can't use the above solution we're
>> reduced to using bjam to build for a single toolset at a time on those
>> systems, which really isn't a problem.
>>
>> I'll definitely use this technique for the systems we control though.
>
> How about creating a script for each g++ version, in some location you
> control, for example, in /home/noel/local/bin/g++-4.0.2.
>
> The script would contain:
>
>    #!/bin/bash
>
>    gcc-select 4.0.2  
>    g++ "$@"
>
>
> and then given the path to this script in "using"?

What's wrong with site-config.jam?

--
Dave Abrahams
Boost Consulting
www.boost-consulting.com

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build