B2 C++ engine development.

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

B2 C++ engine development.

Boost - Build mailing list
One of my goals in the future of B2 is to move the engine to something easier to maintain and improve. As such it's my goal to incrementally rewrite the engine component of B2 in C++. Yesterday I completed an initial step in that by having B2 compile as C++ instead of ANSI C. The code changes are in the "feature/cxx" branch and from outward appearances it's exactly the same as before except that the build scripts now invoke the C++ compiler (only lightly tested for the compilers I have available). First con of this is that the executable is slightly fatter (about 20K on OSX). But also the first pro, it's about 4% faster.

The idea is to merge this to develop, and eventually master, and continue the C++ work based from those (as PRs).

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
AMDG

On 10/24/2018 09:58 PM, Rene Rivera via Boost-build wrote:

> One of my goals in the future of B2 is to move the engine to something
> easier to maintain and improve. As such it's my goal to incrementally
> rewrite the engine component of B2 in C++. Yesterday I completed an initial
> step in that by having B2 compile as C++ instead of ANSI C. The code
> changes are in the "feature/cxx" branch and from outward appearances it's
> exactly the same as before except that the build scripts now invoke the C++
> compiler (only lightly tested for the compilers I have available). First
> con of this is that the executable is slightly fatter (about 20K on OSX).
> But also the first pro, it's about 4% faster.
>

Any explanation for this?  Are the build options
exactly the same other than -x c++?  The code
changes don't look significant, so is it just
that the stricter rules of C++ allow the optimizer
more freedom?  It looks like you're not linking
the C++ runtime (which was my first guess about
the binary size increase).

> The idea is to merge this to develop, and eventually master, and continue
> the C++ work based from those (as PRs).
>

- Is there a reason that you eliminated the two stage
  bootstrap process?  Among other issues, you've made it
  impossible to create a debug build of b2.
- While I support removing bjam, it's an entirely
  separate change.
- Before this goes live, I'd like to see the files
  renamed to *.cpp.
- What language standard are we targeting?  filesystem
  is C++17.

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
On Thu, Oct 25, 2018 at 12:41 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
AMDG

On 10/24/2018 09:58 PM, Rene Rivera via Boost-build wrote:
> One of my goals in the future of B2 is to move the engine to something
> easier to maintain and improve. As such it's my goal to incrementally
> rewrite the engine component of B2 in C++. Yesterday I completed an initial
> step in that by having B2 compile as C++ instead of ANSI C. The code
> changes are in the "feature/cxx" branch and from outward appearances it's
> exactly the same as before except that the build scripts now invoke the C++
> compiler (only lightly tested for the compilers I have available). First
> con of this is that the executable is slightly fatter (about 20K on OSX).
> But also the first pro, it's about 4% faster.
>

Any explanation for this?  Are the build options
exactly the same other than -x c++?  The code
changes don't look significant, so is it just
that the stricter rules of C++ allow the optimizer
more freedom?  It looks like you're not linking
the C++ runtime (which was my first guess about
the binary size increase).

I think the speedup is from the stricter type rules of C++, probably constness and aliasing. But it would take way more investigation and looking at assembly to truly figure that out. And I'm not going to bother doing that :-) I don't prevent linking of the C++ runtime. But since it's not used yet I suspect it's just linked away anyway. My suspicion for size increase would be of the slightly different optimization behavior. In that it might prefer slightly larger code that performs better in many small instances.
 
> The idea is to merge this to develop, and eventually master, and continue
> the C++ work based from those (as PRs).
>

- Is there a reason that you eliminated the two stage
  bootstrap process?  Among other issues, you've made it
  impossible to create a debug build of b2.

I got tired of watching the double builds when it's just as easy to do it once. And I do intend to put in an option for a debug build (and the python build). Although technically you can still do it if you use the basic cxx toolset with CXX and CXXFLAGS. There's almost no benefit to having the build.jam as a large chunk of what it does is packaging. And we can do that many other ways that are less painful (like a python script). I.e. I'm trying to reduce complexity in the bootstrap building.
 
- While I support removing bjam, it's an entirely
  separate change.

I can easily add a copy in there again.
 
- Before this goes live, I'd like to see the files
  renamed to *.cpp.

Sure. I didn't want to rename things yet as I wanted to see the diffs clearly for verification.
 
- What language standard are we targeting?  filesystem
  is C++17.

C++11. As much as I would like to use the latest and greatest using the latest means we would have to immediately move to binary distribution. Maybe in the future we can move further, once we figure out what aspects of newer C++ are essential.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list


On Thu, Oct 25, 2018 at 12:55 PM Rene Rivera <[hidden email]> wrote:
On Thu, Oct 25, 2018 at 12:41 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
AMDG

On 10/24/2018 09:58 PM, Rene Rivera via Boost-build wrote:
> One of my goals in the future of B2 is to move the engine to something
> easier to maintain and improve. As such it's my goal to incrementally
> rewrite the engine component of B2 in C++. Yesterday I completed an initial
> step in that by having B2 compile as C++ instead of ANSI C. The code
> changes are in the "feature/cxx" branch and from outward appearances it's
> exactly the same as before except that the build scripts now invoke the C++
> compiler (only lightly tested for the compilers I have available). First
> con of this is that the executable is slightly fatter (about 20K on OSX).
> But also the first pro, it's about 4% faster.
>

Any explanation for this?  Are the build options
exactly the same other than -x c++?  The code
changes don't look significant, so is it just
that the stricter rules of C++ allow the optimizer
more freedom?  It looks like you're not linking
the C++ runtime (which was my first guess about
the binary size increase).

I think the speedup is from the stricter type rules of C++, probably constness and aliasing. But it would take way more investigation and looking at assembly to truly figure that out. And I'm not going to bother doing that :-) I don't prevent linking of the C++ runtime. But since it's not used yet I suspect it's just linked away anyway. My suspicion for size increase would be of the slightly different optimization behavior. In that it might prefer slightly larger code that performs better in many small instances.
 
> The idea is to merge this to develop, and eventually master, and continue
> the C++ work based from those (as PRs).
>

- Is there a reason that you eliminated the two stage
  bootstrap process?  Among other issues, you've made it
  impossible to create a debug build of b2.

I got tired of watching the double builds when it's just as easy to do it once. And I do intend to put in an option for a debug build (and the python build). Although technically you can still do it if you use the basic cxx toolset with CXX and CXXFLAGS. There's almost no benefit to having the build.jam as a large chunk of what it does is packaging. And we can do that many other ways that are less painful (like a python script). I.e. I'm trying to reduce complexity in the bootstrap building.
 
- While I support removing bjam, it's an entirely
  separate change.

I can easily add a copy in there again.
 
- Before this goes live, I'd like to see the files
  renamed to *.cpp.

Sure. I didn't want to rename things yet as I wanted to see the diffs clearly for verification.
 
- What language standard are we targeting?  filesystem
  is C++17.

C++11. As much as I would like to use the latest and greatest using the latest means we would have to immediately move to binary distribution. Maybe in the future we can move further, once we figure out what aspects of newer C++ are essential.

PS. One of the first items to think about is writing a new bjam language parser. It's about time we stop relying on bison/yacc + the tokenizer :-)

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
AMDG

On 10/25/2018 11:57 AM, Rene Rivera wrote:
> <snip>
> PS. One of the first items to think about is writing a new bjam language
> parser. It's about time we stop relying on bison/yacc + the tokenizer :-)
>

What are you planning to replace it with?  A manual
recursive descent parser?  I'm not sure how much of
a win that is.  Anyway, this is one of the last places
that I would touch as it's very stable and isn't a known
performance bottleneck.  (I have no problem with bison
in the first place).

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
On Thu, Oct 25, 2018 at 1:19 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
AMDG

On 10/25/2018 11:57 AM, Rene Rivera wrote:
> <snip>
> PS. One of the first items to think about is writing a new bjam language
> parser. It's about time we stop relying on bison/yacc + the tokenizer :-)
>

What are you planning to replace it with?  A manual
recursive descent parser? 

Yes. 
 
I'm not sure how much of
a win that is.  Anyway, this is one of the last places
that I would touch as it's very stable and isn't a known
performance bottleneck.  (I have no problem with bison
in the first place).

The one item it can get us is more flexibility in needing spaces to separate tokens. As it's easier to handle this in particular context sensitive locations. But I don't feel particularly tied to this.


--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
AMDG

On 10/25/2018 12:23 PM, Rene Rivera wrote:

> On Thu, Oct 25, 2018 at 1:19 PM Steven Watanabe via Boost-build <
> [hidden email]> wrote:
>
>> On 10/25/2018 11:57 AM, Rene Rivera wrote:
>>> <snip>
>>> PS. One of the first items to think about is writing a new bjam language
>>> parser. It's about time we stop relying on bison/yacc + the tokenizer :-)
>>>
>>
>> What are you planning to replace it with?  A manual
>> recursive descent parser?
>
>
> Yes.
>
>
>> <snip>
>>
>
> The one item it can get us is more flexibility in needing spaces to
> separate tokens. As it's easier to handle this in particular context
> sensitive locations. But I don't feel particularly tied to this.
>

You do realize that I already did the work
for this, and enabling it is just a matter
of flipping a boolean constant?

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
On Thu, Oct 25, 2018 at 1:33 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
AMDG

On 10/25/2018 12:23 PM, Rene Rivera wrote:
> On Thu, Oct 25, 2018 at 1:19 PM Steven Watanabe via Boost-build <
> [hidden email]> wrote:
>
>> On 10/25/2018 11:57 AM, Rene Rivera wrote:
>>> <snip>
>>> PS. One of the first items to think about is writing a new bjam language
>>> parser. It's about time we stop relying on bison/yacc + the tokenizer :-)
>>>
>>
>> What are you planning to replace it with?  A manual
>> recursive descent parser?
>
>
> Yes.
>
>
>> <snip>
>>
>
> The one item it can get us is more flexibility in needing spaces to
> separate tokens. As it's easier to handle this in particular context
> sensitive locations. But I don't feel particularly tied to this.
>

You do realize that I already did the work
for this, and enabling it is just a matter
of flipping a boolean constant?

Yes. Haven't looked at how you did it though :-)

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
In reply to this post by Boost - Build mailing list
On Thu, Oct 25, 2018 at 12:55 PM Rene Rivera <[hidden email]> wrote:
On Thu, Oct 25, 2018 at 12:41 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
AMDG

On 10/24/2018 09:58 PM, Rene Rivera via Boost-build wrote:
> One of my goals in the future of B2 is to move the engine to something
> easier to maintain and improve. As such it's my goal to incrementally
> rewrite the engine component of B2 in C++. Yesterday I completed an initial
> step in that by having B2 compile as C++ instead of ANSI C. The code
> changes are in the "feature/cxx" branch and from outward appearances it's
> exactly the same as before except that the build scripts now invoke the C++
> compiler (only lightly tested for the compilers I have available). First
> con of this is that the executable is slightly fatter (about 20K on OSX).
> But also the first pro, it's about 4% faster.
>

Any explanation for this?  Are the build options
exactly the same other than -x c++?  The code
changes don't look significant, so is it just
that the stricter rules of C++ allow the optimizer
more freedom?  It looks like you're not linking
the C++ runtime (which was my first guess about
the binary size increase).

I think the speedup is from the stricter type rules of C++, probably constness and aliasing. But it would take way more investigation and looking at assembly to truly figure that out. And I'm not going to bother doing that :-) I don't prevent linking of the C++ runtime. But since it's not used yet I suspect it's just linked away anyway. My suspicion for size increase would be of the slightly different optimization behavior. In that it might prefer slightly larger code that performs better in many small instances.

While adding the "--debug" option for building I added LTO to the release build. And that gives an overall 11% perf improvement (with xcode 10 clang).

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
In reply to this post by Boost - Build mailing list
Le jeu. 25 oct. 2018 à 19:55, Rene Rivera via Boost-build
<[hidden email]> a écrit :

>
> On Thu, Oct 25, 2018 at 12:41 PM Steven Watanabe via Boost-build <[hidden email]> wrote:
>>
>> AMDG
>>
>> On 10/24/2018 09:58 PM, Rene Rivera via Boost-build wrote:
>> > One of my goals in the future of B2 is to move the engine to something
>> > easier to maintain and improve. As such it's my goal to incrementally
>> > rewrite the engine component of B2 in C++. Yesterday I completed an initial
>> > step in that by having B2 compile as C++ instead of ANSI C. The code
>> > changes are in the "feature/cxx" branch and from outward appearances it's
>> > exactly the same as before except that the build scripts now invoke the C++
>> > compiler (only lightly tested for the compilers I have available). First
>> > con of this is that the executable is slightly fatter (about 20K on OSX).
>> > But also the first pro, it's about 4% faster.
>> >
>>
>> Any explanation for this?  Are the build options
>> exactly the same other than -x c++?  The code
>> changes don't look significant, so is it just
>> that the stricter rules of C++ allow the optimizer
>> more freedom?  It looks like you're not linking
>> the C++ runtime (which was my first guess about
>> the binary size increase).
>
>
> I think the speedup is from the stricter type rules of C++, probably constness and aliasing. But it would take way more investigation and looking at assembly to truly figure that out. And I'm not going to bother doing that :-) I don't prevent linking of the C++ runtime. But since it's not used yet I suspect it's just linked away anyway. My suspicion for size increase would be of the slightly different optimization behavior. In that it might prefer slightly larger code that performs better in many small instances.

That could also be the result of inline heuristics being different
between C and C++.
Anyway, glad to see some performance improvements in the pipe !

--
Fabien
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
In reply to this post by Boost - Build mailing list
Hi,

Le ven. 26 oct. 2018 à 04:47, Rene Rivera via Boost-build
<[hidden email]> a écrit :
[...]
>> I think the speedup is from the stricter type rules of C++, probably constness and aliasing. But it would take way more investigation and looking at assembly to truly figure that out. And I'm not going to bother doing that :-) I don't prevent linking of the C++ runtime. But since it's not used yet I suspect it's just linked away anyway. My suspicion for size increase would be of the slightly different optimization behavior. In that it might prefer slightly larger code that performs better in many small instances.
>
>
> While adding the "--debug" option for building I added LTO to the release build. And that gives an overall 11% perf improvement (with xcode 10 clang).

Sorry for disrupting this thread with an unrelated remark, but can we
imagine a built-in feature <link-time-optimization> or the like doing
all the magic incantations for us ?

--
Fabien
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
In reply to this post by Boost - Build mailing list

On 2018-10-25 10:46 p.m., Rene Rivera via Boost-build wrote:


While adding the "--debug" option for building I added LTO to the release build. And that gives an overall 11% perf improvement (with xcode 10 clang).

That's quite impressive. I'd be curious to see a heat map showing where the time is spent. I was expecting most of the time being spent in running subprocesses (the actual commands), rather than within b2 itself. So an overall performance improvement of 11% based on work on b2 itself suggests that assumption is wrong.

Thanks,


Stefan
--

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list
On Fri, Oct 26, 2018 at 7:57 AM Stefan Seefeld via Boost-build <[hidden email]> wrote:

On 2018-10-25 10:46 p.m., Rene Rivera via Boost-build wrote:


While adding the "--debug" option for building I added LTO to the release build. And that gives an overall 11% perf improvement (with xcode 10 clang).

That's quite impressive. I'd be curious to see a heat map showing where the time is spent. I was expecting most of the time being spent in running subprocesses (the actual commands), rather than within b2 itself. So an overall performance improvement of 11% based on work on b2 itself suggests that assumption is wrong.

I should have mentioned that I don't measure with build times. The improvements are *only* the b2 parsing and make generation. What I do is run a fake build on the Boost tree like this:

time ../../bfgroup/b2-c --build-type=complete -n -d0 --layout=versioned

I run it multiple times so it's mostly hot code already. And the config checks are all cached.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net


_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build

signature.png (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: B2 C++ engine development.

Boost - Build mailing list


On 2018-10-26 9:58 a.m., Rene Rivera wrote:
On Fri, Oct 26, 2018 at 7:57 AM Stefan Seefeld via Boost-build <[hidden email]> wrote:

On 2018-10-25 10:46 p.m., Rene Rivera via Boost-build wrote:


While adding the "--debug" option for building I added LTO to the release build. And that gives an overall 11% perf improvement (with xcode 10 clang).

That's quite impressive. I'd be curious to see a heat map showing where the time is spent. I was expecting most of the time being spent in running subprocesses (the actual commands), rather than within b2 itself. So an overall performance improvement of 11% based on work on b2 itself suggests that assumption is wrong.

I should have mentioned that I don't measure with build times. The improvements are *only* the b2 parsing and make generation. What I do is run a fake build on the Boost tree like this:

time ../../bfgroup/b2-c --build-type=complete -n -d0 --layout=versioned

I run it multiple times so it's mostly hot code already. And the config checks are all cached.

Ah, OK, that puts things into perspective. :-)

Still, 11% is huge. Congrats !


Stefan
--

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build