New Major-Release Boost.Python Development

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

New Major-Release Boost.Python Development

Jim Bosch-2
I'd like to start work on a new major release of Boost.Python.  While
the library is currently well-maintained in terms of bugfixes, I get the
sense that neither the original developers nor the current maintainer
have the time or inclination to work on new features.  I'd also like to
propose some changes that are slightly backwards-incompatible, as well
as some that mess with the internals to an extent that I'd feel better
about doing it outside Boost itself, to make it easier for adventurous
users to play with the new version without affecting people who depend
on having an extremely stable library in Boost.

To that end, I'm inclined to copy the library to somewhere else
(possibly the boost sandbox, but more likely a separate site), work on
it, produce some minor releases, and re-submit it to Boost for review.
Perhaps the external site would continue on as the home of more
fine-grained releases, or maybe we would fully reintegrate with Boost at
that point (especially if Boost addresses some of its own project
management and release control issues by that point, which I know is
being discussed but to my knowledge doesn't really have a timeline yet).

I am willing to take the lead on this project; I have a number of
features that exist as extensions in the boost sandbox already that
would work better if they could be more fully integrated into the
Boost.Python core, and I think I have the necessary understanding of the
full code base to coordinate things.  I'd like to save a full discussion
of what features a new version would include for another thread, but I
am hoping other people on the list might volunteer some time to work on
aspects they have coded up elsewhere - I know many such extensions exist.

So I have a few questions for anyone who's paying attention:

- For the original Boost.Python developers and current maintainers, and
other people familiar with developing Boost libraries: do you have any
preference on how to approach this?  I don't want to step on any toes,
especially toes attached to people who are responsible for the excellent
library we already have.

- For other Boost.Python experts on this list: do you have existing code
or development time you'd like to contribute?


Thanks!

Jim Bosch

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Stefan Seefeld-2
On 08/25/2011 04:59 PM, Jim Bosch wrote:
>
> To that end, I'm inclined to copy the library to somewhere else
> (possibly the boost sandbox, but more likely a separate site), work on
> it, produce some minor releases, and re-submit it to Boost for review.
> Perhaps the external site would continue on as the home of more
> fine-grained releases, or maybe we would fully reintegrate with Boost
> at that point (especially if Boost addresses some of its own project
> management and release control issues by that point, which I know is
> being discussed but to my knowledge doesn't really have a timeline yet).

Jim,

this is an interesting idea. There has been lots of general (dare I say
generic ?) discussion concerning process improvements (which
unfortunately most of the time diverted into tool discussions). Among
the fundamental issues is a modularization of boost.
I think it would be great if boost.python could follow through on its
own, by becoming a separate entity. Thus, I'm fully supportive of such a
move.

> - For the original Boost.Python developers and current maintainers,
> and other people familiar with developing Boost libraries: do you have
> any preference on how to approach this?  I don't want to step on any
> toes, especially toes attached to people who are responsible for the
> excellent library we already have.

I think branching off and moving to a separate site seems a fair choice.
It would be great to "come back" under the "Boost.org" umbrella once
that will be possible, i.e. once the Boost.org structure allows for
that. (In the sake of progress I'm refraining from voicing my personal
preferences for tools or hosting sites. I'm sure I can learn and adapt
to whatever gets agreed on.)

>
> - For other Boost.Python experts on this list: do you have existing
> code or development time you'd like to contribute?

I'd definitely like to help. I have a wishlist of my own for
improvements that I'd like to see, and which I'd be happy to share and
work on.

Thanks,
        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Ralf Grosse-Kunstleve
In reply to this post by Jim Bosch-2
Hi Jim,

CC to Dave.

This is great news.
My main interests have been stability and not increasing the memory footprint of boost.python extensions. I'm not in a position to further develop boost.python.
Troy and Ravi have done a significant amount of work. I hope they will comment for themselves.
I'd prefer if developments stayed under the boost umbrella, e.g. as boost/python/v3, but I don't feel very strongly about this.

Ralf

On Thu, Aug 25, 2011 at 1:59 PM, Jim Bosch <[hidden email]> wrote:
I'd like to start work on a new major release of Boost.Python.  While the library is currently well-maintained in terms of bugfixes, I get the sense that neither the original developers nor the current maintainer have the time or inclination to work on new features.  I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost.

To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review. Perhaps the external site would continue on as the home of more fine-grained releases, or maybe we would fully reintegrate with Boost at that point (especially if Boost addresses some of its own project management and release control issues by that point, which I know is being discussed but to my knowledge doesn't really have a timeline yet).

I am willing to take the lead on this project; I have a number of features that exist as extensions in the boost sandbox already that would work better if they could be more fully integrated into the Boost.Python core, and I think I have the necessary understanding of the full code base to coordinate things.  I'd like to save a full discussion of what features a new version would include for another thread, but I am hoping other people on the list might volunteer some time to work on aspects they have coded up elsewhere - I know many such extensions exist.

So I have a few questions for anyone who's paying attention:

- For the original Boost.Python developers and current maintainers, and other people familiar with developing Boost libraries: do you have any preference on how to approach this?  I don't want to step on any toes, especially toes attached to people who are responsible for the excellent library we already have.

- For other Boost.Python experts on this list: do you have existing code or development time you'd like to contribute?


Thanks!

Jim Bosch

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Neal Becker
In reply to this post by Jim Bosch-2
What sort of improvements did you have in mind?

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Stefan Seefeld-2
On 08/26/2011 07:17 AM, Neal Becker wrote:
> What sort of improvements did you have in mind?

Two things on my list that are likely going to be somewhat disruptive are:

* Support for subclassing boost.python's own metaclass.
* A per-module type registry, to avoid conflicting converters in
multi-module projects.

        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Niall Douglas
In reply to this post by Jim Bosch-2
On 25 Aug 2011 at 13:59, Jim Bosch wrote:

> - For other Boost.Python experts on this list: do you have existing code
> or development time you'd like to contribute?

Firstly, I must commend you as you're a better man than I for
initiating this. I mostly chase money these past few years, and I
don't like myself for it.

About a year ago I promised someone in here I'd refactor the BPL
interface to native code to support calling a list of functors every
time BPL goes in or out of native code. This would allow the Python
GIL to be released or reacquired, or indeed many other useful things
e.g. benchmarking.

At the time I had a free window of about a month in the near future,
so I thought my promise worth giving. Unfortunately, the project I
was working on at the time had unexpected problems, and the month got
gobbled, so I broke my promise.

I won't make that same mistake this time, not least because I have a
packed schedule for the upcoming year. I might also add that the
number of people willing to fund development in BPL appears to have
dropped off a cliff since 2008, so that doesn't help focus minds
either. Maybe the release of C++1x might reinvigorate things
hopefully.

Nevertheless, I'd like to keep informed. If you put it on github I'll
stick a watch on it :)

BTW, how has the Python3k port worked out? I'm not sure if it's been
mainlined yet has it?

Best of luck!

Niall

--
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no:
472909.



_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Stefan Seefeld-2
On 08/26/2011 01:28 PM, Niall Douglas wrote:
> BTW, how has the Python3k port worked out? I'm not sure if it's been
> mainlined yet has it? Best of luck! Niall

I'm not using Python 3k myself, so I can't comment, but the P3K port
most definitely went into trunk and has been part of the last couple of
releases.

    Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Jim Bosch-2
In reply to this post by Neal Becker
On 08/26/2011 04:17 AM, Neal Becker wrote:
> What sort of improvements did you have in mind?

My list includes:

- Propagating constness to Python (essentially already done as an
extension, but it could have a much nicer interface if I could mess with
class_ itself).

- Make custom registry and template-based conversions more accessible.
The former may just need more documentation, but the rvalue converters
in particular don't seem to have been intended as part of the public API
originally, and I think they're an important part of the library.
Template-based conversions are even more buried in the details - you
have to specialize five or six classes to get it working.  I'd like to
make it possible to create a template-based conversion by (partial)
specializing just one traits class.

- Automatic conversions for newer boost libraries (Fusion, Pointer
Container, and Filesystem are at the top of my list), and more for the
STL and iostreams standard libraries.  I'd like to integrate the
indexing suite (v2) into Boost.Python proper.

- Allow Boost.Python wrapped classes to inherit from Python classes.

- An actual "boost.python" Python module to make exceptions and other
types used in wrappers easily accessible from Python.

- Some limited degree of priority-based overload matching.  Not sure how
best to approach this one yet, though.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Jim Bosch-2
In reply to this post by Ralf Grosse-Kunstleve
On 08/25/2011 04:26 PM, Ralf Grosse-Kunstleve wrote:

> Hi Jim,
>
> CC to Dave.
>
> This is great news.
> My main interests have been stability and not increasing the memory
> footprint of boost.python extensions. I'm not in a position to further
> develop boost.python.
> Troy and Ravi have done a significant amount of work. I hope they will
> comment for themselves.
> I'd prefer if developments stayed under the boost umbrella, e.g. as
> boost/python/v3, but I don't feel very strongly about this.
>

Thanks!

I'd have no problem at all calling it boost/python/v3 (in fact I'd hope
to).  Essentially, my concern is that v3 would fall into a muddy
category between an accepted Boost library and a proposed Boost library,
and I don't have a good example of how that ought to work, with regard
to whether it should exist in the Boost main repository, or even whether
it can even use the Boost label.

And I would like to have both v2 and v3 available simultaneously as
distinct libraries (the way Python 2 and Python 3 are, for instance), at
least while v3 is undergoing lots of changes.  I'm not sure how to fit
that model into the Boost umbrella - I'm happy to do it, but I guess I'm
hoping for someone to tell me how it should fit; I don't want to presume
that v3 is automatically a Boost library without permission, so it
seemed safer to move it outside until it could officially win back its
Boost status through review.

Jim



>
> On Thu, Aug 25, 2011 at 1:59 PM, Jim Bosch <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I'd like to start work on a new major release of Boost.Python.
>       While the library is currently well-maintained in terms of
>     bugfixes, I get the sense that neither the original developers nor
>     the current maintainer have the time or inclination to work on new
>     features.  I'd also like to propose some changes that are slightly
>     backwards-incompatible, as well as some that mess with the internals
>     to an extent that I'd feel better about doing it outside Boost
>     itself, to make it easier for adventurous users to play with the new
>     version without affecting people who depend on having an extremely
>     stable library in Boost.
>
>     To that end, I'm inclined to copy the library to somewhere else
>     (possibly the boost sandbox, but more likely a separate site), work
>     on it, produce some minor releases, and re-submit it to Boost for
>     review. Perhaps the external site would continue on as the home of
>     more fine-grained releases, or maybe we would fully reintegrate with
>     Boost at that point (especially if Boost addresses some of its own
>     project management and release control issues by that point, which I
>     know is being discussed but to my knowledge doesn't really have a
>     timeline yet).
>
>     I am willing to take the lead on this project; I have a number of
>     features that exist as extensions in the boost sandbox already that
>     would work better if they could be more fully integrated into the
>     Boost.Python core, and I think I have the necessary understanding of
>     the full code base to coordinate things.  I'd like to save a full
>     discussion of what features a new version would include for another
>     thread, but I am hoping other people on the list might volunteer
>     some time to work on aspects they have coded up elsewhere - I know
>     many such extensions exist.
>
>     So I have a few questions for anyone who's paying attention:
>
>     - For the original Boost.Python developers and current maintainers,
>     and other people familiar with developing Boost libraries: do you
>     have any preference on how to approach this?  I don't want to step
>     on any toes, especially toes attached to people who are responsible
>     for the excellent library we already have.
>
>     - For other Boost.Python experts on this list: do you have existing
>     code or development time you'd like to contribute?
>
>
>     Thanks!
>
>     Jim Bosch
>
>     _________________________________________________
>     Cplusplus-sig mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://mail.python.org/__mailman/listinfo/cplusplus-sig
>     <http://mail.python.org/mailman/listinfo/cplusplus-sig>
>
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Dave Abrahams
In reply to this post by Jim Bosch-2

Trying to catch up here, so responding to everything all at once.

on Thu Aug 25 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

Just how tall are you, Jimbo?

> I'd like to start work on a new major release of Boost.Python.  

That certainly is welcome news.

> While the library is currently well-maintained in terms of bugfixes, I
> get the sense that neither the original developers nor the current
> maintainer have the time or inclination to work on new features.  

Well, speaking for myself, mostly time.  I'd be inclined to do a rewrite
along the lines of the langbinding ideas if I had time.

Another thing I think should be examined and refreshed is the
documentation style.

> I'd also like to propose some changes that are slightly
> backwards-incompatible, as well as some that mess with the internals
> to an extent that I'd feel better about doing it outside Boost itself,
> to make it easier for adventurous users to play with the new version
> without affecting people who depend on having an extremely stable
> library in Boost.

There's no need to do this "outside of Boost."  A branch in the Boost
repository is a perfect place for exploratory development that will
eventually be released as part of Boost.

> To that end, I'm inclined to copy the library to somewhere else
> (possibly the boost sandbox, but more likely a separate site), work on
> it, produce some minor releases, and re-submit it to Boost for
> review.

If you want to go through another review process, that's up to you.
Getting review feedback definitely has its advantages.  Please, though,
use the sandbox or some other area of the Boost svn repository at least
until we get my hoped-for Git transition completed.

> Perhaps the external site would continue on as the home of more
> fine-grained releases, or maybe we would fully reintegrate with Boost
> at that point (especially if Boost addresses some of its own project
> management and release control issues by that point, which I know is
> being discussed but to my knowledge doesn't really have a timeline
> yet).
>
> I am willing to take the lead on this project;

Someone willing to take the lead is the most important part.

> I have a number of features that exist as extensions in the boost
> sandbox already that would work better if they could be more fully
> integrated into the Boost.Python core,

Good.  Although, if I were you I would also carefully re-examine the
core to see if it has the best design.

> and I think I have the necessary understanding of the full code base
> to coordinate things.

Awesome.

> I'd like to save a full discussion of what features a new version
> would include for another thread, but I am hoping other people on the
> list might volunteer some time to work on aspects they have coded up
> elsewhere - I know many such extensions exist.
>
> So I have a few questions for anyone who's paying attention:
>
> - For the original Boost.Python developers and current maintainers,
> and other people familiar with developing Boost libraries: do you have
> any preference on how to approach this?  I don't want to step on any
> toes, especially toes attached to people who are responsible for the
> excellent library we already have.

See above.  No toe-stepping concerns on my end.  I have a preference
that you build on the ideas we came up with for Boost.Langbinding, but
certainly wouldn't insist on it.

on Thu Aug 25 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:

>
> Jim,
>
> this is an interesting idea. There has been lots of general (dare I
> say generic ?) discussion concerning process improvements (which
> unfortunately most of the time diverted into tool discussions). Among
> the fundamental issues is a modularization of boost.  I think it would
> be great if boost.python could follow through on its own, by becoming
> a separate entity.

Separate from Boost?  I guess that's a possibility but I'm not sure I
see the advantage.

on Thu Aug 25 2011, Ralf Grosse-Kunstleve <rwgrosse-kunstleve-AT-lbl.gov> wrote:

> Hi Jim,
>
> CC to Dave.
>
> This is great news.
> My main interests have been stability and not increasing the memory
> footprint of boost.python extensions. I'm not in a position to further
> develop boost.python.
> Troy and Ravi have done a significant amount of work.

Yes, and hopefully integrating that could be part of any next steps.

> I hope they will comment for themselves.
>
> I'd prefer if developments stayed under the boost umbrella, e.g. as
> boost/python/v3, but I don't feel very strongly about this.

Me too.  We managed a transition from v1 to v2 within Boost, and I think
we could do the same for v3.

> On 08/26/2011 07:17 AM, Neal Becker wrote:
>> What sort of improvements did you have in mind?
>
> Two things on my list that are likely going to be somewhat disruptive are:
>
> * Support for subclassing boost.python's own metaclass.

Cool.

>
> * A per-module type registry, to avoid conflicting converters in
> multi-module projects.

Interesting idea.  How does sharing types across multiple modules work
in that scenario?

on Fri Aug 26 2011, "Niall Douglas" <s_sourceforge-AT-nedprod.com> wrote:

> About a year ago I promised someone in here I'd refactor the BPL
> interface to native code to support calling a list of functors every
> time BPL goes in or out of native code. This would allow the Python
> GIL to be released or reacquired, or indeed many other useful things
> e.g. benchmarking.

Sounds useful.

on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

> On 08/26/2011 04:17 AM, Neal Becker wrote:
>> What sort of improvements did you have in mind?
>
> My list includes:
>
> - Propagating constness to Python (essentially already done as an
> extension, but it could have a much nicer interface if I could mess
> with class_ itself).

Oooh, now that one is tricky.  I'd like to see the design you have in
mind.  Python doesn't have constness; it has immutability, which is
subtly different.

> - Make custom registry and template-based conversions more
> accessible.

+1

> The former may just need more documentation, but the rvalue converters
> in particular don't seem to have been intended as part of the public
> API originally, and I think they're an important part of the
> library. Template-based conversions are even more buried in the
> details - you have to specialize five or six classes to get it
> working.  I'd like to make it possible to create a template-based
> conversion by (partial) specializing just one traits class.

Hmm, well, IIRC anything you do by pure partial specialization does not
go in the registry and can't participate in cross-module communication,
as it's entirely static.  I guess we need a more formal separation of the
two basic techniques for conversion, not to mention documentation, so
people know to what they're opting in.

> - Automatic conversions for newer boost libraries (Fusion, Pointer
> Container, and Filesystem are at the top of my list), and more for the
> STL and iostreams standard libraries.  I'd like to integrate the
> indexing suite (v2) into Boost.Python proper.

Interesting and useful improvements.

> - Allow Boost.Python wrapped classes to inherit from Python classes.

+1

> - An actual "boost.python" Python module to make exceptions and other
> types used in wrappers easily accessible from Python.

Nice.

> - Some limited degree of priority-based overload matching.  Not sure
> how best to approach this one yet, though.

+1
This is a solved problem... just not in Boost.Python.  Daniel Wallin
worked it out for luabind and we were going to incorporate it into
langbinding.  Happy to discuss it further.


--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

[Boost.Python v3] Planning and Logistics

Jim Bosch-2
In the interest of keeping this discussion easy-to-follow, I'm going to
reply to Dave's email twice, with new subjects - I'll stick to questions
about logistics in this email, and talk about features and scope in another.

In summary, I'm getting the sense that a branch in the mainline (not
sandbox) Boost SVN is the way to go.  I imagine most communication would
just happen on the C++-sig list, maybe with the [Boost.Python v3]
subtitle I've used in this email.

On 08/26/2011 01:09 PM, Dave Abrahams wrote:
>
> Trying to catch up here, so responding to everything all at once.
>
> on Thu Aug 25 2011, Jim Bosch<talljimbo-AT-gmail.com>  wrote:
>
> Just how tall are you, Jimbo?

6'4".  Not that tall, in the grand scheme of things, but it was a
teenage internet moniker that stuck with me.

>> I'd also like to propose some changes that are slightly
>> backwards-incompatible, as well as some that mess with the internals
>> to an extent that I'd feel better about doing it outside Boost itself,
>> to make it easier for adventurous users to play with the new version
>> without affecting people who depend on having an extremely stable
>> library in Boost.
>
> There's no need to do this "outside of Boost."  A branch in the Boost
> repository is a perfect place for exploratory development that will
> eventually be released as part of Boost.
>> To that end, I'm inclined to copy the library to somewhere else
>> (possibly the boost sandbox, but more likely a separate site), work on
>> it, produce some minor releases, and re-submit it to Boost for
>> review.
>
> If you want to go through another review process, that's up to you.
> Getting review feedback definitely has its advantages.  Please, though,
> use the sandbox or some other area of the Boost svn repository at least
> until we get my hoped-for Git transition completed.

I'd love to see a Git transition too, but I'm actually more familiar
with SVN at the moment, and I can certainly see the advantages of
working in the same repository as the previous version.

>
> on Thu Aug 25 2011, Stefan Seefeld<stefan-AT-seefeld.name>  wrote:
>
>>
>> Jim,
>>
>> this is an interesting idea. There has been lots of general (dare I
>> say generic ?) discussion concerning process improvements (which
>> unfortunately most of the time diverted into tool discussions). Among
>> the fundamental issues is a modularization of boost.  I think it would
>> be great if boost.python could follow through on its own, by becoming
>> a separate entity.
>
> Separate from Boost?  I guess that's a possibility but I'm not sure I
> see the advantage.
>

 From my perspective, the advantages are mostly just the same reasons
Boost has been talking about increased modularity, with regard to having
stable and development versions for some packages and not for others,
and to allow users to be able to install some components without
installing all of them.  Boost.Python (right now, at least) depends on a
very small core part of Boost, and to my knowledge no other Boost
libraries have real dependencies on Boost.Python (not counting optional
dependencies, like the Boost.Graph python bindings).  If/when Boost as a
whole goes more modular, I think any advantage of being separate would
disappear, and the ideal case for Boost.Python would be for that to happen.

> on Thu Aug 25 2011, Ralf Grosse-Kunstleve<rwgrosse-kunstleve-AT-lbl.gov>  wrote:
>> Troy and Ravi have done a significant amount of work.
>
> Yes, and hopefully integrating that could be part of any next steps.
>
>> I hope they will comment for themselves.
>>
>> I'd prefer if developments stayed under the boost umbrella, e.g. as
>> boost/python/v3, but I don't feel very strongly about this.
>
> Me too.  We managed a transition from v1 to v2 within Boost, and I think
> we could do the same for v3.
>

Good to hear.  I wasn't using Boost.Python back when that happened, but
it's nice to know there's a precedent.

Thanks!

Jim

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

[Boost.Python v3] Features and Scope

Jim Bosch-2
In reply to this post by Dave Abrahams
On 08/26/2011 01:09 PM, Dave Abrahams wrote:
>
> Well, speaking for myself, mostly time.  I'd be inclined to do a rewrite
> along the lines of the langbinding ideas if I had time.
>

I had only been vaguely aware of langbinding until I followed up on your
last email.  It's a very nice separation, though after bad experiences
using SWIG I am a little wary about trying to build a one-size-fits-all
front-end for different languages.

It seems like a reasonable way to proceed would be to try to convert
Boost.Python to the langbinding interface, but not be obsessive about
avoiding Python-specific hacks in the frontend right now.  Once we're
more feature-complete in Python, we can worry about finding
language-agnostic ways to do things that weren't anticipated in the
original langbinding design.

> Another thing I think should be examined and refreshed is the
> documentation style.
>

Agreed.  And I wouldn't just limit it to the official documentation;
there are a lot of little tidbits of useful but often very old
Boost.Python knowledge scattered around the internet (often on
Python-affiliated sites or wikis).  It'd be nice to unify a lot of that,
or at least update the obsolete stuff and add links to a more complete
set of official documentation.

>
> Good.  Although, if I were you I would also carefully re-examine the
> core to see if it has the best design.
>

I wasn't originally planning on doing a full re-evaluation, but after
looking over the langbinding proposal and hearing some of the other
ideas, I think that will probably be necessary.

>
> on Thu Aug 25 2011, Ralf Grosse-Kunstleve<rwgrosse-kunstleve-AT-lbl.gov>  wrote:
> <snip>
>> Troy and Ravi have done a significant amount of work.
>
> Yes, and hopefully integrating that could be part of any next steps.
>
>> I hope they will comment for themselves.

I was under the impression their work had already been integrated into
Boost.Python v2.  Is there a large repository of additional Boost.Python
work elsewhere that I should be aware of?

I am aware of Py++ and the extensions associated with it, and some of
that could definitely go into Boost.Python proper (and I think I've
heard Roman state that he wouldn't have a problem with someone else
doing the work to make it so, and he just didn't have time to do it
himself).

>>
>> * Support for subclassing boost.python's own metaclass.
>
> Cool.
>
>>
>> * A per-module type registry, to avoid conflicting converters in
>> multi-module projects.
>
> Interesting idea.  How does sharing types across multiple modules work
> in that scenario?
>

Hmm.  I'm guessing the global type registry would still be the default,
and per-module registries would override these when available?  It
sounds like Stefan has a clear use case in mind, and I'd be curious to
know what it is.

> on Fri Aug 26 2011, "Niall Douglas"<s_sourceforge-AT-nedprod.com>  wrote:
>
>> About a year ago I promised someone in here I'd refactor the BPL
>> interface to native code to support calling a list of functors every
>> time BPL goes in or out of native code. This would allow the Python
>> GIL to be released or reacquired, or indeed many other useful things
>> e.g. benchmarking.
>
> Sounds useful.
>

This is one of the areas - GIL, threading, etc. - where I'm less
knowledgeable, especially when multiple platforms are involved.  It
sounds like a very useful feature for a lot of people, but I have to
admit I probably wouldn't dive into this without a lot of backup.

> on Fri Aug 26 2011, Jim Bosch<talljimbo-AT-gmail.com>  wrote:
>
>> On 08/26/2011 04:17 AM, Neal Becker wrote:
>>> What sort of improvements did you have in mind?
>>
>> My list includes:
>>
>> - Propagating constness to Python (essentially already done as an
>> extension, but it could have a much nicer interface if I could mess
>> with class_ itself).
>
> Oooh, now that one is tricky.  I'd like to see the design you have in
> mind.  Python doesn't have constness; it has immutability, which is
> subtly different.
>

Essentially, C++ objects returned as const references, pointers, or
smart pointers get converted into a Python proxy object with methods
that forward to the real wrapped object, but only if those methods are
marked as const.  The proxy objects have rvalue converters but no lvalue
converters.  I essentially wrote a wrapper around class_ to check for
constness of member functions and make the proxy class.  When you use
class_ directly, there's no const-correctness, but you can do:

const_aware(class_<T>("class_name"))
    .def(...)

etc., to get const-correct wrappers.  It doesn't work at all well with
visitors, though, and I think I could find a better syntax with the
ability to mess with class_ itself.

Most of the current design can be found in the boost sandbox, in the
python_extensions package.

>> - Make custom registry and template-based conversions more
>> accessible.
>
> +1
>
>> The former may just need more documentation, but the rvalue converters
>> in particular don't seem to have been intended as part of the public
>> API originally, and I think they're an important part of the
>> library. Template-based conversions are even more buried in the
>> details - you have to specialize five or six classes to get it
>> working.  I'd like to make it possible to create a template-based
>> conversion by (partial) specializing just one traits class.
>
> Hmm, well, IIRC anything you do by pure partial specialization does not
> go in the registry and can't participate in cross-module communication,
> as it's entirely static.  I guess we need a more formal separation of the
> two basic techniques for conversion, not to mention documentation, so
> people know to what they're opting in.
>

Yeah, that's exactly right.  To get it to work cross-module you have to
include a header with the specializations in all modules (and worse, in
all wrapper source files in those modules).  It makes things a little
too magical for my taste, but I don't see a way around that, and
template-based converters are really helpful in wrapping array-heavy
numerical code.  Documentation is indeed a big part of this.

>> - Automatic conversions for newer boost libraries (Fusion, Pointer
>> Container, and Filesystem are at the top of my list), and more for the
>> STL and iostreams standard libraries.  I'd like to integrate the
>> indexing suite (v2) into Boost.Python proper.
>
> Interesting and useful improvements.
>
>> - Allow Boost.Python wrapped classes to inherit from Python classes.
>
> +1
>
>> - An actual "boost.python" Python module to make exceptions and other
>> types used in wrappers easily accessible from Python.
>
> Nice.
>
>> - Some limited degree of priority-based overload matching.  Not sure
>> how best to approach this one yet, though.
>
> +1
> This is a solved problem... just not in Boost.Python.  Daniel Wallin
> worked it out for luabind and we were going to incorporate it into
> langbinding.  Happy to discuss it further.
>

That's great to hear.  I'll have to spend some time looking at luabind.
  I'm sure I'll have more questions later.


Thanks!

Jim

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Neal Becker
In reply to this post by Jim Bosch-2
The top of my list is improved interface to numpy.  I know there is work going
on in the form of ndarray, which seems promising.

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Jim Bosch-2
On 08/26/2011 04:47 PM, Neal Becker wrote:
> The top of my list is improved interface to numpy.  I know there is work going
> on in the form of ndarray, which seems promising.

I'm still hesitant to consider ndarray part of Boost.Python; it's really
a separate library, and I think providing a full multidimensional array
class to be outside the scope of what Boost.Python should be.  But I do
intend to keep developing ndarray in parallel with Boost.Python, and
some code may well flow from ndarray to Boost.Python.

This summer, Stefan Seefeld has been mentoring Ankit Daftery on a GSOC
project to clean up the low-level Boost.Python Numpy interface in the
Boost sandbox (which began life as part of ndarray).  While it may not
be an official part of Boost.Python or Boost (it may be submitted as a
separate library) immediately, that will provide an improved interface
to Numpy on a much shorter timescale than the larger Boost.Python
upgrades I'm thinking of for v3.

Ultimately I would like to incorporate this low-level Numpy support as
an optional component in the mainline Boost.Python v3, but I don't want
to tie Boost.Python users to a particular C++ array class, even my own.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Stefan Seefeld-2
In reply to this post by Dave Abrahams
On 08/26/2011 04:09 PM, Dave Abrahams wrote:

> on Thu Aug 25 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:
>> Jim,
>>
>> this is an interesting idea. There has been lots of general (dare I
>> say generic ?) discussion concerning process improvements (which
>> unfortunately most of the time diverted into tool discussions). Among
>> the fundamental issues is a modularization of boost.  I think it would
>> be great if boost.python could follow through on its own, by becoming
>> a separate entity.
> Separate from Boost?  I guess that's a possibility but I'm not sure I
> see the advantage.

Jim already followed up on that, and I fully agree with that. If things
can happen within Boost, all the better.

>> * A per-module type registry, to avoid conflicting converters in
>> multi-module projects.
> Interesting idea.  How does sharing types across multiple modules work
> in that scenario?

That's a good question. I don't have an answer to that. In fact, the
idea of having per-module type registries grew out of discussions I had
with Troy quite a while ago, where we considered all the bad things that
could happen if multiple modules tried to export the same types.  As
always: explicit is better than implicit.

>
>> - Some limited degree of priority-based overload matching.  Not sure
>> how best to approach this one yet, though.
> +1
> This is a solved problem... just not in Boost.Python.  Daniel Wallin
> worked it out for luabind and we were going to incorporate it into
> langbinding.  Happy to discuss it further.
>
>

I'm happy to see some discussion on langbinding in this context. I also
agree with Jim's pragmatic approach to this he is proposing in another mail.

    Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: New Major-Release Boost.Python Development

Stefan Seefeld-2
In reply to this post by Jim Bosch-2
On 08/26/2011 08:21 PM, Jim Bosch wrote:

> On 08/26/2011 04:47 PM, Neal Becker wrote:
>> The top of my list is improved interface to numpy.  I know there is
>> work going
>> on in the form of ndarray, which seems promising.
>
> I'm still hesitant to consider ndarray part of Boost.Python; it's
> really a separate library, and I think providing a full
> multidimensional array class to be outside the scope of what
> Boost.Python should be.  But I do intend to keep developing ndarray in
> parallel with Boost.Python, and some code may well flow from ndarray
> to Boost.Python.
>
> This summer, Stefan Seefeld has been mentoring Ankit Daftery on a GSOC
> project to clean up the low-level Boost.Python Numpy interface in the
> Boost sandbox (which began life as part of ndarray).  While it may not
> be an official part of Boost.Python or Boost (it may be submitted as a
> separate library) immediately, that will provide an improved interface
> to Numpy on a much shorter timescale than the larger Boost.Python
> upgrades I'm thinking of for v3.
>
> Ultimately I would like to incorporate this low-level Numpy support as
> an optional component in the mainline Boost.Python v3, but I don't
> want to tie Boost.Python users to a particular C++ array class, even
> my own.

I really think that this should be a separate module (right now we call
it tentatively "boost.numpy"). It introduces additional dependencies
which not all boost.python users may be interested in.

And yes, I had planned that we could address enough of the remaining
issues to be able to submit boost.numpy for formal review within the
next couple of months, i.e. ideally even before the end of this year.

FWIW,


    Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: [Boost.Python v3] Planning and Logistics

Dave Abrahams
In reply to this post by Jim Bosch-2

on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

> In the interest of keeping this discussion easy-to-follow, I'm going
> to reply to Dave's email twice, with new subjects - I'll stick to
> questions about logistics in this email, and talk about features and
> scope in another.
>
> In summary, I'm getting the sense that a branch in the mainline (not
> sandbox) Boost SVN is the way to go.  

It's all the same repository.  Working in the sandbox is essentially
equivalent to working in a branch of trunk; it's just a matter of where
the code lives in the SVN directory hierarchy.

> I imagine most communication would just happen on the C++-sig list,
> maybe with the [Boost.Python v3] subtitle I've used in this email.

Sure.

> On 08/26/2011 01:09 PM, Dave Abrahams wrote:
>>
>> Trying to catch up here, so responding to everything all at once.
>>
>> on Thu Aug 25 2011, Jim Bosch<talljimbo-AT-gmail.com>  wrote:
>>
>> Just how tall are you, Jimbo?
>
> 6'4".  Not that tall, in the grand scheme of things, but it was a
> teenage internet moniker that stuck with me.

Oh, then I should change my email to [hidden email]

>>> I'd also like to propose some changes that are slightly
>>> backwards-incompatible, as well as some that mess with the internals
>>> to an extent that I'd feel better about doing it outside Boost
>>> itself, to make it easier for adventurous users to play with the new
>>> version without affecting people who depend on having an extremely
>>> stable library in Boost.
>>
>> There's no need to do this "outside of Boost."  A branch in the Boost
>> repository is a perfect place for exploratory development that will
>> eventually be released as part of Boost.
>>
>>
>>> To that end, I'm inclined to copy the library to somewhere else
>>> (possibly the boost sandbox, but more likely a separate site), work on
>>> it, produce some minor releases, and re-submit it to Boost for
>>> review.
>>
>> If you want to go through another review process, that's up to you.
>> Getting review feedback definitely has its advantages.  Please, though,
>> use the sandbox or some other area of the Boost svn repository at least
>> until we get my hoped-for Git transition completed.
>
> I'd love to see a Git transition too, but I'm actually more familiar
> with SVN at the moment, and I can certainly see the advantages of
> working in the same repository as the previous version.
>
>>
>> on Thu Aug 25 2011, Stefan Seefeld<stefan-AT-seefeld.name>  wrote:
>>
>>>
>>> Jim,
>>>
>>> this is an interesting idea. There has been lots of general (dare I
>>> say generic ?) discussion concerning process improvements (which
>>> unfortunately most of the time diverted into tool discussions). Among
>>> the fundamental issues is a modularization of boost.  I think it would
>>> be great if boost.python could follow through on its own, by becoming
>>> a separate entity.
>>
>> Separate from Boost?  I guess that's a possibility but I'm not sure I
>> see the advantage.
>
> From my perspective, the advantages are mostly just the same reasons
> Boost has been talking about increased modularity, with regard to
> having stable and development versions for some packages and not for
> others, and to allow users to be able to install some components
> without installing all of them.  Boost.Python (right now, at least)
> depends on a very small core part of Boost, and to my knowledge no
> other Boost libraries have real dependencies on Boost.Python (not
> counting optional dependencies, like the Boost.Graph python bindings).
> If/when Boost as a whole goes more modular, I think any advantage of
> being separate would disappear, and the ideal case for Boost.Python
> would be for that to happen.

In that case, if I were you, I would actually start using Git with the
modularized / CMake-ified Boost at http://github.com/boost-lib.

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: [Boost.Python v3] Features and Scope

Dave Abrahams
In reply to this post by Jim Bosch-2

on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

> On 08/26/2011 01:09 PM, Dave Abrahams wrote:
>>
>> Well, speaking for myself, mostly time.  I'd be inclined to do a rewrite
>> along the lines of the langbinding ideas if I had time.
>>
>
> I had only been vaguely aware of langbinding until I followed up on
> your last email.  It's a very nice separation, though after bad
> experiences using SWIG I am a little wary about trying to build a
> one-size-fits-all front-end for different languages.
>
> It seems like a reasonable way to proceed would be to try to convert
> Boost.Python to the langbinding interface, but not be obsessive about
> avoiding Python-specific hacks in the frontend right now.  

Absolutely perfect; that's what I was thinking.

> Once we're more feature-complete in Python, we can worry about finding
> language-agnostic ways to do things that weren't anticipated in the
> original langbinding design.
>
>> Another thing I think should be examined and refreshed is the
>> documentation style.
>
> Agreed.  And I wouldn't just limit it to the official documentation;
> there are a lot of little tidbits of useful but often very old
> Boost.Python knowledge scattered around the internet (often on
> Python-affiliated sites or wikis).  It'd be nice to unify a lot of
> that, or at least update the obsolete stuff and add links to a more
> complete set of official documentation.

Sure.

>> Good.  Although, if I were you I would also carefully re-examine the
>> core to see if it has the best design.
>
> I wasn't originally planning on doing a full re-evaluation, but after
> looking over the langbinding proposal and hearing some of the other
> ideas, I think that will probably be necessary.

Happy to discuss with you anywhere you get stuck.  By the way, there's
actually a bunch of langbinding code checked into the SVN repo
somewhere.  Ah, here:
http://svn.boost.org/svn/boost/sandbox/langbinding/


>> on Thu Aug 25 2011, Ralf Grosse-Kunstleve<rwgrosse-kunstleve-AT-lbl.gov>  wrote:
>> <snip>
>>> Troy and Ravi have done a significant amount of work.
>>
>> Yes, and hopefully integrating that could be part of any next steps.
>>
>>> I hope they will comment for themselves.
>
> I was under the impression their work had already been integrated into
> Boost.Python v2.  Is there a large repository of additional
> Boost.Python work elsewhere that I should be aware of?

It's my impression that the integration was stalled due to Ralf's
concerns about object code size.  But I could be wrong.

>> Interesting idea.  How does sharing types across multiple modules work
>> in that scenario?
>
> Hmm.  I'm guessing the global type registry would still be the
> default, and per-module registries would override these when
> available?  It sounds like Stefan has a clear use case in mind, and
> I'd be curious to know what it is.

Me too.

>> on Fri Aug 26 2011, Jim Bosch<talljimbo-AT-gmail.com>  wrote:
>>
>>> On 08/26/2011 04:17 AM, Neal Becker wrote:
>>>> What sort of improvements did you have in mind?
>>>
>>> My list includes:
>>>
>>> - Propagating constness to Python (essentially already done as an
>>> extension, but it could have a much nicer interface if I could mess
>>> with class_ itself).
>>
>> Oooh, now that one is tricky.  I'd like to see the design you have in
>> mind.  Python doesn't have constness; it has immutability, which is
>> subtly different.
>
> Essentially, C++ objects returned as const references, pointers, or
> smart pointers get converted into a Python proxy object with methods
> that forward to the real wrapped object, but only if those methods are
> marked as const.  The proxy objects have rvalue converters but no
> lvalue converters.  

I like it!

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: [Boost.Python v3] Features and Scope

Stefan Seefeld-2
On 08/27/2011 04:40 PM, Dave Abrahams wrote:
>> Hmm.  I'm guessing the global type registry would still be the
>> default, and per-module registries would override these when
>> available?  It sounds like Stefan has a clear use case in mind, and
>> I'd be curious to know what it is.
> Me too.

I believe what we were discussing at the time was a situation where an
extension module would not only define converters for its own types, but
also common types that may are required by the API. This could in
particular include common types from libstdc++.

If multiple extension modules do this, than a Python program that
happens to use them in the same application may end up with undefined
behavior (does this constitute an ODR violation under the hood ?)

To make this work, the common type converters need to be factored out
and shared. This in itself is impractical (since the original authors
may not be aware of this need). Furthermore, a converter may require
module-specific behavior, i.e. converting a std::string in the context
of one module may be different from the desired conversion in another.

Binding converters to particular modules (and requiring to explicitly
import / exporting converters) seems like a solution to the above.

    Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: [Boost.Python v3] Features and Scope

Roman Yakovenko
In reply to this post by Jim Bosch-2
On Sat, Aug 27, 2011 at 2:39 AM, Jim Bosch <[hidden email]> wrote:
> I am aware of Py++ and the extensions associated with it, and some of that
> could definitely go into Boost.Python proper (and I think I've heard Roman
> state that he wouldn't have a problem with someone else doing the work to
> make it so, and he just didn't have time to do it himself).

If you will find them useful, I'll be glad to contribute and help with
the integration.

Py++ also contains the modified version of indexing suite v2 (more
containers, bug fixes and additional methods for existing containers).


Regards,
Roman
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
12