selecting Python version for Boost.Python

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

selecting Python version for Boost.Python

Boost - Build mailing list

Hello,

[Rene, I suppose you are the person most familiar with the Python support in b2, so I suspect you know the answer to the below questions. But this is also about consistency, so I'd like to extend the question to understand how b2 deals with multiple values (such as for the "toolset" feature) in general.]


I'm reviewing the b2 logic in `python.jam` as well as Boost.Python's Jamfiles, and I'm not sure about the intended logic as far as multiple Python versions are concerned:

With my `user-config.jam` file containing two `using python ...` statements (with two distinct versions), is the intent for Boost.Python being built for both ? Or only if the two differ in the major version number (i.e., '2' and '3') ? And what about using "python=N" as command-line option to `b2` ? if I set "N" to be a version string not defined in my `user-config.jam` file, I get an error ("... is not a known value of feature <python"). But if I give a "known value", nothing happens, i.e. `b2` still builds both versions.

So what is the intended semantic for giving multiple python versions in my `user-config.jam` file, and what is the intent for "python=N" on the command line ?

Thanks,

        Stefan


Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
AMDG

On 02/06/2018 11:47 AM, Stefan Seefeld via Boost-build wrote:

> [Rene, I suppose you are the person most familiar with the Python
> support in b2, so I suspect you know the answer to the below questions.
> But this is also about consistency, so I'd like to extend the question
> to understand how b2 deals with multiple values (such as for the
> "toolset" feature) in general.]
>
>
> I'm reviewing the b2 logic in `python.jam` as well as Boost.Python's
> Jamfiles, and I'm not sure about the intended logic as far as multiple
> Python versions are concerned:
>
> With my `user-config.jam` file containing two `using python ...`
> statements (with two distinct versions), is the intent for Boost.Python
> being built for both ? Or only if the two differ in the major version
> number (i.e., '2' and '3') ? And what about using "python=N" as
> command-line option to `b2` ? if I set "N" to be a version string not
> defined in my `user-config.jam` file, I get an error ("... is not a
> known value of feature <python"). But if I give a "known value", nothing
> happens, i.e. `b2` still builds both versions.
>
> So what is the intended semantic for giving multiple python versions in
> my `user-config.jam` file, and what is the intent for "python=N" on the
> command line ?
>

The normal b2 rules are:
- The first version specified is the default, which will
  be used if no version is given explicitly.
- If one or more versions are specified on the
  command line, then exactly those versions will be built.
- All versions used must be initialized in user-config.jam
  (or another configuration file).

  Python does not conform to this.  It only supports one
version of python 2 and one version of python 3 which
are determined solely by the configuration files and
cannot be overridden on the command line.

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
On 06.02.2018 14:30, Steven Watanabe via Boost-build wrote:
AMDG

On 02/06/2018 11:47 AM, Stefan Seefeld via Boost-build wrote:
[Rene, I suppose you are the person most familiar with the Python
support in b2, so I suspect you know the answer to the below questions.
But this is also about consistency, so I'd like to extend the question
to understand how b2 deals with multiple values (such as for the
"toolset" feature) in general.]


I'm reviewing the b2 logic in `python.jam` as well as Boost.Python's
Jamfiles, and I'm not sure about the intended logic as far as multiple
Python versions are concerned:

With my `user-config.jam` file containing two `using python ...`
statements (with two distinct versions), is the intent for Boost.Python
being built for both ? Or only if the two differ in the major version
number (i.e., '2' and '3') ? And what about using "python=N" as
command-line option to `b2` ? if I set "N" to be a version string not
defined in my `user-config.jam` file, I get an error ("... is not a
known value of feature <python"). But if I give a "known value", nothing
happens, i.e. `b2` still builds both versions.

So what is the intended semantic for giving multiple python versions in
my `user-config.jam` file, and what is the intent for "python=N" on the
command line ?

The normal b2 rules are:
- The first version specified is the default, which will
  be used if no version is given explicitly.

So for those "normal" cases, where is a) the list of specified versions and b) the selected version stored ?
(I see the `toolset` variable (feature) holds the full list. Where is the actual selection stored if I run `b2 toolset=gcc` ?)

- If one or more versions are specified on the
  command line, then exactly those versions will be built.

That corresponds to my expectation.

- All versions used must be initialized in user-config.jam
  (or another configuration file).

Can you elaborate ? What about default versions ? What if I have an empty `user-config.jam` file ? "gcc" will still be found, even if I don't explicitly initialize it in any configuration file. Or is there anywhere a list of explicit gcc versions that are supported ?

  Python does not conform to this.  It only supports one
version of python 2 and one version of python 3 which
are determined solely by the configuration files and
cannot be overridden on the command line.

That's exactly what I'd like to fix. In fact, there are multiple issues I'd like to fix, but to do that I need to understand what behaviour would be consistent with the rest of b2.
Here are the issues I want to address:

* `b2 python=<N>` doesn't have any effect, when it really should pick a single python version
* the current logic builds at most two versions, as you say: at most one python 2 and one python 3 version. For consistency we should build as many versions as are selected
* a requirement for the last point: instead of just appending '3' to the library name iff it's built with Python 3, the real Python version, i.e. <major><minor> (without the dot) should be appended.

Where is the looping over all (selected) toolsets being implemented ? And what would be the right location to add the equivalent looping over Python versions ? (I'm now hacking something together within the Boost.Python Jamfile. But that's far from ideal as I need similar loops for the tests...)

Thanks,
        Stefan

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
AMDG

On 02/06/2018 12:46 PM, Stefan Seefeld via Boost-build wrote:
> On 06.02.2018 14:30, Steven Watanabe via Boost-build wrote:
>> <snip>
>> The normal b2 rules are:
>> - The first version specified is the default, which will
>>   be used if no version is given explicitly.
>
> So for those "normal" cases, where is a) the list of specified versions
> and

It's a global list in the feature module:
toolsets = [ feature.get-values <toolset> ] ;

new values are added by feature.extend.

> b) the selected version stored ?

The selected version is stored in a property-set
which is passed as a parameter through all the
internal machinery of Boost.Build.

If you build multiple versions, it will just
cause Boost.Build to run main-target.generate
with each property-set in turn.

> (I see the `toolset` variable (feature) holds the full list. Where is
> the actual selection stored if I run `b2 toolset=gcc` ?)
> >> - If one or more versions are specified on the
>>   command line, then exactly those versions will be built.
>
> That corresponds to my expectation.
>
>> - All versions used must be initialized in user-config.jam
>>   (or another configuration file).
>
> Can you elaborate ? What about default versions ? What if I have an
> empty `user-config.jam` file ? "gcc" will still be found, even if I
> don't explicitly initialize it in any configuration file. Or is there
> anywhere a list of explicit gcc versions that are supported ?
>

toolset gets special case handling in the
core of Boost.Build.  IIRC, python's Jamfile
has

if ! [ python.configured ]
{
    using python ;
}

which sets up a default.  (Boost.Build doesn't
actually care whether `using` is in a configuration
file or a Jamfile.  user-config.jam is essentially
a standalone Jamfile which gets loaded before anything
else.)

>>   Python does not conform to this.  It only supports one
>> version of python 2 and one version of python 3 which
>> are determined solely by the configuration files and
>> cannot be overridden on the command line.
>
> That's exactly what I'd like to fix. In fact, there are multiple issues
> I'd like to fix, but to do that I need to understand what behaviour
> would be consistent with the rest of b2.
> Here are the issues I want to address:
>
> * `b2 python=<N>` doesn't have any effect, when it really should pick a
> single python version
> * the current logic builds at most two versions, as you say: at most one
> python 2 and one python 3 version. For consistency we should build as
> many versions as are selected
> * a requirement for the last point: instead of just appending '3' to the
> library name iff it's built with Python 3, the real Python version, i.e.
> <major><minor> (without the dot) should be appended.
>

The way to do this is to change the <tag> rule to append
the version to the name instead of creating separate
targets for python2 and python3.  This change should
fix all of the above.

> Where is the looping over all (selected) toolsets being implemented ?

It's implemented in two different places:

- build-system.jam processes the command line
  arguments into a list of property-sets.
- A main-target (targets.jam) can specify multiple
  values of a property in its default-build.

> And what would be the right location to add the equivalent looping over
> Python versions ? (I'm now hacking something together within the
> Boost.Python Jamfile. But that's far from ideal as I need similar loops
> for the tests...)
>

You don't need an explicit loop.  Boost.Build can manage it
for you.

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
On 06.02.2018 15:25, Steven Watanabe via Boost-build wrote:
AMDG

On 02/06/2018 12:46 PM, Stefan Seefeld via Boost-build wrote:
  Python does not conform to this.  It only supports one
version of python 2 and one version of python 3 which
are determined solely by the configuration files and
cannot be overridden on the command line.
That's exactly what I'd like to fix. In fact, there are multiple issues
I'd like to fix, but to do that I need to understand what behaviour
would be consistent with the rest of b2.
Here are the issues I want to address:

* `b2 python=<N>` doesn't have any effect, when it really should pick a
single python version
* the current logic builds at most two versions, as you say: at most one
python 2 and one python 3 version. For consistency we should build as
many versions as are selected
* a requirement for the last point: instead of just appending '3' to the
library name iff it's built with Python 3, the real Python version, i.e.
<major><minor> (without the dot) should be appended.

The way to do this is to change the <tag> rule to append
the version to the name instead of creating separate
targets for python2 and python3.  This change should
fix all of the above.

Trying to figure out what the "tag rule" is and how to change (I assume "override" ?) it, I find https://github.com/boostorg/python/blob/develop/build/Jamfile#L110-L111. Can you explain the meaning of those two lines ? (In particular, the second line, featuring "...python-tag".)

Thanks,

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
AMDG

On 02/06/2018 01:53 PM, Stefan Seefeld via Boost-build wrote:
>
> Trying to figure out what the "tag rule" is and how to change (I assume
> "override" ?) it, I find
> https://github.com/boostorg/python/blob/develop/build/Jamfile#L110-L111.
> Can you explain the meaning of those two lines ? (In particular, the
> second line, featuring "...python-tag".)
>

The -<tag> line removes the <tag> that was set in
Jamroot.  python-tag is a function defined in
boostcpp.jam.  (It's imported into Jamroot.  The
ugly mess with BOOST_JAMROOT_MODULE is to handle
scoping rules.)  Actually <tag>@python-tag would
also work because rules are inherited from parent projects.
(The -<tag> line can't be changed because -<property>
is very fragile and relies an exact textual match)

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
On 06.02.2018 16:05, Steven Watanabe via Boost-build wrote:
AMDG

On 02/06/2018 01:53 PM, Stefan Seefeld via Boost-build wrote:
Trying to figure out what the "tag rule" is and how to change (I assume
"override" ?) it, I find
https://github.com/boostorg/python/blob/develop/build/Jamfile#L110-L111.
Can you explain the meaning of those two lines ? (In particular, the
second line, featuring "...python-tag".)

The -<tag> line removes the <tag> that was set in
Jamroot.  python-tag is a function defined in
boostcpp.jam.  (It's imported into Jamroot.  The
ugly mess with BOOST_JAMROOT_MODULE is to handle
scoping rules.)  Actually <tag>@python-tag would
also work because rules are inherited from parent projects.
(The -<tag> line can't be changed because -<property>
is very fragile and relies an exact textual match)

Yuck, that sounds messy.

OK, I'm attempting to simply change the `python-tag` rule (in Jamroot) to inject the python version (<major><minor>) into the generated name (if it's a library).
Additionally to that change, I modified Boost.Python's Jamfile to remove the explicit looping over versions, as per your suggestion:

https://github.com/boostorg/python/commit/ae1875b3ac3e90f68900c1c8fe88fa8556dac2e1

This works, somewhat: If I have a single "using python" statement in my user-config.jam file, I get libraries "boost_python" and "boost_numpy" with the expected version suffix.
However, I only get one ! With more than one "using python" statements, only the first will be built. What am I missing ?
(Oh, and in addition, the build path also doesn't include the python version any longer, so even if b2 would do the looping implicitly, different versions of object files would overwrite each other, as per a bug we discussed recently.)

What is missing ?

Thanks,

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
AMDG

On 02/06/2018 03:07 PM, Stefan Seefeld via Boost-build wrote:

>
> OK, I'm attempting to simply change the `python-tag` rule (in Jamroot)
> to inject the python version (<major><minor>) into the generated name
> (if it's a library).
> Additionally to that change, I modified Boost.Python's Jamfile to remove
> the explicit looping over versions, as per your suggestion:
>
> https://github.com/boostorg/python/commit/ae1875b3ac3e90f68900c1c8fe88fa8556dac2e1
>
> This works, somewhat: If I have a single "using python" statement in my
> user-config.jam file, I get libraries "boost_python" and "boost_numpy"
> with the expected version suffix.
> However, I only get one ! With more than one "using python" statements,
> only the first will be built. What am I missing ?

Did you ask for multiple versions on the command line?

> (Oh, and in addition, the build path also doesn't include the python
> version any longer, so even if b2 would do the looping implicitly,
> different versions of object files would overwrite each other, as per a
> bug we discussed recently.)
>

The version is not included in the path when it is equal to the default.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
On 06.02.2018 17:30, Steven Watanabe via Boost-build wrote:
AMDG

On 02/06/2018 03:07 PM, Stefan Seefeld via Boost-build wrote:
OK, I'm attempting to simply change the `python-tag` rule (in Jamroot)
to inject the python version (<major><minor>) into the generated name
(if it's a library).
Additionally to that change, I modified Boost.Python's Jamfile to remove
the explicit looping over versions, as per your suggestion:

https://github.com/boostorg/python/commit/ae1875b3ac3e90f68900c1c8fe88fa8556dac2e1

This works, somewhat: If I have a single "using python" statement in my
user-config.jam file, I get libraries "boost_python" and "boost_numpy"
with the expected version suffix.
However, I only get one ! With more than one "using python" statements,
only the first will be built. What am I missing ?
Did you ask for multiple versions on the command line?

No. I see I misunderstood. Unless versions are explicitly specified on the command-line, only the first one found is being built. OK, that works indeed now.
(Oh, and in addition, the build path also doesn't include the python
version any longer, so even if b2 would do the looping implicitly,
different versions of object files would overwrite each other, as per a
bug we discussed recently.)

The version is not included in the path when it is equal to the default.

Indeed, so that works too.

So now that the library name has changed, I suppose there is at least one other change I need to add: Windows auto-linking support requires the library name to be encoded in some source (header) file, doesn't it ?

Any other changes I need to make, either to code or build logic, somewhere ?

Thanks !

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
AMDG

On 02/06/2018 03:40 PM, Stefan Seefeld via Boost-build wrote:
>
> So now that the library name has changed, I suppose there is at least
> one other change I need to add: Windows auto-linking support requires
> the library name to be encoded in some source (header) file, doesn't it ?
>

  Right.  It's in a header somewhere... probably named
config.hpp or autolink.hpp

> Any other changes I need to make, either to code or build logic, somewhere ?
>

- python-tag is used by libraries that depend on python (I think
  mpi has python bindings).  Make sure that the library naming
  makes sense for those as well.
- --layout=system should probably continue using the current
  naming scheme (python/python3)
- boostcpp.jam in develop has code to use an old naming scheme.
  --layout=versioned-1.66 should use the old names.  (This
  is only intended to make updating Boost less painful
  whenever we change library naming.).

In Christ,
Steven Watanabe
_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
On 06.02.2018 18:08, Steven Watanabe via Boost-build wrote:
AMDG

On 02/06/2018 03:40 PM, Stefan Seefeld via Boost-build wrote:
So now that the library name has changed, I suppose there is at least
one other change I need to add: Windows auto-linking support requires
the library name to be encoded in some source (header) file, doesn't it ?

  Right.  It's in a header somewhere... probably named
config.hpp or autolink.hpp

Any other changes I need to make, either to code or build logic, somewhere ?

- python-tag is used by libraries that depend on python (I think
  mpi has python bindings).  Make sure that the library naming
  makes sense for those as well.

OK, I'll send a note / question to the Boost list to see who else is using the `python-tag` rule.

- --layout=system should probably continue using the current
  naming scheme (python/python3)

Hmm, more and more distributions rename 'python' to 'python2' to prepare for a pending move to Python 3 by default. So I'd like to propose we follow that trend.

- boostcpp.jam in develop has code to use an old naming scheme.
  --layout=versioned-1.66 should use the old names.  (This
  is only intended to make updating Boost less painful
  whenever we change library naming.).

I'm not sure I understand,  but I'll look at the code to figure out what that "versioned-1.66" scheme is.

Thanks,

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
In reply to this post by Boost - Build mailing list
Hi Steven,

here is the first related PR: https://github.com/boostorg/build/pull/290
The next one (probably to Jamroot) will perhaps be a bit contentious, and require some back and forth. But the above is simple and a prerequisite to anything that follows, and thus it would be great to get that in quickly and independently.

Thanks,


Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: selecting Python version for Boost.Python

Boost - Build mailing list
In reply to this post by Boost - Build mailing list
On 06.02.2018 18:08, Steven Watanabe via Boost-build wrote:

Any other changes I need to make, either to code or build logic, somewhere ?

- python-tag is used by libraries that depend on python (I think
  mpi has python bindings).  Make sure that the library naming
  makes sense for those as well.

For now I intend to overwrite the `python-tag` rule inside Boost.Python, so no other Boost component will be affected. If that change is successfully completed, I may submit a similar PR to Boost.MPI (so the build logic change and the auto-linking code change are combined into one). Once both are complete, it's time to move the `python-tag` upstream, into the root repo.
- --layout=system should probably continue using the current
  naming scheme (python/python3)
- boostcpp.jam in develop has code to use an old naming scheme.
  --layout=versioned-1.66 should use the old names.  (This
  is only intended to make updating Boost less painful
  whenever we change library naming.).
I haven't found any code related to "--layout=versioned-1.66", I only see https://github.com/boostorg/boost/blob/master/boostcpp.jam#L124-L130. Can you point me to any code (or documentation) describing what you have in mind ?

Thanks,


Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build