allowing targets to fail

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

allowing targets to fail

Boost - Build mailing list
Hello,

the bjam C code presently allows for actions to fail if the "xfail" flag
is set. This means that actions are either expected to pass (default) or
fail (if the xfail flag is set). `b2` will report an error if it wasn't
able to rebuild any targets.

How does that map to tests, which may pass or fail (at various stages of
the process) ? Is there a way to indicate to b2 that a (test) target may
fail to be built or be run ? And similarly, is it possible to still
collect the status of these tests so a proper test report can be generated ?

Consider for example a dependency graph where D depends on C depends on
B depends on A. A and B are ordinary (file) targets, but C and D are
'test targets' (for example, C may compile a test, and D may run it).
How could I allow C or D to fail (i.e. tell b2 not to generate an error
message), but still record the error for a later report generation step
? In addition, how could I write another target E that should only be
run after all the others, but without depending on the tests having run
successfully ?

Thanks,

        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 03/24/2017 01:00 PM, Stefan Seefeld via Boost-build wrote:

>
> the bjam C code presently allows for actions to fail if the "xfail" flag
> is set. This means that actions are either expected to pass (default) or
> fail (if the xfail flag is set). `b2` will report an error if it wasn't
> able to rebuild any targets.
>
> How does that map to tests, which may pass or fail (at various stages of
> the process) ? Is there a way to indicate to b2 that a (test) target may
> fail to be built or be run ? And similarly, is it possible to still
> collect the status of these tests so a proper test report can be generated ?
>
> Consider for example a dependency graph where D depends on C depends on
> B depends on A. A and B are ordinary (file) targets, but C and D are
> 'test targets' (for example, C may compile a test, and D may run it).
> How could I allow C or D to fail (i.e. tell b2 not to generate an error
> message), but still record the error for a later report generation step
> ?

  I'm not quite sure I understand.  If you don't want an
error message, then where exactly is the error supposed to
be recorded?

> In addition, how could I write another target E that should only be
> run after all the others, but without depending on the tests having run
> successfully ?
>

  Make E depend on a NOTFILE NOCARE target, which
has a dependency on all the tests.  In theory, I
think this is supposed to work, but the implementation
seems buggy.  (When I tried it, E silently disappears)

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 24.03.2017 15:56, Steven Watanabe via Boost-build wrote:

> AMDG
>
> On 03/24/2017 01:00 PM, Stefan Seefeld via Boost-build wrote:
>> the bjam C code presently allows for actions to fail if the "xfail" flag
>> is set. This means that actions are either expected to pass (default) or
>> fail (if the xfail flag is set). `b2` will report an error if it wasn't
>> able to rebuild any targets.
>>
>> How does that map to tests, which may pass or fail (at various stages of
>> the process) ? Is there a way to indicate to b2 that a (test) target may
>> fail to be built or be run ? And similarly, is it possible to still
>> collect the status of these tests so a proper test report can be generated ?
>>
>> Consider for example a dependency graph where D depends on C depends on
>> B depends on A. A and B are ordinary (file) targets, but C and D are
>> 'test targets' (for example, C may compile a test, and D may run it).
>> How could I allow C or D to fail (i.e. tell b2 not to generate an error
>> message), but still record the error for a later report generation step
>> ?
>   I'm not quite sure I understand.  If you don't want an
> error message, then where exactly is the error supposed to
> be recorded?

Somewhere internally, so I can capture it programmatically (and
post-process it in step E below, but without `b2` command itself to
report it to stdout / stderr.
>> In addition, how could I write another target E that should only be
>> run after all the others, but without depending on the tests having run
>> successfully ?
>>
>   Make E depend on a NOTFILE NOCARE target, which
> has a dependency on all the tests.  In theory, I
> think this is supposed to work, but the implementation
> seems buggy.  (When I tried it, E silently disappears)

Would that be a bug in the engine, or inside the boost.build jam /
python layer above it ?

Thanks,
        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 03/24/2017 02:02 PM, Stefan Seefeld via Boost-build wrote:

> On 24.03.2017 15:56, Steven Watanabe via Boost-build wrote:
>>
>> On 03/24/2017 01:00 PM, Stefan Seefeld via Boost-build wrote:
>>>
>>> Consider for example a dependency graph where D depends on C depends on
>>> B depends on A. A and B are ordinary (file) targets, but C and D are
>>> 'test targets' (for example, C may compile a test, and D may run it).
>>> How could I allow C or D to fail (i.e. tell b2 not to generate an error
>>> message), but still record the error for a later report generation step
>>> ?
>>   I'm not quite sure I understand.  If you don't want an
>> error message, then where exactly is the error supposed to
>> be recorded?
>
> Somewhere internally, so I can capture it programmatically (and
> post-process it in step E below, but without `b2` command itself to
> report it to stdout / stderr.
  There isn't really a way.  NOCARE is almost what you
need, but its behavior for targets with updating
actions isn't clearly specified.  I think it's
better just to let the targets fail.  Boost.Build
doesn't stop on errors by default, so I don't see
that avoiding a failed exit status buys you much.

>>> In addition, how could I write another target E that should only be
>>> run after all the others, but without depending on the tests having run
>>> successfully ?
>>>
>>   Make E depend on a NOTFILE NOCARE target, which
>> has a dependency on all the tests.  In theory, I
>> think this is supposed to work, but the implementation
>> seems buggy.  (When I tried it, E silently disappears)
>
> Would that be a bug in the engine, or inside the boost.build jam /
> python layer above it ?
>
  It's in the engine.  I think it was a symptom
of a different problem.  When I reduce the test
case to something basic, it seems to work.  I've
attached the sample script (run with b2 -f test.jam).

In Christ,
Steven Watanabe


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build

test.jam (390 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 24.03.2017 18:14, Steven Watanabe via Boost-build wrote:

> AMDG
>
> On 03/24/2017 02:02 PM, Stefan Seefeld via Boost-build wrote:
>> On 24.03.2017 15:56, Steven Watanabe via Boost-build wrote:
>>> On 03/24/2017 01:00 PM, Stefan Seefeld via Boost-build wrote:
>>>> Consider for example a dependency graph where D depends on C depends on
>>>> B depends on A. A and B are ordinary (file) targets, but C and D are
>>>> 'test targets' (for example, C may compile a test, and D may run it).
>>>> How could I allow C or D to fail (i.e. tell b2 not to generate an error
>>>> message), but still record the error for a later report generation step
>>>> ?
>>>   I'm not quite sure I understand.  If you don't want an
>>> error message, then where exactly is the error supposed to
>>> be recorded?
>> Somewhere internally, so I can capture it programmatically (and
>> post-process it in step E below, but without `b2` command itself to
>> report it to stdout / stderr.
>   There isn't really a way.  NOCARE is almost what you
> need, but its behavior for targets with updating
> actions isn't clearly specified.  I think it's
> better just to let the targets fail.  Boost.Build
> doesn't stop on errors by default, so I don't see
> that avoiding a failed exit status buys you much.

Can you elaborate on "not specified" a bit ? Or, would it be possible to
do specify it (and then adjust the implementation accordingly) ?
What effect does NOCARE have for dependent targets ? Will they be
skipped if a NOCARE target "fails" ?
I really do want to distinguish between build failures and test
failures, and would prefer a real solution rather than a work around in
the scripting layer.

>>>> In addition, how could I write another target E that should only be
>>>> run after all the others, but without depending on the tests having run
>>>> successfully ?
>>>>
>>>   Make E depend on a NOTFILE NOCARE target, which
>>> has a dependency on all the tests.  In theory, I
>>> think this is supposed to work, but the implementation
>>> seems buggy.  (When I tried it, E silently disappears)
>> Would that be a bug in the engine, or inside the boost.build jam /
>> python layer above it ?
>>
>   It's in the engine.  I think it was a symptom
> of a different problem.  When I reduce the test
> case to something basic, it seems to work.  I've
> attached the sample script (run with b2 -f test.jam).

Thanks, I'll have a look.

        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 03/24/2017 08:48 PM, Stefan Seefeld via Boost-build wrote:

> On 24.03.2017 18:14, Steven Watanabe via Boost-build wrote:
>>
>>   There isn't really a way.  NOCARE is almost what you
>> need, but its behavior for targets with updating
>> actions isn't clearly specified.  I think it's
>> better just to let the targets fail.  Boost.Build
>> doesn't stop on errors by default, so I don't see
>> that avoiding a failed exit status buys you much.
>
> Can you elaborate on "not specified" a bit ?

  The documentation just mentions that it will
continue to build dependent targets.  NOCARE
was originally to allow the inexact header search
algorithm to work without erroring out, and
I suspect that no one really considered what
it should mean for non-leaf targets.

> Or, would it be possible to
> do specify it (and then adjust the implementation accordingly) ?

Yes.

> What effect does NOCARE have for dependent targets ? Will they be
> skipped if a NOCARE target "fails" ?

They will not be skipped.

> I really do want to distinguish between build failures and test
> failures, and would prefer a real solution rather than a work around in
> the scripting layer.

  The easy solution is to split up the build and test
so that they run separately.  After updating the
regular build you can call UPDATE_NOW a second time
to run the tests, sending the output to a log file
instead of stdout.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
Here is a related question:

I'm looking at the "try_compile" example, which seems to use a similar
feature: It runs a configure check ("check_target_builds foo") and
allows it to fail, i.e.
the result is captured in a log file rather than on the console, in
addition to a conditional definition of feature sets.

How is this implemented in the engine, i.e. what is the underlying
target definition (as seen by the engine) and what causes the engine not
to report it as an "updated" or "failed target" on the console ?

Oh, and how does the caching work, i.e. where is that implemented ?


Thanks,
        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 03/31/2017 08:22 AM, Stefan Seefeld via Boost-build wrote:

> Here is a related question:
>
> I'm looking at the "try_compile" example, which seems to use a similar
> feature: It runs a configure check ("check_target_builds foo") and
> allows it to fail, i.e.
> the result is captured in a log file rather than on the console, in
> addition to a conditional definition of feature sets.
>
> How is this implemented in the engine, i.e. what is the underlying
> target definition (as seen by the engine) and what causes the engine not
> to report it as an "updated" or "failed target" on the console ?
>

  The target as seen by the engine is a normal target.
The only thing special is that it's built using a
call to UPDATE_NOW with a file descriptor for the log file.
See the documentation of UPDATE_NOW here:
http://www.boost.org/build/doc/html/jam/language.html#jam.language.rules.builtins.utility

> Oh, and how does the caching work, i.e. where is that implemented ?
>

config-cache.jam

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 31.03.2017 10:47, Steven Watanabe via Boost-build wrote:

> AMDG
>
> On 03/31/2017 08:22 AM, Stefan Seefeld via Boost-build wrote:
>> Here is a related question:
>>
>> I'm looking at the "try_compile" example, which seems to use a similar
>> feature: It runs a configure check ("check_target_builds foo") and
>> allows it to fail, i.e.
>> the result is captured in a log file rather than on the console, in
>> addition to a conditional definition of feature sets.
>>
>> How is this implemented in the engine, i.e. what is the underlying
>> target definition (as seen by the engine) and what causes the engine not
>> to report it as an "updated" or "failed target" on the console ?
>>
>   The target as seen by the engine is a normal target.
> The only thing special is that it's built using a
> call to UPDATE_NOW with a file descriptor for the log file.
> See the documentation of UPDATE_NOW here:
> http://www.boost.org/build/doc/html/jam/language.html#jam.language.rules.builtins.utility

Ah, interesting ! So this provides a mechanism to capture a target's
result while still letting if fail. But it seems this comes at a price:
To be able to capture results of individual target builds, I need to
submit them individually (via "UPDATE_NOW"), which precludes
parallelization.
If I submit multiple targets at once, I only get the overall success
back. Correct ?
>> Oh, and how does the caching work, i.e. where is that implemented ?
>>
> config-cache.jam

Thanks,
        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 03/31/2017 11:14 AM, Stefan Seefeld via Boost-build wrote:
>
> Ah, interesting ! So this provides a mechanism to capture a target's
> result while still letting if fail. But it seems this comes at a price:
> To be able to capture results of individual target builds, I need to
> submit them individually (via "UPDATE_NOW"), which precludes
> parallelization.
> If I submit multiple targets at once, I only get the overall success
> back. Correct ?

  Yep.  As a workaround, you can batch them, and
then call UPDATE_NOW again individually to get
the results.  UPDATE_NOW never tries to build
any target more than once, no matter how many times
it is called from the same b2 process.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 31.03.2017 13:35, Steven Watanabe via Boost-build wrote:

> AMDG
>
> On 03/31/2017 11:14 AM, Stefan Seefeld via Boost-build wrote:
>> Ah, interesting ! So this provides a mechanism to capture a target's
>> result while still letting if fail. But it seems this comes at a price:
>> To be able to capture results of individual target builds, I need to
>> submit them individually (via "UPDATE_NOW"), which precludes
>> parallelization.
>> If I submit multiple targets at once, I only get the overall success
>> back. Correct ?
>   Yep.  As a workaround, you can batch them, and
> then call UPDATE_NOW again individually to get
> the results.  UPDATE_NOW never tries to build
> any target more than once, no matter how many times
> it is called from the same b2 process.

I'm actually wondering whether that's (necessary) true: Is the bjam
engine thread-safe ? If I run `b2 -j 8`, may I invoke UPDATE_NOW from
one action and expect other targets to continue to be built in parallel
? Would that break ? Or would that serialize the execution ?

Thanks,
        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On Sat, Apr 1, 2017 at 6:53 AM, Stefan Seefeld via Boost-build <[hidden email]> wrote:
On <a href="tel:31.03.2017%2013" value="+13103201713">31.03.2017 13:35, Steven Watanabe via Boost-build wrote:
> AMDG
>
> On 03/31/2017 11:14 AM, Stefan Seefeld via Boost-build wrote:
>> Ah, interesting ! So this provides a mechanism to capture a target's
>> result while still letting if fail. But it seems this comes at a price:
>> To be able to capture results of individual target builds, I need to
>> submit them individually (via "UPDATE_NOW"), which precludes
>> parallelization.
>> If I submit multiple targets at once, I only get the overall success
>> back. Correct ?
>   Yep.  As a workaround, you can batch them, and
> then call UPDATE_NOW again individually to get
> the results.  UPDATE_NOW never tries to build
> any target more than once, no matter how many times
> it is called from the same b2 process.

I'm actually wondering whether that's (necessary) true: Is the bjam
engine thread-safe ? If I run `b2 -j 8`, may I invoke UPDATE_NOW from
one action and expect other targets to continue to be built in parallel
? Would that break ? Or would that serialize the execution ?

None of the above. Yes, the engine is parallel-safe. But that is irrelevant for this question. All targets are built with the same mechanism at the low level. The differences for UPDATE_NOW are when it executes the actions and how it waits for them. Regular action execution happens after the parsing/evaluation of the b2 files during the make stages. UPDATE_NOW happens during the parsing/evaluation and blocks it until the actions complete. So there's no question about thread safety, or parallel safety, since b2 is single threaded anyway.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 01.04.2017 08:51, Rene Rivera wrote:

> On Sat, Apr 1, 2017 at 6:53 AM, Stefan Seefeld via Boost-build
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     On 31.03.2017 13 <tel:31.03.2017%2013>:35, Steven Watanabe via
>     Boost-build wrote:
>     > AMDG
>     >
>     > On 03/31/2017 11:14 AM, Stefan Seefeld via Boost-build wrote:
>     >> Ah, interesting ! So this provides a mechanism to capture a
>     target's
>     >> result while still letting if fail. But it seems this comes at
>     a price:
>     >> To be able to capture results of individual target builds, I
>     need to
>     >> submit them individually (via "UPDATE_NOW"), which precludes
>     >> parallelization.
>     >> If I submit multiple targets at once, I only get the overall
>     success
>     >> back. Correct ?
>     >   Yep.  As a workaround, you can batch them, and
>     > then call UPDATE_NOW again individually to get
>     > the results.  UPDATE_NOW never tries to build
>     > any target more than once, no matter how many times
>     > it is called from the same b2 process.
>
>     I'm actually wondering whether that's (necessary) true: Is the bjam
>     engine thread-safe ? If I run `b2 -j 8`, may I invoke UPDATE_NOW from
>     one action and expect other targets to continue to be built in
>     parallel
>     ? Would that break ? Or would that serialize the execution ?
>
>
> None of the above. Yes, the engine is parallel-safe. But that is
> irrelevant for this question. All targets are built with the same
> mechanism at the low level. The differences for UPDATE_NOW are when it
> executes the actions and how it waits for them. Regular action
> execution happens after the parsing/evaluation of the b2 files during
> the make stages. UPDATE_NOW happens during the parsing/evaluation and
> blocks it until the actions complete.

So all those config targets are executed as they are parsed, i.e.
independent on what targets are specified on the command-line ? Even if
I call `./b2 clean` ??

> So there's no question about thread safety, or parallel safety, since
> b2 is single threaded anyway.

OK, then I'm not talking about b2 :-)
Really, I'm asking whether I could (in my own Python frontend) invoke
UPDATE_NOW from within an action, and expect the rest of the evaluation
of the dependency graph to continue undisturbed.


To put it a different way: As I'm composing the dependency graph of an
arbitrary project, I expect to wanting to "build" certain targets that
are allowed to fail (such as configure checks that should result in some
values being set or not set, or tests that may fail, but which shouldn't
cause the `./b2` invocation itself to fail, so wonder whether the
required mechanism to implement that is already there (in the engine
API) or not.

Thanks,
        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
In reply to this post by Boost - Build mailing list
On 01.04.2017 08:51, Rene Rivera wrote:

> On Sat, Apr 1, 2017 at 6:53 AM, Stefan Seefeld via Boost-build
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     On 31.03.2017 13 <tel:31.03.2017%2013>:35, Steven Watanabe via
>     Boost-build wrote:
>     > AMDG
>     >
>     > On 03/31/2017 11:14 AM, Stefan Seefeld via Boost-build wrote:
>     >> Ah, interesting ! So this provides a mechanism to capture a
>     target's
>     >> result while still letting if fail. But it seems this comes at
>     a price:
>     >> To be able to capture results of individual target builds, I
>     need to
>     >> submit them individually (via "UPDATE_NOW"), which precludes
>     >> parallelization.
>     >> If I submit multiple targets at once, I only get the overall
>     success
>     >> back. Correct ?
>     >   Yep.  As a workaround, you can batch them, and
>     > then call UPDATE_NOW again individually to get
>     > the results.  UPDATE_NOW never tries to build
>     > any target more than once, no matter how many times
>     > it is called from the same b2 process.
>
>     I'm actually wondering whether that's (necessary) true: Is the bjam
>     engine thread-safe ? If I run `b2 -j 8`, may I invoke UPDATE_NOW from
>     one action and expect other targets to continue to be built in
>     parallel
>     ? Would that break ? Or would that serialize the execution ?
>
>
> None of the above. Yes, the engine is parallel-safe. But that is
> irrelevant for this question. All targets are built with the same
> mechanism at the low level. The differences for UPDATE_NOW are when it
> executes the actions and how it waits for them. Regular action
> execution happens after the parsing/evaluation of the b2 files during
> the make stages. UPDATE_NOW happens during the parsing/evaluation and
> blocks it until the actions complete. So there's no question about
> thread safety, or parallel safety, since b2 is single threaded anyway.

I'm still trying to understand how this works...

Consider this little snippet (from example/try_compile/)

  obj foo : foo.cpp ;
  exe main : main.cpp : [ check-target-builds foo : <define>FOO=1 :
<define>FOO=0 ] ;

I can add an "ECHO done parsing ;" at the end of the Jamfile to
demonstrate that the "check-target-builds" rule isn't executed during
the parsing. But when is it executed ? Or more generally: what does the
overall data- and workflow look like ? The "define" feature (or property
?) can only be defined once the build result of "foo" is known, which
seems to rule out that it is done during the normal make stage. Or does it ?
Could someone please spell out all the build / evaluation stages for the
above snippet ? When is "foo" built ? When is the "define" property set
on the "main" target ? Etc. ? And given the above conditional definition
of properties, what else may be conditionalized on the outcome of a
target build such as "foo" ?

Many thanks,
        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 04/08/2017 06:31 AM, Stefan Seefeld via Boost-build wrote:

>
> I'm still trying to understand how this works...
>
> Consider this little snippet (from example/try_compile/)
>
>   obj foo : foo.cpp ;
>   exe main : main.cpp : [ check-target-builds foo : <define>FOO=1 :
> <define>FOO=0 ] ;
>
> I can add an "ECHO done parsing ;" at the end of the Jamfile to
> demonstrate that the "check-target-builds" rule isn't executed during
> the parsing. But when is it executed ? Or more generally: what does the
> overall data- and workflow look like ? The "define" feature (or property
> ?) can only be defined once the build result of "foo" is known, which
> seems to rule out that it is done during the normal make stage. Or does it ?
> Could someone please spell out all the build / evaluation stages for the
> above snippet ? When is "foo" built ? When is the "define" property set
> on the "main" target ? Etc. ? And given the above conditional definition
> of properties, what else may be conditionalized on the outcome of a
> target build such as "foo" ?
>

Phase 1: Load Jamfiles
Phase 2: Generate actual targets
Phase 3: Build targets

All properties including check-target-build are handled
in phase 2.

When loading Jamfiles we create a data structure that
looks something like this:

class basic-target {
    # returns a list of targets to build and the usage-requirements
    pair<target-list, property-set> construct(property-set);
}
# Constructing a project builds all targets in the project
class project-target : basic-target {}

# Corresponds to a single named target in a Jamfile
class main-target : basic-target {}

Then pseudo-code for the build looks something like:

project = load-jamfile(".");
(targets, usage-requirements) =
  project.construct(command-line-features);
bjam.update-now $(targets) ;

In Christ,
Steven Watanabe


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
Hi Steven,

thanks for the quick reply !

On 08.04.2017 10:13, Steven Watanabe via Boost-build wrote:

> AMDG
>
> On 04/08/2017 06:31 AM, Stefan Seefeld via Boost-build wrote:
>> I'm still trying to understand how this works...
>>
>> Consider this little snippet (from example/try_compile/)
>>
>>   obj foo : foo.cpp ;
>>   exe main : main.cpp : [ check-target-builds foo : <define>FOO=1 :
>> <define>FOO=0 ] ;
>>
>> I can add an "ECHO done parsing ;" at the end of the Jamfile to
>> demonstrate that the "check-target-builds" rule isn't executed during
>> the parsing. But when is it executed ? Or more generally: what does the
>> overall data- and workflow look like ? The "define" feature (or property
>> ?) can only be defined once the build result of "foo" is known, which
>> seems to rule out that it is done during the normal make stage. Or does it ?
>> Could someone please spell out all the build / evaluation stages for the
>> above snippet ? When is "foo" built ? When is the "define" property set
>> on the "main" target ? Etc. ? And given the above conditional definition
>> of properties, what else may be conditionalized on the outcome of a
>> target build such as "foo" ?
>>
> Phase 1: Load Jamfiles
> Phase 2: Generate actual targets
> Phase 3: Build targets

OK. And the parallel execution (requested via the "-j" option) would
only affect phase 3 ?

> All properties including check-target-build are handled
> in phase 2.

I see ! That means that the target argument passed to the
"check-target-build" ("foo" in my case above) isn't itself considered a
regular target, or somehow gets special treatment. Right ? Are there any
restrictions on what "foo" might be ? (In the above case it's itself a
target created with the "obj" meta-target rule, so I'd assume it could
be anything.

Conceptually this seems like a two-phase build, where a first pass
filters out all those "check-target-build" targets and builds those, to
evaluate dependent properties (what other effects may these have other
than defining properties ?). And the second phase then would be the
regular build, with all properties fixed.

In the autotools world this corresponds of course to the separation of
the "configure" and "make" steps, so this isn't unusual.

>
> When loading Jamfiles we create a data structure that
> looks something like this:
>
> class basic-target {
>     # returns a list of targets to build and the usage-requirements
>     pair<target-list, property-set> construct(property-set);
> }
> # Constructing a project builds all targets in the project
> class project-target : basic-target {}
>
> # Corresponds to a single named target in a Jamfile
> class main-target : basic-target {}
>
> Then pseudo-code for the build looks something like:
>
> project = load-jamfile(".");
> (targets, usage-requirements) =
>   project.construct(command-line-features);
> bjam.update-now $(targets) ;
>
> In Christ,
> Steven Watanabe

Thanks,
        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 04/08/2017 08:46 AM, Stefan Seefeld via Boost-build wrote:

>
> On 08.04.2017 10:13, Steven Watanabe via Boost-build wrote:
>>
>> Phase 1: Load Jamfiles
>> Phase 2: Generate actual targets
>> Phase 3: Build targets
>
> OK. And the parallel execution (requested via the "-j" option) would
> only affect phase 3 ?
>

  Yes.  In theory, if check-target-builds has
to update multiple targets it can build them
in parallel.  However, it's usually used to
build one target at a time.  I've considered
ways to allow check-target-builds to be fully
asynchronous and allow these checks to build
in parallel, but this would require almost a
total rewrite.

>> All properties including check-target-build are handled
>> in phase 2.
>
> I see ! That means that the target argument passed to the
> "check-target-build" ("foo" in my case above) isn't itself considered a
> regular target, or somehow gets special treatment. Right ?

  This is just a high level view.  The implementation
allows both generating targets and building targets
to be handled on a per-target basis.  The special treatment
is just that foo gets processed immediately instead of
being built later.

> Are there any
> restrictions on what "foo" might be ? (In the above case it's itself a
> target created with the "obj" meta-target rule, so I'd assume it could
> be anything.
>

It can be any target.

> Conceptually this seems like a two-phase build,

  It's actually more of a N-phase build.
The targets used by check-target-builds
can themselves use check-target-builds.
The process of building the targets used
by check-target-builds is identical to the
main build.

> where a first pass
> filters out all those "check-target-build" targets and builds those, to
> evaluate dependent properties (what other effects may these have other
> than defining properties ?).

  It's implemented as a <conditional> which can
only affect properties.  However, properties
can affect anything.

> And the second phase then would be the
> regular build, with all properties fixed.
>
> In the autotools world this corresponds of course to the separation of
> the "configure" and "make" steps, so this isn't unusual.
>

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 08.04.2017 11:28, Steven Watanabe via Boost-build wrote:

> AMDG
>
> On 04/08/2017 08:46 AM, Stefan Seefeld via Boost-build wrote:
>> On 08.04.2017 10:13, Steven Watanabe via Boost-build wrote:
>>> Phase 1: Load Jamfiles
>>> Phase 2: Generate actual targets
>>> Phase 3: Build targets
>> OK. And the parallel execution (requested via the "-j" option) would
>> only affect phase 3 ?
>>
>   Yes.  In theory, if check-target-builds has
> to update multiple targets it can build them
> in parallel.  However, it's usually used to
> build one target at a time.  I've considered
> ways to allow check-target-builds to be fully
> asynchronous and allow these checks to build
> in parallel, but this would require almost a
> total rewrite.

(I'm experimenting with a new Python frontend, only reusing the existing
engine. There, I'm able to call `engine.update_now(...)` from within an
action, which seems to get close to what you are describing.)

>>> All properties including check-target-build are handled
>>> in phase 2.
>> I see ! That means that the target argument passed to the
>> "check-target-build" ("foo" in my case above) isn't itself considered a
>> regular target, or somehow gets special treatment. Right ?
>   This is just a high level view.  The implementation
> allows both generating targets and building targets
> to be handled on a per-target basis.  The special treatment
> is just that foo gets processed immediately instead of
> being built later.

Can you qualify "immediately" and "later" a little more ? Normal targets
(those being built "later", that is) are only built if either explicitly
requested or needed by explicitly requested dependent targets.
But, for these "immediate" targets this doesn't seem to be possible, or
does it ? If these immediate targets are used while evaluating which
(regular) targets need building, it seems impossible to figure out
whether they need to be built or not. Thus, it sounds like immediate
targets are always unconditionally built. (This also corresponds to what
Rene suggested earlier.) Am I understanding this correctly ? (I now see
that even if I run `.../b2 clean` these immediate targets are built (or
taken from cache). I somehow had missed that earlier.)

Speaking of caching: How does b2 handle that ? Is the caching tied to
all "immediate" targets ? And more generally: how does b2 recognize
"immediate" targets ? Structurally (by being referenced in property
expressions, say), or by type (e.g. by virtue of the check-target-build
rule) ?

>> Are there any
>> restrictions on what "foo" might be ? (In the above case it's itself a
>> target created with the "obj" meta-target rule, so I'd assume it could
>> be anything.
>>
> It can be any target.
>
>> Conceptually this seems like a two-phase build,
>   It's actually more of a N-phase build.
> The targets used by check-target-builds
> can themselves use check-target-builds.
> The process of building the targets used
> by check-target-builds is identical to the
> main build.

What mechanism detects these dependencies (and thus triggers
prerequisite targets to be built) ? I assumed all this scheduling only
happens in phase 3 (in your original description), but it now seems
that's not the case.

>> where a first pass
>> filters out all those "check-target-build" targets and builds those, to
>> evaluate dependent properties (what other effects may these have other
>> than defining properties ?).
>   It's implemented as a <conditional> which can
> only affect properties.  However, properties
> can affect anything.


Thanks,
        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
AMDG

On 04/08/2017 11:00 AM, Stefan Seefeld via Boost-build wrote:

> On 08.04.2017 11:28, Steven Watanabe via Boost-build wrote:
>>
>>   Yes.  In theory, if check-target-builds has
>> to update multiple targets it can build them
>> in parallel.  However, it's usually used to
>> build one target at a time.  I've considered
>> ways to allow check-target-builds to be fully
>> asynchronous and allow these checks to build
>> in parallel, but this would require almost a
>> total rewrite.
>
> (I'm experimenting with a new Python frontend, only reusing the existing
> engine. There, I'm able to call `engine.update_now(...)` from within an
> action, which seems to get close to what you are describing.)

  It doesn't.  What is necessary is a non-blocking version
of UPDATE_NOW that notifies you when the targets have
finished updating.

>>>> All properties including check-target-build are handled
>>>> in phase 2.
>>> I see ! That means that the target argument passed to the
>>> "check-target-build" ("foo" in my case above) isn't itself considered a
>>> regular target, or somehow gets special treatment. Right ?
>>   This is just a high level view.  The implementation
>> allows both generating targets and building targets
>> to be handled on a per-target basis.  The special treatment
>> is just that foo gets processed immediately instead of
>> being built later.
>
> Can you qualify "immediately" and "later" a little more ?

  Immediately means that we call UPDATE_NOW
directly.  Later means that we add the target
to dependency graph and assume that someone
else will call UPDATE_NOW.

> Normal targets
> (those being built "later", that is) are only built if either explicitly
> requested or needed by explicitly requested dependent targets.
> But, for these "immediate" targets this doesn't seem to be possible, or
> does it ?

  Boost.Build only generates the actual targets
which are needed.  check-target-builds will only
be evaluated if we are building the target that
uses it.

> If these immediate targets are used while evaluating which
> (regular) targets need building, it seems impossible to figure out
> whether they need to be built or not. Thus, it sounds like immediate
> targets are always unconditionally built.

  Immediate targets are only built if they are out-of-date,
just like any other target.

> (This also corresponds to what
> Rene suggested earlier.) Am I understanding this correctly ? (I now see
> that even if I run `.../b2 clean` these immediate targets are built (or
> taken from cache). I somehow had missed that earlier.)
>

Right.  clean interacts somewhat poorly with check-target-builds.

> Speaking of caching: How does b2 handle that ? Is the caching tied to
> all "immediate" targets ?

  check-target-builds saves the result in a special
file which records whether it passed or failed.

> And more generally: how does b2 recognize
> "immediate" targets ? Structurally (by being referenced in property
> expressions, say), or by type (e.g. by virtue of the check-target-build
> rule) ?
>

  b2 itself doesn't care.  It happily updates targets
whenever you call UPDATE_NOW.

>>> Are there any
>>> restrictions on what "foo" might be ? (In the above case it's itself a
>>> target created with the "obj" meta-target rule, so I'd assume it could
>>> be anything.
>>>
>> It can be any target.
>>
>>> Conceptually this seems like a two-phase build,
>>   It's actually more of a N-phase build.
>> The targets used by check-target-builds
>> can themselves use check-target-builds.
>> The process of building the targets used
>> by check-target-builds is identical to the
>> main build.
>
> What mechanism detects these dependencies (and thus triggers
> prerequisite targets to be built) ? I assumed all this scheduling only
> happens in phase 3 (in your original description), but it now seems
> that's not the case.
>

  You're thinking about it wrong.  b2 doesn't break
things up into multiple passes.  All the engine does
is to update targets when you ask it to (which you
can do as many times as you want).  check-target-builds
requests updating a target when it is evaluated.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: allowing targets to fail

Boost - Build mailing list
On 08.04.2017 13:31, Steven Watanabe via Boost-build wrote:
> AMDG
>
> On 04/08/2017 11:00 AM, Stefan Seefeld via Boost-build wrote:
>
>> Can you qualify "immediately" and "later" a little more ?
>   Immediately means that we call UPDATE_NOW
> directly.

Directly where, in what phase of the workflow ? This may seem pedantic,
but I think it's important to spell out in full detail so users of those
features understand the implications.

>   Later means that we add the target
> to dependency graph and assume that someone
> else will call UPDATE_NOW.

To rephrase my question above: I understand the there is a top-level
call to UPDATE_NOW, where the arguments are the targets explicitly
requested from command-line options (and associated properties). So all
those other calls to UPDATE_NOW must be triggered through some other
mechanism by which the (or another ?) dependency graph is evaluated,
eventually leading to those "immediate" targets.
In other words, it seems there are multiple ways in which the (or a)
dependency graph is traversed: the one used by the bjam engine itself
("phase 3" as per your description), and the others sometime earlier,
while that dependency graph is constructed ("phase 2" as per your
description).

Am I capturing the situation correctly with that description ?

>> Normal targets
>> (those being built "later", that is) are only built if either explicitly
>> requested or needed by explicitly requested dependent targets.
>> But, for these "immediate" targets this doesn't seem to be possible, or
>> does it ?
>   Boost.Build only generates the actual targets
> which are needed.  check-target-builds will only
> be evaluated if we are building the target that
> uses it.

This is where I'm (slightly) confused: to me it doesn't look like any
real target "uses" the check-target-builds target. Rather, it's the
'main' *meta*-target that requires it in the target instantiation process.

Right ?

Thanks,
        Stefan


--

      ...ich hab' noch einen Koffer in Berlin...

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
12
Loading...