bjam handling of temps

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

bjam handling of temps

Boost - Build mailing list

Hi Steven,

I'm investigating an issue with bjam's handling of temporaries. It seems I'm still not understanding how bjam handles them. Consider this code snippet:

actions make {
  touch $(<)
}

actions cat {
  cat $(>) > $(<)
}

make A ;
cat B : A ;
cat C : A ;
cat D : B C ;

TEMPORARY B ;

DEPENDS B : A ;
DEPENDS D : B C ;

UPDATE D ;


Running `bjam` on this the first time creates A, B, C, and D, as expected. Calling it again updates D again, and I don't understand why. If I remove the TEMPORARY flag on B, D isn't updated, so this forceful update of D seems to be caused by B being a temp.

The second issue seems even more serious: if I remove B, and call bjam again, nothing is updated, as expected. But if I remove B and C and call bjam, I get an error as D is attempted to be updated after C is remade, but B wasn't updated (it's marked "temporary stable" in the debug output). Why isn't B updated first ?

The logic in make.c suggest's that, despite B's binding is set to PARENT (i.e., D), its fate is determined before that of D, and thus is found to be stable, even though a little later D's fate is set to update. Shouldn't a temporary's fate be determined *after* its parent ?

What am I missing ?

Thanks,


Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: bjam handling of temps

Boost - Build mailing list
AMDG

On 11/03/2017 09:35 AM, Stefan Seefeld via Boost-build wrote:

>
> I'm investigating an issue with bjam's handling of temporaries. It seems I'm
> still not understanding how bjam handles them. Consider this code snippet:
>
> actions make {
>    touch $(<)
> }
>
> actions cat {
>    cat $(>) > $(<)
> }
>
> make A ;
> cat B : A ;
> cat C : A ;
> cat D : B C ;
>
> TEMPORARY B ;
>
> DEPENDS B : A ;
> DEPENDS D : B C ;
>
> UPDATE D ;
>
>
> Running `bjam` on this the first time creates A, B, C, and D, as expected.
> Calling it again updates D again, and I don't understand why. If I remove the
> TEMPORARY flag on B, D isn't updated, so this forceful update of D seems to be
> caused by B being a temp.
>
> The second issue seems even more serious: if I remove B, and call bjam again,
> nothing is updated, as expected. But if I remove B and C and call bjam, I get an
> error as D is attempted to be updated after C is remade, but B wasn't updated
> (it's marked "temporary stable" in the debug output). Why isn't B updated first ?
>
> The logic in make.c suggest's that, despite B's binding is set to PARENT (i.e.,
> D), its fate is determined before that of D, and thus is found to be stable,
> even though a little later D's fate is set to update. Shouldn't a temporary's
> fate be determined *after* its parent ?
>
> What am I missing ?
>

Two things:

First of all, a TEMPORARY target is supposed to be
deleted:

actions RmTemps {
  rm $(>)
}
...
cat D : B C ;
RmTemps D : B ;

  Rebuilding when a temporary is present seems to be
deliberate, though I'm not quite clear on the logic
behind this.
#define T_FATE_SPOIL          4       /* >= SPOIL rebuilds parents */
#define T_FATE_ISTMP          4       /* unneeded temp target oddly
present */

Second, TEMPORARY is really designed to be used with the
`updated` actions flag:

actions piecemeal updated ar {
  ar rc $(<) $(>)
}

TEMPORARY x.o y.o z.o ;
ar x.a : x.o y.o z.o ;
RmTemps x.a : x.o y.o z.o ;

In this case, rebuilding x.a because of one changed source,
should not force the other object files to be created.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: bjam handling of temps

Boost - Build mailing list
On 03.11.2017 12:43, Steven Watanabe via Boost-build wrote:
AMDG

On 11/03/2017 09:35 AM, Stefan Seefeld via Boost-build wrote:
I'm investigating an issue with bjam's handling of temporaries. It seems I'm 
still not understanding how bjam handles them. Consider this code snippet:

actions make {
   touch $(<)
}

actions cat {
   cat $(>) > $(<)
}

make A ;
cat B : A ;
cat C : A ;
cat D : B C ;

TEMPORARY B ;

DEPENDS B : A ;
DEPENDS D : B C ;

UPDATE D ;


Running `bjam` on this the first time creates A, B, C, and D, as expected. 
Calling it again updates D again, and I don't understand why. If I remove the 
TEMPORARY flag on B, D isn't updated, so this forceful update of D seems to be 
caused by B being a temp.

The second issue seems even more serious: if I remove B, and call bjam again, 
nothing is updated, as expected. But if I remove B and C and call bjam, I get an 
error as D is attempted to be updated after C is remade, but B wasn't updated 
(it's marked "temporary stable" in the debug output). Why isn't B updated first ?

The logic in make.c suggest's that, despite B's binding is set to PARENT (i.e., 
D), its fate is determined before that of D, and thus is found to be stable, 
even though a little later D's fate is set to update. Shouldn't a temporary's 
fate be determined *after* its parent ?

What am I missing ?

Two things:

First of all, a TEMPORARY target is supposed to be
deleted:

actions RmTemps {
  rm $(>)
}
...
cat D : B C ;
RmTemps D : B ;

  Rebuilding when a temporary is present seems to be
deliberate, though I'm not quite clear on the logic
behind this.
#define T_FATE_SPOIL          4       /* >= SPOIL rebuilds parents */
#define T_FATE_ISTMP          4       /* unneeded temp target oddly
present */

Yeah, I have wondered about that, too. I think this is manifest in code as https://github.com/boostorg/build/blob/develop/src/engine/make.c#L621-L624, which I have tentatively commented out, as I don't follow the rationale of existing temps being considered tainted. (Quite the opposite, actually: if I keep a temp around, perhaps I'm in the middle of a debug session where I do want to inject code into the otherwise automated process. So I'd rather treat existing "temps" as ordinary (non-temp) targets.)
Second, TEMPORARY is really designed to be used with the
`updated` actions flag:

actions piecemeal updated ar {
  ar rc $(<) $(>)
}

TEMPORARY x.o y.o z.o ;
ar x.a : x.o y.o z.o ;
RmTemps x.a : x.o y.o z.o ;

In this case, rebuilding x.a because of one changed source,
should not force the other object files to be created.

Are you saying that TEMPORARY isn't used anywhere else ? What about the case of a library metatarget built from sources, where object files are marked as TEMPORARY ?
In my original (faber) case, I build a "binary" B from a source B.c and a library L, with a temporary object file B.o being removed after a successful build. If I then touch the library sources L.c, the library is rebuilt, and the binary relinked, but its object file B.o not being recompiled (this is the symptom I narrowed down to the above minimal test case), this fails.
Are you saying I shouldn't be using the TEMPORARY flag for B.o (et al.) ? How is b2 solving this case ?

Thanks,

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: bjam handling of temps

Boost - Build mailing list
AMDG

On 11/03/2017 11:00 AM, Stefan Seefeld via Boost-build wrote:

> On 03.11.2017 12:43, Steven Watanabe via Boost-build wrote:
>> <snip>
>>    Rebuilding when a temporary is present seems to be
>> deliberate, though I'm not quite clear on the logic
>> behind this.
>> #define T_FATE_SPOIL          4       /* >= SPOIL rebuilds parents */
>> #define T_FATE_ISTMP          4       /* unneeded temp target oddly
>> present */
>
> Yeah, I have wondered about that, too. I think this is manifest in code as
> https://github.com/boostorg/build/blob/develop/src/engine/make.c#L621-L624,
> which I have tentatively commented out, as I don't follow the rationale of
> existing temps being considered tainted. (Quite the opposite, actually: if I
> keep a temp around, perhaps I'm in the middle of a debug session where I do want
> to inject code into the otherwise automated process. So I'd rather treat
> existing "temps" as ordinary (non-temp) targets.)

  That shouldn't be an issue.  An existing temporary file
isn't overwritten.  Only its parents are forcibly updated.
Anyway, I didn't write this and I didn't even know about this
behavior until you pointed it out.  You're guess is as
good as mine, here.  Maybe the purpose of this is just
to make sure that the action that deletes the temp file
is run?

>> Second, TEMPORARY is really designed to be used with the
>> `updated` actions flag:
>>
>> actions piecemeal updated ar {
>>    ar rc $(<) $(>)
>> }
>>
>> TEMPORARY x.o y.o z.o ;
>> ar x.a : x.o y.o z.o ;
>> RmTemps x.a : x.o y.o z.o ;
>>
>> In this case, rebuilding x.a because of one changed source,
>> should not force the other object files to be created.
>
> Are you saying that TEMPORARY isn't used anywhere else ?

  It's sometimes also used in a linear dependency chain
where the only way for the parent to be out-of-date
is because of the TEMPORARY target.

> What about the case of
> a library metatarget built from sources, where object files are marked as
> TEMPORARY ?
> In my original (faber) case, I build a "binary" B from a source B.c and a
> library L, with a temporary object file B.o being removed after a successful
> build. If I then touch the library sources L.c, the library is rebuilt, and the
> binary relinked, but its object file B.o not being recompiled (this is the
> symptom I narrowed down to the above minimal test case), this fails.
> Are you saying I shouldn't be using the TEMPORARY flag for B.o (et al.) ? How is
> b2 solving this case ?
>

  In general, you probably shouldn't be using TEMPORARY in cases
where updating the parent requires the TEMPORARY to be created,
as this tends to cause unnecessary rebuilding.  If you really
need it, the problem can be worked around using REBUILDS B : B.o ;

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: bjam handling of temps

Boost - Build mailing list
On 03.11.2017 13:44, Steven Watanabe via Boost-build wrote:
AMDG

On 11/03/2017 11:00 AM, Stefan Seefeld via Boost-build wrote:

Are you saying I shouldn't be using the TEMPORARY flag for B.o (et al.) ? How is 
b2 solving this case ?

  In general, you probably shouldn't be using TEMPORARY in cases
where updating the parent requires the TEMPORARY to be created,
as this tends to cause unnecessary rebuilding.
Yeah, it's definitely a tradeoff.
Hmm, bummer, I really misunderstood what TEMPORARY was for, then (and have invested quite a bit of time to get it working for my use-cases).

  If you really
need it, the problem can be worked around using REBUILDS B : B.o ;

I'm not sure this has the same effect. Consider my case of binary B being built from source S via object file O, with O being flagged as TEMPORARY.
As far as B's fate is concerned, that can be computed purely by looking at S, unless O exists, in which case it depends on that, too. But whenever B needs to be updated, O should be updated first, unless it already is up-to-date.

As far as I can tell, "REBUILDS" is used to force-update one target whenever another one is updated. It does not (as far as I can tell) enforce an order, such that B.o would be made before B is attempted to be updated, or does it ?

Thanks,

Stefan
-- 

      ...ich hab' noch einen Koffer in Berlin...
    

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build
Reply | Threaded
Open this post in threaded view
|

Re: bjam handling of temps

Boost - Build mailing list
AMDG

On 11/03/2017 12:14 PM, Stefan Seefeld via Boost-build wrote:

> On 03.11.2017 13:44, Steven Watanabe via Boost-build wrote:
>>    If you really
>> need it, the problem can be worked around using REBUILDS B : B.o ;
>
> I'm not sure this has the same effect. Consider my case of binary B being built
> from source S via object file O, with O being flagged as TEMPORARY.
> As far as B's fate is concerned, that can be computed purely by looking at S,
> unless O exists, in which case it depends on that, too. But whenever B needs to
> be updated, O should be updated first, unless it already is up-to-date.
>
> As far as I can tell, "REBUILDS" is used to force-update one target whenever
> another one is updated. It does not (as far as I can tell) enforce an order,
> such that B.o would be made before B is attempted to be updated, or does it ?
>

  The build order is handled by the the dependency of B on O.
In fact, the lack of build order is the reason for using REBUILDS
in the first place.  B should force rebuild O, but that doesn't
mean that B needs to be updated before O (quite the contrary).

I think the three rules together should give the right result:
- DEPENDS causes B to be updated after O and causes
  B and O to update if S is modified.
- TEMPORARY causes O to use the timestamp of B and does not
  rebuild O when O is missing.
- REBUILDS makes sure that O is created when B is
  updated for any reason.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: https://lists.boost.org/mailman/listinfo.cgi/boost-build