A modest proposal

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

A modest proposal

Robert Ramey
Some change in spirit/class or boost.move has rippled through to generate a huge number of test failures in the serialization library on the develop branch.  OK that is what the develop branch is for.  But it would be much better if the test script were enhanced to use the following (pseudo code) procedure

for each library
    set branch to master

for each library x
    set branch to develop
    run tests on library x
    set branch back to master


Then and error/problem would show up in only on the pages for tests of that library x which contains the the error a not on all the libraries which use  library x.  This would provide a number of benefits:

a) It would save all the authors whose libraries which use library x from the trouble of chasing down the reasons why a bunch of tests in their own library have spontaneously started to fail.

b) It would permit author of library x to make an error in the development branch without having to create problems for everyone else

The current system does not scale to more libraries.

Robert Ramey

PS - On my own  machine, I test the serialization library develop branch against the master branch of all libraries.  This simplifies my life and minimizes the possibility that merging my changes into the master will create problems.
Reply | Threaded
Open this post in threaded view
|

Re: A modest proposal

Rene Rivera-2
On Fri, Jan 2, 2015 at 11:36 PM, Robert Ramey <[hidden email]> wrote:
Some change in spirit/class or boost.move has rippled through to generate a
huge number of test failures in the serialization library on the develop
branch.  OK that is what the develop branch is for.  But it would be much
better if the test script were enhanced to use the following (pseudo code)
procedure

for each library
    set branch to master

for each library x
    set branch to develop
    run tests on library x
    set branch back to master

I know, you know, that doing this is something I also want. But.. It's actually hard to accomplish at the moment, although not impossible. Switching branches for every library it's against how the build and test systems work. It would mean calling N different build invocations (N == libraries). And the test system would need to adjust to parsing all the individual invocations.


--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

_______________________________________________
Boost-Testing mailing list
[hidden email]
http://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: A modest proposal

Robert Ramey
Rene Rivera-2 wrote
On Fri, Jan 2, 2015 at 11:36 PM, Robert Ramey <[hidden email]> wrote:

> Some change in spirit/class or boost.move has rippled through to generate a
> huge number of test failures in the serialization library on the develop
> branch.  OK that is what the develop branch is for.  But it would be much
> better if the test script were enhanced to use the following (pseudo code)
> procedure
>
> for each library
>     set branch to master
>
> for each library x
>     set branch to develop
>     run tests on library x
>     set branch back to master
>

I know, you know, that doing this is something I also want. But.. It's
actually hard to accomplish at the moment, although not impossible.
Switching branches for every library it's against how the build and test
systems work. It would mean calling N different build invocations (N ==
libraries).
But wouldn't each build/test be ~1/N the size of the current one?

When I test on my own machine, I run boost build from inside my libs/serialization/test directory
and it builds just what is needed for the serialization library tests.  In particular, i builds system and filesystem libraries in order to run the tests.  Since the file system and system libraries are checked out
on the master, that's where they get built.  Which seems fine to me.  I another library were tested later, the most recent builds would be re-used.  So it seems to me that the total amount of compiling, linking and testing going on would be the same as the current system.

In other words, isn't testing each library individually with it's own instance of bjam more or less the same as running bjam from the top of the tree? So would the times be comparable?

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: A modest proposal

Rene Rivera-2
On Sat, Jan 3, 2015 at 10:09 PM, Robert Ramey <[hidden email]> wrote:
Rene Rivera-2 wrote
> On Fri, Jan 2, 2015 at 11:36 PM, Robert Ramey &lt;
>> branch.  OK that is what the develop branch is for.  But it would be much
>> better if the test script were enhanced to use the following (pseudo
>> code)
>> procedure
>>
>> for each library
>>     set branch to master
>>
>> for each library x
>>     set branch to develop
>>     run tests on library x
>>     set branch back to master
>>
>
> I know, you know, that doing this is something I also want. But.. It's
> actually hard to accomplish at the moment, although not impossible.
> Switching branches for every library it's against how the build and test
> systems work. It would mean calling N different build invocations (N ==
> libraries).

But wouldn't each build/test be ~1/N the size of the current one?

As far as disk space is concerned, yes. But not as far as anything else. Consider all the steps involved to do that for each library:

1. Delete [root]/boost tree.
2. Switch target library from master to develop.
3. Run in [root]/status "b2 --limit-tests=<libname> --toolsets=a,b,c". And collecting the b2 output to an overall output log.
4. Switch target library back to master.

(1) Is needed to correctly create new headers when we switch the library from master to develop.

(1) Also means that no existing binaries are reused since time stamps are changed. So we incur the cost of rebuilding any dependencies for each library. Although I'm not sure what happens with timestamps and b2 when symlinks/hardlinks are in use.

(2) Means that we have to keep around both master and develop branches at the testers. Currently we only keep one branch (and from the first obtained revision to current). It also means more git invocations which increases time overhead.

And at the end we run process_jam_log. Which has to change to deal with the fact that it's going to have N concatenate invocations of a b2 output log to generate the results XML file. And getting that working correctly means a lot of testing and iteration, and a lot of waiting around. I.e. it's slow, tedious, and detailed work.

Of course there will be changes to regression.py. Which will also be slow, tedious, detailed work. I'm only saying it's going to be that way because that's how it's been in the past for the SVN->GIT code changes. And the rather simpler, boostorg/boost repo to boostorg/regression repo move.

When I test on my own machine, I run boost build from inside my
libs/serialization/test directory
and it builds just what is needed for the serialization library tests.  In
particular, i builds system and filesystem libraries in order to run the
tests.  Since the file system and system libraries are checked out
on the master, that's where they get built.  Which seems fine to me.  I
another library were tested later, the most recent builds would be re-used.
So it seems to me that the total amount of compiling, linking and testing
going on would be the same as the current system.

In other words, isn't testing each library individually with it's own
instance of bjam more or less the same as running bjam from the top of the
tree? So would the times be comparable?

No, it's not the same. But yes the times would be comparable from an order of complexity point of view. I.e. it's still O(K+N).. just a larger K.

And.. This of course doesn't take into consideration the problems with inter-library master vs. develop dependencies that may exists and may have to be dealt with. But that is something I would only worry about *after* having master/develop mixed testing implemented.


--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

_______________________________________________
Boost-Testing mailing list
[hidden email]
http://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

difference between master and develop branches

Aparna Kumta
Could someone let me know the difference between the master and develop
branches?


Thanks,

Aparna



_______________________________________________
Boost-Testing mailing list
[hidden email]
http://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: difference between master and develop branches

Beman Dawes
On Tue, Jan 6, 2015 at 12:52 PM, Aparna Kumta <[hidden email]> wrote:
> Could someone let me know the difference between the master and develop
> branches?

Roughly, for both the main boost super-project and the individual
libraries, master is the release-ready branch releases are built from.
Develop is the main development branch.

They are our adaptations of Vincent Driessen's GitFlow branching model.

See https://svn.boost.org/trac/boost/wiki/StartModWorkflow

and http://nvie.com/posts/a-successful-git-branching-model/

--Beman
_______________________________________________
Boost-Testing mailing list
[hidden email]
http://lists.boost.org/mailman/listinfo.cgi/boost-testing
Reply | Threaded
Open this post in threaded view
|

Re: A modest proposal

Robert Ramey
In reply to this post by Rene Rivera-2
Rene Rivera-2 wrote
On Sat, Jan 3, 2015 at 10:09 PM, Robert Ramey <[hidden email]> wrote:

> But wouldn't each build/test be ~1/N the size of the current one?
>

As far as disk space is concerned, yes. But not as far as anything else.
Consider all the steps involved to do that for each library:

1. Delete [root]/boost tree.
2. Switch target library from master to develop.
3. Run in [root]/status "b2 --limit-tests=<libname> --toolsets=a,b,c". And
collecting the b2 output to an overall output log.
4. Switch target library back to master.

(1) Is needed to correctly create new headers when we switch the library
from master to develop.
Library X - the library under test

Hmmm - but don't we need only new headers for the library x? Can't all
the others stay the same?.  Perhaps we need to run b2 headers on
library X - but not the rest.
(1) Also means that no existing binaries are reused since time stamps are
changed. So we incur the cost of rebuilding any dependencies for each
library. Although I'm not sure what happens with timestamps and b2 when
symlinks/hardlinks are in use.
I don't see a problem here.  Only library X has changed from the master. Any
existing binaries on the master branch can (and should!) be used.
(2) Means that we have to keep around both master and develop branches at
the testers. Currently we only keep one branch (and from the first obtained
revision to current). It also means more git invocations which increases
time overhead.
Hmm - I've always thought that my local GIT repo has both the master and
develop branches held locally. So when I switch - which I do all the time - it's
quite fast operation which doesn't go to any server.  So I'm really missing
something here.
And at the end we run process_jam_log. Which has to change to deal with the
fact that it's going to have N concatenate invocations of a b2 output log
to generate the results XML file. And getting that working correctly means
a lot of testing and iteration, and a lot of waiting around. I.e. it's
slow, tedious, and detailed work.
(3) rather than do this I run b2 in the local test directory.  So it produces
a local b2.log.  The time this takes is only dependent on that number
of test and the time time it takes to build the one library under test.
(library dependencies in the master might also take time - but that only
needs to be done when the master changes - which is infrequent)
Of course there will be changes to regression.py. Which will also be slow,
tedious, detailed work. I'm only saying it's going to be that way because
that's how it's been in the past for the SVN->GIT code changes. And the
rather simpler, boostorg/boost repo to boostorg/regression repo move.
LOL - I'm very, very sympathetic here.  I realize that setting up and
maintaining the regression testing is a huge job.   In no way would I ask
you to consider it if I thought it was going to be another death march.

But when I do this by hand on my own system - it works smooth as
silk.  In fact, I'm thinking that doing N tests runs on 1/N the system
will workout even faster than the current way.
> In other words, isn't testing each library individually with it's own
> instance of bjam more or less the same as running bjam from the top of the
> tree? So would the times be comparable?

No, it's not the same. But yes the times would be comparable from an order
of complexity point of view. I.e. it's still O(K+N).. just a larger K.

And.. This of course doesn't take into consideration the problems with
inter-library master vs. develop dependencies that may exists and may have
to be dealt with. But that is something I would only worry about *after*
having master/develop mixed testing implemented.
Actually, this is the potentially the most problematic - but very hard to predict.

Regardless of whether or not you decide to spend some time experimenting
with this, I want to thank you for creating and maintaining the Boost Regression
testing infrastructure over these many years.  To me, it is one of the
key reasons for the success of Boost.  I don't believe that the importance
of this effort and your dedication to keeping running and reliable can be
understated.

Robert Ramey