Best Practices for Surviving the Boost Test Gauntlet?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
Beast won the first battle, but it looks like it is losing the war:

<http://www.boost.org/development/tests/develop/developer/beast.html>

I've never heard of many of these platforms let alone know how to go
about fixing the compile, link, and runtime errors. It seems like
deallocate() is not part of some standard library's allocator_traits?
Link error on operator delete from Boost.System? A lot of little
thing.

Is there a document or resource which explains the various
peculiarities and idiosyncrasies of these platforms and toolchains
that I might have a means of fixing them without actually installing
them locally (which is unlikely)?

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit :

> Beast won the first battle, but it looks like it is losing the war:
>
> <http://www.boost.org/development/tests/develop/developer/beast.html>
>
> I've never heard of many of these platforms let alone know how to go
> about fixing the compile, link, and runtime errors. It seems like
> deallocate() is not part of some standard library's allocator_traits?
> Link error on operator delete from Boost.System? A lot of little
> thing.
>
> Is there a document or resource which explains the various
> peculiarities and idiosyncrasies of these platforms and toolchains
> that I might have a means of fixing them without actually installing
> them locally (which is unlikely)?
>
> Thanks

It seems that you redeveloped a testing framework, and this is causing
some of the issues we can read there.

I do not know how important is this internal framework to you, but you
may go for another test framework and focus on the added value of your
library instead.

I am obviously biased toward boost.test :)

Best,
Raffi



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On Wed, Aug 9, 2017 at 2:15 PM, Raffi Enficiaud via Boost
<[hidden email]> wrote:
> Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit :
> It seems that you redeveloped a testing framework, and this is causing some
> of the issues we can read there.

Is there a link to the evidence of this so that I can come up with a fix?

> I do not know how important is this internal framework to you, but you may
> go for another test framework and focus on the added value of your library
> instead.

Changing test frameworks means refactoring all the tests - I'm sure
you can understand why I might be reluctant to do that right now.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On 8/9/2017 5:34 PM, Vinnie Falco via Boost wrote:

> On Wed, Aug 9, 2017 at 2:15 PM, Raffi Enficiaud via Boost
> <[hidden email]> wrote:
>> Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit :
>> It seems that you redeveloped a testing framework, and this is causing some
>> of the issues we can read there.
>
> Is there a link to the evidence of this so that I can come up with a fix?
>
>> I do not know how important is this internal framework to you, but you may
>> go for another test framework and focus on the added value of your library
>> instead.
>
> Changing test frameworks means refactoring all the tests - I'm sure
> you can understand why I might be reluctant to do that right now.

It is understandable but Boost comes with two testing frameworks
built-in. There is Boost Test, which Raffi Enficiaud mentions and is
pretty full-featured, and there is also Boost's lightweight_test.hpp,
which I have found adequate for testing vmd and tti. It is always easier
using what is already available, if it has the testing facilities you
need, than to create your own.

>
> Thanks


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Le 09.08.17 à 23:34, Vinnie Falco via Boost a écrit :
> On Wed, Aug 9, 2017 at 2:15 PM, Raffi Enficiaud via Boost
> <[hidden email]> wrote:
>> Le 08.08.17 à 16:12, Vinnie Falco via Boost a écrit :
>> It seems that you redeveloped a testing framework, and this is causing some
>> of the issues we can read there.
>
> Is there a link to the evidence of this so that I can come up with a fix?

I was randomly clicking on a link, like this one:

http://www.boost.org/development/tests/develop/developer/output/teeks99-09-m-win2012R2-32on64-boost-bin-v2-libs-beast-test-beast-core-buffer_cat-test-msvc-12-0-dbg-lnk-sttc-thrd-mlt.html

Now, I do not know if this is relevant. For preventing running the tests
on unsupported compilers, you may use b2 features, which will clear out
some of the columns:

http://www.boost.org/build/doc/html/bbv2/overview/targets.html#bbv2.overview.targets.requirements

example in project:
https://github.com/boostorg/test/blob/develop/test/Jamfile.v2#L30

>> I do not know how important is this internal framework to you, but you may
>> go for another test framework and focus on the added value of your library
>> instead.
>
> Changing test frameworks means refactoring all the tests - I'm sure
> you can understand why I might be reluctant to do that right now.

When is now and when is later?
If now means boost 1.65, then yes, this is not possible. If this is
1.66, I would say it is not such a big deal.
OTOH, if later is never ...

Your call anyway :)

Raffi



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On Wed, Aug 9, 2017 at 3:11 PM, Raffi Enficiaud via Boost
<[hidden email]> wrote:
> Le 09.08.17 à 23:34, Vinnie Falco via Boost a écrit :
> I was randomly clicking on a link, like this one:
>
> http://www.boost.org/development/tests/develop/developer/output/teeks99-09-m-win2012R2-32on64-boost-bin-v2-libs-beast-test-beast-core-buffer_cat-test-msvc-12-0-dbg-lnk-sttc-thrd-mlt.html

This has nothing to do with the test framework, it just happens that
suite.hpp is one of the first files included and therefore the first
to generate an error.

The problem in this case is that the toolchain is using Visual Studio
2012 which does not have sufficient support for C++11, and in
particular insufficient support for "noexcept" as indicated here:

<https://msdn.microsoft.com/en-us/library/hh567368.aspx>

> For preventing running the tests on unsupported compilers, you may
> use b2 features, which will clear out some of the columns:
>http://www.boost.org/build/doc/html/bbv2/overview/targets.html#bbv2.overview.targets.requirements

Thanks, this is helpful!

> When is now and when is later?
> If now means boost 1.65, then yes, this is not possible. If this is 1.66, I
> would say it is not such a big deal.

My schedule for 2017 is very tight. There is a backlog of Beast issues
that I want to address for its debut in 1.66.0 as some of them may
introduce API changes which I would rather do now instead of after
putting out the first non-beta version. Boost.Asio is going to be
updated at some random time in 2017 and I will have to quickly port
Beast for those changes. And Beast needs to have some time without any
changes prior to the freeze for 1.66.0, so that any lingering defects
can be flushed out.

There's a fair amount of test code and it makes heavy use of
coroutines, threads, and helper classes (Beast's unit test framework
is class-driven where each test is represented by a derived class). I
have in mind to try out the lightweight framework after 1.66.0, which
will be part of a larger push to bring the coverage as close to 100%
as possible but I don't have the resources for it this year. So I am
very much stuck with what I have. So far there is nothing in the build
output to suggest that the testing framework is at fault but I will
keep a close eye on it.

What I'm looking for is the secret document that I am SURE is stashed
away somewhere on a Google Drive account, or maybe a USB stick, which
explains all of the common errors that happen in the Boost test
environment and how they might be worked around. For example, that
early versions of the standard library do not have full support for
std::allocator_traits even though they claim to. I've since fixed that
issue here:

<https://github.com/boostorg/beast/pull/725/commits/017510215122c013579f6c74631468a92e7a4d6d>

But as you know the turnaround time on cycling a library through the
tests is quite long, so if I could get a copy of that document which
has solutions to common problems ahead of time, then I could save
several length iterations.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On 10/08/2017 10:25, Vinnie Falco wrote:
> The problem in this case is that the toolchain is using Visual Studio
> 2012 which does not have sufficient support for C++11, and in
> particular insufficient support for "noexcept" as indicated here:
>
> <https://msdn.microsoft.com/en-us/library/hh567368.aspx>

Are you using the BOOST_NOEXCEPT* macros?  They resolve to nothing (or
in some cases to throw()) on older compilers.

In general for most compatibility issues there is a macro somewhere,
either in Boost.Config or elsewhere.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On Wed, Aug 9, 2017 at 4:04 PM, Gavin Lambert via Boost
<[hidden email]> wrote:
> Are you using the BOOST_NOEXCEPT* macros?  They resolve to nothing (or in
> some cases to throw()) on older compilers.

No, because Beast requires C++11.

> In general for most compatibility issues there is a macro somewhere, either
> in Boost.Config or elsewhere.

I know, and its amazing the amount of work and dedication that some
authors go through to keep their libraries working with old compilers.
Unfortunately that level of support is not something I have resources
to maintain in Beast.

I can only support Beast on Visual Studio 2015 with Update 3 or later.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
> I can only support Beast on Visual Studio 2015 with Update 3 or later.

The way I handled this in AFIO v1 was to add C++ feature requirements in
Boost.Build sufficient to exclude any Visual Studio before some version X.

I also strongly recommend you add #error trapping of ancient compilers
in your config.hpp. Otherwise you'll be plagued in years to come by
users complaining about unobvious error messages when they do weird
things with some older version of your library.

I personally would even go so far as to #error out on any compiler which
is not MSVC, clang nor GCC, and versions before specific hardcoded
minimum versions.

Niall

--
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On Wed, Aug 9, 2017 at 4:13 PM, Niall Douglas via Boost
<[hidden email]> wrote:
> I also strongly recommend you add #error trapping of ancient compilers

Yes

<https://github.com/boostorg/beast/blob/1366302a29595b7300f8bd3481c459f8f4d0fca4/include/boost/beast/config.hpp#L27>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 10/08/2017 11:07, Vinnie Falco wrote:

> On Wed, Aug 9, 2017 at 4:04 PM, Gavin Lambert wrote:
>> Are you using the BOOST_NOEXCEPT* macros?  They resolve to nothing (or in
>> some cases to throw()) on older compilers.
>
> No, because Beast requires C++11.
>
>> In general for most compatibility issues there is a macro somewhere, either
>> in Boost.Config or elsewhere.
>
> I know, and its amazing the amount of work and dedication that some
> authors go through to keep their libraries working with old compilers.
> Unfortunately that level of support is not something I have resources
> to maintain in Beast.
>
> I can only support Beast on Visual Studio 2015 with Update 3 or later.

Then it sounds like you should be putting something in the Jamfile that
tells it not to attempt to build on older compilers that you intend to
be incompatible with.  (I don't know the specifics, though, but you
could perhaps look at some of the other existing C++11/14-only libraries
for inspiration.)

Also, FYI, VS 12.0 is actually Visual Studio 2013.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/8/17 7:12 AM, Vinnie Falco via Boost wrote:

> Beast won the first battle, but it looks like it is losing the war:
>
> <http://www.boost.org/development/tests/develop/developer/beast.html>
>
> I've never heard of many of these platforms let alone know how to go
> about fixing the compile, link, and runtime errors. It seems like
> deallocate() is not part of some standard library's allocator_traits?
> Link error on operator delete from Boost.System? A lot of little
> thing.
>
> Is there a document or resource which explains the various
> peculiarities and idiosyncrasies of these platforms and toolchains
> that I might have a means of fixing them without actually installing
> them locally (which is unlikely)?

I looked at your results, and they don't look half bad for a first attempt.

I can share you the benefit of years of experience dealing with this.
It's much better now that it used to be.

a) Make sure your code aims for standard C++.

b) Any failures are now due to compilers/libraries failing to meet the
standard.

c) Now study documentation on config.hpp.  It's very complete, readable
and not at all difficult to use.  It basically entails tweaking your
code as suggested by the documentation.  You'll find that a lot of
problems disappear after a relatively short session as many problems are
in a header show up in every test.

d) You can tweak your test/jamfile so that it skips tests which are not
applicable.  For example if your library doesn't have header X you can
arrange to skip those tests which require header X.  The information is
in the documentation for config.hpp.  Also the test/bjam file for the
serialization library does this and you see the syntax there.

e) If you follow the above to the end, you will find that your tests
will run on all versions of C++ since you've excluded those
tests/compiler combinations which cannot pass.

f) FWIW I use my own variation of lightweight test.  As long as your
test returns a 1 (failure) or 0 (success) I don't think it matters which
test framework you use.

g) When you get all done with the above, you'll have most of the tests
running on most compilers.  The failures are just that - things that
can't be made to pass without mucking up your library.  Just note that
it's the vendors problem and move on.

h) When all this is done, get down on your knees say a prayer of thanks
to the almighty that your library is header only.

Robert Ramey


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/9/2017 7:07 PM, Vinnie Falco via Boost wrote:

> On Wed, Aug 9, 2017 at 4:04 PM, Gavin Lambert via Boost
> <[hidden email]> wrote:
>> Are you using the BOOST_NOEXCEPT* macros?  They resolve to nothing (or in
>> some cases to throw()) on older compilers.
>
> No, because Beast requires C++11.
>
>> In general for most compatibility issues there is a macro somewhere, either
>> in Boost.Config or elsewhere.
>
> I know, and its amazing the amount of work and dedication that some
> authors go through to keep their libraries working with old compilers.
> Unfortunately that level of support is not something I have resources
> to maintain in Beast.

This is almost exclusively for libraries which have been around for a
long time. Boost fully accepts the fact that your library may need a
particular level of c++ compliance. The important thing is to document
the level of c++ compliance needed by your library. Once you have done
that then it is the end-user's problem if they are using a compiler that
does not meet the level of c++ that your library needs if they wish to
use your library.

>
> I can only support Beast on Visual Studio 2015 with Update 3 or later.
>
> Thanks


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On 10/08/2017 13:57, Edward Diener wrote:
> This is almost exclusively for libraries which have been around for a
> long time. Boost fully accepts the fact that your library may need a
> particular level of c++ compliance. The important thing is to document
> the level of c++ compliance needed by your library. Once you have done
> that then it is the end-user's problem if they are using a compiler that
> does not meet the level of c++ that your library needs if they wish to
> use your library.

While that's true, it's also polite to have your Jamfile configured to
skip compiling on the unsupported platforms, so that the user doesn't
have to explicitly opt out with --without-xxx.

Although it's not as big a deal for header-only libraries, since the
default build doesn't build the tests anyway, IIRC.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best Practices for Surviving the Boost Test Gauntlet?

Boost - Dev mailing list
On 8/10/2017 6:56 PM, Gavin Lambert via Boost wrote:

> On 10/08/2017 13:57, Edward Diener wrote:
>> This is almost exclusively for libraries which have been around for a
>> long time. Boost fully accepts the fact that your library may need a
>> particular level of c++ compliance. The important thing is to document
>> the level of c++ compliance needed by your library. Once you have done
>> that then it is the end-user's problem if they are using a compiler
>> that does not meet the level of c++ that your library needs if they
>> wish to use your library.
>
> While that's true, it's also polite to have your Jamfile configured to
> skip compiling on the unsupported platforms, so that the user doesn't
> have to explicitly opt out with --without-xxx.

I agree with you.

>
> Although it's not as big a deal for header-only libraries, since the
> default build doesn't build the tests anyway, IIRC.

Actually you can do this for tests also, but I agree that it is more
important to do this for library builds.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Loading...