[test] BOOST_TEST - universal testing tool

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
Hi,

Based on a discussion on mini-review thread I make some changes to the
universal testing tool interface and now it look as follows (similar
interfaces on WARN and REQUIRE levels):

BOOST_CHECK_ASSERTION(A) - new expression template based testing tool

BOOST_TEST(A,M) := BOOST_CHECK_ASSERTION(A) if M is not supplied or
BOOST_CHECK_MESSAGE(A,M) if M is supplied.

BOOST_TEST (and it's siblings BOOST_TEST_WARN and BOOST_TEST_REQUIRE) is
intended to become a primary (only?) testing tools from now on. It replaces
and deprecates right away following tools:

BOOST_CHECK
BOOST_CHECK_MESSAGE
BOOST_CHECK_EQUAL,
BOOST_CHECK_NE,
BOOST_CHECK_LT,
BOOST_CHECK_LE,
BOOST_CHECK_GT,
BOOST_CHECK_GE

There are still some tools which are not directly replaceable. I have some
ideas how we can get there, but your input is appreciated:

1. BOOST_CHECK_THROW, BOOST_CHECK_EXCEPTION, BOOST_CHECK_NO_THROW

We can probably leave these as is. Alternatively we'll need to postpone
expression evaluation with separate BOOST_TEST_DELAYED tool. We also need
to introduce special tags unit_test::throws, unit_test::no_throw:

BOOST_TEST_DELAYED(foo(), unit_test::throws<my_exception>() );
BOOST_TEST_DELAYED(foo(), unit_test::throws<my_exception>(predicate) );
BOOST_TEST_DELAYED(foo(), unit_test::throws( my_exception(...) ) );
BOOST_TEST_DELAYED(foo(), unit_test::no_throw() );

Later form will match to exact exception value.

We might be able to fit into BOOST_TEST interface, but that would require
somehow we need to recognize executable entity as special case:

BOOST_TEST( foo, unit_test::throws<my_exception>() );

Not quite sure if this is possible.

2. BOOST_CHECK_CLOSE, BOOST_CHECK_CLOSE_FRACTION, BOOST_CHECK_SMALL

We can deal with these by introducing couple special tags:

BOOST_TEST(a == 1.5, unit_test::tolerance(1e-6) );
BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
BOOST_TEST(a, unit_test::small(1e-3) );

On a plus side we'll be able to use these in new previously unavailable way:
BOOST_TEST(a > 1.1, unit_test::tolerance(1e-6) );

This would test that value of a either > 1.1 or within a fraction tolerance
of it.

3. BOOST_CHECK_PREDICATE

Similar to (1) we need to delay predicate execution. Might be easier to leave
it be.

4. BOOST_CHECK_EQUAL_COLLECTIONS

I think using overloading of equality comparisons we should be able to support
this directly:

BOOST_TEST( expr == std::vector<int>{1,2,3,4});

5. BOOST_CHECK_BITWISE_EQUAL

We can introduce yet another tag to deal with this:

BOOST_TEST(a == 0xA1FB, unit_test::bitwise );

As usual any comments are welcome.

Gennadiy



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Alexander Lamaison
Gennadiy Rozental <[hidden email]> writes:

> Hi,
>
> Based on a discussion on mini-review thread I make some changes to the
> universal testing tool interface and now it look as follows (similar
> interfaces on WARN and REQUIRE levels):
>
> BOOST_CHECK_ASSERTION(A) - new expression template based testing tool
>
> BOOST_TEST(A,M) := BOOST_CHECK_ASSERTION(A) if M is not supplied or
> BOOST_CHECK_MESSAGE(A,M) if M is supplied.
>
> BOOST_TEST (and it's siblings BOOST_TEST_WARN and BOOST_TEST_REQUIRE) is
> intended to become a primary (only?) testing tools from now on. It replaces
> and deprecates right away following tools:
>
> BOOST_CHECK
> BOOST_CHECK_MESSAGE
> BOOST_CHECK_EQUAL,
> BOOST_CHECK_NE,
> BOOST_CHECK_LT,
> BOOST_CHECK_LE,
> BOOST_CHECK_GT,
> BOOST_CHECK_GE

This should simplify test code a lot.  But, as it's now the default
tool, has any progress been made on using it with C++03 compilers?

Alex


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
Alexander Lamaison <awl03 <at> doc.ic.ac.uk> writes:
> This should simplify test code a lot.  But, as it's now the default
> tool, has any progress been made on using it with C++03 compilers?

It actually works for quite some number of cases. You can see in test_tools_test
unit test in here:

http://beta.boost.org/development/tests/trunk/developer/test.html.

There are some use cases which are not supported (you can see them ifdefed out).

Gennadiy


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Vicente Botet
Le 05/11/12 08:16, Gennadiy Rozental a écrit :

> Alexander Lamaison <awl03 <at> doc.ic.ac.uk> writes:
>> This should simplify test code a lot.  But, as it's now the default
>> tool, has any progress been made on using it with C++03 compilers?
> It actually works for quite some number of cases. You can see in test_tools_test
> unit test in here:
>
> http://beta.boost.org/development/tests/trunk/developer/test.html.
>
> There are some use cases which are not supported (you can see them ifdefed out).
>
> Gennadiy
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>
Hi,

there is a regression with some compilers ( intel- darwin- 11.1, clang-
darwin- trunk, clang- darwin- trunk c++11, gcc- 4.7.1_0x), all of them
having the same symptoms

Run [2012-11-05 13:48:26 UTC]: fail

terminate called after throwing an instance of '__gnu_cxx::recursive_init_error'
   what():  std::exception

All the test using Boost.Test are failing. Please, could you take care of this issue?

Best,
Vicente



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Andrzej Krzemienski
In reply to this post by Gennadiy Rozental-2
Hi Gennadiy,
The new interface looks very attractive. Below I enclose two suggestions
that are orthogonal (I think) to your changes, but I thought it was a good
time to suggest them.

2012/11/4 Gennadiy Rozental <[hidden email]>

>
> 2. BOOST_CHECK_CLOSE, BOOST_CHECK_CLOSE_FRACTION, BOOST_CHECK_SMALL
>
> We can deal with these by introducing couple special tags:
>
> BOOST_TEST(a == 1.5, unit_test::tolerance(1e-6) );
> BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
> BOOST_TEST(a, unit_test::small(1e-3) );
>
> On a plus side we'll be able to use these in new previously unavailable
> way:
> BOOST_TEST(a > 1.1, unit_test::tolerance(1e-6) );
>
> This would test that value of a either > 1.1 or within a fraction tolerance
> of it.
>

In my job, we are using Boost.Test in a commercial app. We are doing (and
testing) some basic computations on physical quantities. It turned out that
for around 95% of the checks on floating-point values we need the very same
(percent) tolerance. And it is inconvenient to have to type the same
constant time and again. It would be nice if there was a way to define (as
an option) a "default tolerance". With the solution you propose above,
having to type this tag "unit_test::percent_tolerance" would incur even
more "syntactic noise" in cases where the tolerance is "obvious" to the
reader. Here, the concept of default tolerance would be even more useful.
Although I am not sure if this is doable. But perhaps it is: if I am able
to configure the framework so that when it sees a check on floating-point
numbers it assumes we check with tolerance: default one.

What do you think?

4. BOOST_CHECK_EQUAL_COLLECTIONS
>
> I think using overloading of equality comparisons we should be able to
> support
> this directly:
>
> BOOST_TEST( expr == std::vector<int>{1,2,3,4});
>

Will that work? I.e., in case of any two ranges, not necessarily
containers? If there is some even more clever (than my idea) suggestion to
do it it is even better. What I was missing from Boost.Test for quite a
while was some additional check that would accept as input two ranges
rather than four iterators. "Range" R in this sense is any object for which
expressions begin(R) and end(R) are valid: a container, an array, a pair of
iterators. I often wanted to compare a vector returned by value from one
expression with an array of values, and could not do it without introducing
unnecessary additional variables:

I had to write:

  int reference[] = {1, 2, 4};
  auto&& answer = FindAnswer();
  BOOST_CHECK_EQUAL_COLLECTIONS(begin(reference), end(reference),
begin(answer), end(answer));

I would like to write:

  int reference[] = {1, 2, 4};
  BOOST_CHECK_EQUAL_COLLECTIONS(reference, FindAnswer());

Or (if this is doable):

  BOOST_CHECK_EQUAL_COLLECTIONS(FindAnswer(), {1, 2, 4});

Regards,
&rzej

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Vicente Botet
Andrzej Krzemienski wrote
Hi Gennadiy,
The new interface looks very attractive. Below I enclose two suggestions
that are orthogonal (I think) to your changes, but I thought it was a good
time to suggest them.

2012/11/4 Gennadiy Rozental <[hidden email]>

>
> 2. BOOST_CHECK_CLOSE, BOOST_CHECK_CLOSE_FRACTION, BOOST_CHECK_SMALL
>
> We can deal with these by introducing couple special tags:
>
> BOOST_TEST(a == 1.5, unit_test::tolerance(1e-6) );
> BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
> BOOST_TEST(a, unit_test::small(1e-3) );
>
> On a plus side we'll be able to use these in new previously unavailable
> way:
> BOOST_TEST(a > 1.1, unit_test::tolerance(1e-6) );
>
> This would test that value of a either > 1.1 or within a fraction tolerance
> of it.
>

In my job, we are using Boost.Test in a commercial app. We are doing (and
testing) some basic computations on physical quantities. It turned out that
for around 95% of the checks on floating-point values we need the very same
(percent) tolerance. And it is inconvenient to have to type the same
constant time and again. It would be nice if there was a way to define (as
an option) a "default tolerance". With the solution you propose above,
having to type this tag "unit_test::percent_tolerance" would incur even
more "syntactic noise" in cases where the tolerance is "obvious" to the
reader. Here, the concept of default tolerance would be even more useful.
Although I am not sure if this is doable. But perhaps it is: if I am able
to configure the framework so that when it sees a check on floating-point
numbers it assumes we check with tolerance: default one.

What do you think?
it seems easy for you to define your specific macro

#define MY_FP_TEST(E) BOOST_TEST(E, my_tolerance );

and change the my_tolerance as you want.

Vicente
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Rhys Ulerich-2
In reply to this post by Andrzej Krzemienski
>> BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );

> It turned out that
> for around 95% of the checks on floating-point values we need the very same
> (percent) tolerance. And it is inconvenient to have to type the same
> constant time and again. It would be nice if there was a way to define (as
> an option) a "default tolerance".

Could you simply macro-up the default tolerance?

#define MY_TEST_WITH_DEFAULT_TOLERANCE(expr) \
    BOOST_TEST(expr, unit_test::percent_tolerance(whatever))

- Rhys

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Rhys Ulerich-2
In reply to this post by Gennadiy Rozental-2
> BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
>
> ...
>
> BOOST_TEST( expr == std::vector<int>{1,2,3,4});

Can combine these constructs to perform floating point tests with
tolerance on a collection?

That is,
BOOST_TEST( expr == std::vector<double>{1,2,3,4},
unit_test::percent_tolerance(1e-6));

I ask because, for my numerics codes, I've kludged together the Test
floating point tolerance algorithms to produce something like a
BOOST_CHECK_CLOSE_COLLECTIONS.  Direct support within Test would be
much appreciated.

Thanks,
Rhys

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
Rhys Ulerich <rhys.ulerich <at> gmail.com> writes:

>
> > BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
> >
> > ...
> >
> > BOOST_TEST( expr == std::vector<int>{1,2,3,4});
>
> Can combine these constructs to perform floating point tests with
> tolerance on a collection?
>
> That is,
> BOOST_TEST( expr == std::vector<double>{1,2,3,4},
> unit_test::percent_tolerance(1e-6));

Hmmm... Good point. I'll see if this is possible.

As for "default tolerance", I am afraid it might be a bit surprising if plain  
comparison of doubles suddenly will start to use tolerance without user say so.
Unless it is set to zero by default in which case we are back to usual semantic.
Once you set it to something non zero we can probably switch to "close"
comparison.

What about the scope of the tolerance? Should it be global or per test case?

Gennadiy



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Andrzej Krzemienski
2012/11/6 Gennadiy Rozenal <[hidden email]>

> Rhys Ulerich <rhys.ulerich <at> gmail.com> writes:
>
> >
> > > BOOST_TEST(a == 0.003, unit_test::percent_tolerance(1e-6) );
> > >
> > > ...
> > >
> > > BOOST_TEST( expr == std::vector<int>{1,2,3,4});
> >
> > Can combine these constructs to perform floating point tests with
> > tolerance on a collection?
> >
> > That is,
> > BOOST_TEST( expr == std::vector<double>{1,2,3,4},
> > unit_test::percent_tolerance(1e-6));
>
> Hmmm... Good point. I'll see if this is possible.
>
> As for "default tolerance", I am afraid it might be a bit surprising if
> plain
> comparison of doubles suddenly will start to use tolerance without user
> say so.
> Unless it is set to zero by default in which case we are back to usual
> semantic.
> Once you set it to something non zero we can probably switch to "close"
> comparison.
>
> What about the scope of the tolerance? Should it be global or per test
> case?
>

>From the perspective of my usage, I guess it could be set per suite, if
this is possible. Per-test would require too much repetition, unless I
could define it in my fixture and have the test framework somehow read it
from the fixture.

Regards,
&rzej

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
In reply to this post by Vicente Botet
Vicente J. Botet Escriba <vicente.botet <at> wanadoo.fr> writes:

> All the test using Boost.Test are failing. Please,
> could you take care of this issue?

Should be fixed in trunk now.

Gennadiy


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
In reply to this post by Andrzej Krzemienski
Andrzej Krzemienski <akrzemi1 <at> gmail.com> writes:
> From the perspective of my usage, I guess it could be set per suite, if
> this is possible. Per-test would require too much repetition, unless I
> could define it in my fixture and have the test framework somehow read it
> from the fixture.

What about interface like this:

BOOST_TEST_DECORATORS(
+ decorator::tolerance(1e-6)
)
BOOST_AUTO_TEST_CASE( foo )
{
    BOOST_TEST( foo == 1.5 );
}

BOOST_TEST_DECORATORS(
+ decorator::percent_tolerance(1e-5)
)
BOOST_AUTO_TEST_SUITE(s1)

BOOST_AUTO_TEST_CASE( foo )
{
    BOOST_TEST( foo == 1.5 );
}

BOOST_AUTO_TEST_SUITE_END

Similar functionality can be implemented using fixture decorator, which already
exists.

Gennadiy


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Andrzej Krzemienski
2012/11/6 Gennadiy Rozental <[hidden email]>

> Andrzej Krzemienski <akrzemi1 <at> gmail.com> writes:
> > From the perspective of my usage, I guess it could be set per suite, if
> > this is possible. Per-test would require too much repetition, unless I
> > could define it in my fixture and have the test framework somehow read it
> > from the fixture.
>
> What about interface like this:
>
> BOOST_TEST_DECORATORS(
> + decorator::tolerance(1e-6)
> )
> BOOST_AUTO_TEST_CASE( foo )
> {
>     BOOST_TEST( foo == 1.5 );
> }
>
> BOOST_TEST_DECORATORS(
> + decorator::percent_tolerance(1e-5)
> )
> BOOST_AUTO_TEST_SUITE(s1)
>
> BOOST_AUTO_TEST_CASE( foo )
> {
>     BOOST_TEST( foo == 1.5 );
> }
>
> BOOST_AUTO_TEST_SUITE_END
>
> Similar functionality can be implemented using fixture decorator, which
> already
> exists.
>

This is nice. How does this work if I spread my suite across multiple
source files? do I need to repeat the decorators in each file?

Regards,
&rzej

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Rhys Ulerich-2
In reply to this post by Gennadiy Rozental-2
> What about interface like this:
>
> BOOST_TEST_DECORATORS(
> + decorator::tolerance(1e-6)
> )
> BOOST_AUTO_TEST_CASE( foo )
> {
>     BOOST_TEST( foo == 1.5 );
> }

This just feels like a fixture-- any reason a regular ol' per-suite or
per-test fixture providing a "default_tolerance" member wouldn't
suffice?

- Rhys

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
Rhys Ulerich <rhys.ulerich <at> gmail.com> writes:

>
> > What about interface like this:
> >
> > BOOST_TEST_DECORATORS(
> > + decorator::tolerance(1e-6)
> > )
> > BOOST_AUTO_TEST_CASE( foo )
> > {
> >     BOOST_TEST( foo == 1.5 );
> > }
>
> This just feels like a fixture-- any reason a regular ol' per-suite or
> per-test fixture providing a "default_tolerance" member wouldn't
> suffice?

It is (and it can be implemented like one), but if this is common need why not
make cleaner interface for it?

Gennadiy



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Gennadiy Rozental-2
In reply to this post by Andrzej Krzemienski
Andrzej Krzemienski <akrzemi1 <at> gmail.com> writes:

> This is nice. How does this work if I spread my suite across multiple
> source files? do I need to repeat the decorators in each file?

No. Decorators will stick to the test suite.

Gennadiy



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [test] BOOST_TEST - universal testing tool

Rhys Ulerich-2
In reply to this post by Gennadiy Rozental-2
>> This just feels like a fixture-- any reason a regular ol' per-suite or
>> per-test fixture providing a "default_tolerance" member wouldn't
>> suffice?
>
> It is (and it can be implemented like one), but if this is common need why not
> make cleaner interface for it?

Because expanding the Boost.Test API seems to cause users to ask for
very, very specific features rather than to realize that the existing
functionality supports vastly richer use cases than what most folks
need.

And because fixtures are a wonderful gateway drug.  :)

- Rhys

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost