Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
Hello Boost Users,

for my project I'm testing using BOOST_AUTO_TEST_SUITE and data driven
test sets (BOOST_DATA_TEST_CASE). The goal is to check the proper
working of a parser by checking success as such and compare the result
with a "expected" data (string) vector. In case of parse error, I want
to write the captured cout/cerr by boost.test into a file. In case of
compare error the parsed result also shall be written to a file. The
fragments show the concept/current state:

namespace x3_test { // aka utils ....

/* [Detect if boost test case failed](
  *
https://stackoverflow.com/questions/18466857/detect-if-boost-test-case-failed?answertab=active#tab-top)
*/
bool current_test_passing()
{
   using namespace boost::unit_test;
   auto const id = framework::current_test_case().p_id;
   auto const test_results = results_collector.results(id);
   return test_results.passed();
}

std::string report_diagnostic(fs::path const& test_case_name,
std::string const& input, std::string const& result)
{
     ...
     if(!x3_test::current_test_passing()) {     // only write in case of
failed test
         test_case_result_writer result_writer(test_case_name);
         result_writer.write(result);
     }
     return ss.str();
}

} // x3_test

...

BOOST_AUTO_TEST_SUITE(....

BOOST_DATA_TEST_CASE( declaration,
        dataset.input()
     ^ dataset.expect()
     ^ dataset.test_case_name(),
     code, expect_AST, test_case_name)
{
     using attribute_type = ast::declaration;
     auto const parser = parser::declaration;

     x3_test::testing_parser<attribute_type> parse;
     auto [parse_ok, parsed_AST] = parse(code, parser);

     BOOST_TEST(parse_ok);
     BOOST_REQUIRE_MESSAGE(parse_ok,
         x3_test::report_diagnostic(test_case_name, code, parsed_AST)
     );

     BOOST_TEST(parsed_AST == expect_AST, btt::per_element());
     BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(),
         x3_test::report_diagnostic(test_case_name, code, parsed_AST)
     );
}

BOOST_AUTO_TEST_SUITE_END()

After some testing, the error count is twice the real (obviously here)
but it's all above a work arround.

It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message
predicate (shown by using --log_level=all) nevertheless the test case
successed or failed. If failed it is streamed to the console.

My first thoughs where that the message is called only in case of test
failure which isn't true.

The problem is now, that my report_diagnostic function write the test
output (parsed_AST) to a special dir/file for later evaluating etc. The
intuitive way would be:

std::string report_diagnostic(fs::path const& test_case_name,
std::string const& input, std::string const& result)
{
     ...
     test_case_result_writer result_writer(test_case_name);
     result_writer.write(result);

     return ss.str();
}

BOOST_REQUIRE_MESSAGE(parse_ok,
         // call diagnostic only in failure case and write then
         x3_test::report_diagnostic(test_case_name, code, parsed_AST)
     );
BOOST_REQUIRE_MESSAGE((parsed_AST == expect_AST, btt::per_element()),
         // call diagnostic only in failure case and write then
         x3_test::report_diagnostic(test_case_name, code, parsed_AST)
     );

Hence not depending on current_test_passing() any more.

How to get this?


Further, how can I get the captured std::cerr in case of '!parse_ok'
inside report_diagnostic?

Thanks,

Olaf

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
Le 20/05/2018 à 19:42, Olaf Peter via Boost-users a écrit :

> Hello Boost Users,
>
> for my project I'm testing using BOOST_AUTO_TEST_SUITE and data driven
> test sets (BOOST_DATA_TEST_CASE). The goal is to check the proper
> working of a parser by checking success as such and compare the result
> with a "expected" data (string) vector. In case of parse error, I want
> to write the captured cout/cerr by boost.test into a file. In case of
> compare error the parsed result also shall be written to a file. The
> fragments show the concept/current state:
>
> namespace x3_test { // aka utils ....
>
> /* [Detect if boost test case failed](
>   *
> https://stackoverflow.com/questions/18466857/detect-if-boost-test-case-failed?answertab=active#tab-top)
> */
> bool current_test_passing()
> {
>    using namespace boost::unit_test;
>    auto const id = framework::current_test_case().p_id;
>    auto const test_results = results_collector.results(id);
>    return test_results.passed();
> }
>
> std::string report_diagnostic(fs::path const& test_case_name,
> std::string const& input, std::string const& result)
> {
>      ...
>      if(!x3_test::current_test_passing()) {     // only write in case of
> failed test
>          test_case_result_writer result_writer(test_case_name);
>          result_writer.write(result);
>      }
>      return ss.str();
> }
>
> } // x3_test
>
> ...
>
> BOOST_AUTO_TEST_SUITE(....
>
> BOOST_DATA_TEST_CASE( declaration,
>         dataset.input()
>      ^ dataset.expect()
>      ^ dataset.test_case_name(),
>      code, expect_AST, test_case_name)
> {
>      using attribute_type = ast::declaration;
>      auto const parser = parser::declaration;
>
>      x3_test::testing_parser<attribute_type> parse;
>      auto [parse_ok, parsed_AST] = parse(code, parser);
>
>      BOOST_TEST(parse_ok);
>      BOOST_REQUIRE_MESSAGE(parse_ok,
>          x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>      );
>
>      BOOST_TEST(parsed_AST == expect_AST, btt::per_element());
>      BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(),
>          x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>      );
> }
>
> BOOST_AUTO_TEST_SUITE_END()
>
> After some testing, the error count is twice the real (obviously here)
> but it's all above a work arround.
>
> It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message
> predicate (shown by using --log_level=all) nevertheless the test case
> successed or failed. If failed it is streamed to the console.

That sounds odd to me. Would it be possible to check if the messages get
evaluated in case you do not pass --log_level=all ?

Otherwise, if you be kind of you if you file a bug.


>
> My first thoughs where that the message is called only in case of test
> failure which isn't true.
>
> The problem is now, that my report_diagnostic function write the test
> output (parsed_AST) to a special dir/file for later evaluating etc. The
> intuitive way would be:
>
> std::string report_diagnostic(fs::path const& test_case_name,
> std::string const& input, std::string const& result)
> {
>      ...
>      test_case_result_writer result_writer(test_case_name);
>      result_writer.write(result);
>
>      return ss.str();
> }
>
> BOOST_REQUIRE_MESSAGE(parse_ok,
>          // call diagnostic only in failure case and write then
>          x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>      );
> BOOST_REQUIRE_MESSAGE((parsed_AST == expect_AST, btt::per_element()),
>          // call diagnostic only in failure case and write then
>          x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>      );
>
> Hence not depending on current_test_passing() any more.
>
> How to get this?

What about :

1- using test-case fixtures for writing things at the end of the test
2- registering an observer for getting the failure signal?

The observer is (kind of) described here:
https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html

You may install and remove it globally by using a global fixture:

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html

Once registered, it will capture the events that you want, especially
the failure cases.

 From the observer, you can get the current test cases with this

https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html


>
>
> Further, how can I get the captured std::cerr in case of '!parse_ok'
> inside report_diagnostic?

Currently you have to do it manually. See this
https://stackoverflow.com/a/5419388/1617295

This can again go to a fixture of your own. There is a feature request
on the backlog though:

https://svn.boost.org/trac10/ticket/12452

Best,
Raffi

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
Hello Raffi,

thank you for 'reloading' the topic. The next 14 days I'm on vacation;
after this I will check your suggestions in detail and reduce the
problem to minimal example if required.

Thank you,
Olaf
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
Hello Raffi,

>
>> BOOST_AUTO_TEST_SUITE(....
>>
>> BOOST_DATA_TEST_CASE( declaration,
>>         dataset.input()
>>      ^ dataset.expect()
>>      ^ dataset.test_case_name(),
>>      code, expect_AST, test_case_name)
>> {
>>      using attribute_type = ast::declaration;
>>      auto const parser = parser::declaration;
>>
>>      x3_test::testing_parser<attribute_type> parse;
>>      auto [parse_ok, parsed_AST] = parse(code, parser);
>>
>>      BOOST_TEST(parse_ok);
>>      BOOST_REQUIRE_MESSAGE(parse_ok,
>>          x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>>      );
>>
>>      BOOST_TEST(parsed_AST == expect_AST, btt::per_element());
>>      BOOST_REQUIRE_MESSAGE(x3_test::current_test_passing(),
>>          x3_test::report_diagnostic(test_case_name, code, parsed_AST)
>>      );
>> }
>>
>> BOOST_AUTO_TEST_SUITE_END()
>>
>> After some testing, the error count is twice the real (obviously
>> here) but it's all above a work arround.
>>
>> It seems, that BOOST_REQUIRE_MESSAGE evaluates always the message
>> predicate (shown by using --log_level=all) nevertheless the test case
>> successed or failed. If failed it is streamed to the console.
>
> That sounds odd to me. Would it be possible to check if the messages
> get evaluated in case you do not pass --log_level=all ?
I reduced it and found the original problem. Checking parse_ok twice
results into two messages, this explain what I faced (and misinterpreted
in the last mail). The reason for introducing this was the intention of:

- check on parse_ok success
- if not, generate a diagnostic message - all further checks can be canceled
- compare the result per_element with reference/gold data
- if failed, generate a diagnostic message
- note, report_diagnostic saves the parsed_AST for later inspection

The problem may rise due to my limited understanding about the macros
arround.

>
> What about :
>
> 1- using test-case fixtures for writing things at the end of the test
> 2- registering an observer for getting the failure signal?
>
> The observer is (kind of) described here:
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html 
>
>
> You may install and remove it globally by using a global fixture:
>
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html 
>
>
> Once registered, it will capture the events that you want, especially
> the failure cases.
>
> From the observer, you can get the current test cases with this
>
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html 
>
I will check/evaluate it next time, it simplifies diagnostic since the
parsed_AST can be written by the observer

Thank you,
Olaf

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
In reply to this post by Boost - Users mailing list

>> How to get this?
>
> What about :
>
> 1- using test-case fixtures for writing things at the end of the test
> 2- registering an observer for getting the failure signal?
>
> The observer is (kind of) described here:
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html 
>
>
> You may install and remove it globally by using a global fixture:
>
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html 
>
>
> Once registered, it will capture the events that you want, especially
> the failure cases.
>
> From the observer, you can get the current test cases with this
>
> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html 
>

OK, my test_observer_fixture with overloads is ready:

class test_observer_fixture
{
     struct test_observer : public utf::test_observer
     {
         virtual void test_unit_aborted(utf::test_unit const& tu) override;
         virtual void assertion_result(utf::assertion_result result)
override;
         // are the others required for my use case?
     };

     test_observer                                    observer;

public:
     test_observer_fixture();
     ~test_observer_fixture();

public:
     void setup();
     void teardown();
};

and get called by my BOOST_DATA_TEST_CASE. What is the prefered way to
use this Fixture to save the test result in case of failure/abort?


BOOST_DATA_TEST_CASE( wave,
     utf_data::make_delayed<testsuite::dataset_loader>( "test_case/wave" ),
     input, expected, orig_file_name)
{
    ...
     auto [parse_ok, parse_result] = parse(input, parser, test_case_name);

     BOOST_REQUIRE(parse_ok);

     BOOST_TEST(parse_result == expected, btt::per_element());

        ??? my_diagnostic_saving .... (orig_file_name, input, parse_result);
     );
}

Thanks,
Olaf



_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
On 23.07.18 19:28, Olaf Peter via Boost-users wrote:

>
>>> How to get this?
>>
>> What about :
>>
>> 1- using test-case fixtures for writing things at the end of the test
>> 2- registering an observer for getting the failure signal?
>>
>> The observer is (kind of) described here:
>> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html 
>>
>>
>> You may install and remove it globally by using a global fixture:
>>
>> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html 
>>
>>
>> Once registered, it will capture the events that you want, especially
>> the failure cases.
>>
>> From the observer, you can get the current test cases with this
>>
>> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html 
>>
>
> OK, my test_observer_fixture with overloads is ready:
>
> class test_observer_fixture
> {
>      struct test_observer : public utf::test_observer
>      {
>          virtual void test_unit_aborted(utf::test_unit const& tu) override;
>          virtual void assertion_result(utf::assertion_result result)
> override;
>          // are the others required for my use case?
>      };
>
>      test_observer                                    observer;
>
> public:
>      test_observer_fixture();
>      ~test_observer_fixture();
>
> public:
>      void setup();
>      void teardown();
> };
>
> and get called by my BOOST_DATA_TEST_CASE. What is the prefered way to
> use this Fixture to save the test result in case of failure/abort?
>
>
> BOOST_DATA_TEST_CASE( wave,
>      utf_data::make_delayed<testsuite::dataset_loader>( "test_case/wave" ),
>      input, expected, orig_file_name)
> {
>     ...
>      auto [parse_ok, parse_result] = parse(input, parser, test_case_name);
>
>      BOOST_REQUIRE(parse_ok);
>
>      BOOST_TEST(parse_result == expected, btt::per_element());
>
>         ??? my_diagnostic_saving .... (orig_file_name, input,
> parse_result);
>      );
> }
>
> Thanks,
> Olaf

Sorry for the late answer,

In the teardown, you may check if the current test had failures. You can
do so by accessing the global singleton "results_collector". If "tc" is
the current test case (that you can get with "current_test_case"):

test_results const& tr = results_collector.results( tc.p_id );

will give you the result.

If the fixture should be applied to each of the test case inside your
suite, you can install it like this:

https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_organization/fixtures/case.html#boost_test.tests_organization.fixtures.case.fixture_for_a_complete_subtree

Accessing via the test result will let you check if this is a failure or
an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated) will fire
an abort if I remember well. Using this way to install the fixture will
also let you access the instance of the fixture directly (see examples
in the documentation).

Since you solved your issue with the double messages, I would just use
the appropriate fixture and forget about the observer.

Raffi

PS: I am not sure that the teardown is called for data test case
(something I forgot to check). You may do that in the dtor of your fixture.


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
Am 08.08.2018 um 13:22 schrieb Raffi Enficiaud via Boost-users:

> On 23.07.18 19:28, Olaf Peter via Boost-users wrote:
>>
>>>> How to get this?
>>>
>>> What about :
>>>
>>> 1- using test-case fixtures for writing things at the end of the test
>>> 2- registering an observer for getting the failure signal?
>>>
>>> The observer is (kind of) described here:
>>> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/register_observer.html 
>>>
>>>
>>> You may install and remove it globally by using a global fixture:
>>>
>>> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost_test/utf_reference/test_org_reference/test_org_boost_test_global_fixture.html 
>>>
>>>
>>> Once registered, it will capture the events that you want,
>>> especially the failure cases.
>>>
>>> From the observer, you can get the current test cases with this
>>>
>>> https://www.boost.org/doc/libs/master/libs/test/doc/html/boost/unit_test/framework/current_test_case.html 
>>>
>>
>> OK, my test_observer_fixture with overloads is ready:
>>
>> class test_observer_fixture
>> {
>>      struct test_observer : public utf::test_observer
>>      {
>>          virtual void test_unit_aborted(utf::test_unit const& tu)
>> override;
>>          virtual void assertion_result(utf::assertion_result result)
>> override;
>>          // are the others required for my use case?
>>      };
>>
>>      test_observer                                    observer;
>>
>> public:
>>      test_observer_fixture();
>>      ~test_observer_fixture();
>>
>> public:
>>      void setup();
>>      void teardown();
>> };
>>
>> and get called by my BOOST_DATA_TEST_CASE. What is the prefered way
>> to use this Fixture to save the test result in case of failure/abort?
>>
>>
>> BOOST_DATA_TEST_CASE( wave,
>>      utf_data::make_delayed<testsuite::dataset_loader>(
>> "test_case/wave" ),
>>      input, expected, orig_file_name)
>> {
>>     ...
>>      auto [parse_ok, parse_result] = parse(input, parser,
>> test_case_name);
>>
>>      BOOST_REQUIRE(parse_ok);
>>
>>      BOOST_TEST(parse_result == expected, btt::per_element());
>>
>>         ??? my_diagnostic_saving .... (orig_file_name, input,
>> parse_result);
>>      );
>> }
>>
>> Thanks,
>> Olaf
>
> Sorry for the late answer,
>
> In the teardown, you may check if the current test had failures. You
> can do so by accessing the global singleton "results_collector". If
> "tc" is the current test case (that you can get with
> "current_test_case"):
>
> test_results const& tr = results_collector.results( tc.p_id );
>
> will give you the result.
>
> If the fixture should be applied to each of the test case inside your
> suite, you can install it like this:
>
> https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_organization/fixtures/case.html#boost_test.tests_organization.fixtures.case.fixture_for_a_complete_subtree 
>
>
> Accessing via the test result will let you check if this is a failure
> or an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated) will
> fire an abort if I remember well. Using this way to install the
> fixture will also let you access the instance of the fixture directly
> (see examples in the documentation).
>
> Since you solved your issue with the double messages, I would just use
> the appropriate fixture and forget about the observer.
>
> Raffi
>
> PS: I am not sure that the teardown is called for data test case
> (something I forgot to check). You may do that in the dtor of your
> fixture.
thanks for your answer - I will implement and  check it next.

Regards,
Olaf
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
In reply to this post by Boost - Users mailing list

>> OK, my test_observer_fixture with overloads is ready:
>>
>> class test_observer_fixture
>> {
>>      struct test_observer : public utf::test_observer
>>      {
>>          virtual void test_unit_aborted(utf::test_unit const& tu)
>> override;
>>          virtual void assertion_result(utf::assertion_result result)
>> override;
>>          // are the others required for my use case?
>>      };
>>
>>      test_observer                                    observer;
>>
>> public:
>>      test_observer_fixture();
>>      ~test_observer_fixture();
>>
>> public:
>>      void setup();
>>      void teardown();
>> };
>>
>> and get called by my BOOST_DATA_TEST_CASE. What is the prefered way
>> to use this Fixture to save the test result in case of failure/abort?
>>
>>
>> BOOST_DATA_TEST_CASE( wave,
>>      utf_data::make_delayed<testsuite::dataset_loader>(
>> "test_case/wave" ),
>>      input, expected, orig_file_name)
>> {
>>     ...
>>      auto [parse_ok, parse_result] = parse(input, parser,
>> test_case_name);
>>
>>      BOOST_REQUIRE(parse_ok);
>>
>>      BOOST_TEST(parse_result == expected, btt::per_element());
>>
>>         ??? my_diagnostic_saving .... (orig_file_name, input,
>> parse_result);
>>      );
>> }
>>
>> Thanks,
>> Olaf
>
> Sorry for the late answer,
>
> In the teardown, you may check if the current test had failures. You
> can do so by accessing the global singleton "results_collector". If
> "tc" is the current test case (that you can get with
> "current_test_case"):
>
> test_results const& tr = results_collector.results( tc.p_id );
>
> will give you the result.
>
> If the fixture should be applied to each of the test case inside your
> suite, you can install it like this:
>
> https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_organization/fixtures/case.html#boost_test.tests_organization.fixtures.case.fixture_for_a_complete_subtree 
>
>
> Accessing via the test result will let you check if this is a failure
> or an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated) will
> fire an abort if I remember well. Using this way to install the
> fixture will also let you access the instance of the fixture directly
> (see examples in the documentation).
>
Since my global testobserver works for all events which may interested
even in the future (e.g exceptions), I have problems to apply them in my
use case.

I've added a member function to my testobserer fixture

void test_obserer_fixture::failure_diagnostic(std::string const&, ...) 
which I can activate on failed tests using the events I've got from
utf::observer::assertion_result()

Now, in file test_main.cpp I have:
-----------------------------------------
#define BOOST_TEST_MODULE "Test Suite"
#include <boost/test/included/unit_test.hpp>
#include <testsuite/test_observer_fixture.hpp>

using test_observer_fixture = testsuite::test_observer_fixture;
BOOST_TEST_GLOBAL_FIXTURE(test_observer_fixture);
-----------------------------------------

and in (I have several)  concrete test case file:
------------------------------------------
#include <boost/test/unit_test.hpp>
#include <boost/test/output_test_stream.hpp>
#include <testsuite/test_observer_fixture.hpp>
...
BOOST_AUTO_TEST_SUITE( my_testsuite )

BOOST_DATA_TEST_CASE( failure_test,...
     ...,
     input, expected, test_case_name)
{
...
}
-------------------------------------------

and I can see/log:
# assertion failed: failure_test/test_case/_0

But how to access the global fixture now?

I've changed the test case to:
------------------------------------------
...
BOOST_DATA_TEST_CASE_F(test_observer_fixture, failure_test,
     ...,
     input, expected, test_case_name)
{
     ...
     failure_diagnostic(...);
}
-------------------------------------------


Boost.Test ships a BOOST_DATA_TEST_CASE_F() macro for this case, which
is a 'local' fixture, each time constructed and destroyed which is
expensive - so  I like the idea of a global fixture to save to
filesystem. Even I've installed it as global one too the events are
doubled... but I can see that the test observer shows the test status
followed by the call to failure_diagnostic - so the concept works.

To reduce it: How can I access a member function of global fixture on a
compilation unit layout shown above.

Thanks,
Olaf







_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Test BOOST_TEST/BOOST_REQUIRE_MESSAGE message arg problems

Boost - Users mailing list
Am 18.08.2018 um 20:38 schrieb Olaf Peter via Boost-users:

>
>>> OK, my test_observer_fixture with overloads is ready:
>>>
>>> class test_observer_fixture
>>> {
>>>      struct test_observer : public utf::test_observer
>>>      {
>>>          virtual void test_unit_aborted(utf::test_unit const& tu)
>>> override;
>>>          virtual void assertion_result(utf::assertion_result result)
>>> override;
>>>          // are the others required for my use case?
>>>      };
>>>
>>>      test_observer observer;
>>>
>>> public:
>>>      test_observer_fixture();
>>>      ~test_observer_fixture();
>>>
>>> public:
>>>      void setup();
>>>      void teardown();
>>> };
>>>
>>> and get called by my BOOST_DATA_TEST_CASE. What is the prefered way
>>> to use this Fixture to save the test result in case of failure/abort?
>>>
>>>
>>> BOOST_DATA_TEST_CASE( wave,
>>>      utf_data::make_delayed<testsuite::dataset_loader>(
>>> "test_case/wave" ),
>>>      input, expected, orig_file_name)
>>> {
>>>     ...
>>>      auto [parse_ok, parse_result] = parse(input, parser,
>>> test_case_name);
>>>
>>>      BOOST_REQUIRE(parse_ok);
>>>
>>>      BOOST_TEST(parse_result == expected, btt::per_element());
>>>
>>>         ??? my_diagnostic_saving .... (orig_file_name, input,
>>> parse_result);
>>>      );
>>> }
>>>
>>> Thanks,
>>> Olaf
>>
>> Sorry for the late answer,
>>
>> In the teardown, you may check if the current test had failures. You
>> can do so by accessing the global singleton "results_collector". If
>> "tc" is the current test case (that you can get with
>> "current_test_case"):
>>
>> test_results const& tr = results_collector.results( tc.p_id );
>>
>> will give you the result.
>>
>> If the fixture should be applied to each of the test case inside your
>> suite, you can install it like this:
>>
>> https://www.boost.org/doc/libs/1_67_0/libs/test/doc/html/boost_test/tests_organization/fixtures/case.html#boost_test.tests_organization.fixtures.case.fixture_for_a_complete_subtree 
>>
>>
>> Accessing via the test result will let you check if this is a failure
>> or an abort: the BOOST_TEST_REQUIRE (BOOST_REQUIRE is deprecated)
>> will fire an abort if I remember well. Using this way to install the
>> fixture will also let you access the instance of the fixture directly
>> (see examples in the documentation).
>>
> Since my global testobserver works for all events which may interested
> even in the future (e.g exceptions), I have problems to apply them in
> my use case.
>
> I've added a member function to my testobserer fixture
>
> void test_obserer_fixture::failure_diagnostic(std::string const&,
> ...)  which I can activate on failed tests using the events I've got
> from utf::observer::assertion_result()
>
> Now, in file test_main.cpp I have:
> -----------------------------------------
> #define BOOST_TEST_MODULE "Test Suite"
> #include <boost/test/included/unit_test.hpp>
> #include <testsuite/test_observer_fixture.hpp>
>
> using test_observer_fixture = testsuite::test_observer_fixture;
> BOOST_TEST_GLOBAL_FIXTURE(test_observer_fixture);
> -----------------------------------------
>
> and in (I have several)  concrete test case file:
> ------------------------------------------
> #include <boost/test/unit_test.hpp>
> #include <boost/test/output_test_stream.hpp>
> #include <testsuite/test_observer_fixture.hpp>
> ...
> BOOST_AUTO_TEST_SUITE( my_testsuite )
>
> BOOST_DATA_TEST_CASE( failure_test,...
>     ...,
>     input, expected, test_case_name)
> {
> ...
> }
> -------------------------------------------
>
> and I can see/log:
> # assertion failed: failure_test/test_case/_0
>
> But how to access the global fixture now?
>
> I've changed the test case to:
> ------------------------------------------
> ...
> BOOST_DATA_TEST_CASE_F(test_observer_fixture, failure_test,
>     ...,
>     input, expected, test_case_name)
> {
>     ...
>     failure_diagnostic(...);
> }
> -------------------------------------------
>
>
> Boost.Test ships a BOOST_DATA_TEST_CASE_F() macro for this case, which
> is a 'local' fixture, each time constructed and destroyed which is
> expensive - so  I like the idea of a global fixture to save to
> filesystem. Even I've installed it as global one too the events are
> doubled... but I can see that the test observer shows the test status
> followed by the call to failure_diagnostic - so the concept works.
>
> To reduce it: How can I access a member function of global fixture on
> a compilation unit layout shown above.
even the use of

BOOST_FIXTURE_TEST_SUITE( my_testsuite, test_observer_fixture )

seems to create/destroy the fixture for each test case.


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users