[1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list
This change:
https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5c10cde5bdd8b#diff-efad11c1b810a644deec92a42abe330e

To Boost test (which I'm sure has been long signaled, I just wasn't
aware of it), breaks one Math lib test (develop and master), and quite a
number of Fibre tests (also develop and master), the Math failure should
be fixed in develop
(https://github.com/boostorg/math/commit/1d04bf714e809d386088ce135e3f9ea04caac2b8)
and will get merged to master once CI is green.  Not sure if anything is
in the works for Fiber.

Just a wee heads-up.

Best, John.


--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list
Am Mo., 4. Nov. 2019 um 19:59 Uhr schrieb John Maddock via Boost
<[hidden email]>:
>  Not sure if anything is in the works for Fiber.

wasn't aware of the changes, so no fixes are in the pipeline for boost.fiber

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 04.11.19 19:59, John Maddock via Boost wrote:
> This change:
> https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5c10cde5bdd8b#diff-efad11c1b810a644deec92a42abe330e

Oh, sorry about that. Reverting the lines right now.

>
>
> To Boost test (which I'm sure has been long signaled, I just wasn't
> aware of it), breaks one Math lib test (develop and master), and quite a
> number of Fibre tests (also develop and master), the Math failure should
> be fixed in develop
> (https://github.com/boostorg/math/commit/1d04bf714e809d386088ce135e3f9ea04caac2b8)
> and will get merged to master once CI is green.  Not sure if anything is
> in the works for Fiber.
>
> Just a wee heads-up.
>
> Best, John.
>
>



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list


> -----Original Message-----
> From: Boost <[hidden email]> On Behalf Of Raffi Enficiaud via
> Boost
> Sent: 4 November 2019 22:57
> To: [hidden email]
> Cc: Raffi Enficiaud <[hidden email]>
> Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test
> breaks fiber/math tests
>
> On 04.11.19 19:59, John Maddock via Boost wrote:
> > This change:
> > https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5c1
> > 0cde5bdd8b#diff-efad11c1b810a644deec92a42abe330e
>
> Oh, sorry about that. Reverting the lines right now.

Actually I like the change - did we need 'framework'?

Anyone using this is perhaps relying on a detail?

(The need for it in the Math test case was to ensure that the BOOST_TEST_MESSAGE("my_message") were actually visible.  I think it would be better if messages were visible by default, not something one needs to ask for specially.  I also was unconvinced the setting the environment variable BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)

But definitely a note in the release notes would be good.

The 'fix' is to rename
  unit_test_framework::
to
  unit_test::

Paul

> > To Boost test (which I'm sure has been long signaled, I just wasn't
> > aware of it), breaks one Math lib test (develop and master), and quite
> > a number of Fibre tests (also develop and master), the Math failure
> > should be fixed in develop
> > (https://github.com/boostorg/math/commit/1d04bf714e809d386088ce135e3f9
> > ea04caac2b8) and will get merged to master once CI is green.  Not sure
> > if anything is in the works for Fiber.
> >
> > Just a wee heads-up.
> >
> > Best, John.
> >
> >
>
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list


> -----Original Message-----
> From: Boost <[hidden email]> On Behalf Of Paul A Bristow via
> Boost
> Sent: 5 November 2019 12:42
> To: [hidden email]
> Cc: [hidden email]
> Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in
Boost.Test

> breaks fiber/math tests
>
>
>
> > -----Original Message-----
> > From: Boost <[hidden email]> On Behalf Of Raffi
> > Enficiaud via Boost
> > Sent: 4 November 2019 22:57
> > To: [hidden email]
> > Cc: Raffi Enficiaud <[hidden email]>
> > Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change
> > in Boost.Test breaks fiber/math tests
> >
> > On 04.11.19 19:59, John Maddock via Boost wrote:
> > > This change:
> > > https://github.com/boostorg/test/commit/1c4533f7654d4c049cc6a34a43a5
> > > c1 0cde5bdd8b#diff-efad11c1b810a644deec92a42abe330e
> >
> > Oh, sorry about that. Reverting the lines right now.
>
> Actually I like the change - did we need 'framework'?
>
> Anyone using this is perhaps relying on a detail?
>
> (The need for it in the Math test case was to ensure that the
> BOOST_TEST_MESSAGE("my_message") were actually visible.  I think it would be
> better if messages were visible by default, not something one needs to ask for
> specially.  I also was unconvinced the setting the environment variable
> BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)

I've finally found this exchange explaining the behaviour.

http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td467539
8.html

I think it is very counter-intuitive that messages only emerge to the normal
output if there is an error or --log_level="message"

And that

   #define BOOST_TEST_LOG_LEVEL "message"

 does not appear to have the same effect as --log_level="message"

which is also surprising, and in my book a buglet?

I suggest that this behaviour should be changed so that BOOST_TEST_MESSAGE
*always* produces output.

BOOST_AUTO_TEST_CASE(test_message)
{
   BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n");
} // BOOST_AUTO_TEST_CASE(test_message)

boost_test_message_test.vcxproj ->
I:\Cpp\math\x64\Release\boost_test_message_test.exe

should output

Running 1 test case...

Test BOOST_TEST_MESSAGE.

*** No errors detected

Otherwise I am sure I will not be the only person confused.

At the very least the documentation should highlight that ones expectation may
not match what actually happens.

Paul







_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list
On 05.11.19 17:03, Paul A Bristow via Boost wrote:
>> [snip]
>>
>> Actually I like the change - did we need 'framework'?

"#me too" but this is not the right time to introduce any breaking
changes, especially without notice. It is used nowhere in boost.test but
I have not checked within boost (and certainly not outside) before
making that change (that was also unrelated to the issue being solved).
Some legacy code, a proper deprecation should be emitted, at least in
the change logs.

I do not know if namespaces can have deprecation attributes, that would
be handy.

>> Anyone using this is perhaps relying on a detail?
>>
>> (The need for it in the Math test case was to ensure that the
>> BOOST_TEST_MESSAGE("my_message") were actually visible.  I think it would be
>> better if messages were visible by default, not something one needs to ask for
>> specially.  I also was unconvinced the setting the environment variable
>> BOOST_TEST_MESSAGE worked but I'll check this out and raise a bug report.)
>
> I've finally found this exchange explaining the behaviour.
>
> http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td467539
> 8.html
>
> I think it is very counter-intuitive that messages only emerge to the normal
> output if there is an error or --log_level="message"
>
> And that
>
>    #define BOOST_TEST_LOG_LEVEL "message"
>
>  does not appear to have the same effect as --log_level="message"

The current documentation never mentions the usage through a macro as
you are using in the snippet above. Please see below about the relevant
sections.

>
> which is also surprising, and in my book a buglet?
>
> I suggest that this behaviour should be changed so that BOOST_TEST_MESSAGE
> *always* produces output.
>
> BOOST_AUTO_TEST_CASE(test_message)
> {
>    BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n");
> } // BOOST_AUTO_TEST_CASE(test_message)
>
> boost_test_message_test.vcxproj ->
> I:\Cpp\math\x64\Release\boost_test_message_test.exe
>
> should output
>
> Running 1 test case...
>
> Test BOOST_TEST_MESSAGE.
>
> *** No errors detected

I tend to disagree with that. What I am expecting from my tests is the
report of "true positives", ie. assertions that fails for good reasons
like regressions. All the rest I consider as information that distracts,
or information printed only for non-regular/normal use like debugging. I
would for instance put a message or a warning that I can easily activate
if something goes wrong and needs my attention, until I fix the issue.
Then I would go back to the normal mode.

This is also the approach taken by tools like ctest or b2: normal
logging level prints only a very high level status, verbose logs are
redirected to file for potential inspection, details are printed on error.

The behaviour of Boost.Test in that regard is documented in various
places, mostly in the runtime arguments (command line) and in the
logging part:

*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/log_formats.html
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/log_formats/test_log_output.html
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/test_output_macro_message.html
(even with its own section)
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/log_level.html
(indicates the environment variable as well)
*
https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/runtime_config.html
although the section about environment variables can be clarified



>
> Otherwise I am sure I will not be the only person confused.
>
> At the very least the documentation should highlight that ones expectation may
> not match what actually happens.
I have to say that the current doc is sometimes more than complete with
some redundant parts and confusing information. If you think of a better
way to organize it or a better wording ("me not English native"), please
let me know, I would be happy to improve it.

Thanks!
Raffi

PS: and sorry about breaking your code again so close to the beta.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list
There are times when eminently sensible suggestions (like yours and mine) are
not the best thing to do.

I agree that the benefit isn't worth causing downstream trouble.

But I would still like some warning in the documentation.  I've written
notes-to-self:

// #define BOOST_TEST_LOG_LEVEL "all"   or /DBOOST_TEST_LOG_LEVEL // Do not
output unless there is an error!
//
http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAGE-td467539
8.html
// run with --log_level="message" always gives message.

I use a VS IDE Custom build step "$(TargetDir)$(TargetName).exe"
--log_level="message" for testing console apps.

Paul


> -----Original Message-----
> From: Boost <[hidden email]> On Behalf Of Raffi Enficiaud via
> Boost
> Sent: 5 November 2019 18:49
> To: [hidden email]
> Cc: Raffi Enficiaud <[hidden email]>
> Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in
Boost.Test
> breaks fiber/math tests
>
> On 05.11.19 17:03, Paul A Bristow via Boost wrote:
> >> [snip]
> >>
> >> Actually I like the change - did we need 'framework'?
>
> "#me too" but this is not the right time to introduce any breaking changes,
> especially without notice. It is used nowhere in boost.test but I have not
checked
> within boost (and certainly not outside) before making that change (that was
also
> unrelated to the issue being solved).
> Some legacy code, a proper deprecation should be emitted, at least in the
change

> logs.
>
> I do not know if namespaces can have deprecation attributes, that would be
> handy.
>
> >> Anyone using this is perhaps relying on a detail?
> >>
> >> (The need for it in the Math test case was to ensure that the
> >> BOOST_TEST_MESSAGE("my_message") were actually visible.  I think it
> >> would be better if messages were visible by default, not something
> >> one needs to ask for specially.  I also was unconvinced the setting
> >> the environment variable BOOST_TEST_MESSAGE worked but I'll check
> >> this out and raise a bug report.)
> >
> > I've finally found this exchange explaining the behaviour.
> >
> > http://boost.2283326.n4.nabble.com/test-Problem-with-BOOST-TEST-MESSAG
> > E-td467539
> > 8.html
> >
> > I think it is very counter-intuitive that messages only emerge to the
> > normal output if there is an error or --log_level="message"
> >
> > And that
> >
> >    #define BOOST_TEST_LOG_LEVEL "message"
> >
> >  does not appear to have the same effect as --log_level="message"
>
> The current documentation never mentions the usage through a macro as you are
> using in the snippet above. Please see below about the relevant sections.
>
> >
> > which is also surprising, and in my book a buglet?
> >
> > I suggest that this behaviour should be changed so that
> > BOOST_TEST_MESSAGE
> > *always* produces output.
> >
> > BOOST_AUTO_TEST_CASE(test_message)
> > {
> >    BOOST_TEST_MESSAGE("\nTest BOOST_TEST_MESSAGE.\n"); } //
> > BOOST_AUTO_TEST_CASE(test_message)
> >
> > boost_test_message_test.vcxproj ->
> > I:\Cpp\math\x64\Release\boost_test_message_test.exe
> >
> > should output
> >
> > Running 1 test case...
> >
> > Test BOOST_TEST_MESSAGE.
> >
> > *** No errors detected
>
> I tend to disagree with that. What I am expecting from my tests is the report
of
> "true positives", ie. assertions that fails for good reasons like regressions.
All the
> rest I consider as information that distracts, or information printed only for
non-
> regular/normal use like debugging. I would for instance put a message or a
warning
> that I can easily activate if something goes wrong and needs my attention,
until I
> fix the issue.
> Then I would go back to the normal mode.
>
> This is also the approach taken by tools like ctest or b2: normal logging
level prints
> only a very high level status, verbose logs are redirected to file for
potential
> inspection, details are printed on error.
>
> The behaviour of Boost.Test in that regard is documented in various places,
mostly

> in the runtime arguments (command line) and in the logging part:
>
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_outp
> ut/log_formats.html
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_outp
> ut/log_formats/test_log_output.html
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_outp
> ut/test_tools_support_for_logging/test_output_macro_message.html
> (even with its own section)
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_refere
> nce/rt_param_reference/log_level.html
> (indicates the environment variable as well)
> *
> https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/runtime_c
> onfig.html
> although the section about environment variables can be clarified
>
>
>
> >
> > Otherwise I am sure I will not be the only person confused.
> >
> > At the very least the documentation should highlight that ones
> > expectation may not match what actually happens.
> I have to say that the current doc is sometimes more than complete with some
> redundant parts and confusing information. If you think of a better way to
> organize it or a better wording ("me not English native"), please let me know,
I

> would be happy to improve it.
>
> Thanks!
> Raffi
>
> PS: and sorry about breaking your code again so close to the beta.
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list
On 06.11.19 10:49, Paul A Bristow via Boost wrote:
> There are times when eminently sensible suggestions (like yours and mine) are
> not the best thing to do.
>
> I agree that the benefit isn't worth causing downstream trouble.
>
> But I would still like some warning in the documentation.  I've written
> notes-to-self:

I am all for helping you there. My question is: where should I put this
warning? It is not clear to me if you expect this warning in the doc,
and if so, which section of the doc.



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test breaks fiber/math tests

Boost - Dev mailing list


> -----Original Message-----
> From: Boost <[hidden email]> On Behalf Of Raffi Enficiaud via
> Boost
> Sent: 7 November 2019 06:47
> To: [hidden email]
> Cc: Raffi Enficiaud <[hidden email]>
> Subject: Re: [boost] [1.72][Test][Math][Fiber] Recent breaking change in Boost.Test
> breaks fiber/math tests
>
> On 06.11.19 10:49, Paul A Bristow via Boost wrote:
> > There are times when eminently sensible suggestions (like yours and
> > mine) are not the best thing to do.
> >
> > I agree that the benefit isn't worth causing downstream trouble.
> >
> > But I would still like some warning in the documentation.  I've
> > written
> > notes-to-self:
>
> I am all for helping you there. My question is: where should I put this warning? It is
> not clear to me if you expect this warning in the doc, and if so, which section of
> the doc.

https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/log_level.html

Doesn't make clear to me what the default level is 'out of the box'?  And perhaps warn on other page that it isn't output at the default log level.

https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/testout_reference/test_output_macro_message.html

should also suggest that running with --log_level="message2 is required to get this to actually output anything.

https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/test_output_macro_message.html

and this too.

(I am ashamed to report that I did not follow my own injunction to use hyperlinks to get to this page ☹  As they say, "you can lead a horse to water, but you can't make him drink").

I also didn't find that #define BOOST_TEST_LOG_LEVEL message ort "message" had the desired effect.
Many people might expect this (and consider it desirable to force the messages out.

Thanks

HTH


Paul





_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost