Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users
<[hidden email]> > I was wondering whether there is any
interest in exploring
> a C++20 single header/single module, macro-free Unit
> Testing Framework with no dependencies?

I have no interest in a library that requires C++20, especially
considering that C++20 is not even official yet but also because once
C++20 is released, there will be hardly any users for many years. This
project seems very much like it was written "just because", to use the
latest language features, rather than for pragmatic reasons. I don't
see anything compelling to use it over Boost.LightweightTest for
example.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Sat, 23 Nov 2019 at 04:31, Krzysztof Jusiak <[hidden email]> wrote:
>
> Thank you for your feedback. I appreciate it.
>

In my previous post I forgot to add one more point:

IMO, the library like UT should be implemented as a facade for
Catch2, Boost.Test or whatever.
Then, it would offer unique features with real value to users,
other than the modern macro-less syntax sugar.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list

> On 23 Nov 2019, at 13:19, Mateusz Loskot via Boost <[hidden email]> wrote:
>
> On Sat, 23 Nov 2019 at 04:31, Krzysztof Jusiak <[hidden email]> wrote:
>>
>> Thank you for your feedback. I appreciate it.
>>
>
> In my previous post I forgot to add one more point:
>
> IMO, the library like UT should be implemented as a facade for
> Catch2, Boost.Test or whatever.
> Then, it would offer unique features with real value to users,
> other than the modern macro-less syntax sugar.

I agree in principle that common interfaces would be a huge advantage, however I can not agree that an endeavor to make macro less facilities for C++ whenever we do not need macros anymore, are not worth working on.  Whether the time is right, well we will see.  But then we find what is missing, and that is something too.


Bjørn

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list


> On 22 Nov 2019, at 21:53, Vinnie Falco via Boost <[hidden email]> wrote:
>
> On Thu, Nov 21, 2019 at 7:24 AM Krzysztof Jusiak via Boost-users
> <[hidden email]> > I was wondering whether there is any
> interest in exploring
>> a C++20 single header/single module, macro-free Unit
>> Testing Framework with no dependencies?
>
> I have no interest in a library that requires C++20, especially
> considering that C++20 is not even official yet but also because once
> C++20 is released, there will be hardly any users for many years. This
> project seems very much like it was written "just because", to use the
> latest language features, rather than for pragmatic reasons. I don't
> see anything compelling to use it over Boost.LightweightTest for
> example.

Is your lack of interest mostly based on its lack of potential utility for your projects now or your lack of interest for such tools in the future?

What are the chances new libraries depending heavily on macros in their interfaces will ever make in into the standard library?   If they will not I don’t think it is fair to say this is something that look like someone just want to play with the newest language features, and that it has no other merit.  I agree there may be elements of the library that aligns with your assessment, but I suspect that is for good reasons where options where not available other than macros.  If that is the case, we have learned something new, haven’t we?


Bjørn

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Sat, Nov 23, 2019 at 4:56 AM Bjørn Roald via Boost
<[hidden email]> wrote:
> Is your lack of interest mostly based on its lack of potential utility
> for your projects now or your lack of interest for such tools in the future?

I don't see a compelling case for a unit-test library to be
standardized. And I see nothing wrong with macros. They are not going
away, ever.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
> I don't see a compelling case for a unit-test library to be
> standardized.


I respectfully disagree here. Standardisation aids adoption, learning and
portability.

My view is that c++ would be more popular if the standard was wider in
scope. The arguments I hear against using c++ in new projects include “you
need arcane knowledge to actually achieve anything”.

It is for this reason that I regard boost beast as the most pivotal library
to bless the c++ community since the boost project was started.

And I see nothing wrong with macros.


Except that they pollute the global namespace and contribute to unintuitive
bugs?

They are not going
> away, ever.


We can hope and work in that direction.


>
> Thanks
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>
--
Richard Hodges
[hidden email]
office: +442032898513
home: +376841522
mobile: +376380212

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Sat, Nov 23, 2019 at 6:14 AM Richard Hodges via Boost
<[hidden email]> wrote:
> Standardisation aids adoption, learning and portability.

What does being in the standard offer that cannot be achieved as an
independent library?

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list


> On 23 Nov 2019, at 15:15, Vinnie Falco via Boost <[hidden email]> wrote:
>
> On Sat, Nov 23, 2019 at 6:14 AM Richard Hodges via Boost
> <[hidden email]> wrote:
>> Standardisation aids adoption, learning and portability.
>
> What does being in the standard offer that cannot be achieved as an
> independent library?

When adapted into your project, not much.  But as far as getting there, a lot.  In some projects, a lot more than you may believe, even for a unit test tool.


Bjørn


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Sat, Nov 23, 2019 at 6:22 AM Bjørn Roald via Boost
<[hidden email]> wrote:
> When adapted into your project, not much.  But as far as
> getting there, a lot.  In some projects, a lot more than you
> may believe, even for a unit test tool.

This doesn't answer the question.

Here's an example of how the question might be answered for a popular
Boost library, Boost.Asio:

"C++ benefits from standardized networking, because having common
vocabulary types and named requirements allows libraries written
against standardized networking components to interoperate."

The same cannot be said for unit test libraries, since those types
don't appear in public interfaces. So again what is the benefit of
standardization?

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
> So again what is the benefit of standardization?

IMHO, there is always a benefit of standardization best/common practices
because
* It lowers the entry-level to the language (no need for third-party
libraries)
* It improves the education aspect (one standard way of doing it)
* It makes the language more coherent/stable (consistent design with other
features, stable API)
* It makes the feature a first class citizen (shows that the community
cares about this aspect of the language)

to name a few...

On Sat, Nov 23, 2019 at 7:45 AM Vinnie Falco via Boost <
[hidden email]> wrote:

> On Sat, Nov 23, 2019 at 6:22 AM Bjørn Roald via Boost
> <[hidden email]> wrote:
> > When adapted into your project, not much.  But as far as
> > getting there, a lot.  In some projects, a lot more than you
> > may believe, even for a unit test tool.
>
> This doesn't answer the question.
>
> Here's an example of how the question might be answered for a popular
> Boost library, Boost.Asio:
>
> "C++ benefits from standardized networking, because having common
> vocabulary types and named requirements allows libraries written
> against standardized networking components to interoperate."
>
> The same cannot be said for unit test libraries, since those types
> don't appear in public interfaces. So again what is the benefit of
> standardization?
>
> Thanks
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Sat, Nov 23, 2019 at 7:17 AM Krzysztof Jusiak <[hidden email]> wrote:
> * It lowers the entry-level to the language (no need for third-party libraries)

It should be obvious to everyone that external libraries cannot, and
should not be an impediment to adoption. Similarly, adding every
possible library to the standard is obviously not sustainable. Yes
there's a problem with packaging, but the solution is not to propose
everything for the standard. It is to solve the packaging problem
(perhaps that could be the focus of your new efforts?)

"Because being in the standard makes it easier to include" is not a
sufficient justification for being in the standard.

> * It improves the education aspect (one standard way of doing it)

One of the strengths of C++ is that it does not impose any particular
models of computation or paradigms on the user. This is important for
many reasons, performance being one but also to allow problems to be
solved in different ways, with each solution ideally suited to meet
the needs of a particular use-case.

"Because it becomes easier to teach" is not a sufficient justification
for being in the standard.

> * It makes the feature a first class citizen (shows that the community cares about this aspect of the language)

"Showing that the community cares" is not a sufficient justification
for being in the standard.

The C++ Standard is effectively a legal document which mandates a set
of requirements that must be fulfilled for a vendor to claim that
their product is "C++." Everything added to the standard increases the
burden for all implementers. It is very easy for a random Internet
person to propose a new library for the C++ standard, because they do
not bear the cost of implementation on all platforms and toolchains.
They don't even bear the cost of the initial proposed wording (I see
no wording from you, and a dearth of documentation in general). And
they don't bear the cost of refining the wording to resolve conflicts
before getting into the draft.

There must be an overwhelming preponderance of supporting evidence
that anything added to the standard is worth far more than the cost. I
don't see that in your library.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Sat, 23 Nov 2019 at 16:17, Krzysztof Jusiak via Boost
<[hidden email]> wrote:
>
> > So again what is the benefit of standardization?
>
> IMHO, there is always a benefit of standardization best/common
> practices because


Good, then the documentation, in the motivation section, could/should answer:
   What are the best practices the UT captures?
in comparison to popular existing frameworks,
which do not necessarily use the best practices.

My take on the main points listed in the
https://github.com/boost-experimental/ut/blob/master/README.md#overview,

- no dependencies - it's a convenience, not best practice

- single header - header-only libraries are sometimes it is a convenience,
  sometimes a necessity, sometimes a matter of personal preference or choice,
  but header-only is not a best practice.

- macro-free - generally, this is best practice; specifically, it
depends. As Vinnie
  pointed out, macros (polluting global namespace) in test frameworks are not
  a troublemaker, so in case of UT this is a weak argument.

- easy to use - yes, this is a universal best practice.

- minimal API - well, most test framework I've seen or used allow users
  to get the job done using 2-3 macros e.g. using Catch2 you need to
learn just two.
  2-5 macros is minimal API.

- fast to compile - Can this be called a best practice? Well, it
certainly is a nirvana to aim for
  but in real case, it depends. Boost.MP11 compiles faster than Boost.MPL
  but some projects have no choice than to use MPL.

If the UT aims for an ideal of standardization proposable library,
then it (documentation) needs to make the best practices more vivid.

I do like your library. It is a very interesting experiment, a very good
piece of code to read and learn from about C++20 features.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
I wasn't even trying to imply that [Boost].UT should go to the standard
(it's not even a boost library ;) but, as stated, I believe there are
benefits of having things in the standard and I, personally, think that C++
would benefit from having some basic testing primitives but, as always
there are pros and cons to literally anything.

On Sat, Nov 23, 2019 at 8:43 AM Mateusz Loskot via Boost <
[hidden email]> wrote:

> On Sat, 23 Nov 2019 at 16:17, Krzysztof Jusiak via Boost
> <[hidden email]> wrote:
> >
> > > So again what is the benefit of standardization?
> >
> > IMHO, there is always a benefit of standardization best/common
> > practices because
>
>
> Good, then the documentation, in the motivation section, could/should
> answer:
>    What are the best practices the UT captures?
> in comparison to popular existing frameworks,
> which do not necessarily use the best practices.
>
> My take on the main points listed in the
> https://github.com/boost-experimental/ut/blob/master/README.md#overview,
>
> - no dependencies - it's a convenience, not best practice
>
> - single header - header-only libraries are sometimes it is a convenience,
>   sometimes a necessity, sometimes a matter of personal preference or
> choice,
>   but header-only is not a best practice.
>
> - macro-free - generally, this is best practice; specifically, it
> depends. As Vinnie
>   pointed out, macros (polluting global namespace) in test frameworks are
> not
>   a troublemaker, so in case of UT this is a weak argument.
>
> - easy to use - yes, this is a universal best practice.
>
> - minimal API - well, most test framework I've seen or used allow users
>   to get the job done using 2-3 macros e.g. using Catch2 you need to
> learn just two.
>   2-5 macros is minimal API.
>
> - fast to compile - Can this be called a best practice? Well, it
> certainly is a nirvana to aim for
>   but in real case, it depends. Boost.MP11 compiles faster than Boost.MPL
>   but some projects have no choice than to use MPL.
>
> If the UT aims for an ideal of standardization proposable library,
> then it (documentation) needs to make the best practices more vivid.
>
> I do like your library. It is a very interesting experiment, a very good
> piece of code to read and learn from about C++20 features.
>
> Best regards,
> --
> Mateusz Loskot, http://mateusz.loskot.net
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On Sat, 23 Nov 2019 at 16:59, Krzysztof Jusiak <[hidden email]> wrote:
>
> I wasn't even trying to imply that [Boost].UT should go to the standard (it's not even a boost library ;)
> but, as stated, I believe there are benefits of having things in the standard and I,
> personally, think that C++ would benefit from having some basic testing primitives but,
> as always there are pros and cons to literally anything.

Yes, you were not.

Since you touched on best practices as one of goals/roles of the
standardization,
I just took it as an opportunity to briefly review what the current UT
docs present
in terms of the possible best practices.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
My personal take is that C++ strength comes from standardizing
zero-overhead abstractions and not the underlying implementation which
allows for the best possible implementation depending on a variety of
factors.
I also believe that having basic testing primitives in the standard would
benefit C++ (as stated before) but I understand that opinions will vary
here and I'm not going to argue that (there are pros and cons both).
I'm also definitely not proposing [Boost].UT for standardization here; just
checking whether there is an interest in the library, that's it.

On Sat, Nov 23, 2019 at 8:33 AM Vinnie Falco <[hidden email]> wrote:

> On Sat, Nov 23, 2019 at 7:17 AM Krzysztof Jusiak <[hidden email]>
> wrote:
> > * It lowers the entry-level to the language (no need for third-party
> libraries)
>
> It should be obvious to everyone that external libraries cannot, and
> should not be an impediment to adoption. Similarly, adding every
> possible library to the standard is obviously not sustainable. Yes
> there's a problem with packaging, but the solution is not to propose
> everything for the standard. It is to solve the packaging problem
> (perhaps that could be the focus of your new efforts?)
>
> "Because being in the standard makes it easier to include" is not a
> sufficient justification for being in the standard.
>
> > * It improves the education aspect (one standard way of doing it)
>
> One of the strengths of C++ is that it does not impose any particular
> models of computation or paradigms on the user. This is important for
> many reasons, performance being one but also to allow problems to be
> solved in different ways, with each solution ideally suited to meet
> the needs of a particular use-case.
>
> "Because it becomes easier to teach" is not a sufficient justification
> for being in the standard.
>
> > * It makes the feature a first class citizen (shows that the community
> cares about this aspect of the language)
>
> "Showing that the community cares" is not a sufficient justification
> for being in the standard.
>
> The C++ Standard is effectively a legal document which mandates a set
> of requirements that must be fulfilled for a vendor to claim that
> their product is "C++." Everything added to the standard increases the
> burden for all implementers. It is very easy for a random Internet
> person to propose a new library for the C++ standard, because they do
> not bear the cost of implementation on all platforms and toolchains.
> They don't even bear the cost of the initial proposed wording (I see
> no wording from you, and a dearth of documentation in general). And
> they don't bear the cost of refining the wording to resolve conflicts
> before getting into the draft.
>
> There must be an overwhelming preponderance of supporting evidence
> that anything added to the standard is worth far more than the cost. I
> don't see that in your library.
>
> Thanks
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Sat, Nov 23, 2019 at 8:00 AM Krzysztof Jusiak via Boost
<[hidden email]> wrote:
> I wasn't even trying to imply that [Boost].UT should go to the standard

Yeah, you weren't. That was Bjørn Roald. Although you did claim some
benefits to standardization, which I replied to.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list


> On 23 Nov 2019, at 17:16, Vinnie Falco via Boost <[hidden email]> wrote:
>
> On Sat, Nov 23, 2019 at 8:00 AM Krzysztof Jusiak via Boost
> <[hidden email]> wrote:
>> I wasn't even trying to imply that [Boost].UT should go to the standard
>
> Yeah, you weren't. That was Bjørn Roald. Although you did claim some
> benefits to standardization, which I replied to.

Well, I don’t think I was proposing it for standardization. Not that I do not think it may be worth it down the road.  But for now it is most likely far from being ready for that.  What I am proposing is that it may be worth exploring this type of library for possible future standardization, and that is why I think it may be worth considering for Boost.

I will see if I can spend some more time with it, do some testing, and see if it affect my opinion.  I agree with you Vinnie that it does not provide much functionally over alternatives.  I think you only see it as syntactic sugar what I see as a potential path to something new in standard C++.  Similar technicues may be useful not only unit test, but also for diagnostic logging or other tools.


Bjørn

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list

> On 23. Nov 2019, at 04:31, Krzysztof Jusiak via Boost <[hidden email]> wrote:

> However, with [Boost].UT there is nothing stopping anyone from using simple
> macros (one-lines) to achieve other frameworks syntax
> The good bit about it is that it's an opt-in 'feature' as opposed to being
> the only available option (see example below [1]).

But then you have split in the user base, some will use the macros others will use the macro-free syntax, and both will have difficulty in understanding the code of other.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
I see your point, but I'm not advocating here for providing multiple ways
of doing the same thing as I agree it might be confusing although I'd also
argue that it's already a case with boost.test for example (single
header/static/shared library or BOOST_TEST, BOOST_CHECK, BOOST_CHECK_EQ
accomplish pretty much the same things but there are also a trade-offs here
so it depends)
Also, having a possibility for users to do so is valuable IMHO but I
wouldn't expose any macros from the library.

On Sun, Nov 24, 2019 at 9:55 AM Hans Dembinski <[hidden email]>
wrote:

>
> > On 23. Nov 2019, at 04:31, Krzysztof Jusiak via Boost <
> [hidden email]> wrote:
>
> > However, with [Boost].UT there is nothing stopping anyone from using
> simple
> > macros (one-lines) to achieve other frameworks syntax
> > The good bit about it is that it's an opt-in 'feature' as opposed to
> being
> > the only available option (see example below [1]).
>
> But then you have split in the user base, some will use the macros others
> will use the macro-free syntax, and both will have difficulty in
> understanding the code of other.
>
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Boost-users] Interest in a C++20 Unit Testing Framework?

Boost - Dev mailing list
On 24.11.19 19:26, Krzysztof Jusiak via Boost wrote:
> I see your point, but I'm not advocating here for providing multiple ways
> of doing the same thing as I agree it might be confusing although I'd also
> argue that it's already a case with boost.test for example (single
> header/static/shared library or BOOST_TEST, BOOST_CHECK, BOOST_CHECK_EQ
> accomplish pretty much the same things but there are also a trade-offs here
> so it depends)

For info, BOOST_TEST does everything (and more):

https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/testing_tools/boost_test_universal_macro.html
says

"BOOST_TEST: universal and general purpose assertions"

The other macros are mostly kept for C++03 compilers and compatibility.
For the header/static/shared, this is beyond C++ itself, and IMO this is
needed when you want to address various code bases, although it creates
a lot of confusion on the users' side.

Don't get me wrong, I have nothing against or in favor of the UT library.

Since we are talking about "standard" way of testing C++ code, here are
some properties I believe are common or established [*]:

* reporting in various formats, actionable runtime, including the ones
that are understood by CI
* floating point comparison
* xUnit type of testing (fixture, suites, cases)
* "robust" to various type of exceptions

All the rest is bonus.

Raffi

[*] I have not checked what UT provides, I am a user of Boost.Test and
GoogleTest


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
12