CI Testing and failures

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

CI Testing and failures

Boost - Dev mailing list
Sometimes CI testing fails because of a bug in a compiler. It is
possible to designate in a jam file that a particular test should not be
done for a particular compiler version so that CI testing succeeds, and
therefore a PR can be more easily merged because the CI tests "succeed".
But then the fact that the compiler has a bug is masked by such a change
to the jam file. Is there a solution to such a problem, or maybe this is
really not a problem at all ? What do others think about this ?


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: CI Testing and failures

Boost - Dev mailing list
On 2020-04-26 18:17, Edward Diener via Boost wrote:
> Sometimes CI testing fails because of a bug in a compiler. It is
> possible to designate in a jam file that a particular test should not be
> done for a particular compiler version so that CI testing succeeds, and
> therefore a PR can be more easily merged because the CI tests "succeed".
> But then the fact that the compiler has a bug is masked by such a change
> to the jam file. Is there a solution to such a problem, or maybe this is
> really not a problem at all ? What do others think about this ?

If the bug affects the library, and you still intend to support it, it
follows that you have to add a workaround for the bug in the library.
The failing test indicates that users of this compiler have an actual
problem.

If you don't intend to support this compile, just remove it from the CI.
In the old build matrix this can't be done, except to disable it in the
test/Jamfile.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: CI Testing and failures

Boost - Dev mailing list
On 4/26/2020 12:07 PM, Andrey Semashev via Boost wrote:

> On 2020-04-26 18:17, Edward Diener via Boost wrote:
>> Sometimes CI testing fails because of a bug in a compiler. It is
>> possible to designate in a jam file that a particular test should not
>> be done for a particular compiler version so that CI testing succeeds,
>> and therefore a PR can be more easily merged because the CI tests
>> "succeed". But then the fact that the compiler has a bug is masked by
>> such a change to the jam file. Is there a solution to such a problem,
>> or maybe this is really not a problem at all ? What do others think
>> about this ?
>
> If the bug affects the library, and you still intend to support it, it
> follows that you have to add a workaround for the bug in the library.
> The failing test indicates that users of this compiler have an actual
> problem.

So if there is no workaround, should the feature for the particular
compiler/version be disabled ? Should the test "succeed" for the feature
for the particular compiler/version by either being disabled in Boost
Build or by "passing" because the feature is disabled for the
compiler/version ?

>
> If you don't intend to support this compile, just remove it from the CI.
> In the old build matrix this can't be done, except to disable it in the
> test/Jamfile.

I do not think the entire compiler/version should be removed from
testing if just a very small part of the library does not work because
of a bug. What you are saying is that the tests are meant to succeed
rather than show a valid bug in some compiler/version. The opposite
viewpoint is that a failing test is a good indication of a
compiler/version bug and therefore a failing test is not an indication
of a failing library. For the purposes of Github changes, particularly
PRs, with CI tests, it does seem as if the tests should succeed, which
would back your viewpoint.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: CI Testing and failures

Boost - Dev mailing list
On 2020-04-27 19:52, Edward Diener via Boost wrote:

> On 4/26/2020 12:07 PM, Andrey Semashev via Boost wrote:
>> On 2020-04-26 18:17, Edward Diener via Boost wrote:
>>> Sometimes CI testing fails because of a bug in a compiler. It is
>>> possible to designate in a jam file that a particular test should not
>>> be done for a particular compiler version so that CI testing
>>> succeeds, and therefore a PR can be more easily merged because the CI
>>> tests "succeed". But then the fact that the compiler has a bug is
>>> masked by such a change to the jam file. Is there a solution to such
>>> a problem, or maybe this is really not a problem at all ? What do
>>> others think about this ?
>>
>> If the bug affects the library, and you still intend to support it, it
>> follows that you have to add a workaround for the bug in the library.
>> The failing test indicates that users of this compiler have an actual
>> problem.
>
> So if there is no workaround, should the feature for the particular
> compiler/version be disabled ? Should the test "succeed" for the feature
> for the particular compiler/version by either being disabled in Boost
> Build or by "passing" because the feature is disabled for the
> compiler/version ?
>
>>
>> If you don't intend to support this compile, just remove it from the
>> CI. In the old build matrix this can't be done, except to disable it
>> in the test/Jamfile.
>
> I do not think the entire compiler/version should be removed from
> testing if just a very small part of the library does not work because
> of a bug. What you are saying is that the tests are meant to succeed
> rather than show a valid bug in some compiler/version. The opposite
> viewpoint is that a failing test is a good indication of a
> compiler/version bug and therefore a failing test is not an indication
> of a failing library. For the purposes of Github changes, particularly
> PRs, with CI tests, it does seem as if the tests should succeed, which
> would back your viewpoint.

That is, of course, for you to decide. If only a small part of the
library is not supported on the particular compiler while the rest is
still usable, you can document this limitation and exclude the tests of
that part from the CI (e.g. via preprocessor checks or by factoring out
the test to a separate executable and disabling it in the Jamfile).

My main point is that if you want to deal with the compiler bug, you
should find a workaround and keep the tests running. Otherwise, there's
no point in running tests for the unsupported code on that compiler.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost