Changing virtual memory amount in Appveyor and Travis CI tests

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
Is it possible through .yml file for Appveyor and Travis CI to change
the amount of virtual memory when running a CI test. I am running CI
tests for Boost PP which heavily use the preprocessor, and some of the
extended tests I am running fail on both Appveyor and Travis CI  with
out of memory conditions while on my own local computer, with about 32
GB of ram, the tests succeed. I think in the real world most users now
have at least 32 GB of ram or more, while these virtual environments in
Appveyor and/or Travis CI probably run their tests with much less
overall virtual memory than an average user would have.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 21/09/2020 14:14, Edward Diener wrote:
> Is it possible through .yml file for Appveyor and Travis CI to change
> the amount of virtual memory when running a CI test. I am running CI
> tests for Boost PP which heavily use the preprocessor, and some of the
> extended tests I am running fail on both Appveyor and Travis CI  with
> out of memory conditions while on my own local computer, with about 32
> GB of ram, the tests succeed. I think in the real world most users now
> have at least 32 GB of ram or more, while these virtual environments in
> Appveyor and/or Travis CI probably run their tests with much less
> overall virtual memory than an average user would have.

According to their websites, Appveyor VMs have a limit of around 6 GB
and Travis VMs have a limit around 8 GB for Windows and 4 GB for
Linux/MacOS.  Neither appear to be configurable in any way (but I am not
an expert), which is understandable.

I realise you're probably deliberately testing edge cases with
pathologic memory usage, but normally there'd be something seriously
wrong if a build comes remotely close to using that much.

While yes, these days usually developers do have 32 GB RAM or more, the
"popular" RAM size is still 16 GB or even less, and a lot of this space
would be intended for parallel compiles or VMs and any single compile
should assume it must use a lot less.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 9/20/2020 11:59 PM, Gavin Lambert via Boost wrote:

> On 21/09/2020 14:14, Edward Diener wrote:
>> Is it possible through .yml file for Appveyor and Travis CI to change
>> the amount of virtual memory when running a CI test. I am running CI
>> tests for Boost PP which heavily use the preprocessor, and some of the
>> extended tests I am running fail on both Appveyor and Travis CI  with
>> out of memory conditions while on my own local computer, with about 32
>> GB of ram, the tests succeed. I think in the real world most users now
>> have at least 32 GB of ram or more, while these virtual environments
>> in Appveyor and/or Travis CI probably run their tests with much less
>> overall virtual memory than an average user would have.
>
> According to their websites, Appveyor VMs have a limit of around 6 GB
> and Travis VMs have a limit around 8 GB for Windows and 4 GB for
> Linux/MacOS.  Neither appear to be configurable in any way (but I am not
> an expert), which is understandable.
>
> I realise you're probably deliberately testing edge cases with
> pathologic memory usage, but normally there'd be something seriously
> wrong if a build comes remotely close to using that much.

I agree, which makes me think that the compilers/preprocessors involved
limit the amount of memory why they allow themselves to use. But these
"out of memory" type of conditions are occurring in gcc, clang, and
vc++, so all three are limiting preprocessor memory in some way. Oh
well, I do warn the end-user of Boost PP about pushing some of these
limitations, but it is disappointing that the preprocessors can only
reflect dubious "out of memory" type conditions rather than error out on
what situations are really causing the preprocessors to fail.

>
> While yes, these days usually developers do have 32 GB RAM or more, the
> "popular" RAM size is still 16 GB or even less, and a lot of this space
> would be intended for parallel compiles or VMs and any single compile
> should assume it must use a lot less.

Even if 16 GB, these CIs limiting memory to 4 GB or 6 GB seem a
disservice to me.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 21/09/2020 16:20, Edward Diener wrote:
> Even if 16 GB, these CIs limiting memory to 4 GB or 6 GB seem a
> disservice to me.

While virtual memory is non-linear and weird (and virtual machine memory
even more so), it still approximately boils down to however much
physical RAM they can (affordably) fit onto the host, then divided by
how many VMs they want to run per host.

Giving each VM a larger memory allocation means you can fit fewer
concurrent VMs on the same hardware, meaning that you either need more
hardware or less concurrent users.  And more hardware is expensive,
especially at VM-host-server scale.

And these CIs have a lot of users who don't pay for anything.

So they have to draw a line in the sand somewhere.  They have apparently
decided that those particular limits work for most of their users (or at
least the ones that they care sufficiently about).

It's possible that they have some dedicated servers for paying customers
with looser limits.  But it's also possible that they don't, if it
hasn't been an issue in the past.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 9/21/2020 1:29 AM, Gavin Lambert via Boost wrote:

> On 21/09/2020 16:20, Edward Diener wrote:
>> Even if 16 GB, these CIs limiting memory to 4 GB or 6 GB seem a
>> disservice to me.
>
> While virtual memory is non-linear and weird (and virtual machine memory
> even more so), it still approximately boils down to however much
> physical RAM they can (affordably) fit onto the host, then divided by
> how many VMs they want to run per host.
>
> Giving each VM a larger memory allocation means you can fit fewer
> concurrent VMs on the same hardware, meaning that you either need more
> hardware or less concurrent users.  And more hardware is expensive,
> especially at VM-host-server scale.
>
> And these CIs have a lot of users who don't pay for anything.
>
> So they have to draw a line in the sand somewhere.  They have apparently
> decided that those particular limits work for most of their users (or at
> least the ones that they care sufficiently about).
>
> It's possible that they have some dedicated servers for paying customers
> with looser limits.  But it's also possible that they don't, if it
> hasn't been an issue in the past.

I understand that it is a free resource for Boost testing and that they
may not offer much in the way of virtual memory, as well as having
certain time limits when something is running. But nearly all my tests
for intensive functionality in Boost PP and VMD run fine to completion
on my local computer, using gcc, clang, and VC++ on Windows or Linux,
whereas in the CI testing with Appveyor and Travis CI there are numerous
"out of memory" and "process killed on timeouts" errors which prevents
successful CI tests results. So for my purposes of CI testing these CI
facilities are almost useless on the whole to tell me whether some
change is valid or not. That's fine. I just will pay little attention to
the overall result of the CI tests, since they do not reflect real world
results because of the CI limitations.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 25/09/2020 03:49, Edward Diener wrote:

> I understand that it is a free resource for Boost testing and that they
> may not offer much in the way of virtual memory, as well as having
> certain time limits when something is running. But nearly all my tests
> for intensive functionality in Boost PP and VMD run fine to completion
> on my local computer, using gcc, clang, and VC++ on Windows or Linux,
> whereas in the CI testing with Appveyor and Travis CI there are numerous
> "out of memory" and "process killed on timeouts" errors which prevents
> successful CI tests results. So for my purposes of CI testing these CI
> facilities are almost useless on the whole to tell me whether some
> change is valid or not. That's fine. I just will pay little attention to
> the overall result of the CI tests, since they do not reflect real world
> results because of the CI limitations.

Perhaps you should disable those specific tests for CI builds, while
keeping them for the normal test matrix?  That would keep both useful.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 9/24/2020 7:31 PM, Gavin Lambert via Boost wrote:

> On 25/09/2020 03:49, Edward Diener wrote:
>> I understand that it is a free resource for Boost testing and that
>> they may not offer much in the way of virtual memory, as well as
>> having certain time limits when something is running. But nearly all
>> my tests for intensive functionality in Boost PP and VMD run fine to
>> completion on my local computer, using gcc, clang, and VC++ on Windows
>> or Linux, whereas in the CI testing with Appveyor and Travis CI there
>> are numerous "out of memory" and "process killed on timeouts" errors
>> which prevents successful CI tests results. So for my purposes of CI
>> testing these CI facilities are almost useless on the whole to tell me
>> whether some change is valid or not. That's fine. I just will pay
>> little attention to the overall result of the CI tests, since they do
>> not reflect real world results because of the CI limitations.
>
> Perhaps you should disable those specific tests for CI builds, while
> keeping them for the normal test matrix?  That would keep both useful.

How do I tell in my jamfile when CI testing is occurring in order to
disable the tests for CI builds.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
On 25/09/2020 15:39, Edward Diener wrote:
>> Perhaps you should disable those specific tests for CI builds, while
>> keeping them for the normal test matrix?  That would keep both useful.
>
> How do I tell in my jamfile when CI testing is occurring in order to
> disable the tests for CI builds.

I'm not familiar with Jamfiles (or CI, for that matter), so perhaps
someone else should chime in at this point...

... but it does look like both Travis and Appveyor set the environment
variable CI when building (they both set it to "true", but not with
identical case; checking it against empty value is probably sufficient
though), so you can possibly check that.

(They also set TRAVIS or APPVEYOR respectively, if you need to
distinguish them.)

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
Mere moments ago, quoth I:

>> How do I tell in my jamfile when CI testing is occurring in order to
>> disable the tests for CI builds.
>
> I'm not familiar with Jamfiles (or CI, for that matter), so perhaps
> someone else should chime in at this point...
>
> ... but it does look like both Travis and Appveyor set the environment
> variable CI when building (they both set it to "true", but not with
> identical case; checking it against empty value is probably sufficient
> though), so you can possibly check that.
>
> (They also set TRAVIS or APPVEYOR respectively, if you need to
> distinguish them.)

FWIW, Boost.Gil [1] and Boost.Phoenix [2] appears to have an example of
changing compiler settings based on CI or not, which you could crib from.

Most of the other libraries appear to do it differently -- they group
the tests into different suites and their travis/appveyor files specify
particular suites to run, or they have one test "outside" suites, with
comments implying that the test matrix runs all of them but CI only runs
the ones not in suites by default (I don't know if that's an accurate
statement).

[1] https://github.com/boostorg/gil/blob/master/test/Jamfile
[2] https://github.com/boostorg/phoenix/blob/master/test/Jamfile

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 9/25/20 6:39 AM, Edward Diener via Boost wrote:

> On 9/24/2020 7:31 PM, Gavin Lambert via Boost wrote:
>> On 25/09/2020 03:49, Edward Diener wrote:
>>> I understand that it is a free resource for Boost testing and that
>>> they may not offer much in the way of virtual memory, as well as
>>> having certain time limits when something is running. But nearly all
>>> my tests for intensive functionality in Boost PP and VMD run fine to
>>> completion on my local computer, using gcc, clang, and VC++ on
>>> Windows or Linux, whereas in the CI testing with Appveyor and Travis
>>> CI there are numerous "out of memory" and "process killed on
>>> timeouts" errors which prevents successful CI tests results. So for
>>> my purposes of CI testing these CI facilities are almost useless on
>>> the whole to tell me whether some change is valid or not. That's
>>> fine. I just will pay little attention to the overall result of the
>>> CI tests, since they do not reflect real world results because of the
>>> CI limitations.
>>
>> Perhaps you should disable those specific tests for CI builds, while
>> keeping them for the normal test matrix?  That would keep both useful.
>
> How do I tell in my jamfile when CI testing is occurring in order to
> disable the tests for CI builds.

You can set special environment variables in the CI config file, which
you then test in the Jamfile. For example:

https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/appveyor.yml#L52
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/appveyor.yml#L122
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/test/Jamfile.v2#L72-L84
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/test/Jamfile.v2#L102-L105

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, 25 Sep 2020 at 07:19, Gavin Lambert via Boost
<[hidden email]> wrote:

>
> Mere moments ago, quoth I:
> >> How do I tell in my jamfile when CI testing is occurring in order to
> >> disable the tests for CI builds.
> >
> > I'm not familiar with Jamfiles (or CI, for that matter), so perhaps
> > someone else should chime in at this point...
> >
> > ... but it does look like both Travis and Appveyor set the environment
> > variable CI when building (they both set it to "true", but not with
> > identical case; checking it against empty value is probably sufficient
> > though), so you can possibly check that.
> >
> > (They also set TRAVIS or APPVEYOR respectively, if you need to
> > distinguish them.)
>
> FWIW, Boost.Gil [1] and Boost.Phoenix [2] appears to have an example of
> changing compiler settings based on CI or not, which you could crib from.

For GIL, I was piggybacking on Andrey Semashev's solution
that Andrey just suggested in his response.
It's simple and works well.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 9/25/2020 4:15 AM, Andrey Semashev via Boost wrote:

> On 9/25/20 6:39 AM, Edward Diener via Boost wrote:
>> On 9/24/2020 7:31 PM, Gavin Lambert via Boost wrote:
>>> On 25/09/2020 03:49, Edward Diener wrote:
>>>> I understand that it is a free resource for Boost testing and that
>>>> they may not offer much in the way of virtual memory, as well as
>>>> having certain time limits when something is running. But nearly all
>>>> my tests for intensive functionality in Boost PP and VMD run fine to
>>>> completion on my local computer, using gcc, clang, and VC++ on
>>>> Windows or Linux, whereas in the CI testing with Appveyor and Travis
>>>> CI there are numerous "out of memory" and "process killed on
>>>> timeouts" errors which prevents successful CI tests results. So for
>>>> my purposes of CI testing these CI facilities are almost useless on
>>>> the whole to tell me whether some change is valid or not. That's
>>>> fine. I just will pay little attention to the overall result of the
>>>> CI tests, since they do not reflect real world results because of
>>>> the CI limitations.
>>>
>>> Perhaps you should disable those specific tests for CI builds, while
>>> keeping them for the normal test matrix?  That would keep both useful.
>>
>> How do I tell in my jamfile when CI testing is occurring in order to
>> disable the tests for CI builds.
>
> You can set special environment variables in the CI config file, which
> you then test in the Jamfile. For example:
>
> https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/appveyor.yml#L52 
>
> https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/appveyor.yml#L122 
>
> https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/test/Jamfile.v2#L72-L84 
>
> https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17fc/test/Jamfile.v2#L102-L105 

Thanks ! I forgot about the os.environ functionality in Boost Build,
although I have probably used it myself numerous times <g>.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Changing virtual memory amount in Appveyor and Travis CI tests

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 9/25/2020 1:17 AM, Gavin Lambert via Boost wrote:

> Mere moments ago, quoth I:
>>> How do I tell in my jamfile when CI testing is occurring in order to
>>> disable the tests for CI builds.
>>
>> I'm not familiar with Jamfiles (or CI, for that matter), so perhaps
>> someone else should chime in at this point...
>>
>> ... but it does look like both Travis and Appveyor set the environment
>> variable CI when building (they both set it to "true", but not with
>> identical case; checking it against empty value is probably sufficient
>> though), so you can possibly check that.
>>
>> (They also set TRAVIS or APPVEYOR respectively, if you need to
>> distinguish them.)
>
> FWIW, Boost.Gil [1] and Boost.Phoenix [2] appears to have an example of
> changing compiler settings based on CI or not, which you could crib from.
>
> Most of the other libraries appear to do it differently -- they group
> the tests into different suites and their travis/appveyor files specify
> particular suites to run, or they have one test "outside" suites, with
> comments implying that the test matrix runs all of them but CI only runs
> the ones not in suites by default (I don't know if that's an accurate
> statement).
>
> [1] https://github.com/boostorg/gil/blob/master/test/Jamfile
> [2] https://github.com/boostorg/phoenix/blob/master/test/Jamfile

Thanks ! I also have different aliases ( suites ) in my jamfiles, so I
can use that also.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost