[testing][regressions] All my regressions testers failing due to connection errors to GitHub

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[testing][regressions] All my regressions testers failing due to connection errors to GitHub

Boost - Dev mailing list
Hi,

All my 4 testers (I have 4 machines running several Mingw targets) are
always failing pulling new versions from GitHub. This problem started
some days ago. I've tried to reset all machines, network equipment,
disabling antivirus, firewall, modified http attributes, modified the
regression script to use git protocol instead of https, etc. with no luck.

I'm using Windows machines with official Git. Any clue on what's going
on? Maybe Github recently does not like so many submodules?

Example of failure (several libraries can fail during "git pull
"--recurse-submodules""):

-------------------
-------------------
-------------------

Fetching submodule libs/io
fatal: unable to access 'https://github.com/boostorg/io.git/': OpenSSL
SSL_read: SSL_ERROR_SYSCALL, errno 10054

Then:

# git_source_checkout failed with message "GIT command "git pull
"--recurse-submodules"" failed with code 1"
# Retrying (5 more attempts).
# Executing GIT command: c:\boost\develop\boost_root> git submodule
foreach "git submodule deinit ."

-------------------
-------------------

All retries fail and the script fails with error:

-------------------
-------------------
-------------------

# git_source_checkout failed with message "GIT command "git pull
"--recurse-submodules"" failed with code 1"
# Giving up.
Traceback (most recent call last):
    File "run.py", line 71, in <module>
      runner(root)
    File "c:\boost\develop\boost_regression_src\regression.py", line
286, in __init__
      self.main()
    File "c:\boost\develop\boost_regression_src\regression.py", line
623, in main
      getattr(self,action_m)()
    File "c:\boost\develop\boost_regression_src\regression.py", line
578, in command_regression
      self.command_get_source()
    File "c:\boost\develop\boost_regression_src\regression.py", line
330, in command_get_source
      self.retry(self.git_source_checkout)
    File "c:\boost\develop\boost_regression_src\regression.py", line
662, in retry
      return f()
    File "c:\boost\develop\boost_regression_src\regression.py", line
875, in git_source_checkout
      self.git_checkout(git_info['boost'], self.git_branch(), clean)
    File "c:\boost\develop\boost_regression_src\regression.py", line
849, in git_checkout
      self.git_command( 'pull', '--recurse-submodules' )
    File "c:\boost\develop\boost_regression_src\regression.py", line
834, in git_command
      raise Exception( 'GIT command "%s" failed with code %d' %
(git_cli, rc) )
Exception: GIT command "git pull "--recurse-submodules"" failed with code 1
END   24/05/2018 15:28:37,46

-------------------
-------------------
-------------------

Errors are never in the same libraries, very few times the checkout
succeeds and a regression is run. Anyone else having these problems?

I've attached the log of one failure in case someone can see any clue. I
really don't see how to solve this, so I won't be able to run regressions.

Best,

Ion


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

develop_gcc-4.6c++11_output.zip (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [testing][regressions] All my regressions testers failing due to connection errors to GitHub

Boost - Dev mailing list
On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost <
[hidden email]> wrote:

> Hi,
>
> All my 4 testers (I have 4 machines running several Mingw targets) are
> always failing pulling new versions from GitHub. This problem started some
> days ago. I've tried to reset all machines, network equipment, disabling
> antivirus, firewall, modified http attributes, modified the regression
> script to use git protocol instead of https, etc. with no luck.
>
> I'm using Windows machines with official Git. Any clue on what's going on?
> Maybe Github recently does not like so many submodules?
>
> Example of failure (several libraries can fail during "git pull
> "--recurse-submodules""):
>
> -------------------
> -------------------
> -------------------
>
> Fetching submodule libs/io
> fatal: unable to access 'https://github.com/boostorg/io.git/': OpenSSL
> SSL_read: SSL_ERROR_SYSCALL, errno 10054
>
> Then:
>
> # git_source_checkout failed with message "GIT command "git pull
> "--recurse-submodules"" failed with code 1"
> # Retrying (5 more attempts).
> # Executing GIT command: c:\boost\develop\boost_root> git submodule
> foreach "git submodule deinit ."
>
> -------------------
> -------------------
>
> All retries fail and the script fails with error:
>
> -------------------
> -------------------
> -------------------
>
> # git_source_checkout failed with message "GIT command "git pull
> "--recurse-submodules"" failed with code 1"
> # Giving up.
> Traceback (most recent call last):
>    File "run.py", line 71, in <module>
>      runner(root)
>    File "c:\boost\develop\boost_regression_src\regression.py", line 286,
> in __init__
>      self.main()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 623,
> in main
>      getattr(self,action_m)()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 578,
> in command_regression
>      self.command_get_source()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 330,
> in command_get_source
>      self.retry(self.git_source_checkout)
>    File "c:\boost\develop\boost_regression_src\regression.py", line 662,
> in retry
>      return f()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 875,
> in git_source_checkout
>      self.git_checkout(git_info['boost'], self.git_branch(), clean)
>    File "c:\boost\develop\boost_regression_src\regression.py", line 849,
> in git_checkout
>      self.git_command( 'pull', '--recurse-submodules' )
>    File "c:\boost\develop\boost_regression_src\regression.py", line 834,
> in git_command
>      raise Exception( 'GIT command "%s" failed with code %d' % (git_cli,
> rc) )
> Exception: GIT command "git pull "--recurse-submodules"" failed with code 1
> END   24/05/2018 15:28:37,46
>
> -------------------
> -------------------
> -------------------
>
> Errors are never in the same libraries, very few times the checkout
> succeeds and a regression is run. Anyone else having these problems?
>
> I've attached the log of one failure in case someone can see any clue. I
> really don't see how to solve this, so I won't be able to run regressions.
>
> Best,
>
> Ion
>
I've been having horrible problems getting all the submodules from github,
that look similar to what you've got going. I'm not sure where the issue
is, but I see it across linux and windows boxes at my house, but not when I
run in a cloud data center. I'm thinking it has something to do with my
connection (AT&T), but I'm not sure.

I ended up mirroring all the repos to a local machine, with a cron job to
update them every 15 min) and then cloning from that for the regression
runs. The script I use for that can be found here:
https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostorg.py

For more backgroun, I was seeing the same failed connections over https or
ssh. I got the script failure when even one of the submoduels failed to
update...which I guess is necessary. Now with my script almost all runs one
of the repos fails to fetch, but since they are all just standalone repos,
that failures doesn't cause an overall failure.

Tom

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [testing][regressions] All my regressions testers failing due to connection errors to GitHub

Boost - Dev mailing list
On 26/05/2018 19:08, Tom Kent via Boost wrote:
> On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost <

> I've been having horrible problems getting all the submodules from github,
> that look similar to what you've got going. I'm not sure where the issue
> is, but I see it across linux and windows boxes at my house, but not when I
> run in a cloud data center. I'm thinking it has something to do with my
> connection (AT&T), but I'm not sure.

Looks like a GitHub problem, which is the worst case ;-) Maybe it thinks
it's a DoS attack or similar with so many connections.

> I ended up mirroring all the repos to a local machine, with a cron job to
> update them every 15 min) and then cloning from that for the regression
> runs. The script I use for that can be found here:
> https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostorg.py

Looks like I should do something similar. Many thanks for the script.

> For more backgroun, I was seeing the same failed connections over https or
> ssh. I got the script failure when even one of the submoduels failed to
> update...which I guess is necessary. Now with my script almost all runs one
> of the repos fails to fetch, but since they are all just standalone repos,
> that failures doesn't cause an overall failure.

I wonder if other GitHub projects have similar problems, but perhaps not
many projects have so many submodules. I'll have a look at your
workaround and try to do something, or else I won't be able to run
regression tests.

Many thanks!

Ion

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [testing][regressions] All my regressions testers failing due to connection errors to GitHub

Boost - Dev mailing list
On 05/27/18 12:22, Ion Gaztañaga via Boost wrote:

> On 26/05/2018 19:08, Tom Kent via Boost wrote:
>> On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost <
>
>> I've been having horrible problems getting all the submodules from
>> github,
>> that look similar to what you've got going. I'm not sure where the issue
>> is, but I see it across linux and windows boxes at my house, but not
>> when I
>> run in a cloud data center. I'm thinking it has something to do with my
>> connection (AT&T), but I'm not sure.
>
> Looks like a GitHub problem, which is the worst case ;-) Maybe it thinks
> it's a DoS attack or similar with so many connections.
>
>> I ended up mirroring all the repos to a local machine, with a cron job to
>> update them every 15 min) and then cloning from that for the regression
>> runs. The script I use for that can be found here:
>> https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostorg.py 
>>
>
> Looks like I should do something similar. Many thanks for the script.
>
>> For more backgroun, I was seeing the same failed connections over
>> https or
>> ssh. I got the script failure when even one of the submoduels failed to
>> update...which I guess is necessary. Now with my script almost all
>> runs one
>> of the repos fails to fetch, but since they are all just standalone
>> repos,
>> that failures doesn't cause an overall failure.
>
> I wonder if other GitHub projects have similar problems, but perhaps not
> many projects have so many submodules. I'll have a look at your
> workaround and try to do something, or else I won't be able to run
> regression tests.

Maybe try writing to GitHub support?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [testing][regressions] All my regressions testers failing due to connection errors to GitHub

Boost - Dev mailing list
On 27/05/2018 12:37, Andrey Semashev via Boost wrote:
>> I wonder if other GitHub projects have similar problems, but perhaps
>> not many projects have so many submodules. I'll have a look at your
>> workaround and try to do something, or else I won't be able to run
>> regression tests.
>
> Maybe try writing to GitHub support?

Working on it.

Best,

Ion

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost