Docker environment(s) for Boost Development

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Docker environment(s) for Boost Development

Boost - Dev mailing list
I submitted a pull request into the superproject containing some docker
scripts.  The intention of this is to simplify managing a docker build
environment for boost:

https://github.com/boostorg/boost/pull/184

I'm wondering if folks will find this useful, and whether folks think this
should be in its own submodule instead of being in boostorg/boost.  I only
have an Ubuntu Bionic environment in this pull request but it wouldn't be
that difficult to add debian, centos, and others, as well as various
versions of each.

I have tested it on Unix and on Windows.  On Unix the performance is quite
good overall - but on Windows since they use SMB to do the sharing it is
really slow (I would say a good 20 times slower between typing "b2" and
getting the first output as it scans through all the files.  I would
recommend continuing to use a virtual machine on Windows Hosts for unix
boost development, but this option is there and hopefully they will improve
things.

This partially came out of the discussion we had a number of months back
about how we handle third party dependencies.  For the most part one has to
look at README files in various locations to learn about them.  This is an
attempt to pull all the dependencies together for a complete build into one
place (the Dockerfile).

One day when Microsoft supports Visual Studio compilation inside a
reasonably sized windows container (i.e. based on nanoserver and not
Windows Server Core), I'd like to add that as well since managing the
dependencies on Windows is a bit harder (you usually have to build them).

Thanks,

Jim

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost <
[hidden email]> wrote:

> I submitted a pull request into the superproject containing some docker
> scripts.  The intention of this is to simplify managing a docker build
> environment for boost:
>
> https://github.com/boostorg/boost/pull/184
>
> I'm wondering if folks will find this useful, and whether folks think this
> should be in its own submodule instead of being in boostorg/boost.  I only
> have an Ubuntu Bionic environment in this pull request but it wouldn't be
> that difficult to add debian, centos, and others, as well as various
> versions of each.
>
> I have tested it on Unix and on Windows.  On Unix the performance is quite
> good overall - but on Windows since they use SMB to do the sharing it is
> really slow (I would say a good 20 times slower between typing "b2" and
> getting the first output as it scans through all the files.  I would
> recommend continuing to use a virtual machine on Windows Hosts for unix
> boost development, but this option is there and hopefully they will improve
> things.
>
> This partially came out of the discussion we had a number of months back
> about how we handle third party dependencies.  For the most part one has to
> look at README files in various locations to learn about them.  This is an
> attempt to pull all the dependencies together for a complete build into one
> place (the Dockerfile).
>
> One day when Microsoft supports Visual Studio compilation inside a
> reasonably sized windows container (i.e. based on nanoserver and not
> Windows Server Core), I'd like to add that as well since managing the
> dependencies on Windows is a bit harder (you usually have to build them).
>
> Thanks,
>
> Jim
>

I just wanted to mention that I've got a stable of boost docker images
available at:
https://hub.docker.com/r/teeks99/boost-cpp-docker/

built from the git repo:
https://github.com/teeks99/boost-cpp-docker

I initially created these as the source for the linux regression runners
that I manage (teeks99-02*, teeks99-03*), but have used them for a lot
more. They are more focused on having older versions of compilers (gcc and
clang) working, and I think is largely complimentary to what you have here.
Just wanted to mention it in this thread.


Another thought, it might be a nice idea to take what you've done for the
baseline to create an environment, and for each release build the release
version of boost inside it and publish it to docker hub. This would give
users a way to quickly get a boost development environment, without having
to wait for the distros to package (and mangle) the binaries.

Tom

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
Boost - Dev mailing list wrote
> On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost <

> boost@.boost

>> wrote:
>> I submitted a pull request into the superproject containing some docker
>> scripts.  The intention of this is to simplify managing a docker build
>> environment for boost:
>>
>> https://github.com/boostorg/boost/pull/184
>>
>> [...]
>> This partially came out of the discussion we had a number of months back
>> about how we handle third party dependencies.  For the most part one has
>> to
>> look at README files in various locations to learn about them.  This is
>> an
>> attempt to pull all the dependencies together for a complete build into
>> one
>> place (the Dockerfile).
>
> I just wanted to mention that I've got a stable of boost docker images
> available at:
> https://hub.docker.com/r/teeks99/boost-cpp-docker/
>
> built from the git repo:
> https://github.com/teeks99/boost-cpp-docker
>
> I initially created these as the source for the linux regression runners
> that I manage (teeks99-02*, teeks99-03*)

Tom,

Do you also automated running the regression tests?

Could you explain your workflow with those docker images
you perform to run the complete Boost regression tests suite?

Best regards,
Mateusz Loskot



-----
--
Mateusz Loskot, http://mateusz.loskot.net
--
Sent from: http://boost.2283326.n4.nabble.com/Boost-Dev-f2600599.html

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Boost - Dev mailing list wrote

> The intention of this is to simplify managing a docker build
> environment for boost
> [...]
> This partially came out of the discussion we had a number of months back
> about how we handle third party dependencies.  For the most part one has
> to
> look at README files in various locations to learn about them.  This is an
> attempt to pull all the dependencies together for a complete build into
> one
> place (the Dockerfile).

James,

TBH, to me, this gives quite a little idea what is the actual purpose and
target audience of the docker images.

Is it for Boost users, to provide them with Boost itself?
Is it for Boost contributors, to provide them with development and testing
environment?
What do you mean by "all the dependencies", Boost itself or deps required by
Boost?

Best regards,
Mateusz Loskot



-----
--
Mateusz Loskot, http://mateusz.loskot.net
--
Sent from: http://boost.2283326.n4.nabble.com/Boost-Dev-f2600599.html

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Tue, May 8, 2018 at 1:59 PM, James E. King, III via Boost
<[hidden email]> wrote:
> One day when Microsoft supports Visual Studio compilation inside a
> reasonably sized windows container (i.e. based on nanoserver and not
> Windows Server Core), I'd like to add that as well since managing the
> dependencies on Windows is a bit harder (you usually have to build them).

./vcpkg install zlib isn't that hard is it?


--
Olaf

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Wed, Jul 25, 2018 at 5:41 AM Mateusz Loskot via Boost <
[hidden email]> wrote:

> Boost - Dev mailing list wrote
> > The intention of this is to simplify managing a docker build
> > environment for boost
> > [...]
> > This partially came out of the discussion we had a number of months back
> > about how we handle third party dependencies.  For the most part one has
> > to
> > look at README files in various locations to learn about them.  This is
> an
> > attempt to pull all the dependencies together for a complete build into
> > one
> > place (the Dockerfile).
>
> James,
>
> TBH, to me, this gives quite a little idea what is the actual purpose and
> target audience of the docker images.
>
> Is it for Boost users, to provide them with Boost itself?
> Is it for Boost contributors, to provide them with development and testing
> environment?
> What do you mean by "all the dependencies", Boost itself or deps required
> by
> Boost?
>
> Best regards,
> Mateusz Loskot
>
>
It allows for easier boost development and release engineering by providing
docker images that are capable of building, testing, and packaging boost on
a variety of platforms.  Right now two flavors of Debian and Ubuntu each are
provided, and I'd like to get some RedHat variants in there.

Therefore it is really designed for folks working on boost in some way, not
those
consuming it for their apps, although that is possible and will make it
easier
for folks that must build from source to meet requirements for their
projects.

The dependencies are the third party libraries that different boost modules
use - some optionally, and some required, but all are included.  In
addition the
environment to build the documentation is set up as part of the docker image
so one has a complete build environment, however I'm not aware of any
distro-
specific packaging provided by boost itself, so that exercise is left up to
the
distro consumers.

It came about because I was setting up my 5th or 6th boost development
virtual
machine due to hardware changes and I decided I didn't feel like going
through
it again.  Now I can just install docker, build the docker image for the
distro I want
to work in, and off I go.  I wasn't aware of the teeks99 dockerfile
repository at
the time.  Given clang and gcc allow for layered installation, it would be
easy to
modify the dockerfiles here to include all of those compilers in one image,
and
then folks could use "toolset=" to select the one they want - however the
clang
toolset selector logic in b2 is substandard compared to gcc, so that needs
to be
worked on (I filed an issue in Boost.Build for this).

Ideally I'd like to be able to provide a windows based docker build
environment
to make it easier to build on windows with all the required and optional
third party
libraries needed there.  It's a lot more work to set up a windows build
environment,
but until Microsoft figures out they need to provide Visual Studio Build
Tools that
work on Windows Nano (<=600MB) instead of Windows Core (>3 GB), it's not
that compelling.  Of course, a complete environment for windows includes
all of
the compilers (MSVC 10, 11, 12, 14, 14.*) ideally so that one can build
against
all of them to make sure things work.  This is what we've been leveraging in
Appveyor through our repositories with CI.

Having this (docker images and helper scripts) available in the
superproject
seems appropriate.  Certainly the docker files, less certain about the shell
scripts, since people usually customize them anyway, however a fair amount
of work went into discovering the right set of command-line options to
docker
that allow one to debug inside the container effectively.

The workflow I use is to have one shell in the container and one outside the
container.  The shell outside is used for git commands where your git
identity
is stored ( , although we could add another docker virtual mount that puts
your
user-specific git config into the container.  Interesting...)

jking@ubuntu:~$ cd boost
jking@ubuntu:~/boost$ docker/run.sh ubuntu bionic
boost@286b1533dac3:/boost$ cd libs/uuid
boost@286b1533dac3:/boost/libs/uuid$ ../../b2 -q

By default in the Ubuntu Bionic environment this will use the default
toolset
gcc which is gcc-7.3.0 in that environment.  If you use the "debian stretch"
environment it has gcc-6.3.0 in it.  These are the default compilers that
come
with that distro and it is what consumers of boost will likely use.

- Jim

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
On 25 July 2018 at 12:50, James E. King III via Boost
<[hidden email]> wrote:

> On Wed, Jul 25, 2018 at 5:41 AM Mateusz Loskot via Boost <
> [hidden email]> wrote:
>
>> Boost - Dev mailing list wrote
>> > The intention of this is to simplify managing a docker build
>> > environment for boost
>> > [...]
>> > This partially came out of the discussion we had a number of months back
>> > about how we handle third party dependencies.  For the most part one has
>> > to
>> > look at README files in various locations to learn about them.  This is an
>> > attempt to pull all the dependencies together for a complete build into
>> > one
>> > place (the Dockerfile).
>>
>> James,
>>
>> TBH, to me, this gives quite a little idea what is the actual purpose and
>> target audience of the docker images.
>>
>> Is it for Boost users, to provide them with Boost itself?
>> Is it for Boost contributors, to provide them with development and testing
>> environment?
>> What do you mean by "all the dependencies", Boost itself or deps required
>> by Boost?
>
> It allows for easier boost development and release engineering by providing
> docker images that are capable of building, testing, and packaging boost on
> a variety of platforms.  Right now two flavors of Debian and Ubuntu each are
> provided, and I'd like to get some RedHat variants in there.
>
> Therefore it is really designed for folks working on boost in some way

That is exactly what I'm looking for.
The goal seems to be similar to Tom's.

> The dependencies are the third party libraries that different boost modules
> use - some optionally, and some required, but all are included.

eg. libjpeg, libpng for GIL, zlib for others, etc.
Makes sense.

> It came about because I was setting up my 5th or 6th boost development virtual
> machine due to hardware changes and I decided I didn't feel like going
> through it again.

I've been using Vagrant for long time, but docker seems suitable for
that purpose too,
especially after I experienced it via multi-docker build workflow on
CircleCI 2.0
(already used for Geometry and GIL).

> I wasn't aware of the teeks99 dockerfile repository at the time.
> [...]


Ideally, if the community could agree to maintain single boostorg/docker

I'd like to be able to pull docker image(s), run regression tests and
submit to the Boost waterfall.
I'd also like to get some *BSD images - especially after long standing
failures of GIL on those systems.
Ideally, if I could run a script overnight that pulls images one by one
runs the tests and submits the results. I'm sure such automation is feasible.

> It's a lot more work to set up a windows build environment,
> but until Microsoft figures out they need to provide Visual Studio Build
> Tools that work on Windows Nano (<=600MB)
> [...]

I've been using Vagrant to provide various Windows-based development
environments, and I feel the same pain.

> Having this (docker images and helper scripts) available in the
> superproject seems appropriate.

Yes.

> Certainly the docker files, less certain about the shell
> scripts, since people usually customize them anyway, however a fair amount
> of work went into discovering the right set of command-line options to
> docker that allow one to debug inside the container effectively.
> [...]

If there is boosorg/docker, then there will be documentation on boost.org.
I'll try to help.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Wed, Jul 25, 2018 at 4:17 AM, Mateusz Loskot via Boost <
[hidden email]> wrote:

> Boost - Dev mailing list wrote
> > On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost <
>
> > boost@.boost
>
> >> wrote:
> >> I submitted a pull request into the superproject containing some docker
> >> scripts.  The intention of this is to simplify managing a docker build
> >> environment for boost:
> >>
> >> https://github.com/boostorg/boost/pull/184
> >>
> >> [...]
> >> This partially came out of the discussion we had a number of months back
> >> about how we handle third party dependencies.  For the most part one has
> >> to
> >> look at README files in various locations to learn about them.  This is
> >> an
> >> attempt to pull all the dependencies together for a complete build into
> >> one
> >> place (the Dockerfile).
> >
> > I just wanted to mention that I've got a stable of boost docker images
> > available at:
> > https://hub.docker.com/r/teeks99/boost-cpp-docker/
> >
> > built from the git repo:
> > https://github.com/teeks99/boost-cpp-docker
> >
> > I initially created these as the source for the linux regression runners
> > that I manage (teeks99-02*, teeks99-03*)
>
> Tom,
>
> Do you also automated running the regression tests?
>
> Could you explain your workflow with those docker images
> you perform to run the complete Boost regression tests suite?
>
> Best regards,
> Mateusz Loskot
>

I do use those images in the regression tests that I run; teeks99-02-*,
teeks99-03-* . Those images are setup in dockerhub, so when their upstream
images (typically the ubuntu one that was mainstream when that version of
the compiler was released) get updates, docker hub will automatically
rebuild them with the updates.

As far as new versions of the toolset (e.g. when clang drops a .1 build)
that is up to me to re-run the images. Which I'm often forgetful about
doing. For completely new versions (e.g. gcc7 -> gcc8) I have to manually
make up a new config.

I didn't mention it above, but I have two separate docker hub repos for
keeping the gcc and clang toolsets. These don't have anything boost
specific in it (even the boost-cpp-docker doesn't have that much, just some
specific dependencies), and I've used them for other purposes. They can be
found at:
https://github.com/teeks99/gcc-ubuntu-docker
https://github.com/teeks99/clang-ubuntu-docker

As far as how I use them in the regression tests, I've got a bit of tooling
setup here:
https://github.com/teeks99/boost-build/tree/master/Regression

It is mostly just a couple python scripts that continuously cycle between
configurations and the lists of configs that I want to run on each machine,
windows and linux.

Tom

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Docker environment(s) for Boost Development

Boost - Dev mailing list
On 26 July 2018 at 01:36, Tom Kent via Boost <[hidden email]> wrote:

> On Wed, Jul 25, 2018 at 4:17 AM, Mateusz Loskot via Boost <[hidden email]> wrote:
>> Boost - Dev mailing list wrote
>> > On Tue, May 8, 2018 at 6:59 AM, James E. King, III via Boost <boost@.boost> wrote:
>> >> I submitted a pull request into the superproject containing some docker
>> >> scripts.  The intention of this is to simplify managing a docker build
>> >> environment for boost:
>> >>
>> >> https://github.com/boostorg/boost/pull/184
>> >>
>> >> [...]
>> >> This partially came out of the discussion we had a number of months back
>> >> about how we handle third party dependencies.  For the most part one has
>> >> to
>> >> look at README files in various locations to learn about them.  This is
>> >> an
>> >> attempt to pull all the dependencies together for a complete build into
>> >> one
>> >> place (the Dockerfile).
>> >
>> > I just wanted to mention that I've got a stable of boost docker images
>> > available at:
>> > https://hub.docker.com/r/teeks99/boost-cpp-docker/
>> >
>> > built from the git repo:
>> > https://github.com/teeks99/boost-cpp-docker
>> >
>> > I initially created these as the source for the linux regression runners
>> > that I manage (teeks99-02*, teeks99-03*)
>>
>> Tom,
>>
>> Do you also automated running the regression tests?
>>
>> Could you explain your workflow with those docker images
>> you perform to run the complete Boost regression tests suite?
>
>
> [...]
> As far as how I use them in the regression tests, I've got a bit of tooling
> setup here:
> https://github.com/teeks99/boost-build/tree/master/Regression
>
> It is mostly just a couple python scripts that continuously cycle between
> configurations and the lists of configs that I want to run on each machine,
> windows and linux.

This looks like something I've been looking for/thinking of.
I'm going to try it out in action soonish.

Thanks for sharing.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost