[all] Request for out of the box visibility support

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[all] Request for out of the box visibility support

Boost - Dev mailing list
I'd like to draw attention to a problem with Boost binaries for Linux.

There's an awesome -fvisibility=hidden flag for Linux compilers that
improves load times, performance, size of binaries and reduces the
chance of symbol collisions. More info at
https://gcc.gnu.org/wiki/Visibility .

Unfortunately, most of the Boost libraries do not set it by default:
- atomic
- chrono
- container
- context
- contract
- coroutine
- date_time
- exception
- fiber
- filesystem
- graph
- graph_parallel
- iostreams
- locale
- mpi
- program_options
- python
- random
- regex
- signals
- system
- test
- thread
- timer
- type_erasure
- wave

Moreover minority of the above libraries just do not work with the
flag. Users just can not run ./b2 cxxflags="-fvisibility=hidden"
because there's a chance that some library could stop linking.
Actually things are even more ugly. Linux distributions usually do not
tune the build flags for each package so at least Debian based
distributions build Boost with default flags. Users get suboptimal
builds.



If you're a maintainer of one of the above libraries *please do* the
following steps:
* Make sure that all the public symbols are accordingly marked with
appropriate BOOST_SYMBOL_* macro. Instruction is available here:
https://www.boost.org/doc/libs/1_68_0/libs/config/doc/html/boost_config/boost_macro_reference.html#boost_config.boost_macro_reference.macros_for_libraries_with_separate_source_code.macros_controlling_shared_library_symbol_visibility
* Turn on the visibility=hidden by default:
    * by adding <target-os>linux:<cxxflags>"-fvisibility=hidden" to
the Jamfile if you do not care much for antique compilers (Example
https://github.com/boostorg/stacktrace/blob/819f2b1c861dec7530372a990ecabab7f5fef48f/build/Jamfile.v2#L11
)
    * by using a more advanced technique for detecting the flag
availability (For example see
https://github.com/boostorg/math/blob/develop/build/Jamfile.v2#L20 or
https://github.com/boostorg/log/blob/develop/build/Jamfile.v2#L24 )


P.S.: I would appreciate any comments or updates on the feature request.
P.P.S.: Log, Math, Serialization (and Stacktrace) libraries already
use that flag by default. Many thanks!


--
Best regards,
Antony Polukhin

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On 08/17/18 09:21, Antony Polukhin via Boost wrote:

> I'd like to draw attention to a problem with Boost binaries for Linux.
>
> There's an awesome -fvisibility=hidden flag for Linux compilers that
> improves load times, performance, size of binaries and reduces the
> chance of symbol collisions. More info at
> https://gcc.gnu.org/wiki/Visibility .
>
> Unfortunately, most of the Boost libraries do not set it by default:
> - atomic
> - chrono
> - container
> - context
> - contract
> - coroutine
> - date_time
> - exception
> - fiber
> - filesystem
> - graph
> - graph_parallel
> - iostreams
> - locale
> - mpi
> - program_options
> - python
> - random
> - regex
> - signals
> - system
> - test
> - thread
> - timer
> - type_erasure
> - wave
>
> Moreover minority of the above libraries just do not work with the
> flag. Users just can not run ./b2 cxxflags="-fvisibility=hidden"
> because there's a chance that some library could stop linking.
> Actually things are even more ugly. Linux distributions usually do not
> tune the build flags for each package so at least Debian based
> distributions build Boost with default flags. Users get suboptimal
> builds.
>
>
>
> If you're a maintainer of one of the above libraries *please do* the
> following steps:
> * Make sure that all the public symbols are accordingly marked with
> appropriate BOOST_SYMBOL_* macro. Instruction is available here:
> https://www.boost.org/doc/libs/1_68_0/libs/config/doc/html/boost_config/boost_macro_reference.html#boost_config.boost_macro_reference.macros_for_libraries_with_separate_source_code.macros_controlling_shared_library_symbol_visibility
> * Turn on the visibility=hidden by default:
>      * by adding <target-os>linux:<cxxflags>"-fvisibility=hidden" to
> the Jamfile if you do not care much for antique compilers (Example
> https://github.com/boostorg/stacktrace/blob/819f2b1c861dec7530372a990ecabab7f5fef48f/build/Jamfile.v2#L11
> )
>      * by using a more advanced technique for detecting the flag
> availability (For example see
> https://github.com/boostorg/math/blob/develop/build/Jamfile.v2#L20 or
> https://github.com/boostorg/log/blob/develop/build/Jamfile.v2#L24 )
>
>
> P.S.: I would appreciate any comments or updates on the feature request.
> P.P.S.: Log, Math, Serialization (and Stacktrace) libraries already
> use that flag by default. Many thanks!

I wonder if we should update Boost.Build instead and set visibility to
hidden by default. For libraies that need other visibility we could
offer a property.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/16/18 11:21 PM, Antony Polukhin via Boost wrote:

This is a worthy idea.  Having a shared library/dll export only those
symbols which are required is a big improvement for the user.

But, there's much more to it than meets the eye.   I implemented this
for the serialization library as it was a huge PITA.

a) The syntax for marking up exported functions is different for
microsoft and other compilers.

b) when a library is built on a lot of nested/inherited classes it
becomes non-obvious what to export.

c) when a library depends upon another library, it can become even more
problematic.

d) when doing stuff like running gcc compiler creating shared library
under windows things can get confused.

e) Our testing matrix doesn't display or maintain different results base
on whether a test is being run with shared or static linking.  So the
value of the test results is much diminished.  There are so many
combinations that one must really depend upon the test results but these
don't have the required information.

f) There was a paper for the standardization committee which included a
proposal to help addressing this stuff - but it failed to gain enough
interest to get the problem addressed.  In fact, the whole issue of
dynamically loaded shared libraries raises a bunch of interesting
problems - are global static variables unique?, etc. .  The concept can
be very, very powerful in creating large extendable applications, but
the C++ language is silent on the whole issue.  It's a serious problem
but too unpleasant to actually address.

g) In the absense of any guidence from the standard and compilers not
being on the same page, there's an opportunity to create a boost library
of macros, documentation, and macros and who knows what else.  We have a
great starting point with John Maddox's work on the subject - which is
what I depend on.  But still, the situation is so messed up, I would
hope that can be expanded upon.

g) Actually it's unclear to me how the -fvisibility switch enters into
this.  VC doesn't have it.  I don't think it's actually necessary and
I'm not sure how it alters the functioning of the visibility.  I know,
read the rules.  But like a lot of stuff in C++ the understanding
provided by the baroque rules evaporates the minute one has to
investigate the next ball of yarn.  It's just not possible to understand
all of C++ at one time.


Robert Ramey

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, Aug 17, 2018, 11:10 Andrey Semashev via Boost <[hidden email]>
wrote:
<...>

> I wonder if we should update Boost.Build instead and set visibility to
> hidden by default. For libraies that need other visibility we could
> offer a property.
>

That would be a perfect solution!

>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, Aug 17, 2018, 19:38 Robert Ramey via Boost <[hidden email]>
wrote:

> On 8/16/18 11:21 PM, Antony Polukhin via Boost wrote:
>
> This is a worthy idea.  Having a shared library/dll export only those
> symbols which are required is a big improvement for the user.
>
> But, there's much more to it than meets the eye.   I implemented this
> for the serialization library as it was a huge PITA.
>
> a) The syntax for marking up exported functions is different for
> microsoft and other compilers.
>

Yes, but BOOST_SYMBOL_* macro help a lot.


b) when a library is built on a lot of nested/inherited classes it
> becomes non-obvious what to export.
>

Yes, which is not the point for most of the Boost libraries from the above
list.


<...>

> f) There was a paper for the standardization committee which included a
> proposal to help addressing this stuff - but it failed to gain enough
> interest to get the problem addressed.  In fact, the whole issue of
> dynamically loaded shared libraries raises a bunch of interesting
> problems - are global static variables unique?, etc. .  The concept can
> be very, very powerful in creating large extendable applications, but
> the C++ language is silent on the whole issue.  It's a serious problem
> but too unpleasant to actually address.
>

Yes, that was my paper. I was hoping to update it and present again in the
committee next year. Things do not go smooth with shared libraries in the
EWG, so for this year I've changed the approach and made a proposal on
dynamic loading (based on Boost.DLL) for LEWG.


<...>

> g) Actually it's unclear to me how the -fvisibility switch enters into
> this.  VC doesn't have it.  I don't think it's actually necessary and
> I'm not sure how it alters the functioning of the visibility.  I know,
> read the rules.  But like a lot of stuff in C++ the understanding
> provided by the baroque rules evaporates the minute one has to
> investigate the next ball of yarn.  It's just not possible to understand
> all of C++ at one time.
>

You may assume that VC has it enabled by default and there's no way to
disable it. There are actually many differences in border cases, but
trivial cases work in the same way.


Robert Ramey
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/17/2018 11:14 AM, Robert Ramey via Boost wrote:

> On 8/16/18 11:21 PM, Antony Polukhin via Boost wrote:
>
> This is a worthy idea.  Having a shared library/dll export only those
> symbols which are required is a big improvement for the user.
>
> But, there's much more to it than meets the eye.   I implemented this
> for the serialization library as it was a huge PITA.
>
> a) The syntax for marking up exported functions is different for
> microsoft and other compilers.
>
> b) when a library is built on a lot of nested/inherited classes it
> becomes non-obvious what to export.
>
> c) when a library depends upon another library, it can become even more
> problematic.
>
> d) when doing stuff like running gcc compiler creating shared library
> under windows things can get confused.
>
> e) Our testing matrix doesn't display or maintain different results base
> on whether a test is being run with shared or static linking.  So the
> value of the test results is much diminished.  There are so many
> combinations that one must really depend upon the test results but these
> don't have the required information.
>
> f) There was a paper for the standardization committee which included a
> proposal to help addressing this stuff - but it failed to gain enough
> interest to get the problem addressed.  In fact, the whole issue of
> dynamically loaded shared libraries raises a bunch of interesting
> problems - are global static variables unique?, etc. .  The concept can
> be very, very powerful in creating large extendable applications, but
> the C++ language is silent on the whole issue.  It's a serious problem
> but too unpleasant to actually address.

I do not see any difference, vis-a-vis the visibility problem discussed,
between dynamically loaded shared libraries or statically loaded shared
libraries. I agree with you that the problem of visibility is often more
complicated than what is supposed, given different compilers having
their own rules regarding visibility on different platforms.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On 8/18/18 6:58 AM, Edward Diener via Boost wrote:
>
> I do not see any difference, vis-a-vis the visibility problem discussed,
> between dynamically loaded shared libraries or statically loaded shared
> libraries.

visibility isn't really an issue with static libraries. visibility
decreases the number of externally visible symbols.  In linking a static
library this might decrease linking times - but I haven't noticed it and
I never received complaints about it.  I think that the BOOST visibility
macros are defined to nothing for static builds.

But for shared libraries it's a whole 'nuther issue.  The huge number of
symbols in libraries can make the shared libraries much, much larger and
slow down dynamic linking time considerably.  This is why this is a
worthy project.

 From personal experience in implementing this for the serialization
library (admittedly a more complex example), anyone who embarks upon
this will be disappointed at the amount of time it ends up consuming.

Robert Ramey

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On 08/18/18 18:19, Robert Ramey via Boost wrote:

> On 8/18/18 6:58 AM, Edward Diener via Boost wrote:
>>
>> I do not see any difference, vis-a-vis the visibility problem
>> discussed, between dynamically loaded shared libraries or statically
>> loaded shared libraries.
>
> visibility isn't really an issue with static libraries. visibility
> decreases the number of externally visible symbols.  In linking a static
> library this might decrease linking times - but I haven't noticed it and
> I never received complaints about it.

Visibility in static libraries doesn't affect linking times (i.e.
linking as the building stage, not the binary loading stage) because
hidden symbols are used in symbol resolution. But visibility markup is
still important in static libraries because it is preserved when the
final executable is produced. So, for example, if you mark a type with
default visibility and define it in a static library, you can link that
library into a shared library and the type would still be
default-visible*. This is useful when you want to ensure that a certain
type, like an exception, is public regardless of how it is compiled into
the final executable.

* I'm ignoring here linker scripts, which can further hide symbols on
the linking stage. AFAIK, we don't use linker scripts in Boost, but some
applications do.

>  I think that the BOOST visibility
> macros are defined to nothing for static builds.

No, BOOST_SYMBOL* macros are defined the same way regardless of the build.

> But for shared libraries it's a whole 'nuther issue.  The huge number of
> symbols in libraries can make the shared libraries much, much larger and
> slow down dynamic linking time considerably.  This is why this is a
> worthy project.

More importantly, hiding internal symbols ensures there won't be a
symbol clash. This is also a security matter.

>  From personal experience in implementing this for the serialization
> library (admittedly a more complex example), anyone who embarks upon
> this will be disappointed at the amount of time it ends up consuming.

I suspect there may be a lot of specifics in Boost.Serialization.
However, most of the time supporting hidden visibility is quite
straightforward and goes in line with supporting dllexport/dllimport on
Windows.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/18/2018 11:19 AM, Robert Ramey via Boost wrote:

> On 8/18/18 6:58 AM, Edward Diener via Boost wrote:
>>
>> I do not see any difference, vis-a-vis the visibility problem
>> discussed, between dynamically loaded shared libraries or statically
>> loaded shared libraries.
>
> visibility isn't really an issue with static libraries. visibility
> decreases the number of externally visible symbols.  In linking a static
> library this might decrease linking times - but I haven't noticed it and
> I never received complaints about it.  I think that the BOOST visibility
> macros are defined to nothing for static builds.

I was not referring to static libraries.

You used the phrase "dynamically loaded shared libraries". Did you just
mean "shared libraries" as opposed to "static libraries" ? Whether a
shared library is dynamically loaded or statically loaded it is still a
shared library. My point is that visibility is not affected by whether a
shared library is dynamically loaded or statically loaded.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On 8/18/18 10:46 AM, Edward Diener via Boost wrote:

> You used the phrase "dynamically loaded shared libraries". Did you just
> mean "shared libraries" as opposed to "static libraries" ? Whether a
> shared library is dynamically loaded or statically loaded it is still a
> shared library. My point is that visibility is not affected by whether a
> shared library is dynamically loaded or statically loaded.

To me, all shared libraries can be considered dynamically loaded.  If
not done explicitly they are loaded "automatically" at start up or
perhaps on first use.  The situation gets very interesting when they
have static variables in them.

There's a lot going on here.  Then there are issues related to contracts
and modules.  And when one shared library refers to another.  etc. ...

Robert Ramey


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/18/18 10:26 AM, Andrey Semashev via Boost wrote:
> So, for example, if you mark a type with
> default visibility and define it in a static library, you can link that
> library into a shared library and the type would still be
> default-visible*. This is useful when you want to ensure that a certain
> type, like an exception, is public regardless of how it is compiled into
> the final executable.
>

Hmmm - don't you run into problems linking static libraries into shared
libraries.  I'm pretty sure that this is a problem on VC compilers a
there are different versions of C runtime - one for dynamic linking and
another for static linking.  It's all very confusing to me.



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On Sat, Aug 18, 2018 at 10:54 PM Robert Ramey via Boost
<[hidden email]> wrote:

>
> On 8/18/18 10:26 AM, Andrey Semashev via Boost wrote:
> > So, for example, if you mark a type with
> > default visibility and define it in a static library, you can link that
> > library into a shared library and the type would still be
> > default-visible*. This is useful when you want to ensure that a certain
> > type, like an exception, is public regardless of how it is compiled into
> > the final executable.
>
> Hmmm - don't you run into problems linking static libraries into shared
> libraries.

There is no problem linking static libraries into shared libraries, I
do it every day.

> I'm pretty sure that this is a problem on VC compilers a
> there are different versions of C runtime - one for dynamic linking and
> another for static linking.  It's all very confusing to me.

MSVC issues with its runtime incompatibilities are fairly specific to
MSVC. And those issues are not related to static libraries per se
since the same problems can occur with shared libraries. Basically,
you should always link with the shared runtime, unless you dead sure
know what you're doing and really have to link with the static one.
I'm not even sure they still ship the static runtime.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Fri, Aug 17, 2018, 11:10 Andrey Semashev via Boost <[hidden email]>
wrote:
<...>

> I wonder if we should update Boost.Build instead and set visibility to
> hidden by default. For libraies that need other visibility we could
> offer a property.
>

I'm not experienced with b2 internals but I'd be glad to help with
testing/documenting/... this feature. Is there a champion to actually
implement it?

>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On Sun, Aug 19, 2018 at 7:48 AM Antony Polukhin via Boost <
[hidden email]> wrote:

> On Fri, Aug 17, 2018, 11:10 Andrey Semashev via Boost <
> [hidden email]>
> wrote:
> <...>
>
> > I wonder if we should update Boost.Build instead and set visibility to
> > hidden by default. For libraies that need other visibility we could
> > offer a property.
> >
>
>
I think a better solution is to remove link libraries where possible.  For
example
in date_time there isn't much of a case to keep the linked library.  If we
look at
the issue it causes like https://github.com/boostorg/date_time/pull/34, one
of the
solutions to that issue of visibility is to remove the link library
entirely.  I'd have to
recommend all libraries attempt to become header-only to avoid these issues.
Obviously that won't work for all libraries, but it's possible for a number
of them.
I opened https://github.com/boostorg/date_time/issues/85 to track that for
date_time.

- Jim

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 08/19/18 14:47, Antony Polukhin via Boost wrote:

> On Fri, Aug 17, 2018, 11:10 Andrey Semashev via Boost <[hidden email]>
> wrote:
> <...>
>
>> I wonder if we should update Boost.Build instead and set visibility to
>> hidden by default. For libraies that need other visibility we could
>> offer a property.
>>
>
> I'm not experienced with b2 internals but I'd be glad to help with
> testing/documenting/... this feature. Is there a champion to actually
> implement it?

https://github.com/boostorg/build/pull/331

I've only tested it on Linux, with gcc and clang. Please, test on other
platforms and see if it causes problems.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 17.08.18 08:21, Antony Polukhin via Boost wrote:

> I'd like to draw attention to a problem with Boost binaries for Linux.
>
> There's an awesome -fvisibility=hidden flag for Linux compilers that
> improves load times, performance, size of binaries and reduces the
> chance of symbol collisions. More info at
> https://gcc.gnu.org/wiki/Visibility .
>
> Unfortunately, most of the Boost libraries do not set it by default:
> - atomic
> - chrono
> - container
> - context
> - contract
> - coroutine
> - date_time
> - exception
> - fiber
> - filesystem
> - graph
> - graph_parallel
> - iostreams
> - locale
> - mpi
> - program_options
> - python
> - random
> - regex
> - signals
> - system
> - test
> - thread
> - timer
> - type_erasure
> - wave
>
> Moreover minority of the above libraries just do not work with the
> flag. Users just can not run ./b2 cxxflags="-fvisibility=hidden"
> because there's a chance that some library could stop linking.
> Actually things are even more ugly. Linux distributions usually do not
> tune the build flags for each package so at least Debian based
> distributions build Boost with default flags. Users get suboptimal
> builds.
>
>
>
> If you're a maintainer of one of the above libraries *please do* the
> following steps:
> * Make sure that all the public symbols are accordingly marked with
> appropriate BOOST_SYMBOL_* macro. Instruction is available here:
> https://www.boost.org/doc/libs/1_68_0/libs/config/doc/html/boost_config/boost_macro_reference.html#boost_config.boost_macro_reference.macros_for_libraries_with_separate_source_code.macros_controlling_shared_library_symbol_visibility
> * Turn on the visibility=hidden by default:
>      * by adding <target-os>linux:<cxxflags>"-fvisibility=hidden" to
> the Jamfile if you do not care much for antique compilers (Example
> https://github.com/boostorg/stacktrace/blob/819f2b1c861dec7530372a990ecabab7f5fef48f/build/Jamfile.v2#L11
> )
>      * by using a more advanced technique for detecting the flag
> availability (For example see
> https://github.com/boostorg/math/blob/develop/build/Jamfile.v2#L20 or
> https://github.com/boostorg/log/blob/develop/build/Jamfile.v2#L24 )
>
>
> P.S.: I would appreciate any comments or updates on the feature request.
> P.P.S.: Log, Math, Serialization (and Stacktrace) libraries already
> use that flag by default. Many thanks!

Hi all,

I personally was unable to reduce the chances for symbol clashes with
the -fvisibility=hidden flag alone: I had to combine it with a map file

{
   global: mexFunction;
   local: *;
};

and the option "-Wl,--exclude-libs,ALL" to achieve some sense of isolation.

On Linux, all symbols from all shared libraries are merged into one
unique namespace for a given process, whether they come from a directly
loaded shared library or an indirect dependency loaded later (shared or
static if the shared lib links to static libs).

If we consider only the issue of symbol clashes, to my experience,
hiding symbols does not help much, as some symbols are still pulled and
merged with little to no control on it. I rather found that
counter-intuitive as one may think that hiding symbols gives the same
namespace isolation for shared libraries as we have on Windows, while it
is not the case at all. In case of a symbol clash, we still have a hard
time debugging.

Also, at that time I had all those issues, it seemed that all libraries
were compiled with weak symbols definitions, which made the problem even
more difficult to debug. I've found this "STV_PROTECTED" attribute on
the visibility (see
https://www.ibm.com/developerworks/aix/library/au-aix-symbol-visibility/index.html)
but I do not know how to use it properly.

All in all,
- Linux is a nightmare for symbol clashes
- I'd love to learn a good way of doing things to avoid clashes, please
educate me :D
- I believe this terrain of discussion is in the grey area of C++, and
left so far to package managers



Raffi

PS: The context: I was creating a .mex file for Matlab (from
https://github.com/MPI-IS/Grassmann-Averages-PCA). The C++ shared
library (the .mex) is using boost.thread, and the problem was that
Matlab has its own version of boost.thread. If I remember, even with the
same version of boost as the one shipped with Matlab, I was having
issues (while technically, all the symbols that were merged should have
been equivalent).




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 8/19/18 9:31 AM, James E. King III via Boost wrote:
> On Sun, Aug 19, 2018 at 7:48 AM Antony Polukhin via Boost <
> [hidden email]> wrote:
>

> I think a better solution is to remove link libraries where possible.  For
> example
> in date_time there isn't much of a case to keep the linked library.  If we
> look at
> the issue it causes like https://github.com/boostorg/date_time/pull/34, one
> of the
> solutions to that issue of visibility is to remove the link library
> entirely.  I'd have to
> recommend all libraries attempt to become header-only to avoid these issues.
> Obviously that won't work for all libraries, but it's possible for a number
> of them.
> I opened https://github.com/boostorg/date_time/issues/85 to track that for
> date_time.


TL;DR;

The appeal of using header only libraries is sign that we have bigger
problems not being addressed.

Basically this amounts to deprecating the separate compilation model
that is a historical feature of C++.  In my view it's an essential
feature.  Many complain of compile times that are "too long" for their
1000000 line programs.  More than anything this is an indicator that
their code hasn't been factored into separately compilable modules.

Not that I'm totally unsympathetic.  The C++ separate compilation model
was extended to shared libraries in an ad hoc way by each vendor.  Now
we have issues with symbol visibility and ABI compatibility.  The tools
vendors havn't really addressed this and so C++ programs are a pain to
configure and build.  It's also unwieldy to incorporate library code and
especially pre-compiled modules.

Just using header only - solves the major convenience problems - but the
build doesn't scale anymore creating 1,000,000 compilations.  Now we're
looking a adding concepts, contracts and modules.  I'm really concerned
that C++ may collapse under it's own weight.

What is really needed is for someone to step back and and re-address the
issues of ODR, visibility, ABI compatibility and versioning, build
tooling and anything else that ripples to.  It's huge task.  C++ growth
is O(n^2) where n is the number of features added.

Basically, C++ has become a victim of it's own success.  In the past,
this has often resulted in new programming languages.  But now I see the
C++ universe as being too big for this.

Robert Ramey

>
> - Jim
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 19/08/2018 08:17, Andrey Semashev wrote:
> There is no problem linking static libraries into shared libraries, I
> do it every day.

This is not entirely true -- while you can do that, and it can work, it
comes with a very big caveat that will trap the unwary.

Let's say you have a library A and another library B, and an executable C.

Compile everything as shared, now C will dynamically load B and A, and
everything will Just Work™.

Compile everything as static, now C will statically contain B and A, and
everything will Just Work™.

Compile A as static and B as shared.  Now things *might* work, or you
might have a problem -- it depends on how the libraries are actually
used; in particular whether A is used by B or C or both.

If it's used only by one or the other (and in particular is *not*
exposed in the public API of B at all), then everything is fine.

If B does a poor job of hiding its use of A (ie. it's included in public
header files, even if not part of public API), and if C uses A as well,
you now have an ODR problem.  And the problem is worse if A's objects
are used in the public API of B.

The key thing to realise is that "A as statically linked into B" and "A
as statically linked into C" are technically separate copies of the
library.  And passing objects around between separate copies of the same
library is highly perilous -- a lot of the time you can get away with it
(provided that the same compiler settings were used in both cases, so
you don't get different memory layouts) -- but some things will be
"wrong" such as separate copies of static variables and the like, and
this can cause misbehaviour.


This is why you're always encouraged to link to the runtime library as
shared -- because since it's a dependency of every other library, the
only time it's safe to use it statically is if you don't use *any*
shared libraries at all.  Otherwise you end up with multiple copies of
the runtime, and thus multiple separate heaps, and hilarity ensues.

Having multiple copies of other libraries is perhaps less dramatic than
that, and you can make it "safe" if you're very careful about
segregation, but it's a highly effective source of potential bugs.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On Tue, 21 Aug 2018 at 02:59, Gavin Lambert via Boost <[hidden email]>
wrote:

> On 19/08/2018 08:17, Andrey Semashev wrote:
> > There is no problem linking static libraries into shared libraries, I
> > do it every day.
>
> This is not entirely true -- while you can do that, and it can work, it
> comes with a very big caveat that will trap the unwary.
>

I'm happy you are commenting on this problem again.

Let's say you have a library A and another library B, and an executable C.

>
> Compile everything as shared, now C will dynamically load B and A, and
> everything will Just Work™.
>
> Compile everything as static, now C will statically contain B and A, and
> everything will Just Work™.
>
> Compile A as static and B as shared.  Now things *might* work, or you
> might have a problem -- it depends on how the libraries are actually
> used; in particular whether A is used by B or C or both.
>
> If it's used only by one or the other (and in particular is *not*
> exposed in the public API of B at all), then everything is fine.
>
> If B does a poor job of hiding its use of A (ie. it's included in public
> header files, even if not part of public API), and if C uses A as well,
> you now have an ODR problem.  And the problem is worse if A's objects
> are used in the public API of B.
>
> The key thing to realise is that "A as statically linked into B" and "A
> as statically linked into C" are technically separate copies of the
> library.  And passing objects around between separate copies of the same
> library is highly perilous -- a lot of the time you can get away with it
> (provided that the same compiler settings were used in both cases, so
> you don't get different memory layouts) -- but some things will be
> "wrong" such as separate copies of static variables and the like, and
> this can cause misbehaviour.
>
>
> This is why you're always encouraged to link to the runtime library as
> shared -- because since it's a dependency of every other library, the
> only time it's safe to use it statically is if you don't use *any*
> shared libraries at all.  Otherwise you end up with multiple copies of
> the runtime, and thus multiple separate heaps, and hilarity ensues.
>

This is I think not what happens in practice, in vcpkg f.e. creation of a
static library will by default also imply that that library is statically
linked to the run-time. I also do this when building Boost, while you are
actually saying that that option/possibility should not even exist as I
presume that the combination of a dynamic library statically linked to the
run-time crt makes even less sense.

Having multiple copies of other libraries is perhaps less dramatic than
> that, and you can make it "safe" if you're very careful about
> segregation, but it's a highly effective source of potential bugs.
>

You can only find out that you weren't careful when you are finding it out,
which might be a case of Seems To Work™, until it doesn't, as it's a full
moon

degski
--
*“If something cannot go on forever, it will stop" - Herbert Stein*
*“No, it isn’t truth. Truth isn’t truth" - Rudolph W. L. Giuliani*

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [all] Request for out of the box visibility support

Boost - Dev mailing list
On 21/08/2018 15:35, degski wrote:

>> This is why you're always encouraged to link to the runtime library as
>> shared -- because since it's a dependency of every other library, the
>> only time it's safe to use it statically is if you don't use *any*
>> shared libraries at all.  Otherwise you end up with multiple copies of
>> the runtime, and thus multiple separate heaps, and hilarity ensues.
>
> This is I think not what happens in practice, in vcpkg f.e. creation of a
> static library will by default also imply that that library is statically
> linked to the run-time. I also do this when building Boost, while you are
> actually saying that that option/possibility should not even exist as I
> presume that the combination of a dynamic library statically linked to the
> run-time crt makes even less sense.

I've never used vcpkg, so I can't really speak to its choices, but
whenever you create a new project in VS (regardless if it is an
executable, static library, or dynamic library), it will select the
dynamic CRT by default, because this is the safest choice in all cases.

And a cursory glance at the vcpkg repository shows that this script:

https://github.com/Microsoft/vcpkg/blob/961cd9effd9a5230f211875bb0e9a6773e0e3fab/scripts/cmake/vcpkg_check_linkage.cmake

attempts to prevent you compiling a dynamic library while linking to the
static CRT.  (I don't know how well it is actually applied; perhaps I'm
misreading something.)


As I said before, there is no problem with linking to the static CRT if
you are *only* using static libraries.

There is also less problem if you are only using DLLs that practice
strict memory segregation (eg. using an agreed memory allocator for all
exchanged objects, such as the COM or Shell allocators), or ensuring
that objects are only ever allocated or deallocated on exactly one "side".

It's a lot harder (though not impossible, with very careful use of smart
pointers and allocators) to achieve this segregation with C++ libraries.

(Having said that, this only discusses one aspect of library
duplication, that of separate heaps.  There can be other issues such as
duplicate singletons and other statics; these can also ruin your day,
but are more library-specific.)

Other "shared" CRT state (locale, exceptions, standard I/O streams, etc)
also won't actually be shared if you are using multiple static CRTs.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
12