[serialization] failing MPI build

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[serialization] failing MPI build

alainm

Hi,

Am I the only one with a (global) failing build on MPI ?

../../../boost/archive/detail/interface_oarchive.hpp(63): error: no
instance of overloaded function
"boost::mpi::detail::content_oarchive::save_override" matches the
argument list
             argument types are: (const boost::archive::tracking_type)
             object type is: boost::mpi::detail::content_oarchive
           this->This()->save_override(t);

I am going to investigate, but maybe I missed something obvious.

Thanks

Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

Robert Ramey
Alain Miniussi wrote
Hi,

Am I the only one with a (global) failing build on MPI ?

../../../boost/archive/detail/interface_oarchive.hpp(63): error: no
instance of overloaded function
"boost::mpi::detail::content_oarchive::save_override" matches the
argument list
             argument types are: (const boost::archive::tracking_type)
             object type is: boost::mpi::detail::content_oarchive
           this->This()->save_override(t);

I am going to investigate, but maybe I missed something obvious.
Feel free to investigate.

Recently I've made updates to the serialization library.  These included
changes to support more/all C++11 library components like std::shared_ptr
etc.  This broke polymorphic archives.  My tests weren't sufficiently broad
to detect this.  So it's entirely possible that something in MPI also
broke.  I've never been able to figure out how to test MPI serialization
as part of the test suite.  So if you want investigate that would be helpful.
Even better would be to come up with some tests which might detect
problems like this sooner.

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

Robert Ramey
In trying to investigate this problem, I've attempted to test MPI in accordance with the procedure described at
http://www.boost.org/doc/libs/master/doc/html/mpi/getting_started.html#mpi.config.  It does'n't seem to work,  On the other hand, it seems it builds the rest of boost. Any ideas?

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
On 11/04/2015 06:08, Robert Ramey wrote:
> In trying to investigate this problem, I've attempted to test MPI in
> accordance with the procedure described at
> http://www.boost.org/doc/libs/master/doc/html/mpi/getting_started.html#mpi.config.
> It does'n't seem to work,  On the other hand, it seems it builds the rest of
> boost. Any ideas?

Can you show the mpi section of user-config.jam or project-config.jam ?
Which MPI implementation are you using ?

Alain


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
In reply to this post by Robert Ramey

I just tried a test from scratch on master with intel MPI. One more or
less expected failure on nonblocking-test (I think a fix is in develop)
in "default" mode.
But even on master, there is a huge problem when setting
BOOST_MPI_HOMOGENEOUS in boost/mpi/config.hpp (which is probably the
default usage, most MPI application run in homogeneous environment).

primitive.hpp:110: void
boost::mpi::binary_buffer_iprimitive::load_impl(void *, int): Assertion
`position+l<=static_cast<int>(buffer_.size())' failed.
../../../boost/test/minimal.hpp(123): exception "signal: SIGABRT
(application abort requested)" caught in function: 'int main(int, char **)'

Basically, packed_iprimitive seems fine but binary_buffer_iprimitive has
been broken.

That's for the master branch (the upcomming release).

Regards

ALain


On 11/04/2015 06:08, Robert Ramey wrote:

> In trying to investigate this problem, I've attempted to test MPI in
> accordance with the procedure described at
> http://www.boost.org/doc/libs/master/doc/html/mpi/getting_started.html#mpi.config.
> It does'n't seem to work,  On the other hand, it seems it builds the rest of
> boost. Any ideas?
>
> Robert Ramey
>
>
>
> --
> View this message in context: http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674395.html
> Sent from the Boost - Dev mailing list archive at Nabble.com.
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

Robert Ramey
In reply to this post by alainm
Alain Miniussi wrote
Can you show the mpi section of user-config.jam or project-config.jam ?
$cat ~/user-config.jam
using darwin : 4.9 : /hpc/usr/local/bin/g++ : <cxxflags>"-std=c++11" ;
using clang : 11 : : <cxxflags>"-std=c++11" <cxxflags>"-Wno-c++11-extensions" ;
using clang : 03 : : <cxxflags>"-std=c++03" ;
using clang : 98 : : <cxxflags>"-std=c++98" ;

using darwin : 8.0~iphone
   : /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
   : <striper> <root>/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer
     <compileflags>"-arch armv7"
     <compileflags>"-arch armv7s"
     <compileflags>-mthumb
     <compileflags>-fvisibility=hidden
     <compileflags>-fvisibility-inlines-hidden
     <compileflags>-std=c++11
     <compileflags>-stdlib=libc++
     <compileflags>-miphoneos-version-min=5.1
     <compileflags>-D_LITTLE_ENDIAN
     <compileflags>"-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.0.sdk"
   : <architecture>arm <target-os>iphone
   ;

using darwin : 8.0~iphonesim
   : /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
   : <striper> <root>/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer
     <compileflags>"-arch i386"
     <compileflags>-fvisibility=hidden
     <compileflags>-fvisibility-inlines-hidden
     <compileflags>-std=c++11
     <compileflags>-stdlib=libc++
     <compileflags>-miphoneos-version-min=5.1
     <compileflags>-D_LITTLE_ENDIAN
     <compileflags>"-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator7.0.sdk"
   : <architecture>x86 <target-os>iphone
   ;
using boostbook
  : /Users/robertramey/WorkingProjects/modular-boost/tools/boostbook/docbook-xsl-1.75.2
  : /Users/robertramey/WorkingProjects/modular-boost/tools/boostbook/docbook-dtd-4.2
  ;
using xsltproc : /usr/bin/xsltproc ;
using fop : /Users/robertramey/WorkingProjects/modular-boost/tools/boostbook/fop-0.94/fop : : /usr/bin/java ;
$
$cat  ../../../project-config.jam
# Boost.Build Configuration
# Automatically generated by bootstrap.sh

import option ;
import feature ;

# Compiler configuration. This definition will be used unless
# you already have defined some toolsets in your user-config.jam
# file.
if ! darwin in [ feature.values <toolset> ]
{
    using darwin ;
}
project : default-build <toolset>darwin ;

# Python configuration
using python : 2.7 : /System/Library/Frameworks/Python.framework/Versions/2.7 ;

# List of --with-<library> and --without-<library>
# options. If left empty, all libraries will be built.
# Options specified on the command line completely
# override this variable.
libraries =  ;

# These settings are equivalent to corresponding command-line
# options.
option.set prefix : /usr/local ;
option.set exec-prefix : /usr/local/bin ;
option.set libdir : /usr/local/lib ;
option.set includedir : /usr/local/include ;

# Stop on first error
option.set keep-going : false ;
$
Which MPI implementation are you using ?
Hmmm - I don't know - I just followed the instructions in the documentation page cited above.

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

Robert Ramey
In an attempt to build the MPI library on my machine I added to my user-config.jam

using mpi ;

I get the following:

$pwd
/Users/robertramey/WorkingProjects/modular-boost/libs/mpi/build

$b2 -n --debug-configuration

warning: toolset mpi initialization: can not find tool mpic++
warning: initialized from /Users/robertramey/user-config.jam:41
MPI auto-detection failed: unknown wrapper compiler mpic++
Please report this error to the Boost mailing list: http://www.boost.org
You will need to manually configure MPI support.
warning: toolset mpi initialization: can not find tool mpirun
warning: initialized from /Users/robertramey/user-config.jam:41
MPI launcher: mpirun -np

What do I have to do to make this work?

OK - the above has pointed me to the fact that I need an MPI implementation
installed to get this to work. I'll look into this.

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
Hi,

I suspect one problem is that the commit
https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
was not taken into account in MPI.

Not sure how the MPI should be adapted to that change yet. Some
parameters that were supposed to be there only to help overload
resolution have been removed (according to comments in pfto.hpp).

I guess I need to remove some int arguments from the
(save|load)_override methods of boost::mpi::packed_(i|o)archive classes ?

Alain

On 12/04/2015 18:15, Robert Ramey wrote:

> In an attempt to build the MPI library on my machine I added to my
> user-config.jam
>
> using mpi ;
>
> I get the following:
>
>
>
> What do I have to do to make this work?
>
> OK - the above has pointed me to the fact that I need an MPI implementation
> installed to get this to work. I'll look into this.
>
> Robert Ramey
>
>
>
>
> --
> View this message in context: http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674414.html
> Sent from the Boost - Dev mailing list archive at Nabble.com.
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

Robert Ramey
Alain Miniussi wrote
Hi,

I suspect one problem is that the commit
https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
was not taken into account in MPI.

Not sure how the MPI should be adapted to that change yet. Some
parameters that were supposed to be there only to help overload
resolution have been removed (according to comments in pfto.hpp).

I guess I need to remove some int arguments from the
(save|load)_override methods of boost::mpi::packed_(i|o)archive classes ?
You are correct.  It's easy to see the changes.  If you do a diff on some other archive like binary or text in the serialization library you'll see that the dummy parameter used to emulate partial function template overload has been eliminated.   And of course eliminate the #include boost/serialization/ptfo.hpp . That's all that is necessary.  This simplifies code at the cost of eliminating support of borland 5x and mdvc 7.0 and below which is no longer considered necessary.  Since these headers are considered implementation features of the serialization library it didn't occur to me that making these changes would break anything else.  Sorry about that.

If you're interested in enhancing, or improving the MPI implementation as it regards serialization, you should let  me know so I could acquaint myself with the MPI version and offer some suggestions.  I've had some ideas in my head for years would simplify and expedite usage of the serialization library for object transmission over a stream but I've never spend any time on them as it didn't seem anyone had any interest.

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
On 14/04/2015 16:49, Robert Ramey wrote:

> Alain Miniussi wrote
>> Hi,
>>
>> I suspect one problem is that the commit
>> https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
>> was not taken into account in MPI.
>>
>> Not sure how the MPI should be adapted to that change yet. Some
>> parameters that were supposed to be there only to help overload
>> resolution have been removed (according to comments in pfto.hpp).
>>
>> I guess I need to remove some int arguments from the
>> (save|load)_override methods of boost::mpi::packed_(i|o)archive classes ?
>
> You are correct.  It's easy to see the changes.  If you do a diff on some
> other archive like binary or text in the serialization library you'll see
> that the dummy parameter used to emulate partial function template overload
> has been eliminated.   And of course eliminate the #include
> boost/serialization/ptfo.hpp . That's all that is necessary.  This
> simplifies code at the cost of eliminating support of borland 5x and mdvc
> 7.0 and below which is no longer considered necessary.  Since these headers
> are considered implementation features of the serialization library it
> didn't occur to me that making these changes would break anything else.
> Sorry about that.

I am relatively new to that part of Boost.MPI so I cannot tell if these
features should have been used in boost mpi in the first place.

> If you're interested in enhancing, or improving the MPI implementation as it
> regards serialization, you should let  me know so I could acquaint myself
> with the MPI version and offer some suggestions.  I've had some ideas in my
> head for years would simplify and expedite usage of the serialization
> library for object transmission over a stream but I've never spend any time
> on them as it didn't seem anyone had any interest.

I started doing some modifications that I will put on a branch on github.
Right now, with those modifications the develop version of Boost.MPI
compiles again with the develop version of serialization. But I still
have some runtime failures so I'd like to fix those before going to
github develop.
The failure have to do with serialization of "complex" stl data
structure (stl collections of non PODs). I suspect it has something to
do with binary serialization optimisation (where non POD are considered
POD).

I'll let you know when current modifications are on a publicly available
on github (I'm not that fluent in git/github yet).

Alain Miniussi


> Robert Ramey
>
>
>
> --
> View this message in context: http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674449.html
> Sent from the Boost - Dev mailing list archive at Nabble.com.
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
The noptfo branch of Boost.MPI contains that modification. But still
fails at runtime.
I did set BOOST_MPI_HOMOGENEOUS on that branch, which is the most common
situation in HPC and allow some optimizations that are probably
triggering that runtime bug.

Alain

On 15/04/2015 08:53, Alain Miniussi wrote:

> On 14/04/2015 16:49, Robert Ramey wrote:
>> Alain Miniussi wrote
>>> Hi,
>>>
>>> I suspect one problem is that the commit
>>> https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
>>>
>>> was not taken into account in MPI.
>>>
>>> Not sure how the MPI should be adapted to that change yet. Some
>>> parameters that were supposed to be there only to help overload
>>> resolution have been removed (according to comments in pfto.hpp).
>>>
>>> I guess I need to remove some int arguments from the
>>> (save|load)_override methods of boost::mpi::packed_(i|o)archive
>>> classes ?
>>
>> You are correct.  It's easy to see the changes.  If you do a diff on some
>> other archive like binary or text in the serialization library you'll see
>> that the dummy parameter used to emulate partial function template
>> overload
>> has been eliminated.   And of course eliminate the #include
>> boost/serialization/ptfo.hpp . That's all that is necessary.  This
>> simplifies code at the cost of eliminating support of borland 5x and mdvc
>> 7.0 and below which is no longer considered necessary.  Since these
>> headers
>> are considered implementation features of the serialization library it
>> didn't occur to me that making these changes would break anything else.
>> Sorry about that.
>
> I am relatively new to that part of Boost.MPI so I cannot tell if these
> features should have been used in boost mpi in the first place.
>
>> If you're interested in enhancing, or improving the MPI implementation
>> as it
>> regards serialization, you should let  me know so I could acquaint myself
>> with the MPI version and offer some suggestions.  I've had some ideas
>> in my
>> head for years would simplify and expedite usage of the serialization
>> library for object transmission over a stream but I've never spend any
>> time
>> on them as it didn't seem anyone had any interest.
>
> I started doing some modifications that I will put on a branch on github.
> Right now, with those modifications the develop version of Boost.MPI
> compiles again with the develop version of serialization. But I still
> have some runtime failures so I'd like to fix those before going to
> github develop.
> The failure have to do with serialization of "complex" stl data
> structure (stl collections of non PODs). I suspect it has something to
> do with binary serialization optimisation (where non POD are considered
> POD).
>
> I'll let you know when current modifications are on a publicly available
> on github (I'm not that fluent in git/github yet).
>
> Alain Miniussi
>
>
>> Robert Ramey
>>
>>
>>
>> --
>> View this message in context:
>> http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674449.html
>>
>> Sent from the Boost - Dev mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> Unsubscribe & other changes:
>> http://lists.boost.org/mailman/listinfo.cgi/boost
>>
>
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
In reply to this post by Robert Ramey

As for the runtime bug (in broadcast_stl_test-2 for example). It start
failing when the broadcasted map count is loaded.
At line 45 of collections_save_impl.hpp:
   ar << BOOST_SERIALIZATION_NVP(count);
gdb indicates count value == 2 (which is correct.

I have no clue what should be in ar at that point (notsure to understand
the nvp thing, why not just send 2 ?) but I'm getting:
(gdb) p ar.buffer_
$66 = std::vector of length 12, capacity 12 = {0 '\000', 0 '\000', 0
'\000', 0 '\000', 2 '\002', 0 '\000', 0 '\000', 0 '\000', 0 '\000', 0
'\000', 0 '\000', 0 '\000'}
(gdb)

but in map.hpp's load_map_collection
       ar >> BOOST_SERIALIZATION_NVP(count);
count contains garbage (and a very big number too) hence an array overflow.

Any idea ?

Thanks

ALain

On 14/04/2015 16:49, Robert Ramey wrote:

> Alain Miniussi wrote
>> Hi,
>>
>> I suspect one problem is that the commit
>> https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
>> was not taken into account in MPI.
>>
>> Not sure how the MPI should be adapted to that change yet. Some
>> parameters that were supposed to be there only to help overload
>> resolution have been removed (according to comments in pfto.hpp).
>>
>> I guess I need to remove some int arguments from the
>> (save|load)_override methods of boost::mpi::packed_(i|o)archive classes ?
>
> You are correct.  It's easy to see the changes.  If you do a diff on some
> other archive like binary or text in the serialization library you'll see
> that the dummy parameter used to emulate partial function template overload
> has been eliminated.   And of course eliminate the #include
> boost/serialization/ptfo.hpp . That's all that is necessary.  This
> simplifies code at the cost of eliminating support of borland 5x and mdvc
> 7.0 and below which is no longer considered necessary.  Since these headers
> are considered implementation features of the serialization library it
> didn't occur to me that making these changes would break anything else.
> Sorry about that.
>
> If you're interested in enhancing, or improving the MPI implementation as it
> regards serialization, you should let  me know so I could acquaint myself
> with the MPI version and offer some suggestions.  I've had some ideas in my
> head for years would simplify and expedite usage of the serialization
> library for object transmission over a stream but I've never spend any time
> on them as it didn't seem anyone had any interest.
>
> Robert Ramey
>
>
>
> --
> View this message in context: http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674449.html
> Sent from the Boost - Dev mailing list archive at Nabble.com.
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
On 15/04/2015 15:55, Alain Miniussi wrote:

>
> As for the runtime bug (in broadcast_stl_test-2 for example). It start
> failing when the broadcasted map count is loaded.
> At line 45 of collections_save_impl.hpp:
>    ar << BOOST_SERIALIZATION_NVP(count);
> gdb indicates count value == 2 (which is correct.
>
> I have no clue what should be in ar at that point (notsure to understand
> the nvp thing, why not just send 2 ?) but I'm getting:
> (gdb) p ar.buffer_
> $66 = std::vector of length 12, capacity 12 = {0 '\000', 0 '\000', 0
> '\000', 0 '\000', 2 '\002', 0 '\000', 0 '\000', 0 '\000', 0 '\000', 0
> '\000', 0 '\000', 0 '\000'}
> (gdb)
>
> but in map.hpp's load_map_collection
>        ar >> BOOST_SERIALIZATION_NVP(count);
> count contains garbage (and a very big number too) hence an array overflow.

4294967297

>
> Any idea ?
>
> Thanks
>
> ALain
>
> On 14/04/2015 16:49, Robert Ramey wrote:
>> Alain Miniussi wrote
>>> Hi,
>>>
>>> I suspect one problem is that the commit
>>> https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
>>>
>>> was not taken into account in MPI.
>>>
>>> Not sure how the MPI should be adapted to that change yet. Some
>>> parameters that were supposed to be there only to help overload
>>> resolution have been removed (according to comments in pfto.hpp).
>>>
>>> I guess I need to remove some int arguments from the
>>> (save|load)_override methods of boost::mpi::packed_(i|o)archive
>>> classes ?
>>
>> You are correct.  It's easy to see the changes.  If you do a diff on some
>> other archive like binary or text in the serialization library you'll see
>> that the dummy parameter used to emulate partial function template
>> overload
>> has been eliminated.   And of course eliminate the #include
>> boost/serialization/ptfo.hpp . That's all that is necessary.  This
>> simplifies code at the cost of eliminating support of borland 5x and mdvc
>> 7.0 and below which is no longer considered necessary.  Since these
>> headers
>> are considered implementation features of the serialization library it
>> didn't occur to me that making these changes would break anything else.
>> Sorry about that.
>>
>> If you're interested in enhancing, or improving the MPI implementation
>> as it
>> regards serialization, you should let  me know so I could acquaint myself
>> with the MPI version and offer some suggestions.  I've had some ideas
>> in my
>> head for years would simplify and expedite usage of the serialization
>> library for object transmission over a stream but I've never spend any
>> time
>> on them as it didn't seem anyone had any interest.
>>
>> Robert Ramey
>>
>>
>>
>> --
>> View this message in context:
>> http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674449.html
>>
>> Sent from the Boost - Dev mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> Unsubscribe & other changes:
>> http://lists.boost.org/mailman/listinfo.cgi/boost
>>
>
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
In reply to this post by alainm
On 15/04/2015 15:55, Alain Miniussi wrote:

>
> As for the runtime bug (in broadcast_stl_test-2 for example). It start
> failing when the broadcasted map count is loaded.
> At line 45 of collections_save_impl.hpp:
>    ar << BOOST_SERIALIZATION_NVP(count);
> gdb indicates count value == 2 (which is correct.
>
> I have no clue what should be in ar at that point (notsure to understand
> the nvp thing, why not just send 2 ?) but I'm getting:
> (gdb) p ar.buffer_
> $66 = std::vector of length 12, capacity 12 = {0 '\000', 0 '\000', 0
> '\000', 0 '\000', 2 '\002', 0 '\000', 0 '\000', 0 '\000', 0 '\000', 0
> '\000', 0 '\000', 0 '\000'}
> (gdb)
>
> but in map.hpp's load_map_collection
>        ar >> BOOST_SERIALIZATION_NVP(count);
> count contains garbage (and a very big number too) hence an array overflow.

It appears that the positions are inconsistent, the buffer size is 12
*after* writing the count and 16 *before* reading it.


> Any idea ?
>
> Thanks
>
> ALain
>
> On 14/04/2015 16:49, Robert Ramey wrote:
>> Alain Miniussi wrote
>>> Hi,
>>>
>>> I suspect one problem is that the commit
>>> https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
>>>
>>> was not taken into account in MPI.
>>>
>>> Not sure how the MPI should be adapted to that change yet. Some
>>> parameters that were supposed to be there only to help overload
>>> resolution have been removed (according to comments in pfto.hpp).
>>>
>>> I guess I need to remove some int arguments from the
>>> (save|load)_override methods of boost::mpi::packed_(i|o)archive
>>> classes ?
>>
>> You are correct.  It's easy to see the changes.  If you do a diff on some
>> other archive like binary or text in the serialization library you'll see
>> that the dummy parameter used to emulate partial function template
>> overload
>> has been eliminated.   And of course eliminate the #include
>> boost/serialization/ptfo.hpp . That's all that is necessary.  This
>> simplifies code at the cost of eliminating support of borland 5x and mdvc
>> 7.0 and below which is no longer considered necessary.  Since these
>> headers
>> are considered implementation features of the serialization library it
>> didn't occur to me that making these changes would break anything else.
>> Sorry about that.
>>
>> If you're interested in enhancing, or improving the MPI implementation
>> as it
>> regards serialization, you should let  me know so I could acquaint myself
>> with the MPI version and offer some suggestions.  I've had some ideas
>> in my
>> head for years would simplify and expedite usage of the serialization
>> library for object transmission over a stream but I've never spend any
>> time
>> on them as it didn't seem anyone had any interest.
>>
>> Robert Ramey
>>
>>
>>
>> --
>> View this message in context:
>> http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674449.html
>>
>> Sent from the Boost - Dev mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> Unsubscribe & other changes:
>> http://lists.boost.org/mailman/listinfo.cgi/boost
>>
>
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
in basic_iarchive::load_preamble the class_id is actually an
class_id_optional_type, and as such ends up being ignored at mpi's
  // input archives need to ignore  the optional information
 

=>void load_override(archive::class_id_optional_type & /*t*/){}

But it was actually inserted in the archive, which ends up in a shift of
2 bytes when reading the size.

On 15/04/2015 16:27, Alain Miniussi wrote:

> On 15/04/2015 15:55, Alain Miniussi wrote:
>>
>> As for the runtime bug (in broadcast_stl_test-2 for example). It start
>> failing when the broadcasted map count is loaded.
>> At line 45 of collections_save_impl.hpp:
>>    ar << BOOST_SERIALIZATION_NVP(count);
>> gdb indicates count value == 2 (which is correct.
>>
>> I have no clue what should be in ar at that point (notsure to understand
>> the nvp thing, why not just send 2 ?) but I'm getting:
>> (gdb) p ar.buffer_
>> $66 = std::vector of length 12, capacity 12 = {0 '\000', 0 '\000', 0
>> '\000', 0 '\000', 2 '\002', 0 '\000', 0 '\000', 0 '\000', 0 '\000', 0
>> '\000', 0 '\000', 0 '\000'}
>> (gdb)
>>
>> but in map.hpp's load_map_collection
>>        ar >> BOOST_SERIALIZATION_NVP(count);
>> count contains garbage (and a very big number too) hence an array
>> overflow.
>
> It appears that the positions are inconsistent, the buffer size is 12
> *after* writing the count and 16 *before* reading it.
Gt it wrong, it's 2 actually, which is also problematic,

>
>
>> Any idea ?
>>
>> Thanks
>>
>> ALain
>>
>> On 14/04/2015 16:49, Robert Ramey wrote:
>>> Alain Miniussi wrote
>>>> Hi,
>>>>
>>>> I suspect one problem is that the commit
>>>> https://github.com/boostorg/serialization/commit/5a94e5e4eb01cd8c941db725e5f89965a91740a9
>>>>
>>>>
>>>> was not taken into account in MPI.
>>>>
>>>> Not sure how the MPI should be adapted to that change yet. Some
>>>> parameters that were supposed to be there only to help overload
>>>> resolution have been removed (according to comments in pfto.hpp).
>>>>
>>>> I guess I need to remove some int arguments from the
>>>> (save|load)_override methods of boost::mpi::packed_(i|o)archive
>>>> classes ?
>>>
>>> You are correct.  It's easy to see the changes.  If you do a diff on
>>> some
>>> other archive like binary or text in the serialization library you'll
>>> see
>>> that the dummy parameter used to emulate partial function template
>>> overload
>>> has been eliminated.   And of course eliminate the #include
>>> boost/serialization/ptfo.hpp . That's all that is necessary.  This
>>> simplifies code at the cost of eliminating support of borland 5x and
>>> mdvc
>>> 7.0 and below which is no longer considered necessary.  Since these
>>> headers
>>> are considered implementation features of the serialization library it
>>> didn't occur to me that making these changes would break anything else.
>>> Sorry about that.
>>>
>>> If you're interested in enhancing, or improving the MPI implementation
>>> as it
>>> regards serialization, you should let  me know so I could acquaint
>>> myself
>>> with the MPI version and offer some suggestions.  I've had some ideas
>>> in my
>>> head for years would simplify and expedite usage of the serialization
>>> library for object transmission over a stream but I've never spend any
>>> time
>>> on them as it didn't seem anyone had any interest.
>>>
>>> Robert Ramey
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674449.html
>>>
>>>
>>> Sent from the Boost - Dev mailing list archive at Nabble.com.
>>>
>>> _______________________________________________
>>> Unsubscribe & other changes:
>>> http://lists.boost.org/mailman/listinfo.cgi/boost
>>>
>>
>>
>
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

Robert Ramey
In reply to this post by alainm
Alain Miniussi wrote
>> I guess I need to remove some int arguments from the
>> (save|load)_override methods of boost::mpi::packed_(i|o)archive classes ?
>
> You are correct.  It's easy to see the changes.  If you do a diff on some
> other archive like binary or text in the serialization library you'll see
> that the dummy parameter used to emulate partial function template overload
> has been eliminated.   And of course eliminate the #include
> boost/serialization/ptfo.hpp . That's all that is necessary.  This
> simplifies code at the cost of eliminating support of borland 5x and mdvc
> 7.0 and below which is no longer considered necessary.  Since these headers
> are considered implementation features of the serialization library it
> didn't occur to me that making these changes would break anything else.
> Sorry about that.

I am relatively new to that part of Boost.MPI so I cannot tell if these
features should have been used in boost mpi in the first place.
These features were copied from the archive implementations included
in the serialization library.  Their function was to support compilers which are
obsolete.  For this reason they were eliminated from the serialization library
and should be eliminated from MPI version as well. These changes did not
cause any test failures in the serialization library nor require any changes
in any tests to pass.

One experiment would be to "resurrect" the ptfo.hpp file from github
and re-run the tests.  If the tests pass then we know that it's because
ptfo.hpp was removed.  Otherwise it would be due to some other change.

> If you're interested in enhancing, or improving the MPI implementation as it
> regards serialization, you should let  me know so I could acquaint myself
> with the MPI version and offer some suggestions.  I've had some ideas in my
> head for years would simplify and expedite usage of the serialization
> library for object transmission over a stream but I've never spend any time
> on them as it didn't seem anyone had any interest.

I started doing some modifications that I will put on a branch on github.
Right now, with those modifications the develop version of Boost.MPI
compiles again with the develop version of serialization. But I still
have some runtime failures so I'd like to fix those before going to
github develop.

The failure have to do with serialization of "complex" stl data
structure (stl collections of non PODs). I suspect it has something to
do with binary serialization optimisation (where non POD are considered
POD).


Removal of pfto support entails changing signatures of some save/load
functions which could easily manifest itself in the problems you're having.
Double check for save/load functions with "extra" arguments.

Robert Ramey
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
Ok, I just missed one related with class_optional_id_type.
The version in the noptfo branch works fine.

I'll merge those changes in develop.

Alain

On 15/04/2015 17:42, Robert Ramey wrote:

> Alain Miniussi wrote
>>>> I guess I need to remove some int arguments from the
>>>> (save|load)_override methods of boost::mpi::packed_(i|o)archive classes
>>>> ?
>>>
>>> You are correct.  It's easy to see the changes.  If you do a diff on some
>>> other archive like binary or text in the serialization library you'll see
>>> that the dummy parameter used to emulate partial function template
>>> overload
>>> has been eliminated.   And of course eliminate the #include
>>> boost/serialization/ptfo.hpp . That's all that is necessary.  This
>>> simplifies code at the cost of eliminating support of borland 5x and mdvc
>>> 7.0 and below which is no longer considered necessary.  Since these
>>> headers
>>> are considered implementation features of the serialization library it
>>> didn't occur to me that making these changes would break anything else.
>>> Sorry about that.
>>
>> I am relatively new to that part of Boost.MPI so I cannot tell if these
>> features should have been used in boost mpi in the first place.
>
> These features were copied from the archive implementations included
> in the serialization library.  Their function was to support compilers which
> are
> obsolete.  For this reason they were eliminated from the serialization
> library
> and should be eliminated from MPI version as well. These changes did not
> cause any test failures in the serialization library nor require any changes
> in any tests to pass.
>
> One experiment would be to "resurrect" the ptfo.hpp file from github
> and re-run the tests.  If the tests pass then we know that it's because
> ptfo.hpp was removed.  Otherwise it would be due to some other change.
>
>> If you're interested in enhancing, or improving the MPI implementation as
>> it
>> regards serialization, you should let  me know so I could acquaint myself
>> with the MPI version and offer some suggestions.  I've had some ideas in
>> my
>> head for years would simplify and expedite usage of the serialization
>> library for object transmission over a stream but I've never spend any
>> time
>> on them as it didn't seem anyone had any interest.
>
> I started doing some modifications that I will put on a branch on github.
> Right now, with those modifications the develop version of Boost.MPI
> compiles again with the develop version of serialization. But I still
> have some runtime failures so I'd like to fix those before going to
> github develop.
>
> The failure have to do with serialization of "complex" stl data
> structure (stl collections of non PODs). I suspect it has something to
> do with binary serialization optimisation (where non POD are considered
> POD).
>
>
> Removal of pfto support entails changing signatures of some save/load
> functions which could easily manifest itself in the problems you're having.
> Double check for save/load functions with "extra" arguments.
>
> Robert Ramey
>
>
>
> --
> View this message in context: http://boost.2283326.n4.nabble.com/serialization-failing-MPI-build-tp4674375p4674493.html
> Sent from the Boost - Dev mailing list archive at Nabble.com.
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


--
---
Alain

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [serialization] failing MPI build

alainm
In reply to this post by Robert Ramey

On 14/04/2015 16:49, Robert Ramey wrote:
  .....
>
> If you're interested in enhancing, or improving the MPI implementation as it
> regards serialization, you should let  me know so I could acquaint myself
> with the MPI version and offer some suggestions.  I've had some ideas in my
> head for years would simplify and expedite usage of the serialization
> library for object transmission over a stream but I've never spend any time
> on them as it didn't seem anyone had any interest.

Maybe we could move the discussion to the boost mpi mailing list (I see
you subscribed to it).

Maybe the mpi specific archives could be moved to serialize in the long
run ? At least that part could be tested without an MPI installation.

Alain


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost