Boost Serialization: Serializing large objects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Boost Serialization: Serializing large objects

Boost - Dev mailing list
We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.

This is on a 32-bit QNX system.


  *   Is there a limit to the size of the object that can be serialized?
  *   Is there anything we can do to fix or work around this?

Here is the code we are using:

    std::stringstream ss_;
    boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
    oa << &object;

Adlai

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
On 10/5/17 10:41 AM, Adlai Shawareb via Boost wrote:

> We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
>
> This is on a 32-bit QNX system.
>
>
>    *   Is there a limit to the size of the object that can be serialized?
>    *   Is there anything we can do to fix or work around this?
>
> Here is the code we are using:
>
>      std::stringstream ss_;
>      boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
>      oa << &object;
>
> Adlai
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>

There is no known limitation on file size in the boost serialization
library.

Robert Ramey

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 10/5/17 10:41 AM, Adlai Shawareb via Boost wrote:

> We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
>
> This is on a 32-bit QNX system.
>
>
>    *   Is there a limit to the size of the object that can be serialized?
>    *   Is there anything we can do to fix or work around this?
>
> Here is the code we are using:
>
>      std::stringstream ss_;
>      boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
>      oa << &object;
>
> Adlai
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>

suggestion: Try using a text file stream rather than a stringstream.

Robert Ramey

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
> We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
>
> This is on a 32-bit QNX system.
>
>  Is there a limit to the size of the object that can be serialized?
>  Is there anything we can do to fix or work around this?
>
> Here is the code we are using:
>
>    std::stringstream ss_;
>    boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
>    oa << &object;
>
> Adlai

Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream?

Adlai

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
On 10/9/17 4:21 PM, Adlai Shawareb via Boost wrote:

> Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream?

Boost doesn't do it.  std::stringstream does it.

It's not too hard to make your own archive which is just a giant memory
buffer.  I think there's an example in the documentation which does
this.  But I might be wrong.

Robert Ramey

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
>
> Boost doesn't do it. std::stringstream does it.
>
> It's not too hard to make your own archive which is just a giant memory
> buffer. I think there's an example in the documentation which does
> this. But I might be wrong.

Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream?

I looked at the Serialization Tutorial and didn't see an example of making my own archive. If you are inclined, I would appreciate an example.

Adlai

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
On 10/10/17 10:41 AM, Adlai Shawareb via Boost wrote:
>>
>> Boost doesn't do it. std::stringstream does it.
>>
>> It's not too hard to make your own archive which is just a giant memory
>> buffer. I think there's an example in the documentation which does
>> this. But I might be wrong.
>
> Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream?

as far as I know there are no such calls supported by std::stringstream.
  But maybe I'm wrong about that.  Boost serialization uses the
interface provide by std::basic_stream - no more than that.
>
> I looked at the Serialization Tutorial and didn't see an example of making my own archive. If you are inclined, I would appreciate an example.
>
> Adlai
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
> as far as I know there are no such calls supported by std::stringstream.
> But maybe I'm wrong about that. Boost serialization uses the
> interface provide by std::basic_stream - no more than that.

Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added?
I would like to know so that I can look into why using that method we get an error.

Adlai

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
On 10/10/17 11:29 AM, Adlai Shawareb via Boost wrote:
>> as far as I know there are no such calls supported by std::stringstream.
>> But maybe I'm wrong about that. Boost serialization uses the
>> interface provide by std::basic_stream - no more than that.
>
> Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added?
> I would like to know so that I can look into why using that method we get an error.
>
> Adlai

I'm sorry, I can't give short answers to these questions.  You'll have
to study the documentation.

Robert Ramey
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
>
>
> Thanks Robert. Does Boost use operator<< to add data to the stringstream?
> If not, how is data added?
> I would like to know so that I can look into why using that method we get
> an error.
>
> Adlai
>

You say the object is 700 megs. Is that 700 megs in memory or in text? The
text representation is almost certainly bigger than the in memory
representation so you might just be running out of memory.

Alternatively, you could just be running out of address space if
stringstream requires a contiguous buffer. That's an implementation detail,
but I would not be surprised if stringstream allocates a contiguous buffer,
in which case you almost certainly would be bumping into a memory
fragmentation issue due to the 32bit address space.

If you serialize directly to a file instead of stringstream, as Robert
suggested, the problem might go away.

-- chris

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
> You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory
> representation so you might just be running out of memory.
>
> Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I
> would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory
> fragmentation issue due to the 32bit address space.
>
> If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away.
>
> -- chris

Thanks for responding Chris. The unserialized data is 700 megs. You are correct, the text representation is closer to 1.2 GB. We are using extended addressing on QNX, so I haven't focused on the 32bit address space / fragmentation issue, but it can't be ruled out. The contiguous buffer issue seems likely. The questions I was asking of Robert were to look for something that we could take to QNX to get some feedback.

Adlai

________________________________

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost Serialization: Serializing large objects

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 05-10-17 19:41, Adlai Shawareb via Boost wrote:
>     std::stringstream ss_;
>     boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
>     oa << &object;

If you can't write directly to a file, just use `back_insert_device`
with a properly reserved container (std::string or std::vector<char> for
example):
http://www.boost.org/doc/libs/1_65_1/libs/iostreams/doc/classes/back_inserter.html




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost