Combining Boost Serialization with Boost Python

[James Amundson-2]

I have a use case involving Boost Serialization and Boost Python. I have a (mostly) C++ library wrapped with Python. The C++ code uses Boost Serialization to periodically create checkpoints from which the user can resume later. Getting the serialization and python libraries to work together didn't present any special problems until I had a case in which the user had a derived Python class (from a C++ base). In searching the web, I have seen people asking about similar cases, but never an answer. See, e.g.,
     http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-python-two-way-pickle
.

I have a partial solution to the problem using Boost Serialization and Python Pickle together. I would appreciate any advice on how to improve it. My solution is limited in that the state of the base C++ object is not restored. I also get an extra copy of the C++ object. Since my current use case has no state in the base object, I can live with these limitations. I would however, like to find a full solution.

In the files I have attached, I can successfully run three variations on a simple test case. A class of type Base is passed to a class of type Caller. Caller calls the passed class's doit method twice, saves (checkpoints) its state, then calls the doit method again. The resume script restores the state of the Caller and Base classes, then calls the doit method again. If successful, the resumed doit call should be the same as the third doit call from the original script.

In testcheckpoint1.py, the Base class is the original (C++) Base. It has no state, so the example is pretty trivial. Its doit method prints "Base::doit"
--------------------------------------------------------------------------
% python testcheckpoint1.py                                                                                                            
Base::doit
Base::doit
checkpointed here
Base::doit
% python testresume.py
resuming...
Base::doit
--------------------------------------------------------------------------

In testcheckpoint2.py I pass a derived Python class to Caller. The derived doit method displays "Derived:doit" and a counter:
--------------------------------------------------------------------------
% python testcheckpoint2.py                                                                                                            
Derived::doit count = 1
Derived::doit count = 2
checkpointed here
Derived::doit count = 3
% python testresume.py                                                                                                            
resuming...
Derived::doit count = 3
--------------------------------------------------------------------------

In testcheckpoint3,py, I use a derived Python class with a non-trivial constructor, which is used to set the initial value of the counter:
--------------------------------------------------------------------------
% python testcheckpoint3.py                                                                                                            
Derived2::doit count = 5
Derived2::doit count = 6
checkpointed here
Derived2::doit count = 7
% python testresume.py                                                                                                            
resuming...
Derived2::doit count = 7
--------------------------------------------------------------------------

As you can see, many things are possible with the code the way it currently exists. I would still like to figure out how to preserve the state of the base class. Any and all advice will be appreciated.

--Jim Amundson

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Neal Becker]

I have used this myself for some time.
Here is an example:

typedef boost::mt19937 rng_t;

struct mt_pickle_suite : bp::pickle_suite {
   
  static bp::object getstate (const rng_t& rng) {
    std::ostringstream os;
    boost::archive::binary_oarchive oa(os);
    oa << rng;
    return bp::str (os.str());
  }

  static void
  setstate(rng_t& rng, bp::object entries) {
    bp::str s = bp::extract<bp::str> (entries)();
    std::string st = bp::extract<std::string> (s)();
    std::istringstream is (st);
   
    boost::archive::binary_iarchive ia (is);
    ia >> rng;
  }
};

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Jim Bosch-2]

On 03/14/2012 04:37 PM, James Amundson wrote:

> I have a use case involving Boost Serialization and Boost Python. I have
> a (mostly) C++ library wrapped with Python. The C++ code uses Boost
> Serialization to periodically create checkpoints from which the user can
> resume later. Getting the serialization and python libraries to work
> together didn't present any special problems until I had a case in which
> the user had a derived Python class (from a C++ base). In searching the
> web, I have seen people asking about similar cases, but never an answer.
> See, e.g.,
> http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-python-two-way-pickle
> <http://stackoverflow.com/questions/7289897/boost-serialization-and-boost-python-two-way-pickle>
> .
>

It's been a long while since I last tried to pickle Boost.Python
objects, but I do recall being a lot happier with the level of control I
had when I just implemented my own __reduce__ methods rather than
relying on the __getstate__ and __setstate__ defined by
enable_pickling().  In many cases, it was most convenient to actually
just write __reduce__ in pure-Python and add it to the wrapped classes
in the __init__.py file.  It would have to delegate to wrapped C++
methods to do the Boost.Serialization calls, of course.

Using __reduce__ would allow you to provide a specific callable to
reconstruct the Python derived class, which could then ensure it does
exactly the right combination of regular unpickling and C++ deserialization.

Sorry that's not the review of your "almost there" solution you were
looking for, but I do think you might find the problem easier to solve
with __reduce__.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig