pickle the unpickleable

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

pickle the unpickleable

Neal Becker
I have a pickling problem I suspect is just not possible to solve.

I have some wrappers around boost::random objects.

There is a rng object, which is a single underlying mersenne twister.

There are various distributions.  For example, uniform_real.

Each uniform_real holds a reference to the single, external mt object.

I don't think there's any way to unpickle several instance of uniform_real so
that their references to the (shared) mt object get correctly restored.

Imagine something like this:

class mt;

struct uniform_real {
  mt & _mt;

  uniform_real (mt &, min, max) ...
};

When unpickled, the internal _mt states are saved and restored correctly, but 2
instances of uniform_real will not share a reference to a single mt object, they
each have their own copy which starts with a copy of the same state (no good).

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: pickle the unpickleable

Jim Bosch-2
On 09/03/2011 04:58 AM, Neal Becker wrote:

> I have a pickling problem I suspect is just not possible to solve.
>
> I have some wrappers around boost::random objects.
>
> There is a rng object, which is a single underlying mersenne twister.
>
> There are various distributions.  For example, uniform_real.
>
> Each uniform_real holds a reference to the single, external mt object.
>
> I don't think there's any way to unpickle several instance of uniform_real so
> that their references to the (shared) mt object get correctly restored.
>
> Imagine something like this:
>
> class mt;
>
> struct uniform_real {
>    mt&  _mt;
>
>    uniform_real (mt&, min, max) ...
> };
>
> When unpickled, the internal _mt states are saved and restored correctly, but 2
> instances of uniform_real will not share a reference to a single mt object, they
> each have their own copy which starts with a copy of the same state (no good).
>

You might be able to get something working if you use Pickler and
Unpickler objects instead of the convenience functions in
pickle/cPickle.  Check out the "memo" and "persistent_id" attributes in
particular (in the Python library reference docs).

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Solved (pickle multiple refs to objects)

Neal Becker
The solution to my problem is a combination of boost::serialization and
boost::shared_ptr.

Rather than holding refs to the contained shared objects, if the objects are
share_ptr, then serialization will do the correct thing.

For example, I have

struct boost_uniform_int_wrap {
  boost::shared_ptr<rng_t> rng_obj;
  boost::uniform_int<> gen;

  boost_uniform_int_wrap (boost::shared_ptr<rng_t> _rng_obj, int min, int max) :
    rng_obj (_rng_obj),
    gen (min, max) {}
...

struct uniform_int_pickle_suite : bp::pickle_suite {
  static bp::tuple getinitargs (boost_uniform_int_wrap const& gen) {
    return bp::make_tuple (gen.rng_obj, gen.gen.min(), gen.gen.max());
  }
};

Also, I have modified mersenne_twister to make it serializable (very simple to
add).

Then to make mersenne_twister pickleable:

typedef boost::mt19937 rng_t;

struct mt_pickle_suite : bp::pickle_suite {
   
  static bp::object getstate (const rng_t& rng) {
    std::ostringstream os;
    boost::archive::binary_oarchive oa(os);
    oa << rng;
    return bp::str (os.str());
  }

  static void
  setstate(rng_t& rng, bp::object entries) {
    bp::str s = bp::extract<bp::str> (entries)();
    std::string st = bp::extract<std::string> (s)();
    std::istringstream is (st);
   
    boost::archive::binary_iarchive ia (is);
    ia >> rng;
  }
};

Now:
from boost_rand import rng, normal
from sys import getrefcount

rng1 = rng()
print getrefcount(rng1)
n1 = normal (rng1, 1, 0)
print getrefcount(rng1)
n2 = normal (rng1, 1, 0)
print getrefcount(rng1)

from cPickle import dumps, loads
rng2, n3, n4 = loads (dumps ((rng1, n1, n2), -1))

In [10]: ## working on region in file /usr/tmp/python-18983NT.py...
2
3
4

In [11]: rng2
Out[11]: <boost_rand.rng at 0x307c838>

In [12]: n3.rng
Out[12]: <boost_rand.rng at 0x307c838>

In [13]: n4.rng
Out[13]: <boost_rand.rng at 0x307c838>

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig