Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

Adam Preble
I am trying to break a smart pointer cycle in my application.  There is an interface that can be implemented in both c++ and Python, and it is used for messaging callbacks.  I can contain a list of these as shared pointers, but it only makes sense for cases where the owner of the container has exclusive control of the object in question.  These work fine in both c++ and Python.  I decided for the messaging callback list I would switch them to weak_ptr's. 

I get into trouble if I register a Python implementation of the callback with ownership staying within Python.  When it comes time to communicate a message back, I see that the weak_ptr has:

use_count = 0
weak_count = 1

The pointer fails to cast.  Meanwhile, the object is looks very much alive in Python.  I have a destructor in the c++ definition interface for debugging and I see it never gets called.  Similarly, __del__ doesn't get called on the Python implementation, although I am lead to believe I can't trust it.  Perhaps most compelling is that I can continue to poke and prod at the object in the Python shell well after this failed weak_ptr cast.  

I am wondering if there is anything I can do for the cast to succeed.  At this point all I can think of is making the owner of the container get a specific reference to the blasted thing, which I'd rather not have to do.


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

Holger Brandsmeier
Adam,

you probably run into a similar problem that I ran into (not with
boost::shared_ptr but with our custom implementation which strongly
mimics boost::shared_ptr). I wanted to have a class in C++ that is
extended in python, and it should store a weak_ptr to itself.
Unfortunately I didn't solve this yet, I stored a shared_ptr which
means that the object gets never deleted. I have to revisit this at a
later point, but I'll try to explain you what's going on.

You probably have a function like this

void addCallback(share_ptr<> arg) {
callbacks.push_back(weak_ptr<>(arg);
}

Or you immediately take the argument as a weak_ptr, that doesn't
matter. The more important fact is, you created that object in some
C++, you have that object in python and then you call addCallback from
the python context.

So, before the function exits your object `arg` exists at least in three places:
 1) somewhere in C++ where it was created
 2) in the python context
 3) in the context of addCallback

To understand what is going wrong, I need to explain that the shared
pointer in context 1 and 3 are not the same! They share the same
deallocator and the same object, but they are not the same shared
pointer, i.e. their use count and weak_count are not the same. That is
due to the way boost python handles step 2, that magic is in these
three files (in boost/python):

converter/shared_ptr_to_python.hpp
converter/shared_ptr_from_python.hpp
converter/shared_ptr_deleter.hpp

In `shared_ptr_deleter.hpp` there is a certain magic implemented, that
does the following:
Assume you have class `Parent` and a class `Child` derived from it.
Now you can do:
 - create an instance of Child C++ and bring it to python as shared_ptr<Child>
 - pass that instance to C++ (via shared_ptr<Child> or shared_ptr<Parent>)
 - get it later back from C++ but as a shared_ptr<Parent>
 - magic: you can treat that instance as a shared_ptr<Child>
In C++ you would need to do a dynamic cast to get this functionallity,
but because that object has been known to python to be an instance of
Child, boost::python automatically makes it an instance of Child, nice
right?

Unfortunately your (and my) problem are a consequence of this. When
you go from 2->3 boost::python prepares for doing its magic. It
doesn't just return a copy of the shared_ptr from 1), it creates a new
shared_ptr with a special Deallocator object. The use_count at that
moment is 2: one for python 2) and one for addCallback() 3). When the
function addCallback() finishes, the use_count=1 (from python) and
weak_count=1 from 3). Once the python context ends then use_count=0
and weak_count=1, and I believe that is exactly what you observe.

In this case as use_count drops to 0 the boost custom Deallocator gets
called. This is usally not bad, as he just deregisters (decreases the
use cound by 1) in the shared_ptr for context 1.Only if that use_count
in context 1) would drop to 0 the object would get deleted (that's why
you don't observe it to be deleted). The problem is now, that the two
weak/shared ptrs (which still point to a healthy and alive object) are
now disconnected. So when you try to turn the weak_ptr in context3 to
a strong pointer you would get serious problems.

I hope this is a correct explanation of the sitation. To solve this
would need to revist the magic for shared_ptrs in boost::python. I
plan to try and solve it some point later, but I am no regular
developer for boost::python and I can not promise that I will succeed,
nor when.

-Holger

On Tue, Mar 6, 2012 at 03:23, Adam Preble <[hidden email]> wrote:

> I am trying to break a smart pointer cycle in my application.  There is an
> interface that can be implemented in both c++ and Python, and it is used for
> messaging callbacks.  I can contain a list of these as shared pointers, but
> it only makes sense for cases where the owner of the container has exclusive
> control of the object in question.  These work fine in both c++ and Python.
>  I decided for the messaging callback list I would switch them to
> weak_ptr's.
>
> I get into trouble if I register a Python implementation of the callback
> with ownership staying within Python.  When it comes time to communicate a
> message back, I see that the weak_ptr has:
>
> use_count = 0
> weak_count = 1
>
> The pointer fails to cast.  Meanwhile, the object is looks very much alive
> in Python.  I have a destructor in the c++ definition interface for
> debugging and I see it never gets called.  Similarly, __del__ doesn't get
> called on the Python implementation, although I am lead to believe I can't
> trust it.  Perhaps most compelling is that I can continue to poke and prod
> at the object in the Python shell well after this failed weak_ptr cast.
>
> I am wondering if there is anything I can do for the cast to succeed.  At
> this point all I can think of is making the owner of the container get a
> specific reference to the blasted thing, which I'd rather not have to do.
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

Adam Preble


On Tue, Mar 6, 2012 at 3:47 AM, Holger Brandsmeier <[hidden email]> wrote:
Adam,

You probably have a function like this

void addCallback(share_ptr<> arg) {
callbacks.push_back(weak_ptr<>(arg);
}


That's pretty much what I have happening.
 

So, before the function exits your object `arg` exists at least in three places:
 1) somewhere in C++ where it was created
 2) in the python context
 3) in the context of addCallback


Technically, it was declared and constructed from Python, but everything you say is a consequence of this is consistent with what I'm seeing.  We could get into semantics here.  If I create an object implementing a C++ interface, do we consider that created in Python or would it be regarded as created in the C++ runtime?  
 
Assume you have class `Parent` and a class `Child` derived from it.
Now you can do:
 - create an instance of Child C++ and bring it to python as shared_ptr<Child>
 - pass that instance to C++ (via shared_ptr<Child> or shared_ptr<Parent>)
 - get it later back from C++ but as a shared_ptr<Parent>
 - magic: you can treat that instance as a shared_ptr<Child>
In C++ you would need to do a dynamic cast to get this functionallity,
but because that object has been known to python to be an instance of
Child, boost::python automatically makes it an instance of Child, nice
right?

Unfortunately your (and my) problem are a consequence of this. When
you go from 2->3 boost::python prepares for doing its magic. It
doesn't just return a copy of the shared_ptr from 1), it creates a new
shared_ptr with a special Deallocator object. The use_count at that
moment is 2: one for python 2) and one for addCallback() 3). When the
function addCallback() finishes, the use_count=1 (from python) and
weak_count=1 from 3). Once the python context ends then use_count=0
and weak_count=1, and I believe that is exactly what you observe.

In this case as use_count drops to 0 the boost custom Deallocator gets
called. This is usally not bad, as he just deregisters (decreases the
use cound by 1) in the shared_ptr for context 1.Only if that use_count
in context 1) would drop to 0 the object would get deleted (that's why
you don't observe it to be deleted). The problem is now, that the two
weak/shared ptrs (which still point to a healthy and alive object) are
now disconnected. So when you try to turn the weak_ptr in context3 to
a strong pointer you would get serious problems.


Sounds about like what I'm dealing with here.  Perhaps of particular note is that the pointers I'm moving around are typed for a parent class, but the actual reference is to a child.  
 
I hope this is a correct explanation of the sitation. To solve this
would need to revist the magic for shared_ptrs in boost::python. I
plan to try and solve it some point later, but I am no regular
developer for boost::python and I can not promise that I will succeed,
nor when.


Originally I was using shared_ptr instead of weak_ptr for the callback managers, and found some stuff never got deleted.  Could this process also cause the disconnect there? 

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

Holger Brandsmeier
Adam,

>> So, before the function exits your object `arg` exists at least in three
>> places:
>>  1) somewhere in C++ where it was created
>>  2) in the python context
>>  3) in the context of addCallback
>>
>
> Technically, it was declared and constructed from Python, but everything you
> say is a consequence of this is consistent with what I'm seeing.  We could
> get into semantics here.  If I create an object implementing a C++
> interface, do we consider that created in Python or would it be regarded as
> created in the C++ runtime?

Regard it as beein created from C++, as it has been created from boost
python. If it would be a pure python object, then it would be a
different story as there would not be a shared_ptr in the first place.

>> Assume you have class `Parent` and a class `Child` derived from it.
>> Now you can do:
>>  - create an instance of Child C++ and bring it to python as
>> shared_ptr<Child>
>>  - pass that instance to C++ (via shared_ptr<Child> or shared_ptr<Parent>)
>>  - get it later back from C++ but as a shared_ptr<Parent>
>>  - magic: you can treat that instance as a shared_ptr<Child>
>> In C++ you would need to do a dynamic cast to get this functionallity,
>> but because that object has been known to python to be an instance of
>> Child, boost::python automatically makes it an instance of Child, nice
>> right?
>>
>> Unfortunately your (and my) problem are a consequence of this. When
>> you go from 2->3 boost::python prepares for doing its magic. It
>> doesn't just return a copy of the shared_ptr from 1), it creates a new
>> shared_ptr with a special Deallocator object. The use_count at that
>> moment is 2: one for python 2) and one for addCallback() 3). When the
>> function addCallback() finishes, the use_count=1 (from python) and
>> weak_count=1 from 3). Once the python context ends then use_count=0
>> and weak_count=1, and I believe that is exactly what you observe.
>>
>> In this case as use_count drops to 0 the boost custom Deallocator gets
>> called. This is usally not bad, as he just deregisters (decreases the
>> use cound by 1) in the shared_ptr for context 1.Only if that use_count
>> in context 1) would drop to 0 the object would get deleted (that's why
>> you don't observe it to be deleted). The problem is now, that the two
>> weak/shared ptrs (which still point to a healthy and alive object) are
>> now disconnected. So when you try to turn the weak_ptr in context3 to
>> a strong pointer you would get serious problems.
>>
>
> Sounds about like what I'm dealing with here.  Perhaps of particular note is
> that the pointers I'm moving around are typed for a parent class, but the
> actual reference is to a child.

I don't think that it matters if you actually use that feature or not.
Just because the functionality is there, you get the problem. I just
wanted to give you an idea of why it is very nice that we have that
functionality.

>> I hope this is a correct explanation of the sitation. To solve this
>> would need to revist the magic for shared_ptrs in boost::python. I
>> plan to try and solve it some point later, but I am no regular
>> developer for boost::python and I can not promise that I will succeed,
>> nor when.
>>
>
> Originally I was using shared_ptr instead of weak_ptr for the callback
> managers, and found some stuff never got deleted.  Could this process also
> cause the disconnect there?

If you are storing the result as a shared_ptr in the problem, then you
will not run into problem, that is what I am doing at the moment. This
implies that my class never gets deleted, which I can accept at the
moment, but its actually quite bad.

Maybe someone else on the list has a solution for this. At the moment
I can only explain what I believe is causing the error.

-Holger
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Retaining a useful weak_ptr to a wrapped C++ interface implemented in Python

Adam Preble
Do have a way to unregister the callback?  When I was using a shared_ptr originally, I was trying to use that, yet the objects still hung around.  I was thinking that I properly didn't completely unregister it from all listeners, but I wonder now if I actually managed to cover all my tracks.  I'm working through some unit tests goofs from my most recent refactoring related to all this, so I can't reliably test this out.  When I get the whole thing working together again, I'm thinking of trying again with some callbacks and seeing that the objects in the shared pointers are getting destroyed.

On Tue, Mar 6, 2012 at 9:46 AM, Holger Brandsmeier <[hidden email]> wrote:
If you are storing the result as a shared_ptr in the problem, then you
will not run into problem, that is what I am doing at the moment. This
implies that my class never gets deleted, which I can accept at the
moment, but its actually quite bad.

Maybe someone else on the list has a solution for this. At the moment
I can only explain what I believe is causing the error.


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig