Boost.Python: Callbacks to class functions

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Boost.Python: Callbacks to class functions

paulcmnt
I have an `EventManager` class written in C++ and exposed to Python. This is how I intended for it to be used from the Python side:

    class Something:
        def __init__(self):
            EventManager.addEventHandler(FooEvent, self.onFooEvent)
        def __del__(self):
            EventManager.removeEventHandler(FooEvent, self.onFooEvent)
        def onFooEvent(self, event):
            pass

(The `add-` and `remove-` are exposed as static functions of `EventManager`.)

The problem with the above code is that the callbacks are captured inside `boost::python::object` instances; when I do `self.onFooEvent` these will increase the reference count of `self`, which will prevent it from being deleted, so the destructor never gets called, so the event handlers never get removed (except at the end of the application).

The code works well for functions that don't have a `self` argument (i.e. free or static functions). How should I capture Python function objects such that I won't increase their reference count? I only need a weak reference to the objects.
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python: Callbacks to class functions

Nat Goodspeed-2
On Aug 1, 2011, at 10:50 AM, diego_pmc <[hidden email]> wrote:

> How should I capture Python function objects such
> that I won't increase their reference count? I only need a weak reference to the objects.

http://docs.python.org/library/weakref.html#module-weakref

I don't know how to access a Python weakref from Boost.Python.
>
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python: Callbacks to class functions

Jim Bosch-2
In reply to this post by paulcmnt
On 08/01/2011 07:50 AM, diego_pmc wrote:

> I have an `EventManager` class written in C++ and exposed to Python. This is
> how I intended for it to be used from the Python side:
>
>      class Something:
>          def __init__(self):
>              EventManager.addEventHandler(FooEvent, self.onFooEvent)
>          def __del__(self):
>              EventManager.removeEventHandler(FooEvent, self.onFooEvent)
>          def onFooEvent(self, event):
>              pass
>
> (The `add-` and `remove-` are exposed as static functions of
> `EventManager`.)
>
> The problem with the above code is that the callbacks are captured inside
> `boost::python::object` instances; when I do `self.onFooEvent` these will
> increase the reference count of `self`, which will prevent it from being
> deleted, so the destructor never gets called, so the event handlers never
> get removed (except at the end of the application).
>
> The code works well for functions that don't have a `self` argument (i.e.
> free or static functions). How should I capture Python function objects such
> that I won't increase their reference count? I only need a weak reference to
> the objects.

Are these cycles actually a problem in practice?  Python does do garbage
collection, so it might be that it knows about all these dependencies
and just hasn't bothered to try to delete them because it doesn't need
the memory yet.

Anyhow, as the other reply suggested, one option is clearly weakref (in
addition, look at http://docs.python.org/c-api/weakref.html for how to
use those in C/C++; there's no special Boost.Python interface for them).

Unfortunately, what could have been a better option - implementing the
special functions that tell Python how to break cycles in your C++
classes (http://docs.python.org/c-api/gcsupport.html) - is pretty low
level, and probably impossible with Boost.Python.

Good luck!

Jim Bosch
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python: Callbacks to class functions

paulcmnt

On Tue, Aug 2, 2011 at 3:00 AM, Nat Goodspeed <[hidden email]> wrote:

http://docs.python.org/library/weakref.html#module-weakref

 
Unfortunately, I can't use `weakref` I already tried that, but the problem is that the weak references get deleted (and their value set to `None`) before `__del__` is called. So when `removeEventHandler` is called, it won't be able to find the weakref it's supposed to remove (because its value has been changed to `None`).

To fix, this would require some special handling of Python function objects in the `EventManager` — I don't want that, I wish to keep the manager agnostic to Python since I'm going to be pushing events into it from both C++ and Python. I'd like to solve this problem exclusively through the `EventManager` wrapper class I wrote.


On Tue, Aug 2, 2011 at 3:24 AM, Jim Bosch-2 [via Boost] <[hidden email]> wrote:

Are these cycles actually a problem in practice?  Python does do garbage
collection, so it might be that it knows about all these dependencies
and just hasn't bothered to try to delete them because it doesn't need
the memory yet.


 Yes, they are problematic. When the object gets removed it should not receive any more events or it will very likely result in some odd behavior on the screen. These objects are entity states in a game. A state dictates how an entity reacts to game events and when the entity changes its state, the old one should stop giving the entity instructions, or they will conflict with the instructions given by the new state.
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python: Callbacks to class functions

Jim Bosch-2
On 08/01/2011 09:44 PM, diego_pmc wrote:

>
>     Are these cycles actually a problem in practice?  Python does do
>     garbage
>     collection, so it might be that it knows about all these dependencies
>     and just hasn't bothered to try to delete them because it doesn't need
>     the memory yet.
>
>
>   Yes, they are problematic. When the object gets removed it should not
> receive any more events or it will very likely result in some odd
> behavior on the screen. These objects are entity states in a game. A
> state dictates how an entity reacts to game events and when the entity
> changes its state, the old one should stop giving the entity
> instructions, or they will conflict with the instructions given by the
> new state.

Hmm.  That might mean you need to do a big design change; while it often
works, one really isn't supposed to rely on __del__ being called when a
Python object first could be garbage-collected - when cycles are
involved, Python doesn't even guarantee that it will ever call __del__.
  It sounds like you'd be much better off with a named destructor-like
method that would be called explicitly when you want to remove an object
from the game.

Jim

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python: Callbacks to class functions

paulcmnt
On Tue, Aug 2, 2011 at 8:24 AM, Jim Bosch <[hidden email]> wrote:

Hmm.  That might mean you need to do a big design change; while it often works, one really isn't supposed to rely on __del__ being called when a Python object first could be garbage-collected - when cycles are involved, Python doesn't even guarantee that it will ever call __del__.  It sounds like you'd be much better off with a named destructor-like method that would be called explicitly when you want to remove an object from the game.


I am aware of that. :) Still, I'd like to leave it only as a last resort, I don't like complicating the API (who does? :P ). If I get rid of the strong reference that the `EventManager` has over the callback, there will be no cycles. My dependency graph would look something like this:


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig