Abstract class instances to-python conversion

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Abstract class instances to-python conversion

Valentin Perrelle
Hi,

I'm currently embedding and extending Python using Boost.Python. I'm
trying to expose some c++ library (OIS) classes to my Python scripts.
This library mainly exposes abstract classes since the actual
implementation are derived classes specialized for each operating system.

So, i have to wrap those abstract classes. Then, eventually i will
convert some existing C++ object of this class to its Python equivalent.
Ideally, I would think that the HeldType is a pointer to the existing
C++ object. There is no pointer managment problem since the lifetime of
the object is greater than the execution time of the script.

There is probably something i don't understand in the design of
Boost.Python. At this point, my problem is that i need a to_python
conversion which requires the abscence of noncopyable attribute which
then implies to be able to build an instance of the object (which i
cannot provide since the class is abstract, and i don't have any
concrete derived class). I don't yet understand why holding a pointer in
the Python object requires the ability to build instances of the wrapped
class. The exact compile-time error i get is:

boost_1_44/boost/python/object/pointer_holder.hpp:194:14: error: cannot
allocate an object of abstract type 'OIS::Keyboard'

I'm using boost 1.44 (required by some other library) Python 3.2 and
MinGW/Msys. Module declaration:

BOOST_PYTHON_MODULE(OIS)
{
        class_<Keyboard, Keyboard*>("Keyboard", no_init);
}

Python init:

try {
        PyImport_AppendInittab("OIS", PyInit_OIS);
        Py_InitializeEx(0);

        object main(import("__main__"));
        dict globals(main.attr("__dict__"));
        globals["keyboard"] = ptr(keyboard);
}
catch (error_already_set&) {
        PyErr_Print();
}

Did i choose the wrong desing ? Did i do something wrong ? What should i
do to solve this problem ?


Valentin Perrelle.
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Jim Bosch-2
On 07/30/2011 01:15 PM, Valentin Perrelle wrote:

> Hi,
>
> I'm currently embedding and extending Python using Boost.Python. I'm
> trying to expose some c++ library (OIS) classes to my Python scripts.
> This library mainly exposes abstract classes since the actual
> implementation are derived classes specialized for each operating system.
>
> So, i have to wrap those abstract classes. Then, eventually i will
> convert some existing C++ object of this class to its Python equivalent.
> Ideally, I would think that the HeldType is a pointer to the existing
> C++ object. There is no pointer managment problem since the lifetime of
> the object is greater than the execution time of the script.
>

In this case, I don't you need to specify a HeldType, because that only
affects what happens when you construct the object in Python, and you
aren't every doing that.  But you do need to specify "noncopyable" if
the class doesn't have a copy constructor...

> There is probably something i don't understand in the design of
> Boost.Python. At this point, my problem is that i need a to_python
> conversion which requires the abscence of noncopyable attribute which
> then implies to be able to build an instance of the object (which i
> cannot provide since the class is abstract, and i don't have any
> concrete derived class). I don't yet understand why holding a pointer in
> the Python object requires the ability to build instances of the wrapped
> class. The exact compile-time error i get is:
>
> boost_1_44/boost/python/object/pointer_holder.hpp:194:14: error: cannot
> allocate an object of abstract type 'OIS::Keyboard'
>

Why do you think you can't have noncopyable?  You can certainly support
some to-python conversions (including the one you've invoked below with
"ptr(keyboard)") on a noncopyable class.  Are you hoping to support some
specific one, or is this the root of your misunderstanding?

Anyhow, I'd recommend trying

class_<Keyboard, noncopyable>("Keyboard", no_init);

with the rest of what you have.

Good luck!

Jim Bosch




>
> BOOST_PYTHON_MODULE(OIS)
> {
> class_<Keyboard, Keyboard*>("Keyboard", no_init);
> }
>
> Python init:
>
> try {
> PyImport_AppendInittab("OIS", PyInit_OIS);
> Py_InitializeEx(0);
>
> object main(import("__main__"));
> dict globals(main.attr("__dict__"));
> globals["keyboard"] = ptr(keyboard);
> }
> catch (error_already_set&) {
> PyErr_Print();
> }
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Valentin Perrelle
Thank you for your answer.

 > Why do you think you can't have noncopyable?

I understood it prevents the registration of converters for the class. I
just read the manual again and now i understand it only remove converter
which copy instances, i.e. conversion of values.

> Anyhow, I'd recommend trying
>
> class_<Keyboard, noncopyable>("Keyboard", no_init);
>
> with the rest of what you have.

That's what I initially tried. It produced runtime error:

TypeError: No Python class registered for C++ class OIS::Keyboard

Thanks to you, I know understand some points i couldn't catch reading
the manual 3-4 times. I'm now able to fix the current problem by
importing my module before assigning the variable 'keyboard'. Which
leads me to another question : is there a way to register converters
without importing the module ?


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Jim Bosch-2
On 07/30/2011 03:18 PM, Valentin Perrelle wrote:
> Thanks to you, I know understand some points i couldn't catch reading
> the manual 3-4 times. I'm now able to fix the current problem by
> importing my module before assigning the variable 'keyboard'. Which
> leads me to another question : is there a way to register converters
> without importing the module ?
>

Hmm.  That's a little tricky.  You can of course put your "class_"
definitions in an arbitrary C++ function, and call that somehow before
importing the module, but then you really shouldn't call that function
again when importing because that would register some bits of the class
twice.

I guess I don't really understand how your program flow is supposed to
work - how did you plan to invoke C++ code from Python before importing
your Boost.Python module?  Usually the natural place to register
converters is during module import after you've registered the classes.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Valentin Perrelle
Le 31/07/2011 07:38, Jim Bosch a écrit :
> I guess I don't really understand how your program flow is supposed to
> work - how did you plan to invoke C++ code from Python before
> importing your Boost.Python module?  Usually the natural place to
> register converters is during module import after you've registered
> the classes.

I don't plan to invoke c++ code from Python before importing the module:
I need to import the module before exposing some instance of some class
of the module. Does this mean that the module should be already imported
when the script start running ? Can't i expose the object and then let
decide the Python script wheter it need to use it or not ?


Valentin Perrelle.
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Jim Bosch-2
On 07/31/2011 12:52 AM, Valentin Perrelle wrote:

> Le 31/07/2011 07:38, Jim Bosch a écrit :
>> I guess I don't really understand how your program flow is supposed to
>> work - how did you plan to invoke C++ code from Python before
>> importing your Boost.Python module? Usually the natural place to
>> register converters is during module import after you've registered
>> the classes.
>
> I don't plan to invoke c++ code from Python before importing the module:
> I need to import the module before exposing some instance of some class
> of the module. Does this mean that the module should be already imported
> when the script start running ? Can't i expose the object and then let
> decide the Python script wheter it need to use it or not ?
>

All of that sounds sounds.  But at what point were you trying to
register a to-python converter?  It sounded like you were trying to do
that before importing the module, and since a to-python converter is by
definition C++, I didn't understand how you could do it in a Python
script before importing the module.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Valentin Perrelle

> All of that sounds sounds.  But at what point were you trying to
> register a to-python converter?  It sounded like you were trying to do
> that before importing the module, and since a to-python converter is
> by definition C++, I didn't understand how you could do it in a Python
> script before importing the module.

I'm registering the converter by calling the boost::python::import
function in C++ code. I don't know any other way to do that.

Now, this give me another error. I'm trying to implement a "reload
script" feature. I thought that all i had to do was to call Py_Finalize,
then Py_Initialize again, and to remove any references i was holding to
wrapping classes. But whenever i'm importing my extension again, i get
the runtime error:

Assertion failed: slot->m_to_python == 0, file
libs\python\src\converter\registry.cpp, line 212

which means my to_python converter have been registered once again. Is
there a way to unregister them ? should i find a to not initialize the
extension again ?

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

mrmclovin
You could use the reload() function in python 2.7 or imp.reload() in python 3. It takes a module object as argument.

// Simon

On 1 aug 2011, at 12:00, Valentin Perrelle <[hidden email]> wrote:

>
>> All of that sounds sounds.  But at what point were you trying to register a to-python converter?  It sounded like you were trying to do that before importing the module, and since a to-python converter is by definition C++, I didn't understand how you could do it in a Python script before importing the module.
>
> I'm registering the converter by calling the boost::python::import function in C++ code. I don't know any other way to do that.
>
> Now, this give me another error. I'm trying to implement a "reload script" feature. I thought that all i had to do was to call Py_Finalize, then Py_Initialize again, and to remove any references i was holding to wrapping classes. But whenever i'm importing my extension again, i get the runtime error:
>
> Assertion failed: slot->m_to_python == 0, file libs\python\src\converter\registry.cpp, line 212
>
> which means my to_python converter have been registered once again. Is there a way to unregister them ? should i find a to not initialize the extension again ?
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Valentin Perrelle

> You could use the reload() function in python 2.7 or imp.reload() in python 3. It takes a module object as argument.
Thanks. However it wouldn't reset to the initial state in the general
case. All modules needs to be unloaded. I don't know a safe way to it
yet. I'm just sure i want to do it in c++ not in Python.

I've just found that the call of Py_Finalize with Boost.Python is a
known issue, already discussed on this mailing list and that the manual
says not to call it :
http://www.boost.org/doc/libs/1_47_0/libs/python/doc/tutorial/doc/html/python/embedding.html

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

mrmclovin
In My program I need to unload modules as well. What I do is remove all references to the particular module and it will be unloaded.

Are you using boost python for python 2 or 3? If it's the latter it is safe to use Py_Finalize()! I use it myself!

// Simon

On 1 aug 2011, at 12:58, Valentin Perrelle <[hidden email]> wrote:

>
>> You could use the reload() function in python 2.7 or imp.reload() in python 3. It takes a module object as argument.
> Thanks. However it wouldn't reset to the initial state in the general case. All modules needs to be unloaded. I don't know a safe way to it yet. I'm just sure i want to do it in c++ not in Python.
>
> I've just found that the call of Py_Finalize with Boost.Python is a known issue, already discussed on this mailing list and that the manual says not to call it : http://www.boost.org/doc/libs/1_47_0/libs/python/doc/tutorial/doc/html/python/embedding.html
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Valentin Perrelle
Le 01/08/2011 13:19, Simon Warg a écrit :
> In My program I need to unload modules as well. What I do is remove all references to the particular module and it will be unloaded.
It seems i didn't achieve to do that. There should be some references i
can't remove, i don't know why yet.

>
> Are you using boost python for python 2 or 3? If it's the latter it is safe to use Py_Finalize()! I use it myself!
I'm using Python 3. But the problem of unregistered converters is still
there. See
http://mail.python.org/pipermail/cplusplus-sig/2009-August/014736.html


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

mrmclovin
You can remove one reference from the sys module:

Import sys
del sys.modules['mymodule']

In cpp it would be like:
import('sys').attr('modules')['mymodule'].del()

I can give you my code later. Don't have it here!

// Simon

On 1 aug 2011, at 13:38, Valentin Perrelle <[hidden email]> wrote:

> Le 01/08/2011 13:19, Simon Warg a écrit :
>> In My program I need to unload modules as well. What I do is remove all references to the particular module and it will be unloaded.
> It seems i didn't achieve to do that. There should be some references i can't remove, i don't know why yet.
>
>>
>> Are you using boost python for python 2 or 3? If it's the latter it is safe to use Py_Finalize()! I use it myself!
> I'm using Python 3. But the problem of unregistered converters is still there. See http://mail.python.org/pipermail/cplusplus-sig/2009-August/014736.html
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Abstract class instances to-python conversion

Valentin Perrelle

> In cpp it would be like:
> import('sys').attr('modules')['mymodule'].del()
Thank you, it worked. However, I believe it would only remove one
module, not any other module imported. I tried to clear the dictionnary,
which worked in Python, i didn't achieve to reproduce this in c++.
Anyway, this would free memory occupied by cycles in the python
reference graph. But this is only a matter of memory leak, not really
relevent in my context.

I tried another solution, which doesn't seem at first very suitable. I
created a subinterpreter for each new execution of the script. Since
there is never 2 execution at the same time, this ensure a good
independance between distinct executions. It worked well. This may be a
good solution, at least until the Py_Finalize issue is solved.

Thank you again for your help.

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig