question about implicit type conversion of vector to a custom type

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

question about implicit type conversion of vector to a custom type

Grant Tang
This is a question about using implicitly_convertible to convert vector to another custom class type.

I already have boost python type conversion code to convert Python list/tuple to c++ vector, which works fine. There is another custom class EMObject, which is a thin wrapper to many builit-in type. It takes a built-in type like int, float, string, and vector as a single constructor argument and return type. So those built-in type can convert to and from EMObject implicitly. The functions in c++ which takes this EMObject as an argument are expose to Python. Now I need call these functions in Python with argument from Python list or tuple. Since I already convert python list/tuple to c++ vector, I just need declare implicit type conversion from vector to EMObject like following:

implicitly_convertible<vector<float>, EMAN::EMObject>();
implicitly_convertible<vector<int>, EMAN::EMObject>();
implicitly_convertible<vector<std::string>, EMAN::EMObject>();

This seems a perfect solution to my question. But unfortunately I find out only the first one working, regardless which one is the first. Could anybody tell me why this happen?

--
Grant

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: question about implicit type conversion of vector to a custom type

Jim Bosch-2
On 08/16/2011 12:43 PM, Grant Tang wrote:

> This is a question about using implicitly_convertible to convert vector
> to another custom class type.
>
> I already have boost python type conversion code to convert Python
> list/tuple to c++ vector, which works fine. There is another custom
> class EMObject, which is a thin wrapper to many builit-in type. It takes
> a built-in type like int, float, string, and vector as a single
> constructor argument and return type. So those built-in type can convert
> to and from EMObject implicitly. The functions in c++ which takes this
> EMObject as an argument are expose to Python. Now I need call these
> functions in Python with argument from Python list or tuple. Since I
> already convert python list/tuple to c++ vector, I just need declare
> implicit type conversion from vector to EMObject like following:
>
> implicitly_convertible<vector<float>, EMAN::EMObject>();
> implicitly_convertible<vector<int>, EMAN::EMObject>();
> implicitly_convertible<vector<std::string>, EMAN::EMObject>();
>
> This seems a perfect solution to my question. But unfortunately I find
> out only the first one working, regardless which one is the first. Could
> anybody tell me why this happen?
>

Unfortunately, there's no "best-match" type checking in Boost.Python;
when trying to convert a Python object to a particular C++ type, it
simply runs over a list of all the converters registered (using RTTI)
for that C++ type.

Each of those converters then gets a chance to check whether it
"matches" the Python object; if it thinks it does, none of the other
converters get tried.

I'd guess that's what's happening to you - implicitly_convertible
doesn't really do enough work in that match stage, so the first
conversion you've registered says "I match!" and then fails by throwing
an exception.

The fix might be in how you've written your vector<> conversions; when
given a list or tuple that contains the wrong element type, they need to
report that they "don't match", rather than matching any list or tuple
and then throwing an exception when the element type is incorrect.

Good luck!

Jim Bosch
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: question about implicit type conversion of vector to a custom type

Grant Tang


"Jim Bosch"  wrote in message news:[hidden email]...

On 08/16/2011 12:43 PM, Grant Tang wrote:

> This is a question about using implicitly_convertible to convert vector
> to another custom class type.
>
> I already have boost python type conversion code to convert Python
> list/tuple to c++ vector, which works fine. There is another custom
> class EMObject, which is a thin wrapper to many builit-in type. It takes
> a built-in type like int, float, string, and vector as a single
> constructor argument and return type. So those built-in type can convert
> to and from EMObject implicitly. The functions in c++ which takes this
> EMObject as an argument are expose to Python. Now I need call these
> functions in Python with argument from Python list or tuple. Since I
> already convert python list/tuple to c++ vector, I just need declare
> implicit type conversion from vector to EMObject like following:
>
> implicitly_convertible<vector<float>, EMAN::EMObject>();
> implicitly_convertible<vector<int>, EMAN::EMObject>();
> implicitly_convertible<vector<std::string>, EMAN::EMObject>();
>
> This seems a perfect solution to my question. But unfortunately I find
> out only the first one working, regardless which one is the first. Could
> anybody tell me why this happen?
>

Unfortunately, there's no "best-match" type checking in Boost.Python;
when trying to convert a Python object to a particular C++ type, it
simply runs over a list of all the converters registered (using RTTI)
for that C++ type.

Each of those converters then gets a chance to check whether it
"matches" the Python object; if it thinks it does, none of the other
converters get tried.

I'd guess that's what's happening to you - implicitly_convertible
doesn't really do enough work in that match stage, so the first
conversion you've registered says "I match!" and then fails by throwing
an exception.

The fix might be in how you've written your vector<> conversions; when
given a list or tuple that contains the wrong element type, they need to
report that they "don't match", rather than matching any list or tuple
and then throwing an exception when the element type is incorrect.

Good luck!

Jim Bosch


Thank you for your reply, Jim.
My vector conversion code is pretty simple and straightforward.
This is the conversion class in typeconversion.h file:

    template <class T>
    struct vector_from_python
    {
        vector_from_python()
        {
            python::converter::registry::push_back(&convertible, &construct,
                                                   python::type_id<vector<T>
>());
        }

        static void* convertible(PyObject* obj_ptr)
        {
            if (!(PyList_Check(obj_ptr) || PyTuple_Check(obj_ptr)
                  || PyIter_Check(obj_ptr)  || PyRange_Check(obj_ptr))) {
                return 0;
            }

            return obj_ptr;
        }


        static void construct(PyObject* obj_ptr,
                              python::converter::rvalue_from_python_stage1_data*
data)
        {
            void* storage =
((python::converter::rvalue_from_python_storage<vector<T> >*)
                             data)->storage.bytes;
            new (storage) vector<T>();

            data->convertible = storage;

            vector<T>& result = *((vector<T>*) storage);

            python::handle<> obj_iter(PyObject_GetIter(obj_ptr));

            while(1) {
                python::handle<>
py_elem_hdl(python::allow_null(PyIter_Next(obj_iter.get())));
                if (PyErr_Occurred()) {
                    python::throw_error_already_set();
                }

                if (!py_elem_hdl.get()) {
                    break;
                }

                python::object py_elem_obj(py_elem_hdl);
                python::extract<T> elem_proxy(py_elem_obj);
                result.push_back(elem_proxy());
            }
        }
    };

I specialize this template with different type in BOOST_PYTHON_MODULE:

vector_from_python<int>();
vector_from_python<float>();
vector_from_python<string>();

This solution looks like work fine for converting list/tuple to vector<>.
But I got this problem when need implicitly convert vector<> to EMObject. Do
you mean I should write a conversion class for each type of vector instead
of use a template class?

Grant


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: question about implicit type conversion of vector to a custom type

Grant Tang
In reply to this post by Jim Bosch-2

"Jim Bosch"  wrote in message news:[hidden email]...

>Unfortunately, there's no "best-match" type checking in Boost.Python; when
>trying to convert a Python object to a particular C++ type, it simply runs
>over a list of all the converters registered (using RTTI) for that C++
>type.
>
>Each of those converters then gets a chance to check whether it "matches"
>the Python object; if it thinks it does, none of the other converters get
>tried.
>
>I'd guess that's what's happening to you - implicitly_convertible doesn't
>really do enough work in that match stage, so the first conversion you've
>registered says "I match!" and then fails by throwing an exception.
>
>The fix might be in how you've written your vector<> conversions; when
>given a list or tuple that contains the wrong element type, they need to
>report that they "don't match", rather than matching any list or tuple and
>then throwing an exception when the element type is incorrect.
>
>Good luck!
>
>Jim Bosch

I change my vector<> conversion(see it in my last reply) to add
specialization for int, float, string etc.
I added type check in convertible() function:

    template <>
    struct vector_from_python <float>
    {
        vector_from_python()
        {
            python::converter::registry::push_back(&convertible, &construct,
                                                   python::type_id<vector<float>
 >());
        }

        static void* convertible(PyObject* obj_ptr)
        {
            if (!(PyList_Check(obj_ptr) || PyTuple_Check(obj_ptr)
                  || PyIter_Check(obj_ptr)  || PyRange_Check(obj_ptr))) {
                return 0;
            }

            PyObject * first_obj = PyObject_GetItem(obj_ptr,
PyInt_FromLong(0));
            if( !PyObject_TypeCheck(first_obj, &PyFloat_Type) ) {
                return 0;
            }

            return obj_ptr;
        }


        static void construct(PyObject* obj_ptr,
                              python::converter::rvalue_from_python_stage1_data*
data)
        {
            void* storage =
((python::converter::rvalue_from_python_storage<vector<float> >*)
                             data)->storage.bytes;
            new (storage) vector<float>();

            data->convertible = storage;

            vector<float>& result = *((vector<float>*) storage);

            python::handle<> obj_iter(PyObject_GetIter(obj_ptr));

            while(1) {
                python::handle<>
py_elem_hdl(python::allow_null(PyIter_Next(obj_iter.get())));
                if (PyErr_Occurred()) {
                    python::throw_error_already_set();
                }

                if (!py_elem_hdl.get()) {
                    break;
                }

                python::object py_elem_obj(py_elem_hdl);
                python::extract<float> elem_proxy(py_elem_obj);
                result.push_back(elem_proxy());
            }
        }
    };

This time the implicit type conversion works perfectly. But I got a new
problem: the memory leak!
The memory leak happens only for float type, whenever I convert the float
list in python to vector
of float in c++, the memory for float list is leaked. I put the call in a
function, and called the gc.collect()
after exit the function, the memory is still not recycled.

Why is the memory of the python list is not freed after exit the scope?

Grant


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: question about implicit type conversion of vector to a custom type

Jim Bosch-2
On 08/18/2011 09:37 AM, Grant Tang wrote:
>
> I change my vector<> conversion(see it in my last reply) to add
> specialization for int, float, string etc.
> I added type check in convertible() function:
>

Sorry I didn't reply to your last email earlier, but it looks like
you're on the right track.

Here's a slightly modified version of your convertible function:

static void* convertible(PyObject* obj_ptr)
{
   // Unfortunately, this no longer works on pure iterators, because
   // they don't necessarily support "obj[0]".
   // PySequence_Check should work on lists, tuples, and other
   // things that support __getitem__ with integer arguments.
   if (!(PySequence_Check(obj_ptr)) {
     return 0;
   }
   // Succeed if there are no items in the sequence.
   if (PySequence_Size(obj_ptr) == 0) {
     return obj_ptr;
   }
   // PySequence_GetItem takes an int directly; otherwise you'd need
   // to DECREF the int object you pass to PyObject_GetItem as well.
   PyObject * first_obj = PySequence_GetItem(obj_ptr, 0);
   if( !PyObject_TypeCheck(first_obj, &PyFloat_Type) ) {
     Py_DECREF(first_obj); // avoid memory leaks on failure
     return 0;
   }
   Py_DECREF(first_obj); // avoid memory leaks on success
   return obj_ptr;
}

> This time the implicit type conversion works perfectly. But I got a new
> problem: the memory leak!
> The memory leak happens only for float type, whenever I convert the
> float list in python to vector
> of float in c++, the memory for float list is leaked. I put the call in
> a function, and called the gc.collect()
> after exit the function, the memory is still not recycled.
>
> Why is the memory of the python list is not freed after exit the scope?
>

It's the missing Py_DECREF calls; it's only because of a Python
implementation detail that you didn't have leaks with the other types.
The Python C-API requires you to call Py_DECREF whenever you're done
with an object.  Alternately, you can put your PyObject pointers in
bp::handle<> or bp::object, and they'll be managed automatically.

Jim

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: question about implicit type conversion of vector to a custom type

Grant Tang

"Jim Bosch"  wrote in message news:[hidden email]...

On 08/18/2011 09:37 AM, Grant Tang wrote:
>
> I change my vector<> conversion(see it in my last reply) to add
> specialization for int, float, string etc.
> I added type check in convertible() function:
>

Sorry I didn't reply to your last email earlier, but it looks like
you're on the right track.

Here's a slightly modified version of your convertible function:

static void* convertible(PyObject* obj_ptr)
{
   // Unfortunately, this no longer works on pure iterators, because
   // they don't necessarily support "obj[0]".
   // PySequence_Check should work on lists, tuples, and other
   // things that support __getitem__ with integer arguments.
   if (!(PySequence_Check(obj_ptr)) {
     return 0;
   }
   // Succeed if there are no items in the sequence.
   if (PySequence_Size(obj_ptr) == 0) {
     return obj_ptr;
   }
   // PySequence_GetItem takes an int directly; otherwise you'd need
   // to DECREF the int object you pass to PyObject_GetItem as well.
   PyObject * first_obj = PySequence_GetItem(obj_ptr, 0);
   if( !PyObject_TypeCheck(first_obj, &PyFloat_Type) ) {
     Py_DECREF(first_obj); // avoid memory leaks on failure
     return 0;
   }
   Py_DECREF(first_obj); // avoid memory leaks on success
   return obj_ptr;
}

> This time the implicit type conversion works perfectly. But I got a new
> problem: the memory leak!
> The memory leak happens only for float type, whenever I convert the
> float list in python to vector
> of float in c++, the memory for float list is leaked. I put the call in
> a function, and called the gc.collect()
> after exit the function, the memory is still not recycled.
>
> Why is the memory of the python list is not freed after exit the scope?
>

It's the missing Py_DECREF calls; it's only because of a Python
implementation detail that you didn't have leaks with the other types.
The Python C-API requires you to call Py_DECREF whenever you're done
with an object.  Alternately, you can put your PyObject pointers in
bp::handle<> or bp::object, and they'll be managed automatically.

Jim



Bingo! This fixes my problem. Thank you very much Jim!

Grant

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig