wrapped std::vector slice affecting items in a python list

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

wrapped std::vector slice affecting items in a python list

babak
Hi,

I've come across some unusual behaviour that I was hoping some one might be able to explain to me.

In python I have a list and a wrapped std::vector<MyCustomType> (using the vector_indexing_suite) where the list contains the items in the vector. If I slice insert items into the vector then the items in the python list change. The attached example illustrates this.

Is this behaviour expected ?

I'm quite confused as to what's going on so any help clarifying this would be greatly appreciated.
thanks,
Babak

test.py
source.cpp
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

Jim Bosch-2
On 08/12/2011 04:28 AM, babak wrote:

> Hi,
>
> I've come across some unusual behaviour that I was hoping some one might be
> able to explain to me.
>
> In python I have a list and a wrapped std::vector<MyCustomType>  (using the
> vector_indexing_suite) where the list contains the items in the vector. If I
> slice insert items into the vector then the items in the python list change.
> The attached example illustrates this.
>
> Is this behaviour expected ?
>
> I'm quite confused as to what's going on so any help clarifying this would
> be greatly appreciated.
> thanks,
> Babak
>
> http://boost.2283326.n4.nabble.com/file/n3738972/test.py test.py
> http://boost.2283326.n4.nabble.com/file/n3738972/source.cpp source.cpp
>

Some of the stuff vector_indexing_suite is sufficiently complex that at
least I'd say this behavior is not unexpected.  After all, when you
slice a Python list, and modify the elements in the sliced bit, you
modify the original list as well (unless your elements are immutable
things like strings or numbers, and in that case you aren't really
modifying them, you're replacing them).

I think the thing you get back from slicing a std::vector is probably a
proxy object designed to have exactly that behavior (note: I haven't
checked; that's just how I'd have implemented it).

In any case, I'd recommend against wrapping a std::vector of a mutable
C++ type directly in Python.  It's almost always unsafe, because any
operation on the vector that invalidates iterators will destroy the
elements of the vector, even if there are Python objects that point to
those elements elsewhere with nonzero reference counts.  It's much safer
to use a vector of shared_ptr, or always convert to Python lists and
avoid vector_indexing_suite.  This isn't a Boost.Python-specific
problem, it's an inherent incompatibility between C++'s value-based
containers and Python's shared-reference ones.

Jim Bosch

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

babak
Hi Jim

Thanks for your response.

I'm abit surprised that this is the case, I kind of thought that indexing_suite was there to bridge this incompatibility. Is it normal then for large projects reliant on value based containers to copy data when going from python to c++ ?

thanks
babak
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

Jim Bosch-2
On 08/16/2011 07:32 AM, babak wrote:
> Hi Jim
>
> Thanks for your response.
>
> I'm abit surprised that this is the case, I kind of thought that
> indexing_suite was there to bridge this incompatibility. Is it normal then
> for large projects reliant on value based containers to copy data when going
> from python to c++ ?
>

I don't really know what normal is; my particular area is scientific
computing, so I actually use numpy arrays in almost all the cases where
I'm dealing with huge containers.  I do tend to copy data when dealing
with small containers of more complex objects, and when it's really an
issue, I make my C++ objects cheap to copy (via copy on write or
something).  Things are also a lot safer if you only wrap vectors as
const; it's exposing the mutators to Python that gets really difficult.
  Finally, if your C++ code works on iterators rather than containers,
you can avoid copying by using stl_input_iterator to iterate directly
over the Python containers.

One large project I'm a part of uses vectors of shared_ptr extensively
on the C++ side to combat this problem (even though they use SWIG, not
Boost.Python).  If you have control over the C++, that's often the best
solution, regardless of how you're exposing them to Python.

I would that any large project that uses SWIG with value-based
containers is probably mostly ignoring the safety issues; unlike
Boost.Python, SWIG simply doesn't have the facilities to protect you
from null pointers and dangling references, so its STL wrappers don't do
anything to prevent such things.  The onus is on the Python user to be safe.

On the Boost.Python side, indexing_suite essentially does the best it
can to make things safe and full-featured, and while it can't totally
foolproof your code, it does make it a lot more difficult to segfault
accidentally.  If you find yourself needing it a lot, you may want to
looking into indexing suite v2, which I believe is best obtained through
the Py++ package.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

babak
Okay thanks for the info.

Things are also a lot safer if you only wrap vectors as const; it's exposing the mutators to Python that gets really difficult. 

Is the const-ness of a wrapped object just a side effect of not exposing methods which modify it or is it due to something else more explicit ?

thanks!

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

Jim Bosch-2
On 08/16/2011 11:29 AM, Babak Khataee wrote:
> Okay thanks for the info.
>
>     Things are also a lot safer if you only wrap vectors as const; it's
>     exposing the mutators to Python that gets really difficult.
>
>
> Is the const-ness of a wrapped object just a side effect of not exposing
> methods which modify it or is it due to something else more explicit ?

The former.  It's really only the methods that can invalidate iterators
(i.e. those that add or remove elements) that you need to worry about.
Non-const access to individual elements is totally safe on its own.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

babak
okay cool. I think that's a reasonable compromise. It should also (hopefully) make it possible to move away from the indexing suite as well.

thanks for your help!

On 16 August 2011 21:41, Jim Bosch <[hidden email]> wrote:
On 08/16/2011 11:29 AM, Babak Khataee wrote:
Okay thanks for the info.

   Things are also a lot safer if you only wrap vectors as const; it's
   exposing the mutators to Python that gets really difficult.


Is the const-ness of a wrapped object just a side effect of not exposing
methods which modify it or is it due to something else more explicit ?

The former.  It's really only the methods that can invalidate iterators (i.e. those that add or remove elements) that you need to worry about. Non-const access to individual elements is totally safe on its own.

Jim


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapped std::vector slice affecting items in a python list

babak
Hi,

Thought you might be interested to know that I think the cause of the problem in my example is the iter method that the indexing suite adds. Unlike getitem it doesn't seem to return proxy objects, so when I slice insert into the cpp container some of the references made by iterating through it change, instead of becoming detached.

babak
Loading...