Iterating over small objects

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Iterating over small objects

babak
Hi guys,

Has anyone encountered (what feels like) quite slow iteration when looping over small containers ? I'm in a situation where I'm iterating over 100000+ 3d vectors and its taking much longer than I'd expect. The vector type is bound via boost::python and has an iter method which uses range(being, end) for the iteration.

I've attached a simple example binding and python test to illustrate what I mean more clearly. I'm under the impression that in a situation like this the overhead of creating the iterator is greater than the actual iteration, but thats just a guess. Alternatively my perspective on what the iteration speed should be might be completely skewed, so any help in clarifying what's going on here would be greatly appreciated.

thanks
Babak

source.cpp
test.py
Reply | Threaded
Open this post in threaded view
|

Re: Iterating over small objects

Jim Bosch-2
On 01/19/2012 02:08 PM, babak wrote:

> Hi guys,
>
> Has anyone encountered (what feels like) quite slow iteration when looping
> over small containers ? I'm in a situation where I'm iterating over 100000+
> 3d vectors and its taking much longer than I'd expect. The vector type is
> bound via boost::python and has an iter method which uses range(being, end)
> for the iteration.
>
> I've attached a simple example binding and python test to illustrate what I
> mean more clearly. I'm under the impression that in a situation like this
> the overhead of creating the iterator is greater than the actual iteration,
> but thats just a guess. Alternatively my perspective on what the iteration
> speed should be might be completely skewed, so any help in clarifying what's
> going on here would be greatly appreciated.
>

My sense is that this is something that really can't be expected to be
efficient across the C++/Python boundary.  There is some overhead
associated with creating the exposures, and there's also some overhead
in converting the types, especially if those vectors aren't builtin
numeric types.

If you can redesign the C++/Python boundary to convert your vectors
directly into Python lists or tuples instead of wrapped C++ vectors, I
expect you'll see an improvement, because you'll be able to take
advantage of whatever optimization the Python developers have put into
iteration over those classes.  Of course, it's best by far to do
whatever iteration you can in C++ if performance is important, but I'm
guessing that's not an option here.

Jim

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Iterating over small objects

babak
Okay cool thanks your input.

For any one else has a similar problem - writing a simple iterator function in python (which is a generator) and assigning that to the __iter__ member works well as an alternative. For me this turned out to be considerably faster than using a c++ iterator/getitem iteration.

On 19 January 2012 19:37, Jim Bosch <[hidden email]> wrote:
On 01/19/2012 02:08 PM, babak wrote:
Hi guys,

Has anyone encountered (what feels like) quite slow iteration when looping
over small containers ? I'm in a situation where I'm iterating over 100000+
3d vectors and its taking much longer than I'd expect. The vector type is
bound via boost::python and has an iter method which uses range(being, end)
for the iteration.

I've attached a simple example binding and python test to illustrate what I
mean more clearly. I'm under the impression that in a situation like this
the overhead of creating the iterator is greater than the actual iteration,
but thats just a guess. Alternatively my perspective on what the iteration
speed should be might be completely skewed, so any help in clarifying what's
going on here would be greatly appreciated.


My sense is that this is something that really can't be expected to be efficient across the C++/Python boundary.  There is some overhead associated with creating the exposures, and there's also some overhead in converting the types, especially if those vectors aren't builtin numeric types.

If you can redesign the C++/Python boundary to convert your vectors directly into Python lists or tuples instead of wrapped C++ vectors, I expect you'll see an improvement, because you'll be able to take advantage of whatever optimization the Python developers have put into iteration over those classes.  Of course, it's best by far to do whatever iteration you can in C++ if performance is important, but I'm guessing that's not an option here.

Jim


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig