# Iterating over small objects

 Classic List Threaded
3 messages
Reply | Threaded
Open this post in threaded view
|

## Iterating over small objects

 Hi guys, Has anyone encountered (what feels like) quite slow iteration when looping over small containers ? I'm in a situation where I'm iterating over 100000+ 3d vectors and its taking much longer than I'd expect. The vector type is bound via boost::python and has an iter method which uses range(being, end) for the iteration. I've attached a simple example binding and python test to illustrate what I mean more clearly. I'm under the impression that in a situation like this the overhead of creating the iterator is greater than the actual iteration, but thats just a guess. Alternatively my perspective on what the iteration speed should be might be completely skewed, so any help in clarifying what's going on here would be greatly appreciated. thanks Babak source.cpptest.py
Reply | Threaded
Open this post in threaded view
|

## Re: Iterating over small objects

 On 01/19/2012 02:08 PM, babak wrote: > Hi guys, > > Has anyone encountered (what feels like) quite slow iteration when looping > over small containers ? I'm in a situation where I'm iterating over 100000+ > 3d vectors and its taking much longer than I'd expect. The vector type is > bound via boost::python and has an iter method which uses range(being, end) > for the iteration. > > I've attached a simple example binding and python test to illustrate what I > mean more clearly. I'm under the impression that in a situation like this > the overhead of creating the iterator is greater than the actual iteration, > but thats just a guess. Alternatively my perspective on what the iteration > speed should be might be completely skewed, so any help in clarifying what's > going on here would be greatly appreciated. > My sense is that this is something that really can't be expected to be efficient across the C++/Python boundary.  There is some overhead associated with creating the exposures, and there's also some overhead in converting the types, especially if those vectors aren't builtin numeric types. If you can redesign the C++/Python boundary to convert your vectors directly into Python lists or tuples instead of wrapped C++ vectors, I expect you'll see an improvement, because you'll be able to take advantage of whatever optimization the Python developers have put into iteration over those classes.  Of course, it's best by far to do whatever iteration you can in C++ if performance is important, but I'm guessing that's not an option here. Jim _______________________________________________ Cplusplus-sig mailing list [hidden email] http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

## Re: Iterating over small objects

 Okay cool thanks your input. For any one else has a similar problem - writing a simple iterator function in python (which is a generator) and assigning that to the __iter__ member works well as an alternative. For me this turned out to be considerably faster than using a c++ iterator/getitem iteration. On 19 January 2012 19:37, Jim Bosch wrote: On 01/19/2012 02:08 PM, babak wrote: Hi guys, Has anyone encountered (what feels like) quite slow iteration when looping over small containers ? I'm in a situation where I'm iterating over 100000+ 3d vectors and its taking much longer than I'd expect. The vector type is bound via boost::python and has an iter method which uses range(being, end) for the iteration. I've attached a simple example binding and python test to illustrate what I mean more clearly. I'm under the impression that in a situation like this the overhead of creating the iterator is greater than the actual iteration, but thats just a guess. Alternatively my perspective on what the iteration speed should be might be completely skewed, so any help in clarifying what's going on here would be greatly appreciated. My sense is that this is something that really can't be expected to be efficient across the C++/Python boundary.  There is some overhead associated with creating the exposures, and there's also some overhead in converting the types, especially if those vectors aren't builtin numeric types. If you can redesign the C++/Python boundary to convert your vectors directly into Python lists or tuples instead of wrapped C++ vectors, I expect you'll see an improvement, because you'll be able to take advantage of whatever optimization the Python developers have put into iteration over those classes.  Of course, it's best by far to do whatever iteration you can in C++ if performance is important, but I'm guessing that's not an option here. Jim _______________________________________________ Cplusplus-sig mailing list [hidden email] http://mail.python.org/mailman/listinfo/cplusplus-sig _______________________________________________ Cplusplus-sig mailing list [hidden email] http://mail.python.org/mailman/listinfo/cplusplus-sig