copy constructors and multiple instances

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

copy constructors and multiple instances

Josh Stratton
In my particular scene I have a Scene class, which operates as kind of
a context for my operations and holds all the meshes in the scene.  I
send my scene down to the python side so import methods will import
into the containers in the scene object.  For example,
addMeshFromFile(scene, fileName) which puts the data from the file
into the scene object.  However, if the scene object isn't be
returned, it seems I'm just copying the scene object (which might be
memory expensive), adding the data to the copy, and throwing the copy
away as I don't return it and it doesn't affect my original scene
object in C++ that I passed in.  I assume this is a fairly common
problem and I need to rework my API somehow like providing a more
robust copy constructor that does a shallow copy of my data
containers.  Still, I might have to reorganize it so I'm not passing
my scene to python.  If an object is copied every time, I think that
might be memory prohibitive.  I'm including my current source from
github.

// header file handling my scene and it's data
https://github.com/strattonbrazil/Sunshine/blob/master/src/scene.h

// functions for creating the bindings to python
https://github.com/strattonbrazil/Sunshine/blob/master/src/python_bindings.cpp

// python file which reads a file and adds it to the scene
https://github.com/strattonbrazil/Sunshine/blob/master/src/objImporter.py

Do I need to change need to change my copy constructor to do a more
shallow copy?  Maybe change my internal scene objects to pointers?
Would this alleviate the memory issue?
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Jim Bosch-2
On 09/01/2011 07:03 AM, Josh Stratton wrote:

> In my particular scene I have a Scene class, which operates as kind of
> a context for my operations and holds all the meshes in the scene.  I
> send my scene down to the python side so import methods will import
> into the containers in the scene object.  For example,
> addMeshFromFile(scene, fileName) which puts the data from the file
> into the scene object.  However, if the scene object isn't be
> returned, it seems I'm just copying the scene object (which might be
> memory expensive), adding the data to the copy, and throwing the copy
> away as I don't return it and it doesn't affect my original scene
> object in C++ that I passed in.  I assume this is a fairly common
> problem and I need to rework my API somehow like providing a more
> robust copy constructor that does a shallow copy of my data
> containers.  Still, I might have to reorganize it so I'm not passing
> my scene to python.  If an object is copied every time, I think that
> might be memory prohibitive.  I'm including my current source from
> github.
>
> // header file handling my scene and it's data
> https://github.com/strattonbrazil/Sunshine/blob/master/src/scene.h
>
> // functions for creating the bindings to python
> https://github.com/strattonbrazil/Sunshine/blob/master/src/python_bindings.cpp
>
> // python file which reads a file and adds it to the scene
> https://github.com/strattonbrazil/Sunshine/blob/master/src/objImporter.py
>
> Do I need to change need to change my copy constructor to do a more
> shallow copy?  Maybe change my internal scene objects to pointers?
> Would this alleviate the memory issue?

A couple of comments:

- I'm a bit confused by your python_bindings.cpp file; did you paste
together several files?  If not, I've never seen anyone try to define
more than one module in a single shared library, and I don't it's
supposed to work.  You can wrap all of your objects in a single Python
module - the body of a BOOST_PYTHON_MODULE block is just a regular
function block, so you can include as many calls to class_ (and other
things) in there as you like.  This might be part of the confusion about
needing to import modules from one of your earlier emails.

- You shouldn't have to do anything to the copy constructor to make this
work.  Boost.Python is perfectly happy to pass things by reference or by
pointer.  It could be that your QSharedPointers are getting in the way
of this.  What is the signature of Mesh::buildByIndex?  If the Scene is
passed by QSharedPointer, that might force a copy, because Boost.Python
only knows how to dereference the pointer, not copy it (in other words,
it doesn't want to assume it can copy the smart pointer, because some
smart pointers like auto_ptr can't be copied, so it's making a new
QSharedPointer from a copy-constructed object).  If you can change it,
just make it take a Scene by reference - that will almost certainly
avoid making copies.  Alternately, if you can use boost::shared_ptr
instead, that will also solve all of your problems.  Unfortunately
support for non-boost shared pointers isn't great in Boost.Python right now.

- If you really want to ensure Boost.Python doesn't make unnecessary
copies of Scene, wrap it with:

class_<Scene,QSharedPointer<Scene>,boost::noncopyable>(...)

That will cause things that require copies to produce compiler errors,
and you can follow those errors to see exactly where the copy is
required.  It will also keep you from returning Scene objects by value,
however, but it sounds like that might be desirable.

Jim


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Josh Stratton
>> In my particular scene I have a Scene class, which operates as kind of
>> a context for my operations and holds all the meshes in the scene.  I
>> send my scene down to the python side so import methods will import
>> into the containers in the scene object.  For example,
>> addMeshFromFile(scene, fileName) which puts the data from the file
>> into the scene object.  However, if the scene object isn't be
>> returned, it seems I'm just copying the scene object (which might be
>> memory expensive), adding the data to the copy, and throwing the copy
>> away as I don't return it and it doesn't affect my original scene
>> object in C++ that I passed in.  I assume this is a fairly common
>> problem and I need to rework my API somehow like providing a more
>> robust copy constructor that does a shallow copy of my data
>> containers.  Still, I might have to reorganize it so I'm not passing
>> my scene to python.  If an object is copied every time, I think that
>> might be memory prohibitive.  I'm including my current source from
>> github.
>>
>> // header file handling my scene and it's data
>> https://github.com/strattonbrazil/Sunshine/blob/master/src/scene.h
>>
>> // functions for creating the bindings to python
>>
>> https://github.com/strattonbrazil/Sunshine/blob/master/src/python_bindings.cpp
>>
>> // python file which reads a file and adds it to the scene
>> https://github.com/strattonbrazil/Sunshine/blob/master/src/objImporter.py
>>
>> Do I need to change need to change my copy constructor to do a more
>> shallow copy?  Maybe change my internal scene objects to pointers?
>> Would this alleviate the memory issue?
>
> A couple of comments:
>
> - I'm a bit confused by your python_bindings.cpp file; did you paste
> together several files?  If not, I've never seen anyone try to define more
> than one module in a single shared library, and I don't it's supposed to
> work.  You can wrap all of your objects in a single Python module - the body
> of a BOOST_PYTHON_MODULE block is just a regular function block, so you can
> include as many calls to class_ (and other things) in there as you like.
>  This might be part of the confusion about needing to import modules from
> one of your earlier emails.

I'm currently just embedding python, but I eventually wanted to extend
it down the road.  I guess putting everything under the same macro
makes more sense.  One thing I could find briefly skimming over the
docs is creating submodules.  If the BOOST_PYTHON_MODULE is named
"foo", is there some macro/function I can put inside of that handles
submodules, so classes imported under it are imported in python such
as "from foo.bar import SomeClass"?

> - You shouldn't have to do anything to the copy constructor to make this
> work.  Boost.Python is perfectly happy to pass things by reference or by
> pointer.  It could be that your QSharedPointers are getting in the way of
> this.  What is the signature of Mesh::buildByIndex?  If the Scene is passed
> by QSharedPointer, that might force a copy, because Boost.Python only knows
> how to dereference the pointer, not copy it (in other words, it doesn't want
> to assume it can copy the smart pointer, because some smart pointers like
> auto_ptr can't be copied, so it's making a new QSharedPointer from a
> copy-constructed object).  If you can change it, just make it take a Scene
> by reference - that will almost certainly avoid making copies.  Alternately,
> if you can use boost::shared_ptr instead, that will also solve all of your
> problems.  Unfortunately support for non-boost shared pointers isn't great
> in Boost.Python right now.

Switching to boost's shared_ptr shouldn't be difficult.
Mesh::buildByIndex does take a shared-pointer, but if I understand
correctly, switching to boost::shared_ptr<Scene> should solve my
issue.

> - If you really want to ensure Boost.Python doesn't make unnecessary copies
> of Scene, wrap it with:
>
> class_<Scene,QSharedPointer<Scene>,boost::noncopyable>(...)
>
> That will cause things that require copies to produce compiler errors, and
> you can follow those errors to see exactly where the copy is required.  It
> will also keep you from returning Scene objects by value, however, but it
> sounds like that might be desirable.

I saw this in the docs and seems to be what I need.  Like you said, at
least I'd get compiling error instead of guessing what is going on in
the background.  Thanks.
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Jim Bosch-2
On 09/01/2011 01:01 PM, Josh Stratton wrote:

>> - I'm a bit confused by your python_bindings.cpp file; did you paste
>> together several files?  If not, I've never seen anyone try to define more
>> than one module in a single shared library, and I don't it's supposed to
>> work.  You can wrap all of your objects in a single Python module - the body
>> of a BOOST_PYTHON_MODULE block is just a regular function block, so you can
>> include as many calls to class_ (and other things) in there as you like.
>>   This might be part of the confusion about needing to import modules from
>> one of your earlier emails.
>
> I'm currently just embedding python, but I eventually wanted to extend
> it down the road.  I guess putting everything under the same macro
> makes more sense.  One thing I could find briefly skimming over the
> docs is creating submodules.  If the BOOST_PYTHON_MODULE is named
> "foo", is there some macro/function I can put inside of that handles
> submodules, so classes imported under it are imported in python such
> as "from foo.bar import SomeClass"?
>

To do that, you just need to add some pure Python and a directory
structure.  If "foo" is a directory, and bar.so is a Boost.Python
extension module (suffix is different on Windows, of course), you just
need to have:

foo/__init__.py
foo/bar.so

But just like you can't have a pure-Python package+module structure in a
single file, you can't do it in C++ either; you need to have a directory
and the __init__.py file.

>> - You shouldn't have to do anything to the copy constructor to make this
>> work.  Boost.Python is perfectly happy to pass things by reference or by
>> pointer.  It could be that your QSharedPointers are getting in the way of
>> this.  What is the signature of Mesh::buildByIndex?  If the Scene is passed
>> by QSharedPointer, that might force a copy, because Boost.Python only knows
>> how to dereference the pointer, not copy it (in other words, it doesn't want
>> to assume it can copy the smart pointer, because some smart pointers like
>> auto_ptr can't be copied, so it's making a new QSharedPointer from a
>> copy-constructed object).  If you can change it, just make it take a Scene
>> by reference - that will almost certainly avoid making copies.  Alternately,
>> if you can use boost::shared_ptr instead, that will also solve all of your
>> problems.  Unfortunately support for non-boost shared pointers isn't great
>> in Boost.Python right now.
>
> Switching to boost's shared_ptr shouldn't be difficult.
> Mesh::buildByIndex does take a shared-pointer, but if I understand
> correctly, switching to boost::shared_ptr<Scene>  should solve my
> issue.

Yup.  With boost::shared_ptr, you don't even need to include it as a
template argument of class_, if you instead add the line:

boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();

This allows you to return shared_ptr to Python, but because a
boost::shared_ptr can use an arbitrary deleter, Boost.Python can convert
any wrapped Scene object (even one that doesn't hold a
shared_ptr<Scene>) into a shared_ptr<Scene> with correct reference counting.

If you do include boost::shared_ptr<Scene> in the class_ template
arguments, all Scene objects constructed in Python will be inside a
shared_ptr, but because of the above, that isn't necessary unless you
have functions that need to reset the shared_ptr itself.

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Josh Stratton
BOOST_PYTHON_MODULE(scene)
{
    class_<Scene, boost::noncopyable>("Scene");
}

// later calling this
boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
PyImport_AppendInittab("scene", &initscene);

So is this all I really need for the mapping then?  I'm getting an
error trying to send the Scene object down to python.

Error in Python: <type 'exceptions.TypeError'>: No to_python
(by-value) converter found for C++ type: Scene

On Thu, Sep 1, 2011 at 1:52 PM, Jim Bosch <[hidden email]> wrote:

> On 09/01/2011 01:01 PM, Josh Stratton wrote:
>>>
>>> - I'm a bit confused by your python_bindings.cpp file; did you paste
>>> together several files?  If not, I've never seen anyone try to define
>>> more
>>> than one module in a single shared library, and I don't it's supposed to
>>> work.  You can wrap all of your objects in a single Python module - the
>>> body
>>> of a BOOST_PYTHON_MODULE block is just a regular function block, so you
>>> can
>>> include as many calls to class_ (and other things) in there as you like.
>>>  This might be part of the confusion about needing to import modules from
>>> one of your earlier emails.
>>
>> I'm currently just embedding python, but I eventually wanted to extend
>> it down the road.  I guess putting everything under the same macro
>> makes more sense.  One thing I could find briefly skimming over the
>> docs is creating submodules.  If the BOOST_PYTHON_MODULE is named
>> "foo", is there some macro/function I can put inside of that handles
>> submodules, so classes imported under it are imported in python such
>> as "from foo.bar import SomeClass"?
>>
>
> To do that, you just need to add some pure Python and a directory structure.
>  If "foo" is a directory, and bar.so is a Boost.Python extension module
> (suffix is different on Windows, of course), you just need to have:
>
> foo/__init__.py
> foo/bar.so
>
> But just like you can't have a pure-Python package+module structure in a
> single file, you can't do it in C++ either; you need to have a directory and
> the __init__.py file.
>
>>> - You shouldn't have to do anything to the copy constructor to make this
>>> work.  Boost.Python is perfectly happy to pass things by reference or by
>>> pointer.  It could be that your QSharedPointers are getting in the way of
>>> this.  What is the signature of Mesh::buildByIndex?  If the Scene is
>>> passed
>>> by QSharedPointer, that might force a copy, because Boost.Python only
>>> knows
>>> how to dereference the pointer, not copy it (in other words, it doesn't
>>> want
>>> to assume it can copy the smart pointer, because some smart pointers like
>>> auto_ptr can't be copied, so it's making a new QSharedPointer from a
>>> copy-constructed object).  If you can change it, just make it take a
>>> Scene
>>> by reference - that will almost certainly avoid making copies.
>>>  Alternately,
>>> if you can use boost::shared_ptr instead, that will also solve all of
>>> your
>>> problems.  Unfortunately support for non-boost shared pointers isn't
>>> great
>>> in Boost.Python right now.
>>
>> Switching to boost's shared_ptr shouldn't be difficult.
>> Mesh::buildByIndex does take a shared-pointer, but if I understand
>> correctly, switching to boost::shared_ptr<Scene>  should solve my
>> issue.
>
> Yup.  With boost::shared_ptr, you don't even need to include it as a
> template argument of class_, if you instead add the line:
>
> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>
> This allows you to return shared_ptr to Python, but because a
> boost::shared_ptr can use an arbitrary deleter, Boost.Python can convert any
> wrapped Scene object (even one that doesn't hold a shared_ptr<Scene>) into a
> shared_ptr<Scene> with correct reference counting.
>
> If you do include boost::shared_ptr<Scene> in the class_ template arguments,
> all Scene objects constructed in Python will be inside a shared_ptr, but
> because of the above, that isn't necessary unless you have functions that
> need to reset the shared_ptr itself.
>
> Jim
>
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Hans Meine-2
In reply to this post by Jim Bosch-2
Am Donnerstag, 1. September 2011, 22:52:56 schrieb Jim Bosch:

> Yup.  With boost::shared_ptr, you don't even need to include it as a
> template argument of class_, if you instead add the line:
>
> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>
> This allows you to return shared_ptr to Python, but because a
> boost::shared_ptr can use an arbitrary deleter, Boost.Python can convert
> any wrapped Scene object (even one that doesn't hold a
> shared_ptr<Scene>) into a shared_ptr<Scene> with correct reference
> counting.
>
> If you do include boost::shared_ptr<Scene> in the class_ template
> arguments, all Scene objects constructed in Python will be inside a
> shared_ptr, but because of the above, that isn't necessary unless you
> have functions that need to reset the shared_ptr itself.

Jim, this is the best write-up on the shared_ptr support I read _in all the
years_.  The above matches my current mental model of the topic, but it took
me a long time to understand this.  The custom deleter part (rarely mentioned
and probably not well known even among shared_ptr users) is absolutely crucial
information to people with a solid C++ (and Python) background trying to
understand BPL behavior in this respect.  Looking forward to you improving the
docs. :-)

Best,
  Hans
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Hans Meine-2
In reply to this post by Josh Stratton
Am Freitag, 2. September 2011, 04:31:24 schrieb Josh Stratton:

> BOOST_PYTHON_MODULE(scene)
> {
>     class_<Scene, boost::noncopyable>("Scene");
> }
>
> // later calling this
> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
> PyImport_AppendInittab("scene", &initscene);
>
> So is this all I really need for the mapping then?

register_ptr_to_python deals with the boost::shared_ptr around a Scene, but of
course you still need a converter for the latter, e.g. as registered by using
class_ to describe your class.

HTH
  Hans
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Josh Stratton
This is where I'm calling the function inside Scene, by the way, and
getting that error.

object processFileFunc = _pyMainModule.attr("MeshImporter").attr("processFile");
processFileFunc(this, fileName); // right here, where the python func
takes a scene object and filename

I haven't touched this part of the code when since I was using
QSharedPointers, so I assume there's something I'm still not doing to
setup this mapping.

On Fri, Sep 2, 2011 at 1:24 AM, Hans Meine <[hidden email]> wrote:

> Am Freitag, 2. September 2011, 04:31:24 schrieb Josh Stratton:
>> BOOST_PYTHON_MODULE(scene)
>> {
>>     class_<Scene, boost::noncopyable>("Scene");
>> }
>>
>> // later calling this
>> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>> PyImport_AppendInittab("scene", &initscene);
>>
>> So is this all I really need for the mapping then?
>
> register_ptr_to_python deals with the boost::shared_ptr around a Scene, but of
> course you still need a converter for the latter, e.g. as registered by using
> class_ to describe your class.
>
> HTH
>  Hans
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Dave Abrahams
In reply to this post by Jim Bosch-2

on Thu Sep 01 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>
> This allows you to return shared_ptr to Python, but because a
> boost::shared_ptr can use an arbitrary deleter, Boost.Python can
> convert any wrapped Scene object (even one that doesn't hold a
> shared_ptr<Scene>) into a shared_ptr<Scene> with correct reference
> counting.

I might have forgotten by now, but I don't think you need to register
shared_ptr, and I don't think the above will have any effect.  IIRC I
made sure shared_ptr support "just works" out-of-the-box.

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Josh Stratton
Well, right now I'm just passing "this" from inside the scene object,
so does that need to be wrapped in a shared pointer?

On Fri, Sep 2, 2011 at 12:19 PM, Dave Abrahams <[hidden email]> wrote:

>
> on Thu Sep 01 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:
>
>> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>>
>> This allows you to return shared_ptr to Python, but because a
>> boost::shared_ptr can use an arbitrary deleter, Boost.Python can
>> convert any wrapped Scene object (even one that doesn't hold a
>> shared_ptr<Scene>) into a shared_ptr<Scene> with correct reference
>> counting.
>
> I might have forgotten by now, but I don't think you need to register
> shared_ptr, and I don't think the above will have any effect.  IIRC I
> made sure shared_ptr support "just works" out-of-the-box.
>
> --
> Dave Abrahams
> BoostPro Computing
> http://www.boostpro.com
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Dave Abrahams

on Fri Sep 02 2011, Josh Stratton <strattonbrazil-AT-gmail.com> wrote:

> Well, right now I'm just passing "this" from inside the scene object,
> so does that need to be wrapped in a shared pointer?

That's a pretty vague description of what you're doing, so it's hard to
say.  I suggest you reduce your question to a minimal case that doesn't
involve any of your domain-specific stuff.  Then it will be easy to
answer.

>
> On Fri, Sep 2, 2011 at 12:19 PM, Dave Abrahams <[hidden email]> wrote:
>>
>> on Thu Sep 01 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:
>>
>>> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>>>
>>> This allows you to return shared_ptr to Python, but because a
>>> boost::shared_ptr can use an arbitrary deleter, Boost.Python can
>>> convert any wrapped Scene object (even one that doesn't hold a
>>> shared_ptr<Scene>) into a shared_ptr<Scene> with correct reference
>>> counting.
>>
>> I might have forgotten by now, but I don't think you need to register
>> shared_ptr, and I don't think the above will have any effect.  IIRC I
>> made sure shared_ptr support "just works" out-of-the-box.
>>
>> --
>> Dave Abrahams
>> BoostPro Computing
>> http://www.boostpro.com>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> [hidden email]
>> http://mail.python.org/mailman/listinfo/cplusplus-sig>

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Josh Stratton
Here's a really short example of what I don't understand.  Basically I
can setup the python function to accept shared pointers, but can I
setup it to also use normal pointers?  In the example code attached,
if I uncomment scene->sendYourselfToPython(); it fails because it's
using "this" as the scene pointer.

Error in Python: <type 'exceptions.TypeError'>: No to_python
(by-value) converter found for C++ type: Scene

So it appears to only work for shared pointers at this point.  Can I
also get it to work with "this" at the same time?  Jim pointed me to
"enable_shared_from_this" as a way for creating a shared pointer
object, which makes sense from the docs of how to do it, but I was
wondering if I necessarily NEED to send it down as a shared pointer.

Thanks

http://dpaste.com/607969/

OR

#include <iostream>

#include <boost/python.hpp>
#include <boost/python/class.hpp>
#include <boost/python/module.hpp>
#include <boost/python/def.hpp>

using namespace boost::python;

object pyMainModule;
object pyMainNamespace;

#define EXAMPLE_PY_FUNCTION \
  "from scene import Scene\n" \
  "class Foo(object):\n" \
  "  @staticmethod\n" \
  "  def processFile(scene, filename):\n" \
  "    print('here')\n"

std::string parse_python_exception();

class Scene
{
public:
  void sendYourselfToPython()
  {
    try {
      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
      object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
      processFileFunc(this, "test.txt");
    } catch (boost::python::error_already_set const &) {
      std::string perror = parse_python_exception();
      std::cerr << "Error in Python: " << perror << std::endl;
    }
  }
};
typedef boost::shared_ptr<Scene> SceneP;

BOOST_PYTHON_MODULE(scene)
{
  class_<Scene, boost::noncopyable>("Scene");
}

main(int argc, char**argv)
{
  std::cout << "starting program..." << std::endl;

  Py_Initialize();
  pyMainModule = import("__main__");
  pyMainNamespace = pyMainModule.attr("__dict__");

  boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
  PyImport_AppendInittab("scene", &initscene);

  SceneP scene(new Scene());
  scene->sendYourselfToPython(); // confused here

  try {
    object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
    object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
    processFileFunc(scene, "test.txt");
  } catch (boost::python::error_already_set const &) {
    std::string perror = parse_python_exception();
    std::cerr << "Error in Python: " << perror << std::endl;
  }
}


// taken from http://thejosephturner.com/blog/2011/06/15/embedding-python-in-c-applications-with-boostpython-part-2/
namespace py = boost::python;
std::string parse_python_exception() {
    PyObject *type_ptr = NULL, *value_ptr = NULL, *traceback_ptr = NULL;
    PyErr_Fetch(&type_ptr, &value_ptr, &traceback_ptr);
    std::string ret("Unfetchable Python error");
    if (type_ptr != NULL) {
        py::handle<> h_type(type_ptr);
        py::str type_pstr(h_type);
        py::extract<std::string> e_type_pstr(type_pstr);
        if(e_type_pstr.check())
            ret = e_type_pstr();
        else
            ret = "Unknown exception type";
    }

    if (value_ptr != NULL) {
        py::handle<> h_val(value_ptr);
        py::str a(h_val);
        py::extract<std::string> returned(a);
        if(returned.check())
            ret +=  ": " + returned();
        else
            ret += std::string(": Unparseable Python error: ");
    }

    if (traceback_ptr != NULL) {
        py::handle<> h_tb(traceback_ptr);
        py::object tb(py::import("traceback"));
        py::object fmt_tb(tb.attr("format_tb"));
        py::object tb_list(fmt_tb(h_tb));
        py::object tb_str(py::str("\n").join(tb_list));
        py::extract<std::string> returned(tb_str);
        if(returned.check())
            ret += ": " + returned();
        else
            ret += std::string(": Unparseable Python traceback");
    }
    return ret;
}


On Fri, Sep 2, 2011 at 2:15 PM, Dave Abrahams <[hidden email]> wrote:

>
> on Fri Sep 02 2011, Josh Stratton <strattonbrazil-AT-gmail.com> wrote:
>
>> Well, right now I'm just passing "this" from inside the scene object,
>> so does that need to be wrapped in a shared pointer?
>
> That's a pretty vague description of what you're doing, so it's hard to
> say.  I suggest you reduce your question to a minimal case that doesn't
> involve any of your domain-specific stuff.  Then it will be easy to
> answer.
>
>>
>> On Fri, Sep 2, 2011 at 12:19 PM, Dave Abrahams <[hidden email]> wrote:
>>>
>>> on Thu Sep 01 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:
>>>
>>>> boost::python::register_ptr_to_python< boost::shared_ptr<Scene> >();
>>>>
>>>> This allows you to return shared_ptr to Python, but because a
>>>> boost::shared_ptr can use an arbitrary deleter, Boost.Python can
>>>> convert any wrapped Scene object (even one that doesn't hold a
>>>> shared_ptr<Scene>) into a shared_ptr<Scene> with correct reference
>>>> counting.
>>>
>>> I might have forgotten by now, but I don't think you need to register
>>> shared_ptr, and I don't think the above will have any effect.  IIRC I
>>> made sure shared_ptr support "just works" out-of-the-box.
>>>
>>> --
>>> Dave Abrahams
>>> BoostPro Computing
>>> http://www.boostpro.com>
>>> _______________________________________________
>>> Cplusplus-sig mailing list
>>> [hidden email]
>>> http://mail.python.org/mailman/listinfo/cplusplus-sig>
>
> --
> Dave Abrahams
> BoostPro Computing
> http://www.boostpro.com
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Dave Abrahams

on Fri Sep 02 2011, Josh Stratton <strattonbrazil-AT-gmail.com> wrote:

> Here's a really short example of what I don't understand.  Basically I
> can setup the python function to accept shared pointers, but can I
> setup it to also use normal pointers?  

Yes, once you've exposed the class to Python, you can pass pointers to
Python.  However, Boost.Python won't let you pass raw pointers around
without telling it (via call policies) what you want to do about
lifetime management... actually what it will do, if you don't tell it
otherwise, is copy the pointee, which is at least safe from an object
lifetime point-of-view.

> In the example code attached, if I uncomment
> scene->sendYourselfToPython(); it fails because it's using "this" as
> the scene pointer.

(It's already uncommented)

> Error in Python: <type 'exceptions.TypeError'>: No to_python
> (by-value) converter found for C++ type: Scene

...because you've said it's noncopyable, there's no way to copy the
pointee, which is what happens when it is passed by value.

> So it appears to only work for shared pointers at this point.  Can I
> also get it to work with "this" at the same time?  Jim pointed me to
> "enable_shared_from_this" as a way for creating a shared pointer
> object, which makes sense from the docs of how to do it, but I was
> wondering if I necessarily NEED to send it down as a shared pointer.

There *are* things you can do to subvert Python's checking, but they're
highly discouraged.  Jim is on the right track: if you want to pass a
mutable reference to the object and you want its lifetime to be properly
managed, you should do what he's saying.  If you don't like
using enable_shared_from_this, of course, you can just do it outside a
member function:

  void sendSceneToPython(SceneP s)
    try {
      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
      object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
      processFileFunc(s, "test.txt");
    } catch (boost::python::error_already_set const &) {
      std::string perror = parse_python_exception();
      std::cerr << "Error in Python: " << perror << std::endl;
    }
  }

HTH,

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: copy constructors and multiple instances

Josh Stratton
Thanks for everyone's help.  I'm posting my updated class that is now working.

class Scene : public boost::enable_shared_from_this<Scene>
{
public:
  void sendYourselfToPython()
  {
    try {
      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
      object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
      processFileFunc(shared_from_this(), "test.txt");
    } catch (boost::python::error_already_set const &) {
      std::string perror = parse_python_exception();
      std::cerr << "Error in Python: " << perror << std::endl;
    }
  }
};
typedef boost::shared_ptr<Scene> SceneP;

On Fri, Sep 2, 2011 at 9:08 PM, Dave Abrahams <[hidden email]> wrote:

>
> on Fri Sep 02 2011, Josh Stratton <strattonbrazil-AT-gmail.com> wrote:
>
>> Here's a really short example of what I don't understand.  Basically I
>> can setup the python function to accept shared pointers, but can I
>> setup it to also use normal pointers?
>
> Yes, once you've exposed the class to Python, you can pass pointers to
> Python.  However, Boost.Python won't let you pass raw pointers around
> without telling it (via call policies) what you want to do about
> lifetime management... actually what it will do, if you don't tell it
> otherwise, is copy the pointee, which is at least safe from an object
> lifetime point-of-view.
>
>> In the example code attached, if I uncomment
>> scene->sendYourselfToPython(); it fails because it's using "this" as
>> the scene pointer.
>
> (It's already uncommented)
>
>> Error in Python: <type 'exceptions.TypeError'>: No to_python
>> (by-value) converter found for C++ type: Scene
>
> ...because you've said it's noncopyable, there's no way to copy the
> pointee, which is what happens when it is passed by value.
>
>> So it appears to only work for shared pointers at this point.  Can I
>> also get it to work with "this" at the same time?  Jim pointed me to
>> "enable_shared_from_this" as a way for creating a shared pointer
>> object, which makes sense from the docs of how to do it, but I was
>> wondering if I necessarily NEED to send it down as a shared pointer.
>
> There *are* things you can do to subvert Python's checking, but they're
> highly discouraged.  Jim is on the right track: if you want to pass a
> mutable reference to the object and you want its lifetime to be properly
> managed, you should do what he's saying.  If you don't like
> using enable_shared_from_this, of course, you can just do it outside a
> member function:
>
>  void sendSceneToPython(SceneP s)
>    try {
>      object ignored = exec(EXAMPLE_PY_FUNCTION, pyMainNamespace);
>      object processFileFunc = pyMainModule.attr("Foo").attr("processFile");
>      processFileFunc(s, "test.txt");
>    } catch (boost::python::error_already_set const &) {
>      std::string perror = parse_python_exception();
>      std::cerr << "Error in Python: " << perror << std::endl;
>    }
>  }
>
> HTH,
>
> --
> Dave Abrahams
> BoostPro Computing
> http://www.boostpro.com
>
> _______________________________________________
> Cplusplus-sig mailing list
> [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig