Boost.Python sandox or multiple interpreters

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Boost.Python sandox or multiple interpreters

David Bellot
Hi,

I'm using Boost.Python to load and use plugins in Python. Each plugin can
define a few functions which are are pre-determined, so that my C++
software knows what to call.

However, the mistake I've made was to think that the Python interpreter
could be initialized many times, so that each plugins was existing and
running into a separate interpreter. It's of course not the case and
therefore, each time I load a new plugin, its functions overwrite the
functions of the same name from the previous plugin (the one which was
loaded just before), and so on.

Is there a possibility to either :

- have as many independent Python interpreters at the same time ?
- or encapsulate each plugin in an independent "environment" ?
- or any other solution which would elegantly solve this problem ? :-D

Thanks for your help,
David

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python sandox or multiple interpreters

Stefan Seefeld-2
On 13.01.2017 11:40, David Bellot wrote:

> Hi,
>
> I'm using Boost.Python to load and use plugins in Python. Each plugin can
> define a few functions which are are pre-determined, so that my C++
> software knows what to call.
>
> However, the mistake I've made was to think that the Python interpreter
> could be initialized many times, so that each plugins was existing and
> running into a separate interpreter. It's of course not the case and
> therefore, each time I load a new plugin, its functions overwrite the
> functions of the same name from the previous plugin (the one which was
> loaded just before), and so on.
>
> Is there a possibility to either :
>
> - have as many independent Python interpreters at the same time ?

That's a question you should ask and discuss with Python developers.
(The current answer is 'no', as there is substantial global state in the
Python runtime.)

> - or encapsulate each plugin in an independent "environment" ?

Hard to tell without knowing these plugins. Right now it sounds like
that's a question only you can answer. :-)

> - or any other solution which would elegantly solve this problem ? :-D

Likewise.

Yes, I think the only possible way to solve this is to instantiate
Python first, and then let the plugins register with it, rather than
each plugin assume it was in control of "its" Python interpreter.
Or, if your plugins are heavyweight enough, you could run them within
their own proper address space (read: each in a separate process). But
you were asking for elegant solutions... :-)


> Thanks for your help,
> David

        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python sandox or multiple interpreters

David Bellot
>
> > - or encapsulate each plugin in an independent "environment" ?
>
> Hard to tell without knowing these plugins. Right now it sounds like
> that's a question only you can answer. :-)
>

​The plugins are quite simple in fact:

The Python code define a few functions like

def on_value_update(​x):
   blah blah blah
   return foobar

and they can have global variables (if it's a problem I can forbid user's
to use those global variables in their plugins).

Each plugin is defined in a .py file and at runtime I load all those files,
one after each other.

I was reading about "scope" but I'm not sure from the documentation if it's
the right solution.
Can I load each of my Python files with all their functions and variables
into such a "scope" and switch from one scope to another depending on the
plugin I want to use ?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python sandox or multiple interpreters

Stefan Seefeld-2
On 13.01.2017 12:46, David Bellot wrote:

>>> - or encapsulate each plugin in an independent "environment" ?
>> Hard to tell without knowing these plugins. Right now it sounds like
>> that's a question only you can answer. :-)
>>
> ​The plugins are quite simple in fact:
>
> The Python code define a few functions like
>
> def on_value_update(​x):
>    blah blah blah
>    return foobar
>
> and they can have global variables (if it's a problem I can forbid user's
> to use those global variables in their plugins).

For avoidance of doubt: "global variables" aren't variables in
module-scope (they are still "local" to the module, and don't cause any
problem.)

> Each plugin is defined in a .py file and at runtime I load all those files,
> one after each other.

That sounds all good. Each plugin corresponds to a Python module (which
is an object), and you can call module-level functions (such as
'on_value_update') on that object as if it was a method. So if you keep
a list of module objects in your C++ runtime, you can access their
methods without any global functions colliding. (See
http://boostorg.github.io/python/doc/html/tutorial/tutorial/embedding.html#tutorial.embedding.using_the_interpreter
for how to embed Python code into your C++ app.)

In case that doesn't help, can you explain where in your current setup
the plugin methods overwrite each other as they are loaded ?

        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python sandox or multiple interpreters

Boost - Dev mailing list
in fact, I found a much better solution, much more element by using the
wrapper<T> class and having the boost::python::object instanciated in C++
rather than in the interpreter (if my interpretation of the interpreted is
correctly interpreted in this context).

So each plugin is a python object and thanks to wrapper<>, the Python user
can now derive from a base Plugin class and implement whatever callbacks
they need, instead of doing the strange file gibberish I did before.

For those wondering how it works:
1- Make a Base class with pure virtual methods that you want the python
user to implement

class Plugin
{
public:
    Plugin(Server&); // Server is whatever C++ code control the plugin in
fact
    virtual ~Plugin() = default;

    // evaluates the plugin
    virtual bool eval(int x) = 0;

    // use some Server function
    void a_function(const std::string& val);

private:
    Server& server;
};

2- Make an intermediate class which inherits from Base and
boost::python::wrapper<Base> in which the user's function are implemented
like in the following example:

class PluginWrapper final : public Plugin, public wrapper<Plugin>
{
    using Plugin::Plugin;

    void eval(int x) override
    {
        return get_override("eval")(x); // This will call the user's Python
code
    }
};

3- Expose your API in Python as usual:

BOOST_PYTHON_MODULE(Foobar)
{
    class_<Server>("Server");

    class_<PluginWrapper, boost::noncopyable>("PyPlugin", init<Server&>())
        .def("eval", &Plugin::eval);
}

4- When you want to initialize your plugins, you can load the user's Python
code as follow, where module is the Python module name (can be foobar.py or
a foobar/ directory with __init__.py inside) and path is the ... path to it
!

    dict locals;
    locals["module_name"] = module;
    locals["path"]        = path;

    exec("import imp\n"
             "new_module = imp.load_module(module_name, open(path), path,
('py', 'U', imp.PY_SOURCE))\n",
             globals,
             locals);

in the end  locals["new_module"] contains the module and that's what you
want to use to instantiate an "object" in your C++ code.

5- Start your C++ code with something like this:
  Py_Initialize();

for Python 2
  PyImport_AppendInittab("Foobar", initFoobar);

or for Python 3:
  PyImport_AppendInittab("Foobar", PyInit_Foobar);

    Server server;

    // import the __main__ module and obtain the globals dict
    object main     = import("__main__");
    object globals  = main.attr("__dict__");

    // *** HERE PUT CODE FROM STEP 4 ***

    object module = locals["new_module"] // This contains the Python module
    object PluginClass = module.attr("MyUserPlugin"); // This contains the
plugin class as defined by the user
    object plugin = PluginClass(server); // This contains the running
plugin itself

// And you can instantiate as many object plugin as you want. They all will
be different even if instantiated from the same class because they live in
the C++ space. It's what solved my problem in the most elegant way. I know
it looks soooo obvious now :-D but I just didn't know about the wrapper<>
class before !

    // eval the plugin
    bool result = plugin.attr("eval")(32); // Yep, as simple as that !!!!!

6- And finally write your plugins:
from Foobar import *

class MyUserPlugin(PyPlugin):
    def __init__(self, server):
        PyPlugin.__init__(self, server)

    def eval(x):
        print "hello, world from the plugin. The value is ", x
        return True

7- one last note: the Server is used for the Python code to call C++ code
or for example send data to the C++ part of your sofware. So you can add
Python functions in you BOOS_PYTHON_MODULE to do that too.

And that's it. Look so simple now :-D

Best regards,
David


ref: https://skebanga.github.io/embedded-python/

On Fri, Jan 13, 2017 at 6:02 PM, Stefan Seefeld <[hidden email]> wrote:

> On 13.01.2017 12:46, David Bellot wrote:
> >>> - or encapsulate each plugin in an independent "environment" ?
> >> Hard to tell without knowing these plugins. Right now it sounds like
> >> that's a question only you can answer. :-)
> >>
> > ​The plugins are quite simple in fact:
> >
> > The Python code define a few functions like
> >
> > def on_value_update(​x):
> >    blah blah blah
> >    return foobar
> >
> > and they can have global variables (if it's a problem I can forbid user's
> > to use those global variables in their plugins).
>
> For avoidance of doubt: "global variables" aren't variables in
> module-scope (they are still "local" to the module, and don't cause any
> problem.)
>
> > Each plugin is defined in a .py file and at runtime I load all those
> files,
> > one after each other.
>
> That sounds all good. Each plugin corresponds to a Python module (which
> is an object), and you can call module-level functions (such as
> 'on_value_update') on that object as if it was a method. So if you keep
> a list of module objects in your C++ runtime, you can access their
> methods without any global functions colliding. (See
> http://boostorg.github.io/python/doc/html/tutorial/tutorial/
> embedding.html#tutorial.embedding.using_the_interpreter
> for how to embed Python code into your C++ app.)
>
> In case that doesn't help, can you explain where in your current setup
> the plugin methods overwrite each other as they are loaded ?
>
>         Stefan
>
> --
>
>       ...ich hab' noch einen Koffer in Berlin...
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman
> /listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python sandox or multiple interpreters

Boost - Dev mailing list
Hi David,

indeed, that's a good solution. Glad you figured that out ! :-)

        Stefan

--

      ...ich hab' noch einen Koffer in Berlin...


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Boost.Python sandox or multiple interpreters

Boost - Dev mailing list
Dear David,

just as a further hint, I found that the tests in boost.python can serve as useful how-tos. Your wrapper solution can be found in python/test/polymorphism2.cpp, for example.

Best regards,
Hans

> On 17 Feb 2017, at 13:00, Stefan Seefeld via Boost <[hidden email]> wrote:
>
> Hi David,
>
> indeed, that's a good solution. Glad you figured that out ! :-)
>
>        Stefan
>
> --
>
>      ...ich hab' noch einen Koffer in Berlin...
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost